public inbox for [email protected]
help / color / mirror / Atom feedRe: [PoC] Improve dead tuple storage for lazy vacuum
78+ messages / 4 participants
[nested] [flat]
* Re: [PoC] Improve dead tuple storage for lazy vacuum
@ 2023-01-31 14:42 Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-01-31 14:42 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Mon, Jan 30, 2023 at 11:30 PM Masahiko Sawada <[email protected]> wrote:
>
> On Mon, Jan 30, 2023 at 1:31 PM John Naylor
> <[email protected]> wrote:
> >
> > On Sun, Jan 29, 2023 at 9:50 PM Masahiko Sawada <[email protected]> wrote:
> > >
> > > On Sat, Jan 28, 2023 at 8:33 PM John Naylor
> > > <[email protected]> wrote:
> >
> > > > The first implementation should be simple, easy to test/verify, easy to understand, and easy to replace. As much as possible anyway.
> > >
> > > Yes, but if a concurrent writer waits for another process to finish
> > > the iteration, it ends up waiting on a lwlock, which is not
> > > interruptible.
> > >
> > > >
> > > > > So the idea is that we set iter_active to true (with the
> > > > > lock in exclusive mode), and prevent concurrent updates when the flag
> > > > > is true.
> > > >
> > > > ...by throwing elog(ERROR)? I'm not so sure users of this API would prefer that to waiting.
> > >
> > > Right. I think if we want to wait rather than an ERROR, the waiter
> > > should wait in an interruptible way, for example, a condition
> > > variable. I did a simpler way in the v22 patch.
> > >
> > > ...but looking at dshash.c, dshash_seq_next() seems to return an entry
> > > while holding a lwlock on the partition. My assumption might be wrong.
> >
> > Using partitions there makes holding a lock less painful on average, I imagine, but I don't know the details there.
> >
> > If we make it clear that the first committed version is not (yet) designed for high concurrency with mixed read-write workloads, I think waiting (as a protocol) is fine. If waiting is a problem for some use case, at that point we should just go all the way and replace the locking entirely. In fact, it might be good to spell this out in the top-level comment and include a link to the second ART paper.
>
> Agreed. Will update the comments.
>
> >
> > > > [thinks some more...] Is there an API-level assumption that hasn't been spelled out? Would it help to have a parameter for whether the iteration function wants to reserve the privilege to perform writes? It could take the appropriate lock at the start, and there could then be multiple read-only iterators, but only one read/write iterator. Note, I'm just guessing here, and I don't want to make things more difficult for future improvements.
> > >
> > > Seems a good idea. Given the use case for parallel heap vacuum, it
> > > would be a good idea to support having multiple read-only writers. The
> > > iteration of the v22 is read-only, so if we want to support read-write
> > > iterator, we would need to support a function that modifies the
> > > current key-value returned by the iteration.
> >
> > Okay, so updating during iteration is not currently supported. It could in the future, but I'd say that can also wait for fine-grained concurrency support. Intermediate-term, we should at least make it straightforward to support:
> >
> > 1) parallel heap vacuum -> multiple read-only iterators
> > 2) parallel heap pruning -> multiple writers
> >
> > It may or may not be worth it for someone to actually start either of those projects, and there are other ways to improve vacuum that may be more pressing. That said, it seems the tid store with global locking would certainly work fine for #1 and maybe "not too bad" for #2. #2 can also mitigate waiting by using larger batching, or the leader process could "pre-warm" the tid store with zero-values using block numbers from the visibility map.
>
> True. Using a larger batching method seems to be worth testing when we
> implement the parallel heap pruning.
>
> In the next version patch, I'm going to update the locking support
> part and incorporate other comments I got.
>
I've attached v24 patches. The locking support patch is separated
(0005 patch). Also I kept the updates for TidStore and the vacuum
integration from v23 separate.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] v24-0005-Add-read-write-lock-to-radix-tree-in-RT_SHMEM-ca.patch (11.0K, ../../CAD21AoCSkEDYAqW3+JtSQQY+R2tDCrXA=PgNAUXUdUQ5gBOp0Q@mail.gmail.com/2-v24-0005-Add-read-write-lock-to-radix-tree-in-RT_SHMEM-ca.patch)
download | inline diff:
From 1085ef0b9b8b31795616abc43063a91b27e7d5a4 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 25 Jan 2023 17:43:29 +0900
Subject: [PATCH v24 5/9] Add read-write lock to radix tree in RT_SHMEM case.
---
src/include/lib/radixtree.h | 102 ++++++++++++++++--
.../modules/test_radixtree/test_radixtree.c | 8 +-
2 files changed, 100 insertions(+), 10 deletions(-)
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index f591d903fc..48134b10e4 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -40,6 +40,18 @@
* There are some optimizations not yet implemented, particularly path
* compression and lazy path expansion.
*
+ * To handle concurrency, we use a single reader-writer lock for the radix
+ * tree. The radix tree is exclusively locked during write operations such
+ * as RT_SET() and RT_DELETE(), and shared locked during read operations
+ * such as RT_SEARCH(). An iteration also holds the shared lock on the radix
+ * tree until it is completed.
+ *
+ * TODO: The current locking mechanism is not optimized for high concurrency
+ * with mixed read-write workloads. In the future it might be worthwhile
+ * to replace it with the Optimistic Lock Coupling or ROWEX mentioned in
+ * the paper "The ART of Practical Synchronization" by the same authors as
+ * the ART paper, 2016.
+ *
* WIP: the radix tree nodes don't shrink.
*
* To generate a radix tree and associated functions for a use case several
@@ -224,7 +236,7 @@ typedef dsa_pointer RT_HANDLE;
#endif
#ifdef RT_SHMEM
-RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx, dsa_area *dsa);
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id);
RT_SCOPE RT_RADIX_TREE * RT_ATTACH(dsa_area *dsa, dsa_pointer dp);
RT_SCOPE void RT_DETACH(RT_RADIX_TREE *tree);
RT_SCOPE RT_HANDLE RT_GET_HANDLE(RT_RADIX_TREE *tree);
@@ -371,6 +383,16 @@ typedef struct RT_NODE
#define RT_INVALID_PTR_ALLOC NULL
#endif
+#ifdef RT_SHMEM
+#define RT_LOCK_EXCLUSIVE(tree) LWLockAcquire(&tree->ctl->lock, LW_EXCLUSIVE)
+#define RT_LOCK_SHARED(tree) LWLockAcquire(&tree->ctl->lock, LW_SHARED)
+#define RT_UNLOCK(tree) LWLockRelease(&tree->ctl->lock);
+#else
+#define RT_LOCK_EXCLUSIVE(tree) ((void) 0)
+#define RT_LOCK_SHARED(tree) ((void) 0)
+#define RT_UNLOCK(tree) ((void) 0)
+#endif
+
/*
* Inner nodes and leaf nodes have analogous structure. To distinguish
* them at runtime, we take advantage of the fact that the key chunk
@@ -596,6 +618,7 @@ typedef struct RT_RADIX_TREE_CONTROL
#ifdef RT_SHMEM
RT_HANDLE handle;
uint32 magic;
+ LWLock lock;
#endif
RT_PTR_ALLOC root;
@@ -1376,7 +1399,7 @@ RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC store
*/
RT_SCOPE RT_RADIX_TREE *
#ifdef RT_SHMEM
-RT_CREATE(MemoryContext ctx, dsa_area *dsa)
+RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id)
#else
RT_CREATE(MemoryContext ctx)
#endif
@@ -1398,6 +1421,7 @@ RT_CREATE(MemoryContext ctx)
tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, dp);
tree->ctl->handle = dp;
tree->ctl->magic = RT_RADIX_TREE_MAGIC;
+ LWLockInitialize(&tree->ctl->lock, tranche_id);
#else
tree->ctl = (RT_RADIX_TREE_CONTROL *) palloc0(sizeof(RT_RADIX_TREE_CONTROL));
@@ -1581,6 +1605,8 @@ RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE value)
Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
#endif
+ RT_LOCK_EXCLUSIVE(tree);
+
/* Empty tree, create the root */
if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
RT_NEW_ROOT(tree, key);
@@ -1606,6 +1632,7 @@ RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE value)
if (!RT_NODE_SEARCH_INNER(child, key, &new_child))
{
RT_SET_EXTEND(tree, key, value, parent, stored_child, child);
+ RT_UNLOCK(tree);
return false;
}
@@ -1620,12 +1647,13 @@ RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE value)
if (!updated)
tree->ctl->num_keys++;
+ RT_UNLOCK(tree);
return updated;
}
/*
* Search the given key in the radix tree. Return true if there is the key,
- * otherwise return false. On success, we set the value to *val_p so it must
+ * otherwise return false. On success, we set the value to *val_p so it must
* not be NULL.
*/
RT_SCOPE bool
@@ -1633,14 +1661,20 @@ RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
{
RT_PTR_LOCAL node;
int shift;
+ bool found;
#ifdef RT_SHMEM
Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
#endif
Assert(value_p != NULL);
+ RT_LOCK_SHARED(tree);
+
if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
return false;
+ }
node = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
shift = node->shift;
@@ -1654,13 +1688,19 @@ RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
break;
if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ {
+ RT_UNLOCK(tree);
return false;
+ }
node = RT_PTR_GET_LOCAL(tree, child);
shift -= RT_NODE_SPAN;
}
- return RT_NODE_SEARCH_LEAF(node, key, value_p);
+ found = RT_NODE_SEARCH_LEAF(node, key, value_p);
+
+ RT_UNLOCK(tree);
+ return found;
}
#ifdef RT_USE_DELETE
@@ -1682,8 +1722,13 @@ RT_DELETE(RT_RADIX_TREE *tree, uint64 key)
Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
#endif
+ RT_LOCK_EXCLUSIVE(tree);
+
if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
return false;
+ }
/*
* Descend the tree to search the key while building a stack of nodes we
@@ -1702,7 +1747,10 @@ RT_DELETE(RT_RADIX_TREE *tree, uint64 key)
node = RT_PTR_GET_LOCAL(tree, allocnode);
if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ {
+ RT_UNLOCK(tree);
return false;
+ }
allocnode = child;
shift -= RT_NODE_SPAN;
@@ -1715,6 +1763,7 @@ RT_DELETE(RT_RADIX_TREE *tree, uint64 key)
if (!deleted)
{
/* no key is found in the leaf node */
+ RT_UNLOCK(tree);
return false;
}
@@ -1726,7 +1775,10 @@ RT_DELETE(RT_RADIX_TREE *tree, uint64 key)
* node.
*/
if (node->count > 0)
+ {
+ RT_UNLOCK(tree);
return true;
+ }
/* Free the empty leaf node */
RT_FREE_NODE(tree, allocnode);
@@ -1748,6 +1800,7 @@ RT_DELETE(RT_RADIX_TREE *tree, uint64 key)
RT_FREE_NODE(tree, allocnode);
}
+ RT_UNLOCK(tree);
return true;
}
#endif
@@ -1812,7 +1865,12 @@ RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
}
}
-/* Create and return the iterator for the given radix tree */
+/*
+ * Create and return the iterator for the given radix tree.
+ *
+ * The radix tree is locked in shared mode during the iteration, so
+ * RT_END_ITERATE needs to be called when finished to release the lock.
+ */
RT_SCOPE RT_ITER *
RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
{
@@ -1826,6 +1884,8 @@ RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
iter = (RT_ITER *) palloc0(sizeof(RT_ITER));
iter->tree = tree;
+ RT_LOCK_SHARED(tree);
+
/* empty tree */
if (!iter->tree->ctl->root)
return iter;
@@ -1846,7 +1906,7 @@ RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
}
/*
- * Return true with setting key_p and value_p if there is next key. Otherwise,
+ * Return true with setting key_p and value_p if there is next key. Otherwise
* return false.
*/
RT_SCOPE bool
@@ -1901,9 +1961,20 @@ RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p)
return false;
}
+/*
+ * Terminate the iteration and release the lock.
+ *
+ * This function needs to be called after finishing or when exiting an
+ * iteration.
+ */
RT_SCOPE void
RT_END_ITERATE(RT_ITER *iter)
{
+#ifdef RT_SHMEM
+ Assert(LWLockHeldByMe(&iter->tree->ctl->lock));
+#endif
+
+ RT_UNLOCK(iter->tree);
pfree(iter);
}
@@ -1915,6 +1986,8 @@ RT_MEMORY_USAGE(RT_RADIX_TREE *tree)
{
Size total = 0;
+ RT_LOCK_SHARED(tree);
+
#ifdef RT_SHMEM
Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
total = dsa_get_total_size(tree->dsa);
@@ -1926,6 +1999,7 @@ RT_MEMORY_USAGE(RT_RADIX_TREE *tree)
}
#endif
+ RT_UNLOCK(tree);
return total;
}
@@ -2010,6 +2084,8 @@ RT_VERIFY_NODE(RT_PTR_LOCAL node)
RT_SCOPE void
RT_STATS(RT_RADIX_TREE *tree)
{
+ RT_LOCK_SHARED(tree);
+
fprintf(stderr, "max_val = " UINT64_FORMAT "\n", tree->ctl->max_val);
fprintf(stderr, "num_keys = " UINT64_FORMAT "\n", tree->ctl->num_keys);
@@ -2029,6 +2105,8 @@ RT_STATS(RT_RADIX_TREE *tree)
tree->ctl->cnt[RT_CLASS_125],
tree->ctl->cnt[RT_CLASS_256]);
}
+
+ RT_UNLOCK(tree);
}
static void
@@ -2222,14 +2300,18 @@ RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key)
RT_STATS(tree);
+ RT_LOCK_SHARED(tree);
+
if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
{
+ RT_UNLOCK(tree);
fprintf(stderr, "empty tree\n");
return;
}
if (key > tree->ctl->max_val)
{
+ RT_UNLOCK(tree);
fprintf(stderr, "key " UINT64_FORMAT "(0x" RT_UINT64_FORMAT_HEX ") is larger than max val\n",
key, key);
return;
@@ -2263,6 +2345,7 @@ RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key)
shift -= RT_NODE_SPAN;
level++;
}
+ RT_UNLOCK(tree);
fprintf(stderr, "%s", buf.data);
}
@@ -2274,8 +2357,11 @@ RT_DUMP(RT_RADIX_TREE *tree)
RT_STATS(tree);
+ RT_LOCK_SHARED(tree);
+
if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
{
+ RT_UNLOCK(tree);
fprintf(stderr, "empty tree\n");
return;
}
@@ -2283,6 +2369,7 @@ RT_DUMP(RT_RADIX_TREE *tree)
initStringInfo(&buf);
RT_DUMP_NODE(tree, tree->ctl->root, 0, true, &buf);
+ RT_UNLOCK(tree);
fprintf(stderr, "%s",buf.data);
}
@@ -2310,6 +2397,9 @@ RT_DUMP(RT_RADIX_TREE *tree)
#undef RT_GET_KEY_CHUNK
#undef BM_IDX
#undef BM_BIT
+#undef RT_LOCK_EXCLUSIVE
+#undef RT_LOCK_SHARED
+#undef RT_UNLOCK
#undef RT_NODE_IS_LEAF
#undef RT_NODE_MUST_GROW
#undef RT_NODE_KIND_COUNT
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
index 2a93e731ae..bbe1a619b6 100644
--- a/src/test/modules/test_radixtree/test_radixtree.c
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -144,7 +144,7 @@ test_empty(void)
dsa_area *dsa;
dsa = dsa_create(tranche_id);
- radixtree = rt_create(CurrentMemoryContext, dsa);
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
#else
radixtree = rt_create(CurrentMemoryContext);
#endif
@@ -195,7 +195,7 @@ test_basic(int children, bool test_inner)
test_inner ? "inner" : "leaf", children);
#ifdef RT_SHMEM
- radixtree = rt_create(CurrentMemoryContext, dsa);
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
#else
radixtree = rt_create(CurrentMemoryContext);
#endif
@@ -363,7 +363,7 @@ test_node_types(uint8 shift)
elog(NOTICE, "testing radix tree node types with shift \"%d\"", shift);
#ifdef RT_SHMEM
- radixtree = rt_create(CurrentMemoryContext, dsa);
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
#else
radixtree = rt_create(CurrentMemoryContext);
#endif
@@ -434,7 +434,7 @@ test_pattern(const test_spec * spec)
MemoryContextSetIdentifier(radixtree_ctx, spec->test_name);
#ifdef RT_SHMEM
- radixtree = rt_create(radixtree_ctx, dsa);
+ radixtree = rt_create(radixtree_ctx, dsa, tranche_id);
#else
radixtree = rt_create(radixtree_ctx);
#endif
--
2.31.1
[application/octet-stream] v24-0008-Update-TidStore-patch-from-v23.patch (10.8K, ../../CAD21AoCSkEDYAqW3+JtSQQY+R2tDCrXA=PgNAUXUdUQ5gBOp0Q@mail.gmail.com/3-v24-0008-Update-TidStore-patch-from-v23.patch)
download | inline diff:
From c76104ba85a5668cfbcb236610bc494127642102 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Tue, 31 Jan 2023 17:41:31 +0900
Subject: [PATCH v24 8/9] Update TidStore patch from v23.
Incorporate the comments, update comments, and add the description of
concurrency support.
---
src/backend/access/common/tidstore.c | 110 +++++++++++++++------------
1 file changed, 62 insertions(+), 48 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 89aea71945..f656de2189 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -11,7 +11,10 @@
* to tidstore_create(). Other backends can attach to the shared TidStore by
* tidstore_attach().
*
- * XXX: Only one process is allowed to iterate over the TidStore at a time.
+ * Regarding the concurrency, it basically relies on the concurrency support in
+ * the radix tree, but we acquires the lock on a TidStore in some cases, for
+ * example, when to reset the store and when to access the number tids in the
+ * store (num_tids).
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -23,7 +26,6 @@
*/
#include "postgres.h"
-#include "access/htup_details.h"
#include "access/tidstore.h"
#include "miscadmin.h"
#include "port/pg_bitutils.h"
@@ -87,14 +89,17 @@
#define RT_VALUE_TYPE uint64
#include "lib/radixtree.h"
-/* The header object for a TidStore */
+/* The control object for a TidStore */
typedef struct TidStoreControl
{
- int64 num_tids; /* the number of Tids stored so far */
+ /* the number of tids in the store */
+ int64 num_tids;
+
+ /* These values are never changed after creation */
size_t max_bytes; /* the maximum bytes a TidStore can use */
int max_offset; /* the maximum offset number */
+ int offset_nbits; /* the number of bits required for max_offset */
bool encode_tids; /* do we use tid encoding? */
- int offset_nbits; /* the number of bits used for offset number */
int offset_key_nbits; /* the number of bits of a offset number
* used for the key */
@@ -117,7 +122,7 @@ struct TidStore
*/
TidStoreControl *control;
- /* Storage for Tids. Use either one depending on TidStoreIsShared() */
+ /* Storage for Tids. Use either one depending on TidStoreIsShared() */
union
{
local_rt_radix_tree *local;
@@ -170,24 +175,24 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
/*
* Create the radix tree for the main storage.
*
- * Memory consumption depends on the number of Tids stored, but also on the
+ * Memory consumption depends on the number of stored tids, but also on the
* distribution of them, how the radix tree stores, and the memory management
* that backed the radix tree. The maximum bytes that a TidStore can
* use is specified by the max_bytes in tidstore_create(). We want the total
- * amount of memory consumption not to exceed the max_bytes.
+ * amount of memory consumption by a TidStore not to exceed the max_bytes.
*
- * In non-shared cases, the radix tree uses slab allocators for each kind of
- * node class. The most memory consuming case while adding Tids associated
- * with one page (i.e. during tidstore_add_tids()) is that we allocate the
- * largest radix tree node in a new slab block, which is approximately 70kB.
- * Therefore, we deduct 70kB from the maximum bytes.
+ * In local TidStore cases, the radix tree uses slab allocators for each kind
+ * of node class. The most memory consuming case while adding Tids associated
+ * with one page (i.e. during tidstore_add_tids()) is that we allocate a new
+ * slab block for a new radix tree node, which is approximately 70kB. Therefore,
+ * we deduct 70kB from the max_bytes.
*
* In shared cases, DSA allocates the memory segments big enough to follow
* a geometric series that approximately doubles the total DSA size (see
* make_new_segment() in dsa.c). We simulated the how DSA increases segment
* size and the simulation revealed, the 75% threshold for the maximum bytes
- * perfectly works in case where it is a power-of-2, and the 60% threshold
- * works for other cases.
+ * perfectly works in case where the max_bytes is a power-of-2, and the 60%
+ * threshold works for other cases.
*/
if (area != NULL)
{
@@ -199,7 +204,7 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
dp = dsa_allocate0(area, sizeof(TidStoreControl));
ts->control = (TidStoreControl *) dsa_get_address(area, dp);
- ts->control->max_bytes =(uint64) (max_bytes * ratio);
+ ts->control->max_bytes = (uint64) (max_bytes * ratio);
ts->area = area;
ts->control->magic = TIDSTORE_MAGIC;
@@ -212,12 +217,16 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
ts->tree.local = local_rt_create(CurrentMemoryContext);
ts->control = (TidStoreControl *) palloc0(sizeof(TidStoreControl));
- ts->control->max_bytes = max_bytes - (1024 * 70);
+ ts->control->max_bytes = max_bytes - (70 * 1024);
}
ts->control->max_offset = max_offset;
ts->control->offset_nbits = pg_ceil_log2_32(max_offset);
+ /*
+ * We use tid encoding if the number of bits for the offset number doesn't
+ * fix in a value, uint64.
+ */
if (ts->control->offset_nbits > TIDSTORE_VALUE_NBITS)
{
ts->control->encode_tids = true;
@@ -311,7 +320,10 @@ tidstore_destroy(TidStore *ts)
pfree(ts);
}
-/* Forget all collected Tids */
+/*
+ * Forget all collected Tids. It's similar to tidstore_destroy but we don't free
+ * entire TidStore but recreate only the radix tree storage.
+ */
void
tidstore_reset(TidStore *ts)
{
@@ -350,15 +362,6 @@ tidstore_reset(TidStore *ts)
}
}
-static inline void
-tidstore_insert_kv(TidStore *ts, uint64 key, uint64 val)
-{
- if (TidStoreIsShared(ts))
- shared_rt_set(ts->tree.shared, key, val);
- else
- local_rt_set(ts->tree.local, key, val);
-}
-
/* Add Tids on a block to TidStore */
void
tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
@@ -371,8 +374,6 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
- ItemPointerSetBlockNumber(&tid, blkno);
-
if (ts->control->encode_tids)
{
key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
@@ -383,9 +384,9 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
key_base = (uint64) blkno;
nkeys = 1;
}
-
values = palloc0(sizeof(uint64) * nkeys);
+ ItemPointerSetBlockNumber(&tid, blkno);
for (int i = 0; i < num_offsets; i++)
{
uint64 key;
@@ -413,7 +414,10 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
{
uint64 key = key_base + i;
- tidstore_insert_kv(ts, key, values[i]);
+ if (TidStoreIsShared(ts))
+ shared_rt_set(ts->tree.shared, key, values[i]);
+ else
+ local_rt_set(ts->tree.local, key, values[i]);
}
}
@@ -449,8 +453,11 @@ tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
}
/*
- * Prepare to iterate through a TidStore. The caller must be certain that
- * no other backend will attempt to update the TidStore during the iteration.
+ * Prepare to iterate through a TidStore. Since the radix tree is locked during the
+ * iteration, so tidstore_end_iterate() needs to called when finished.
+ *
+ * Concurrent updates during the iteration will be blocked when inserting a
+ * key-value to the radix tree.
*/
TidStoreIter *
tidstore_begin_iterate(TidStore *ts)
@@ -482,13 +489,14 @@ tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
{
if (TidStoreIsShared(iter->ts))
return shared_rt_iterate_next(iter->tree_iter.shared, key, val);
- else
- return local_rt_iterate_next(iter->tree_iter.local, key, val);
+
+ return local_rt_iterate_next(iter->tree_iter.local, key, val);
}
/*
- * Scan the TidStore and return a TidStoreIterResult representing Tids
- * in one page. Offset numbers in the result is sorted.
+ * Scan the TidStore and return a pointer to TidStoreIterResult that has tids
+ * in one block. We return the block numbers in ascending order and the offset
+ * numbers in each result is also sorted in ascending order.
*/
TidStoreIterResult *
tidstore_iterate_next(TidStoreIter *iter)
@@ -502,6 +510,7 @@ tidstore_iterate_next(TidStoreIter *iter)
if (BlockNumberIsValid(result->blkno))
{
+ /* Process the previously collected key-value */
result->num_offsets = 0;
tidstore_iter_extract_tids(iter, iter->next_key, iter->next_val);
}
@@ -515,8 +524,8 @@ tidstore_iterate_next(TidStoreIter *iter)
if (BlockNumberIsValid(result->blkno) && result->blkno != blkno)
{
/*
- * Remember the key-value pair for the next block for the
- * next iteration.
+ * We got a key-value pair for a different block. So return the
+ * collected tids, and remember the key-value for the next iteration.
*/
iter->next_key = key;
iter->next_val = val;
@@ -531,7 +540,10 @@ tidstore_iterate_next(TidStoreIter *iter)
return result;
}
-/* Finish an iteration over TidStore */
+/*
+ * Finish an iteration over TidStore. This needs to be called after finishing
+ * or when existing an iteration.
+ */
void
tidstore_end_iterate(TidStoreIter *iter)
{
@@ -544,7 +556,7 @@ tidstore_end_iterate(TidStoreIter *iter)
pfree(iter);
}
-/* Return the number of Tids we collected so far */
+/* Return the number of tids we collected so far */
int64
tidstore_num_tids(TidStore *ts)
{
@@ -552,7 +564,7 @@ tidstore_num_tids(TidStore *ts)
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
- if (TidStoreIsShared(ts))
+ if (!TidStoreIsShared(ts))
return ts->control->num_tids;
LWLockAcquire(&ts->control->lock, LW_SHARED);
@@ -593,9 +605,8 @@ tidstore_memory_usage(TidStore *ts)
*/
if (TidStoreIsShared(ts))
return sizeof(TidStore) + shared_rt_memory_usage(ts->tree.shared);
- else
- return sizeof(TidStore) + sizeof(TidStore) +
- local_rt_memory_usage(ts->tree.local);
+
+ return sizeof(TidStore) + sizeof(TidStore) + local_rt_memory_usage(ts->tree.local);
}
/*
@@ -609,7 +620,7 @@ tidstore_get_handle(TidStore *ts)
return ts->control->handle;
}
-/* Extract Tids from the given key-value pair */
+/* Extract tids from the given key-value pair */
static void
tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
{
@@ -621,7 +632,10 @@ tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
OffsetNumber off;
if (i > iter->ts->control->max_offset)
+ {
+ Assert(!iter->ts->control->encode_tids);
break;
+ }
if ((val & (UINT64CONST(1) << i)) == 0)
continue;
@@ -644,8 +658,8 @@ key_get_blkno(TidStore *ts, uint64 key)
{
if (ts->control->encode_tids)
return (BlockNumber) (key >> ts->control->offset_key_nbits);
- else
- return (BlockNumber) key;
+
+ return (BlockNumber) key;
}
/* Encode a tid to key and offset */
--
2.31.1
[application/octet-stream] v24-0009-Update-vacuum-integration-patch-from-v23.patch (10.0K, ../../CAD21AoCSkEDYAqW3+JtSQQY+R2tDCrXA=PgNAUXUdUQ5gBOp0Q@mail.gmail.com/4-v24-0009-Update-vacuum-integration-patch-from-v23.patch)
download | inline diff:
From fd380a199f38545a56d7fa11c45ec088d62389f4 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Tue, 31 Jan 2023 22:44:40 +0900
Subject: [PATCH v24 9/9] Update vacuum integration patch from v23.
---
src/backend/access/heap/vacuumlazy.c | 64 +++++++++++++--------------
src/backend/commands/vacuumparallel.c | 11 +++--
2 files changed, 37 insertions(+), 38 deletions(-)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 3537df16fd..b4e40423a8 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3,18 +3,18 @@
* vacuumlazy.c
* Concurrent ("lazy") vacuuming.
*
- * The major space usage for vacuuming is storage for the array of dead TIDs
+ * The major space usage for vacuuming is TidStore, a storage for dead TIDs
* that are to be removed from indexes. We want to ensure we can vacuum even
* the very largest relations with finite memory space usage. To do that, we
- * set upper bounds on the number of TIDs we can keep track of at once.
+ * set upper bounds on the maximum memory that can be used for keeping track
+ * of dead TIDs at once.
*
* We are willing to use at most maintenance_work_mem (or perhaps
* autovacuum_work_mem) memory space to keep track of dead TIDs. We initially
- * allocate an array of TIDs of that size, with an upper limit that depends on
- * table size (this limit ensures we don't allocate a huge area uselessly for
- * vacuuming small tables). If the array threatens to overflow, we must call
- * lazy_vacuum to vacuum indexes (and to vacuum the pages that we've pruned).
- * This frees up the memory space dedicated to storing dead TIDs.
+ * create a TidStore with the maximum bytes that can be used by the TidStore.
+ * If the TidStore is full, we must call lazy_vacuum to vacuum indexes (and to
+ * vacuum the pages that we've pruned). This frees up the memory space dedicated
+ * to storing dead TIDs.
*
* In practice VACUUM will often complete its initial pass over the target
* heap relation without ever running out of space to store TIDs. This means
@@ -492,11 +492,11 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
}
/*
- * Allocate dead_items array memory using dead_items_alloc. This handles
- * parallel VACUUM initialization as part of allocating shared memory
- * space used for dead_items. (But do a failsafe precheck first, to
- * ensure that parallel VACUUM won't be attempted at all when relfrozenxid
- * is already dangerously old.)
+ * Allocate dead_items memory using dead_items_alloc. This handles parallel
+ * VACUUM initialization as part of allocating shared memory space used for
+ * dead_items. (But do a failsafe precheck first, to ensure that parallel
+ * VACUUM won't be attempted at all when relfrozenxid is already dangerously
+ * old.)
*/
lazy_check_wraparound_failsafe(vacrel);
dead_items_alloc(vacrel, params->nworkers);
@@ -802,7 +802,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* have collected the TIDs whose index tuples need to be removed.
*
* Finally, invokes lazy_vacuum_heap_rel to vacuum heap pages, which
- * largely consists of marking LP_DEAD items (from collected TID array)
+ * largely consists of marking LP_DEAD items (from vacrel->dead_items)
* as LP_UNUSED. This has to happen in a second, final pass over the
* heap, to preserve a basic invariant that all index AMs rely on: no
* extant index tuple can ever be allowed to contain a TID that points to
@@ -973,7 +973,7 @@ lazy_scan_heap(LVRelState *vacrel)
continue;
}
- /* Collect LP_DEAD items in dead_items array, count tuples */
+ /* Collect LP_DEAD items in dead_items, count tuples */
if (lazy_scan_noprune(vacrel, buf, blkno, page, &hastup,
&recordfreespace))
{
@@ -1015,10 +1015,10 @@ lazy_scan_heap(LVRelState *vacrel)
* Prune, freeze, and count tuples.
*
* Accumulates details of remaining LP_DEAD line pointers on page in
- * dead_items array. This includes LP_DEAD line pointers that we
- * pruned ourselves, as well as existing LP_DEAD line pointers that
- * were pruned some time earlier. Also considers freezing XIDs in the
- * tuple headers of remaining items with storage.
+ * dead_items. This includes LP_DEAD line pointers that we pruned
+ * ourselves, as well as existing LP_DEAD line pointers that were pruned
+ * some time earlier. Also considers freezing XIDs in the tuple headers
+ * of remaining items with storage.
*/
lazy_scan_prune(vacrel, buf, blkno, page, &prunestate);
@@ -1084,7 +1084,7 @@ lazy_scan_heap(LVRelState *vacrel)
}
else if (prunestate.num_offsets > 0)
{
- /* Save details of the LP_DEAD items from the page */
+ /* Save details of the LP_DEAD items from the page in dead_items */
tidstore_add_tids(dead_items, blkno, prunestate.deadoffsets,
prunestate.num_offsets);
@@ -1535,9 +1535,9 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* The approach we take now is to restart pruning when the race condition is
* detected. This allows heap_page_prune() to prune the tuples inserted by
* the now-aborted transaction. This is a little crude, but it guarantees
- * that any items that make it into the dead_items array are simple LP_DEAD
- * line pointers, and that every remaining item with tuple storage is
- * considered as a candidate for freezing.
+ * that any items that make it into the dead_items are simple LP_DEAD line
+ * pointers, and that every remaining item with tuple storage is considered
+ * as a candidate for freezing.
*/
static void
lazy_scan_prune(LVRelState *vacrel,
@@ -1929,7 +1929,7 @@ retry:
* lazy_scan_prune, which requires a full cleanup lock. While pruning isn't
* performed here, it's quite possible that an earlier opportunistic pruning
* operation left LP_DEAD items behind. We'll at least collect any such items
- * in the dead_items array for removal from indexes.
+ * in the dead_items for removal from indexes.
*
* For aggressive VACUUM callers, we may return false to indicate that a full
* cleanup lock is required for processing by lazy_scan_prune. This is only
@@ -2088,7 +2088,7 @@ lazy_scan_noprune(LVRelState *vacrel,
vacrel->NewRelfrozenXid = NoFreezePageRelfrozenXid;
vacrel->NewRelminMxid = NoFreezePageRelminMxid;
- /* Save any LP_DEAD items found on the page in dead_items array */
+ /* Save any LP_DEAD items found on the page in dead_items */
if (vacrel->nindexes == 0)
{
/* Using one-pass strategy (since table has no indexes) */
@@ -2373,9 +2373,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
/*
* lazy_vacuum_heap_rel() -- second pass over the heap for two pass strategy
*
- * This routine marks LP_DEAD items in vacrel->dead_items array as LP_UNUSED.
- * Pages that never had lazy_scan_prune record LP_DEAD items are not visited
- * at all.
+ * This routine marks LP_DEAD items in vacrel->dead_items as LP_UNUSED. Pages
+ * that never had lazy_scan_prune record LP_DEAD items are not visited at all.
*
* We may also be able to truncate the line pointer array of the heap pages we
* visit. If there is a contiguous group of LP_UNUSED items at the end of the
@@ -2461,7 +2460,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
ereport(DEBUG2,
(errmsg("table \"%s\": removed " UINT64_FORMAT "dead item identifiers in %u pages",
- vacrel->relname, tidstore_num_tids(vacrel->dead_items), vacuumed_pages)));
+ vacrel->relname, tidstore_num_tids(vacrel->dead_items),
+ vacuumed_pages)));
/* Revert to the previous phase information for error traceback */
restore_vacuum_error_info(vacrel, &saved_err_info);
@@ -2660,8 +2660,8 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
* lazy_vacuum_one_index() -- vacuum index relation.
*
* Delete all the index tuples containing a TID collected in
- * vacrel->dead_items array. Also update running statistics.
- * Exact details depend on index AM's ambulkdelete routine.
+ * vacrel->dead_items. Also update running statistics. Exact
+ * details depend on index AM's ambulkdelete routine.
*
* reltuples is the number of heap tuples to be passed to the
* bulkdelete callback. It's always assumed to be estimated.
@@ -3067,8 +3067,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
}
/*
- * Allocate dead_items (either using palloc, or in dynamic shared memory).
- * Sets dead_items in vacrel for caller.
+ * Allocate a (local or shared) TidStore for storing dead TIDs. Sets dead_items
+ * in vacrel for caller.
*
* Also handles parallel initialization as part of allocating dead_items in
* DSM when required.
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 5c7e6ed99c..d653683693 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -9,12 +9,11 @@
* In a parallel vacuum, we perform both index bulk deletion and index cleanup
* with parallel worker processes. Individual indexes are processed by one
* vacuum process. ParalleVacuumState contains shared information as well as
- * the memory space for storing dead items allocated in the DSM segment. We
- * launch parallel worker processes at the start of parallel index
- * bulk-deletion and index cleanup and once all indexes are processed, the
- * parallel worker processes exit. Each time we process indexes in parallel,
- * the parallel context is re-initialized so that the same DSM can be used for
- * multiple passes of index bulk-deletion and index cleanup.
+ * the shared TidStore. We launch parallel worker processes at the start of
+ * parallel index bulk-deletion and index cleanup and once all indexes are
+ * processed, the parallel worker processes exit. Each time we process indexes
+ * in parallel, the parallel context is re-initialized so that the same DSM can
+ * be used for multiple passes of index bulk-deletion and index cleanup.
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
--
2.31.1
[application/octet-stream] v24-0007-Use-TIDStore-for-storing-dead-tuple-TID-during-l.patch (40.1K, ../../CAD21AoCSkEDYAqW3+JtSQQY+R2tDCrXA=PgNAUXUdUQ5gBOp0Q@mail.gmail.com/5-v24-0007-Use-TIDStore-for-storing-dead-tuple-TID-during-l.patch)
download | inline diff:
From 850aff99cfddb2e77822d616248a4550cdae269c Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Tue, 17 Jan 2023 17:20:37 +0700
Subject: [PATCH v24 7/9] Use TIDStore for storing dead tuple TID during lazy
vacuum
Previously, we used an array of ItemPointerData to store dead tuple
TIDs, which was not space efficient and slow to lookup. Also, we had
the 1GB limit on its size.
Now we use TIDStore to store dead tuple TIDs. Since the TIDStore,
backed by the radix tree, incrementaly allocates the memory, we get
rid of the 1GB limit.
Since we are no longer able to exactly estimate the maximum number of
TIDs can be stored the pg_stat_progress_vacuum shows the progress
information based on the amount of memory in bytes. The column names
are also changed to max_dead_tuple_bytes and num_dead_tuple_bytes.
In addition, since the TIDStore use the radix tree internally, the
minimum amount of memory required by TIDStore is 1MB, the inital DSA
segment size. Due to that, we increase the minimum value of
maintenance_work_mem (also autovacuum_work_mem) from 1MB to 2MB.
XXX: needs to bump catalog version
---
doc/src/sgml/monitoring.sgml | 8 +-
src/backend/access/heap/vacuumlazy.c | 218 +++++++--------------
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/vacuum.c | 78 +-------
src/backend/commands/vacuumparallel.c | 62 +++---
src/backend/postmaster/autovacuum.c | 6 +-
src/backend/storage/lmgr/lwlock.c | 2 +
src/backend/utils/misc/guc_tables.c | 2 +-
src/include/commands/progress.h | 4 +-
src/include/commands/vacuum.h | 25 +--
src/include/storage/lwlock.h | 1 +
src/test/regress/expected/cluster.out | 2 +-
src/test/regress/expected/create_index.out | 2 +-
src/test/regress/expected/rules.out | 4 +-
src/test/regress/sql/cluster.sql | 2 +-
src/test/regress/sql/create_index.sql | 2 +-
16 files changed, 142 insertions(+), 278 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index d936aa3da3..0230c74e3d 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6870,10 +6870,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>max_dead_tuples</structfield> <type>bigint</type>
+ <structfield>max_dead_tuple_bytes</structfield> <type>bigint</type>
</para>
<para>
- Number of dead tuples that we can store before needing to perform
+ Amount of dead tuple data that we can store before needing to perform
an index vacuum cycle, based on
<xref linkend="guc-maintenance-work-mem"/>.
</para></entry>
@@ -6881,10 +6881,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>num_dead_tuples</structfield> <type>bigint</type>
+ <structfield>num_dead_tuple_bytes</structfield> <type>bigint</type>
</para>
<para>
- Number of dead tuples collected since the last index vacuum cycle.
+ Amount of dead tuple data collected since the last index vacuum cycle.
</para></entry>
</row>
</tbody>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 8f14cf85f3..3537df16fd 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -40,6 +40,7 @@
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
#include "access/multixact.h"
+#include "access/tidstore.h"
#include "access/transam.h"
#include "access/visibilitymap.h"
#include "access/xact.h"
@@ -188,7 +189,7 @@ typedef struct LVRelState
* lazy_vacuum_heap_rel, which marks the same LP_DEAD line pointers as
* LP_UNUSED during second heap pass.
*/
- VacDeadItems *dead_items; /* TIDs whose index tuples we'll delete */
+ TidStore *dead_items; /* TIDs whose index tuples we'll delete */
BlockNumber rel_pages; /* total number of pages */
BlockNumber scanned_pages; /* # pages examined (not skipped via VM) */
BlockNumber removed_pages; /* # pages removed by relation truncation */
@@ -220,11 +221,14 @@ typedef struct LVRelState
typedef struct LVPagePruneState
{
bool hastup; /* Page prevents rel truncation? */
- bool has_lpdead_items; /* includes existing LP_DEAD items */
+
+ /* collected offsets of LP_DEAD items including existing ones */
+ OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
+ int num_offsets;
/*
* State describes the proper VM bit states to set for the page following
- * pruning and freezing. all_visible implies !has_lpdead_items, but don't
+ * pruning and freezing. all_visible implies num_offsets == 0, but don't
* trust all_frozen result unless all_visible is also set to true.
*/
bool all_visible; /* Every item visible to all? */
@@ -259,8 +263,9 @@ static bool lazy_scan_noprune(LVRelState *vacrel, Buffer buf,
static void lazy_vacuum(LVRelState *vacrel);
static bool lazy_vacuum_all_indexes(LVRelState *vacrel);
static void lazy_vacuum_heap_rel(LVRelState *vacrel);
-static int lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
- Buffer buffer, int index, Buffer vmbuffer);
+static void lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
+ OffsetNumber *offsets, int num_offsets,
+ Buffer buffer, Buffer vmbuffer);
static bool lazy_check_wraparound_failsafe(LVRelState *vacrel);
static void lazy_cleanup_all_indexes(LVRelState *vacrel);
static IndexBulkDeleteResult *lazy_vacuum_one_index(Relation indrel,
@@ -825,21 +830,21 @@ lazy_scan_heap(LVRelState *vacrel)
blkno,
next_unskippable_block,
next_fsm_block_to_vacuum = 0;
- VacDeadItems *dead_items = vacrel->dead_items;
+ TidStore *dead_items = vacrel->dead_items;
Buffer vmbuffer = InvalidBuffer;
bool next_unskippable_allvis,
skipping_current_range;
const int initprog_index[] = {
PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_TOTAL_HEAP_BLKS,
- PROGRESS_VACUUM_MAX_DEAD_TUPLES
+ PROGRESS_VACUUM_MAX_DEAD_TUPLE_BYTES
};
int64 initprog_val[3];
/* Report that we're scanning the heap, advertising total # of blocks */
initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
initprog_val[1] = rel_pages;
- initprog_val[2] = dead_items->max_items;
+ initprog_val[2] = tidstore_max_memory(vacrel->dead_items);
pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
/* Set up an initial range of skippable blocks using the visibility map */
@@ -906,8 +911,7 @@ lazy_scan_heap(LVRelState *vacrel)
* dead_items TIDs, pause and do a cycle of vacuuming before we tackle
* this page.
*/
- Assert(dead_items->max_items >= MaxHeapTuplesPerPage);
- if (dead_items->max_items - dead_items->num_items < MaxHeapTuplesPerPage)
+ if (tidstore_is_full(vacrel->dead_items))
{
/*
* Before beginning index vacuuming, we release any pin we may
@@ -1018,7 +1022,7 @@ lazy_scan_heap(LVRelState *vacrel)
*/
lazy_scan_prune(vacrel, buf, blkno, page, &prunestate);
- Assert(!prunestate.all_visible || !prunestate.has_lpdead_items);
+ Assert(!prunestate.all_visible || (prunestate.num_offsets == 0));
/* Remember the location of the last page with nonremovable tuples */
if (prunestate.hastup)
@@ -1034,14 +1038,12 @@ lazy_scan_heap(LVRelState *vacrel)
* performed here can be thought of as the one-pass equivalent of
* a call to lazy_vacuum().
*/
- if (prunestate.has_lpdead_items)
+ if (prunestate.num_offsets > 0)
{
Size freespace;
- lazy_vacuum_heap_page(vacrel, blkno, buf, 0, vmbuffer);
-
- /* Forget the LP_DEAD items that we just vacuumed */
- dead_items->num_items = 0;
+ lazy_vacuum_heap_page(vacrel, blkno, prunestate.deadoffsets,
+ prunestate.num_offsets, buf, vmbuffer);
/*
* Periodically perform FSM vacuuming to make newly-freed
@@ -1078,7 +1080,16 @@ lazy_scan_heap(LVRelState *vacrel)
* with prunestate-driven visibility map and FSM steps (just like
* the two-pass strategy).
*/
- Assert(dead_items->num_items == 0);
+ Assert(tidstore_num_tids(dead_items) == 0);
+ }
+ else if (prunestate.num_offsets > 0)
+ {
+ /* Save details of the LP_DEAD items from the page */
+ tidstore_add_tids(dead_items, blkno, prunestate.deadoffsets,
+ prunestate.num_offsets);
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
+ tidstore_memory_usage(dead_items));
}
/*
@@ -1145,7 +1156,7 @@ lazy_scan_heap(LVRelState *vacrel)
* There should never be LP_DEAD items on a page with PD_ALL_VISIBLE
* set, however.
*/
- else if (prunestate.has_lpdead_items && PageIsAllVisible(page))
+ else if ((prunestate.num_offsets > 0) && PageIsAllVisible(page))
{
elog(WARNING, "page containing LP_DEAD items is marked as all-visible in relation \"%s\" page %u",
vacrel->relname, blkno);
@@ -1193,7 +1204,7 @@ lazy_scan_heap(LVRelState *vacrel)
* Final steps for block: drop cleanup lock, record free space in the
* FSM
*/
- if (prunestate.has_lpdead_items && vacrel->do_index_vacuuming)
+ if ((prunestate.num_offsets > 0) && vacrel->do_index_vacuuming)
{
/*
* Wait until lazy_vacuum_heap_rel() to save free space. This
@@ -1249,7 +1260,7 @@ lazy_scan_heap(LVRelState *vacrel)
* Do index vacuuming (call each index's ambulkdelete routine), then do
* related heap vacuuming
*/
- if (dead_items->num_items > 0)
+ if (tidstore_num_tids(dead_items) > 0)
lazy_vacuum(vacrel);
/*
@@ -1543,13 +1554,11 @@ lazy_scan_prune(LVRelState *vacrel,
HTSV_Result res;
int tuples_deleted,
tuples_frozen,
- lpdead_items,
live_tuples,
recently_dead_tuples;
int nnewlpdead;
HeapPageFreeze pagefrz;
int64 fpi_before = pgWalUsage.wal_fpi;
- OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
@@ -1571,7 +1580,6 @@ retry:
pagefrz.NoFreezePageRelminMxid = vacrel->NewRelminMxid;
tuples_deleted = 0;
tuples_frozen = 0;
- lpdead_items = 0;
live_tuples = 0;
recently_dead_tuples = 0;
@@ -1580,9 +1588,9 @@ retry:
*
* We count tuples removed by the pruning step as tuples_deleted. Its
* final value can be thought of as the number of tuples that have been
- * deleted from the table. It should not be confused with lpdead_items;
- * lpdead_items's final value can be thought of as the number of tuples
- * that were deleted from indexes.
+ * deleted from the table. It should not be confused with
+ * prunestate->deadoffsets; prunestate->deadoffsets's final value can
+ * be thought of as the number of tuples that were deleted from indexes.
*/
tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest,
InvalidTransactionId, 0, &nnewlpdead,
@@ -1593,7 +1601,7 @@ retry:
* requiring freezing among remaining tuples with storage
*/
prunestate->hastup = false;
- prunestate->has_lpdead_items = false;
+ prunestate->num_offsets = 0;
prunestate->all_visible = true;
prunestate->all_frozen = true;
prunestate->visibility_cutoff_xid = InvalidTransactionId;
@@ -1638,7 +1646,7 @@ retry:
* (This is another case where it's useful to anticipate that any
* LP_DEAD items will become LP_UNUSED during the ongoing VACUUM.)
*/
- deadoffsets[lpdead_items++] = offnum;
+ prunestate->deadoffsets[prunestate->num_offsets++] = offnum;
continue;
}
@@ -1875,7 +1883,7 @@ retry:
*/
#ifdef USE_ASSERT_CHECKING
/* Note that all_frozen value does not matter when !all_visible */
- if (prunestate->all_visible && lpdead_items == 0)
+ if (prunestate->all_visible && prunestate->num_offsets == 0)
{
TransactionId cutoff;
bool all_frozen;
@@ -1888,28 +1896,9 @@ retry:
}
#endif
- /*
- * Now save details of the LP_DEAD items from the page in vacrel
- */
- if (lpdead_items > 0)
+ if (prunestate->num_offsets > 0)
{
- VacDeadItems *dead_items = vacrel->dead_items;
- ItemPointerData tmp;
-
vacrel->lpdead_item_pages++;
- prunestate->has_lpdead_items = true;
-
- ItemPointerSetBlockNumber(&tmp, blkno);
-
- for (int i = 0; i < lpdead_items; i++)
- {
- ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
- dead_items->items[dead_items->num_items++] = tmp;
- }
-
- Assert(dead_items->num_items <= dead_items->max_items);
- pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
- dead_items->num_items);
/*
* It was convenient to ignore LP_DEAD items in all_visible earlier on
@@ -1928,7 +1917,7 @@ retry:
/* Finally, add page-local counts to whole-VACUUM counts */
vacrel->tuples_deleted += tuples_deleted;
vacrel->tuples_frozen += tuples_frozen;
- vacrel->lpdead_items += lpdead_items;
+ vacrel->lpdead_items += prunestate->num_offsets;
vacrel->live_tuples += live_tuples;
vacrel->recently_dead_tuples += recently_dead_tuples;
}
@@ -2129,8 +2118,7 @@ lazy_scan_noprune(LVRelState *vacrel,
}
else
{
- VacDeadItems *dead_items = vacrel->dead_items;
- ItemPointerData tmp;
+ TidStore *dead_items = vacrel->dead_items;
/*
* Page has LP_DEAD items, and so any references/TIDs that remain in
@@ -2139,17 +2127,10 @@ lazy_scan_noprune(LVRelState *vacrel,
*/
vacrel->lpdead_item_pages++;
- ItemPointerSetBlockNumber(&tmp, blkno);
+ tidstore_add_tids(dead_items, blkno, deadoffsets, lpdead_items);
- for (int i = 0; i < lpdead_items; i++)
- {
- ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
- dead_items->items[dead_items->num_items++] = tmp;
- }
-
- Assert(dead_items->num_items <= dead_items->max_items);
- pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
- dead_items->num_items);
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
+ tidstore_memory_usage(dead_items));
vacrel->lpdead_items += lpdead_items;
@@ -2198,7 +2179,7 @@ lazy_vacuum(LVRelState *vacrel)
if (!vacrel->do_index_vacuuming)
{
Assert(!vacrel->do_index_cleanup);
- vacrel->dead_items->num_items = 0;
+ tidstore_reset(vacrel->dead_items);
return;
}
@@ -2227,7 +2208,7 @@ lazy_vacuum(LVRelState *vacrel)
BlockNumber threshold;
Assert(vacrel->num_index_scans == 0);
- Assert(vacrel->lpdead_items == vacrel->dead_items->num_items);
+ Assert(vacrel->lpdead_items == tidstore_num_tids(vacrel->dead_items));
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2254,8 +2235,8 @@ lazy_vacuum(LVRelState *vacrel)
* cases then this may need to be reconsidered.
*/
threshold = (double) vacrel->rel_pages * BYPASS_THRESHOLD_PAGES;
- bypass = (vacrel->lpdead_item_pages < threshold &&
- vacrel->lpdead_items < MAXDEADITEMS(32L * 1024L * 1024L));
+ bypass = (vacrel->lpdead_item_pages < threshold) &&
+ tidstore_memory_usage(vacrel->dead_items) < (32L * 1024L * 1024L);
}
if (bypass)
@@ -2300,7 +2281,7 @@ lazy_vacuum(LVRelState *vacrel)
* Forget the LP_DEAD items that we just vacuumed (or just decided to not
* vacuum)
*/
- vacrel->dead_items->num_items = 0;
+ tidstore_reset(vacrel->dead_items);
}
/*
@@ -2373,7 +2354,7 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
* place).
*/
Assert(vacrel->num_index_scans > 0 ||
- vacrel->dead_items->num_items == vacrel->lpdead_items);
+ tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items);
Assert(allindexes || vacrel->failsafe_active);
/*
@@ -2410,10 +2391,11 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
static void
lazy_vacuum_heap_rel(LVRelState *vacrel)
{
- int index = 0;
BlockNumber vacuumed_pages = 0;
Buffer vmbuffer = InvalidBuffer;
LVSavedErrInfo saved_err_info;
+ TidStoreIter *iter;
+ TidStoreIterResult *result;
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2428,7 +2410,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
VACUUM_ERRCB_PHASE_VACUUM_HEAP,
InvalidBlockNumber, InvalidOffsetNumber);
- while (index < vacrel->dead_items->num_items)
+ iter = tidstore_begin_iterate(vacrel->dead_items);
+ while ((result = tidstore_iterate_next(iter)) != NULL)
{
BlockNumber blkno;
Buffer buf;
@@ -2437,7 +2420,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
vacuum_delay_point();
- blkno = ItemPointerGetBlockNumber(&vacrel->dead_items->items[index]);
+ blkno = result->blkno;
vacrel->blkno = blkno;
/*
@@ -2451,7 +2434,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
buf = ReadBufferExtended(vacrel->rel, MAIN_FORKNUM, blkno, RBM_NORMAL,
vacrel->bstrategy);
LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
- index = lazy_vacuum_heap_page(vacrel, blkno, buf, index, vmbuffer);
+ lazy_vacuum_heap_page(vacrel, blkno, result->offsets, result->num_offsets,
+ buf, vmbuffer);
/* Now that we've vacuumed the page, record its available space */
page = BufferGetPage(buf);
@@ -2461,6 +2445,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
vacuumed_pages++;
}
+ tidstore_end_iterate(iter);
vacrel->blkno = InvalidBlockNumber;
if (BufferIsValid(vmbuffer))
@@ -2470,36 +2455,30 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
* We set all LP_DEAD items from the first heap pass to LP_UNUSED during
* the second heap pass. No more, no less.
*/
- Assert(index > 0);
Assert(vacrel->num_index_scans > 1 ||
- (index == vacrel->lpdead_items &&
+ (tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items &&
vacuumed_pages == vacrel->lpdead_item_pages));
ereport(DEBUG2,
- (errmsg("table \"%s\": removed %lld dead item identifiers in %u pages",
- vacrel->relname, (long long) index, vacuumed_pages)));
+ (errmsg("table \"%s\": removed " UINT64_FORMAT "dead item identifiers in %u pages",
+ vacrel->relname, tidstore_num_tids(vacrel->dead_items), vacuumed_pages)));
/* Revert to the previous phase information for error traceback */
restore_vacuum_error_info(vacrel, &saved_err_info);
}
/*
- * lazy_vacuum_heap_page() -- free page's LP_DEAD items listed in the
- * vacrel->dead_items array.
+ * lazy_vacuum_heap_page() -- free page's LP_DEAD items.
*
* Caller must have an exclusive buffer lock on the buffer (though a full
* cleanup lock is also acceptable). vmbuffer must be valid and already have
* a pin on blkno's visibility map page.
- *
- * index is an offset into the vacrel->dead_items array for the first listed
- * LP_DEAD item on the page. The return value is the first index immediately
- * after all LP_DEAD items for the same page in the array.
*/
-static int
-lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
- int index, Buffer vmbuffer)
+static void
+lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
+ OffsetNumber *deadoffsets, int num_offsets, Buffer buffer,
+ Buffer vmbuffer)
{
- VacDeadItems *dead_items = vacrel->dead_items;
Page page = BufferGetPage(buffer);
OffsetNumber unused[MaxHeapTuplesPerPage];
int nunused = 0;
@@ -2518,16 +2497,11 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
START_CRIT_SECTION();
- for (; index < dead_items->num_items; index++)
+ for (int i = 0; i < num_offsets; i++)
{
- BlockNumber tblk;
- OffsetNumber toff;
ItemId itemid;
+ OffsetNumber toff = deadoffsets[i];
- tblk = ItemPointerGetBlockNumber(&dead_items->items[index]);
- if (tblk != blkno)
- break; /* past end of tuples for this block */
- toff = ItemPointerGetOffsetNumber(&dead_items->items[index]);
itemid = PageGetItemId(page, toff);
Assert(ItemIdIsDead(itemid) && !ItemIdHasStorage(itemid));
@@ -2597,7 +2571,6 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
/* Revert to the previous phase information for error traceback */
restore_vacuum_error_info(vacrel, &saved_err_info);
- return index;
}
/*
@@ -3093,46 +3066,6 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
return vacrel->nonempty_pages;
}
-/*
- * Returns the number of dead TIDs that VACUUM should allocate space to
- * store, given a heap rel of size vacrel->rel_pages, and given current
- * maintenance_work_mem setting (or current autovacuum_work_mem setting,
- * when applicable).
- *
- * See the comments at the head of this file for rationale.
- */
-static int
-dead_items_max_items(LVRelState *vacrel)
-{
- int64 max_items;
- int vac_work_mem = IsAutoVacuumWorkerProcess() &&
- autovacuum_work_mem != -1 ?
- autovacuum_work_mem : maintenance_work_mem;
-
- if (vacrel->nindexes > 0)
- {
- BlockNumber rel_pages = vacrel->rel_pages;
-
- max_items = MAXDEADITEMS(vac_work_mem * 1024L);
- max_items = Min(max_items, INT_MAX);
- max_items = Min(max_items, MAXDEADITEMS(MaxAllocSize));
-
- /* curious coding here to ensure the multiplication can't overflow */
- if ((BlockNumber) (max_items / MaxHeapTuplesPerPage) > rel_pages)
- max_items = rel_pages * MaxHeapTuplesPerPage;
-
- /* stay sane if small maintenance_work_mem */
- max_items = Max(max_items, MaxHeapTuplesPerPage);
- }
- else
- {
- /* One-pass case only stores a single heap page's TIDs at a time */
- max_items = MaxHeapTuplesPerPage;
- }
-
- return (int) max_items;
-}
-
/*
* Allocate dead_items (either using palloc, or in dynamic shared memory).
* Sets dead_items in vacrel for caller.
@@ -3143,11 +3076,9 @@ dead_items_max_items(LVRelState *vacrel)
static void
dead_items_alloc(LVRelState *vacrel, int nworkers)
{
- VacDeadItems *dead_items;
- int max_items;
-
- max_items = dead_items_max_items(vacrel);
- Assert(max_items >= MaxHeapTuplesPerPage);
+ int vac_work_mem = IsAutoVacuumWorkerProcess() &&
+ autovacuum_work_mem != -1 ?
+ autovacuum_work_mem * 1024L : maintenance_work_mem * 1024L;
/*
* Initialize state for a parallel vacuum. As of now, only one worker can
@@ -3174,7 +3105,7 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
else
vacrel->pvs = parallel_vacuum_init(vacrel->rel, vacrel->indrels,
vacrel->nindexes, nworkers,
- max_items,
+ vac_work_mem, MaxHeapTuplesPerPage,
vacrel->verbose ? INFO : DEBUG2,
vacrel->bstrategy);
@@ -3187,11 +3118,8 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
}
/* Serial VACUUM case */
- dead_items = (VacDeadItems *) palloc(vac_max_items_to_alloc_size(max_items));
- dead_items->max_items = max_items;
- dead_items->num_items = 0;
-
- vacrel->dead_items = dead_items;
+ vacrel->dead_items = tidstore_create(vac_work_mem, MaxHeapTuplesPerPage,
+ NULL);
}
/*
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 8608e3fa5b..a526e607fe 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1165,7 +1165,7 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuple_bytes, S.param7 AS dead_tuple_bytes
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 7b1a4b127e..d8e680ca20 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -97,7 +97,6 @@ static bool vacuum_rel(Oid relid, RangeVar *relation, VacuumParams *params,
static double compute_parallel_delay(void);
static VacOptValue get_vacoptval_from_boolean(DefElem *def);
static bool vac_tid_reaped(ItemPointer itemptr, void *state);
-static int vac_cmp_itemptr(const void *left, const void *right);
/*
* Primary entry point for manual VACUUM and ANALYZE commands
@@ -2303,16 +2302,16 @@ get_vacoptval_from_boolean(DefElem *def)
*/
IndexBulkDeleteResult *
vac_bulkdel_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat,
- VacDeadItems *dead_items)
+ TidStore *dead_items)
{
/* Do bulk deletion */
istat = index_bulk_delete(ivinfo, istat, vac_tid_reaped,
(void *) dead_items);
ereport(ivinfo->message_level,
- (errmsg("scanned index \"%s\" to remove %d row versions",
+ (errmsg("scanned index \"%s\" to remove " UINT64_FORMAT " row versions",
RelationGetRelationName(ivinfo->index),
- dead_items->num_items)));
+ tidstore_num_tids(dead_items))));
return istat;
}
@@ -2343,82 +2342,15 @@ vac_cleanup_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat)
return istat;
}
-/*
- * Returns the total required space for VACUUM's dead_items array given a
- * max_items value.
- */
-Size
-vac_max_items_to_alloc_size(int max_items)
-{
- Assert(max_items <= MAXDEADITEMS(MaxAllocSize));
-
- return offsetof(VacDeadItems, items) + sizeof(ItemPointerData) * max_items;
-}
-
/*
* vac_tid_reaped() -- is a particular tid deletable?
*
* This has the right signature to be an IndexBulkDeleteCallback.
- *
- * Assumes dead_items array is sorted (in ascending TID order).
*/
static bool
vac_tid_reaped(ItemPointer itemptr, void *state)
{
- VacDeadItems *dead_items = (VacDeadItems *) state;
- int64 litem,
- ritem,
- item;
- ItemPointer res;
-
- litem = itemptr_encode(&dead_items->items[0]);
- ritem = itemptr_encode(&dead_items->items[dead_items->num_items - 1]);
- item = itemptr_encode(itemptr);
-
- /*
- * Doing a simple bound check before bsearch() is useful to avoid the
- * extra cost of bsearch(), especially if dead items on the heap are
- * concentrated in a certain range. Since this function is called for
- * every index tuple, it pays to be really fast.
- */
- if (item < litem || item > ritem)
- return false;
-
- res = (ItemPointer) bsearch((void *) itemptr,
- (void *) dead_items->items,
- dead_items->num_items,
- sizeof(ItemPointerData),
- vac_cmp_itemptr);
-
- return (res != NULL);
-}
-
-/*
- * Comparator routines for use with qsort() and bsearch().
- */
-static int
-vac_cmp_itemptr(const void *left, const void *right)
-{
- BlockNumber lblk,
- rblk;
- OffsetNumber loff,
- roff;
-
- lblk = ItemPointerGetBlockNumber((ItemPointer) left);
- rblk = ItemPointerGetBlockNumber((ItemPointer) right);
-
- if (lblk < rblk)
- return -1;
- if (lblk > rblk)
- return 1;
-
- loff = ItemPointerGetOffsetNumber((ItemPointer) left);
- roff = ItemPointerGetOffsetNumber((ItemPointer) right);
-
- if (loff < roff)
- return -1;
- if (loff > roff)
- return 1;
+ TidStore *dead_items = (TidStore *) state;
- return 0;
+ return tidstore_lookup_tid(dead_items, itemptr);
}
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index bcd40c80a1..5c7e6ed99c 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -103,6 +103,9 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /* Handle of the shared TidStore */
+ tidstore_handle dead_items_handle;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -166,7 +169,8 @@ struct ParallelVacuumState
PVIndStats *indstats;
/* Shared dead items space among parallel vacuum workers */
- VacDeadItems *dead_items;
+ TidStore *dead_items;
+ dsa_area *dead_items_area;
/* Points to buffer usage area in DSM */
BufferUsage *buffer_usage;
@@ -222,20 +226,23 @@ static void parallel_vacuum_error_callback(void *arg);
*/
ParallelVacuumState *
parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
- int nrequested_workers, int max_items,
- int elevel, BufferAccessStrategy bstrategy)
+ int nrequested_workers, int vac_work_mem,
+ int max_offset, int elevel,
+ BufferAccessStrategy bstrategy)
{
ParallelVacuumState *pvs;
ParallelContext *pcxt;
PVShared *shared;
- VacDeadItems *dead_items;
+ TidStore *dead_items;
PVIndStats *indstats;
BufferUsage *buffer_usage;
WalUsage *wal_usage;
+ void *area_space;
+ dsa_area *dead_items_dsa;
bool *will_parallel_vacuum;
Size est_indstats_len;
Size est_shared_len;
- Size est_dead_items_len;
+ Size dsa_minsize = dsa_minimum_size();
int nindexes_mwm = 0;
int parallel_workers = 0;
int querylen;
@@ -283,9 +290,8 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_estimate_chunk(&pcxt->estimator, est_shared_len);
shm_toc_estimate_keys(&pcxt->estimator, 1);
- /* Estimate size for dead_items -- PARALLEL_VACUUM_KEY_DEAD_ITEMS */
- est_dead_items_len = vac_max_items_to_alloc_size(max_items);
- shm_toc_estimate_chunk(&pcxt->estimator, est_dead_items_len);
+ /* Estimate size for dead tuple DSA -- PARALLEL_VACUUM_KEY_DSA */
+ shm_toc_estimate_chunk(&pcxt->estimator, dsa_minsize);
shm_toc_estimate_keys(&pcxt->estimator, 1);
/*
@@ -351,6 +357,16 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_INDEX_STATS, indstats);
pvs->indstats = indstats;
+ /* Prepare DSA space for dead items */
+ area_space = shm_toc_allocate(pcxt->toc, dsa_minsize);
+ shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, area_space);
+ dead_items_dsa = dsa_create_in_place(area_space, dsa_minsize,
+ LWTRANCHE_PARALLEL_VACUUM_DSA,
+ pcxt->seg);
+ dead_items = tidstore_create(vac_work_mem, max_offset, dead_items_dsa);
+ pvs->dead_items = dead_items;
+ pvs->dead_items_area = dead_items_dsa;
+
/* Prepare shared information */
shared = (PVShared *) shm_toc_allocate(pcxt->toc, est_shared_len);
MemSet(shared, 0, est_shared_len);
@@ -360,6 +376,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
+ shared->dead_items_handle = tidstore_get_handle(dead_items);
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
@@ -368,15 +385,6 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_SHARED, shared);
pvs->shared = shared;
- /* Prepare the dead_items space */
- dead_items = (VacDeadItems *) shm_toc_allocate(pcxt->toc,
- est_dead_items_len);
- dead_items->max_items = max_items;
- dead_items->num_items = 0;
- MemSet(dead_items->items, 0, sizeof(ItemPointerData) * max_items);
- shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, dead_items);
- pvs->dead_items = dead_items;
-
/*
* Allocate space for each worker's BufferUsage and WalUsage; no need to
* initialize
@@ -434,6 +442,9 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
istats[i] = NULL;
}
+ tidstore_destroy(pvs->dead_items);
+ dsa_detach(pvs->dead_items_area);
+
DestroyParallelContext(pvs->pcxt);
ExitParallelMode();
@@ -442,7 +453,7 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
}
/* Returns the dead items space */
-VacDeadItems *
+TidStore *
parallel_vacuum_get_dead_items(ParallelVacuumState *pvs)
{
return pvs->dead_items;
@@ -940,7 +951,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
Relation *indrels;
PVIndStats *indstats;
PVShared *shared;
- VacDeadItems *dead_items;
+ TidStore *dead_items;
+ void *area_space;
+ dsa_area *dead_items_area;
BufferUsage *buffer_usage;
WalUsage *wal_usage;
int nindexes;
@@ -984,10 +997,10 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
PARALLEL_VACUUM_KEY_INDEX_STATS,
false);
- /* Set dead_items space */
- dead_items = (VacDeadItems *) shm_toc_lookup(toc,
- PARALLEL_VACUUM_KEY_DEAD_ITEMS,
- false);
+ /* Set dead items */
+ area_space = shm_toc_lookup(toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, false);
+ dead_items_area = dsa_attach_in_place(area_space, seg);
+ dead_items = tidstore_attach(dead_items_area, shared->dead_items_handle);
/* Set cost-based vacuum delay */
VacuumCostActive = (VacuumCostDelay > 0);
@@ -1033,6 +1046,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
InstrEndParallelQuery(&buffer_usage[ParallelWorkerNumber],
&wal_usage[ParallelWorkerNumber]);
+ tidstore_detach(pvs.dead_items);
+ dsa_detach(dead_items_area);
+
/* Pop the error context stack */
error_context_stack = errcallback.previous;
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index f5ea381c53..d88db3e1f8 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -3397,12 +3397,12 @@ check_autovacuum_work_mem(int *newval, void **extra, GucSource source)
return true;
/*
- * We clamp manually-set values to at least 1MB. Since
+ * We clamp manually-set values to at least 2MB. Since
* maintenance_work_mem is always set to at least this value, do the same
* here.
*/
- if (*newval < 1024)
- *newval = 1024;
+ if (*newval < 2048)
+ *newval = 2048;
return true;
}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 55b3a04097..c223a7dc94 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -192,6 +192,8 @@ static const char *const BuiltinTrancheNames[] = {
"LogicalRepLauncherDSA",
/* LWTRANCHE_LAUNCHER_HASH: */
"LogicalRepLauncherHash",
+ /* LWTRANCHE_PARALLEL_VACUUM_DSA: */
+ "ParallelVacuumDSA",
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index c5a95f5dcc..a8e7041395 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2312,7 +2312,7 @@ struct config_int ConfigureNamesInt[] =
GUC_UNIT_KB
},
&maintenance_work_mem,
- 65536, 1024, MAX_KILOBYTES,
+ 65536, 2048, MAX_KILOBYTES,
NULL, NULL, NULL
},
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index e5add41352..b209d3cf84 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -23,8 +23,8 @@
#define PROGRESS_VACUUM_HEAP_BLKS_SCANNED 2
#define PROGRESS_VACUUM_HEAP_BLKS_VACUUMED 3
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
-#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
-#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_MAX_DEAD_TUPLE_BYTES 5
+#define PROGRESS_VACUUM_DEAD_TUPLE_BYTES 6
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 689dbb7702..a3ebb169ef 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -17,6 +17,7 @@
#include "access/htup.h"
#include "access/genam.h"
#include "access/parallel.h"
+#include "access/tidstore.h"
#include "catalog/pg_class.h"
#include "catalog/pg_statistic.h"
#include "catalog/pg_type.h"
@@ -276,21 +277,6 @@ struct VacuumCutoffs
MultiXactId MultiXactCutoff;
};
-/*
- * VacDeadItems stores TIDs whose index tuples are deleted by index vacuuming.
- */
-typedef struct VacDeadItems
-{
- int max_items; /* # slots allocated in array */
- int num_items; /* current # of entries */
-
- /* Sorted array of TIDs to delete from indexes */
- ItemPointerData items[FLEXIBLE_ARRAY_MEMBER];
-} VacDeadItems;
-
-#define MAXDEADITEMS(avail_mem) \
- (((avail_mem) - offsetof(VacDeadItems, items)) / sizeof(ItemPointerData))
-
/* GUC parameters */
extern PGDLLIMPORT int default_statistics_target; /* PGDLLIMPORT for PostGIS */
extern PGDLLIMPORT int vacuum_freeze_min_age;
@@ -339,18 +325,17 @@ extern Relation vacuum_open_relation(Oid relid, RangeVar *relation,
LOCKMODE lmode);
extern IndexBulkDeleteResult *vac_bulkdel_one_index(IndexVacuumInfo *ivinfo,
IndexBulkDeleteResult *istat,
- VacDeadItems *dead_items);
+ TidStore *dead_items);
extern IndexBulkDeleteResult *vac_cleanup_one_index(IndexVacuumInfo *ivinfo,
IndexBulkDeleteResult *istat);
-extern Size vac_max_items_to_alloc_size(int max_items);
/* in commands/vacuumparallel.c */
extern ParallelVacuumState *parallel_vacuum_init(Relation rel, Relation *indrels,
int nindexes, int nrequested_workers,
- int max_items, int elevel,
- BufferAccessStrategy bstrategy);
+ int vac_work_mem, int max_offset,
+ int elevel, BufferAccessStrategy bstrategy);
extern void parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats);
-extern VacDeadItems *parallel_vacuum_get_dead_items(ParallelVacuumState *pvs);
+extern TidStore *parallel_vacuum_get_dead_items(ParallelVacuumState *pvs);
extern void parallel_vacuum_bulkdel_all_indexes(ParallelVacuumState *pvs,
long num_table_tuples,
int num_index_scans);
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index 07002fdfbe..537b34b30c 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -207,6 +207,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_PGSTATS_DATA,
LWTRANCHE_LAUNCHER_DSA,
LWTRANCHE_LAUNCHER_HASH,
+ LWTRANCHE_PARALLEL_VACUUM_DSA,
LWTRANCHE_FIRST_USER_DEFINED
} BuiltinTrancheIds;
diff --git a/src/test/regress/expected/cluster.out b/src/test/regress/expected/cluster.out
index 2eec483eaa..e04f50726f 100644
--- a/src/test/regress/expected/cluster.out
+++ b/src/test/regress/expected/cluster.out
@@ -526,7 +526,7 @@ create index cluster_sort on clstr_4 (hundred, thousand, tenthous);
-- ensure we don't use the index in CLUSTER nor the checking SELECTs
set enable_indexscan = off;
-- Use external sort:
-set maintenance_work_mem = '1MB';
+set maintenance_work_mem = '2MB';
cluster clstr_4 using cluster_sort;
select * from
(select hundred, lag(hundred) over () as lhundred,
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 6cd57e3eaa..d1889b9d10 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -1214,7 +1214,7 @@ DROP TABLE unlogged_hash_table;
-- CREATE INDEX hash_ovfl_index ON hash_ovfl_heap USING hash (x int4_ops);
-- Test hash index build tuplesorting. Force hash tuplesort using low
-- maintenance_work_mem setting and fillfactor:
-SET maintenance_work_mem = '1MB';
+SET maintenance_work_mem = '2MB';
CREATE INDEX hash_tuplesort_idx ON tenk1 USING hash (stringu1 name_ops) WITH (fillfactor = 10);
EXPLAIN (COSTS OFF)
SELECT count(*) FROM tenk1 WHERE stringu1 = 'TVAAAA';
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index e7a2f5856a..f6ae02eb14 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2020,8 +2020,8 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param3 AS heap_blks_scanned,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
- s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param6 AS max_dead_tuple_bytes,
+ s.param7 AS dead_tuple_bytes
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_recovery_prefetch| SELECT stats_reset,
diff --git a/src/test/regress/sql/cluster.sql b/src/test/regress/sql/cluster.sql
index a4cfaae807..a4cb5b98a5 100644
--- a/src/test/regress/sql/cluster.sql
+++ b/src/test/regress/sql/cluster.sql
@@ -258,7 +258,7 @@ create index cluster_sort on clstr_4 (hundred, thousand, tenthous);
set enable_indexscan = off;
-- Use external sort:
-set maintenance_work_mem = '1MB';
+set maintenance_work_mem = '2MB';
cluster clstr_4 using cluster_sort;
select * from
(select hundred, lag(hundred) over () as lhundred,
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index a3738833b2..edb5e4b4f3 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -367,7 +367,7 @@ DROP TABLE unlogged_hash_table;
-- Test hash index build tuplesorting. Force hash tuplesort using low
-- maintenance_work_mem setting and fillfactor:
-SET maintenance_work_mem = '1MB';
+SET maintenance_work_mem = '2MB';
CREATE INDEX hash_tuplesort_idx ON tenk1 USING hash (stringu1 name_ops) WITH (fillfactor = 10);
EXPLAIN (COSTS OFF)
SELECT count(*) FROM tenk1 WHERE stringu1 = 'TVAAAA';
--
2.31.1
[application/octet-stream] v24-0006-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch (34.4K, ../../CAD21AoCSkEDYAqW3+JtSQQY+R2tDCrXA=PgNAUXUdUQ5gBOp0Q@mail.gmail.com/6-v24-0006-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch)
download | inline diff:
From 9bb09e2742c2c8aa21802697c33fb3357f7516d9 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 23 Dec 2022 23:50:39 +0900
Subject: [PATCH v24 6/9] Add TIDStore, to store sets of TIDs (ItemPointerData)
efficiently.
The TIDStore is designed to store large sets of TIDs efficiently, and
is backed by the radix tree. A TID is encoded into 64-bit key and
value and inserted to the radix tree.
The TIDStore is not used for anything yet, aside from the test code,
but the follow up patch integrates the TIDStore with lazy vacuum,
reducing lazy vacuum memory usage and lifting the 1GB limit on its
size, by storing the list of dead TIDs more efficiently.
This includes a unit test module, in src/test/modules/test_tidstore.
---
doc/src/sgml/monitoring.sgml | 4 +
src/backend/access/common/Makefile | 1 +
src/backend/access/common/meson.build | 1 +
src/backend/access/common/tidstore.c | 674 ++++++++++++++++++
src/backend/storage/lmgr/lwlock.c | 2 +
src/include/access/tidstore.h | 49 ++
src/include/storage/lwlock.h | 1 +
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_tidstore/Makefile | 23 +
.../test_tidstore/expected/test_tidstore.out | 13 +
src/test/modules/test_tidstore/meson.build | 35 +
.../test_tidstore/sql/test_tidstore.sql | 7 +
.../test_tidstore/test_tidstore--1.0.sql | 8 +
.../modules/test_tidstore/test_tidstore.c | 195 +++++
.../test_tidstore/test_tidstore.control | 4 +
16 files changed, 1019 insertions(+)
create mode 100644 src/backend/access/common/tidstore.c
create mode 100644 src/include/access/tidstore.h
create mode 100644 src/test/modules/test_tidstore/Makefile
create mode 100644 src/test/modules/test_tidstore/expected/test_tidstore.out
create mode 100644 src/test/modules/test_tidstore/meson.build
create mode 100644 src/test/modules/test_tidstore/sql/test_tidstore.sql
create mode 100644 src/test/modules/test_tidstore/test_tidstore--1.0.sql
create mode 100644 src/test/modules/test_tidstore/test_tidstore.c
create mode 100644 src/test/modules/test_tidstore/test_tidstore.control
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1756f1a4b6..d936aa3da3 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2192,6 +2192,10 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<entry>Waiting to access a shared TID bitmap during a parallel bitmap
index scan.</entry>
</row>
+ <row>
+ <entry><literal>SharedTidStore</literal></entry>
+ <entry>Waiting to access a shared TID store.</entry>
+ </row>
<row>
<entry><literal>SharedTupleStore</literal></entry>
<entry>Waiting to access a shared tuple store during parallel
diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile
index b9aff0ccfd..67b8cc6108 100644
--- a/src/backend/access/common/Makefile
+++ b/src/backend/access/common/Makefile
@@ -27,6 +27,7 @@ OBJS = \
syncscan.o \
toast_compression.o \
toast_internals.o \
+ tidstore.o \
tupconvert.o \
tupdesc.o
diff --git a/src/backend/access/common/meson.build b/src/backend/access/common/meson.build
index f5ac17b498..fce19c09ce 100644
--- a/src/backend/access/common/meson.build
+++ b/src/backend/access/common/meson.build
@@ -15,6 +15,7 @@ backend_sources += files(
'syncscan.c',
'toast_compression.c',
'toast_internals.c',
+ 'tidstore.c',
'tupconvert.c',
'tupdesc.c',
)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
new file mode 100644
index 0000000000..89aea71945
--- /dev/null
+++ b/src/backend/access/common/tidstore.c
@@ -0,0 +1,674 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.c
+ * Tid (ItemPointerData) storage implementation.
+ *
+ * This module provides a in-memory data structure to store Tids (ItemPointer).
+ * Internally, a tid is encoded as a pair of 64-bit key and 64-bit value, and
+ * stored in the radix tree.
+ *
+ * A TidStore can be shared among parallel worker processes by passing DSA area
+ * to tidstore_create(). Other backends can attach to the shared TidStore by
+ * tidstore_attach().
+ *
+ * XXX: Only one process is allowed to iterate over the TidStore at a time.
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/access/common/tidstore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/tidstore.h"
+#include "miscadmin.h"
+#include "port/pg_bitutils.h"
+#include "storage/lwlock.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/*
+ * For encoding purposes, tids are represented as a pair of 64-bit key and
+ * 64-bit value. First, we construct 64-bit unsigned integer by combining
+ * the block number and the offset number. The number of bits used for the
+ * offset number is specified by max_offsets in tidstore_create(). We are
+ * frugal with the bits, because smaller keys could help keeping the radix
+ * tree shallow.
+ *
+ * For example, a tid of heap with 8kB blocks uses the lowest 9 bits for
+ * the offset number and uses the next 32 bits for the block number. That
+ * is, only 41 bits are used:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ *
+ * X = bits used for offset number
+ * Y = bits used for block number
+ * u = unused bit
+ * (high on the left, low on the right)
+ *
+ * 9 bits are enough for the offset number, because MaxHeapTuplesPerPage < 2^9
+ * on 8kB blocks.
+ *
+ * The 64-bit value is the bitmap representation of the lowest 6 bits
+ * (TIDSTORE_VALUE_NBITS) of the integer, and the rest 35 bits are used
+ * as the key:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ * |----| value
+ * |---------------------------------------------| key
+ *
+ * The maximum height of the radix tree is 5 in this case.
+ *
+ * If the number of bits for offset number fits in a 64-bit value, we don't
+ * encode tids but directly use the block number and the offset number as key
+ * and value, respectively.
+ */
+#define TIDSTORE_VALUE_NBITS 6 /* log(64, 2) */
+
+/* A magic value used to identify our TidStores. */
+#define TIDSTORE_MAGIC 0x826f6a10
+
+#define RT_PREFIX local_rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE uint64
+#include "lib/radixtree.h"
+
+#define RT_PREFIX shared_rt
+#define RT_SHMEM
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE uint64
+#include "lib/radixtree.h"
+
+/* The header object for a TidStore */
+typedef struct TidStoreControl
+{
+ int64 num_tids; /* the number of Tids stored so far */
+ size_t max_bytes; /* the maximum bytes a TidStore can use */
+ int max_offset; /* the maximum offset number */
+ bool encode_tids; /* do we use tid encoding? */
+ int offset_nbits; /* the number of bits used for offset number */
+ int offset_key_nbits; /* the number of bits of a offset number
+ * used for the key */
+
+ /* The below fields are used only in shared case */
+
+ uint32 magic;
+ LWLock lock;
+
+ /* handles for TidStore and radix tree */
+ tidstore_handle handle;
+ shared_rt_handle tree_handle;
+} TidStoreControl;
+
+/* Per-backend state for a TidStore */
+struct TidStore
+{
+ /*
+ * Control object. This is allocated in DSA area 'area' in the shared
+ * case, otherwise in backend-local memory.
+ */
+ TidStoreControl *control;
+
+ /* Storage for Tids. Use either one depending on TidStoreIsShared() */
+ union
+ {
+ local_rt_radix_tree *local;
+ shared_rt_radix_tree *shared;
+ } tree;
+
+ /* DSA area for TidStore if used */
+ dsa_area *area;
+};
+#define TidStoreIsShared(ts) ((ts)->area != NULL)
+
+/* Iterator for TidStore */
+typedef struct TidStoreIter
+{
+ TidStore *ts;
+
+ /* iterator of radix tree. Use either one depending on TidStoreIsShared() */
+ union
+ {
+ shared_rt_iter *shared;
+ local_rt_iter *local;
+ } tree_iter;
+
+ /* we returned all tids? */
+ bool finished;
+
+ /* save for the next iteration */
+ uint64 next_key;
+ uint64 next_val;
+
+ /* output for the caller */
+ TidStoreIterResult result;
+} TidStoreIter;
+
+static void tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val);
+static inline BlockNumber key_get_blkno(TidStore *ts, uint64 key);
+static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off);
+
+/*
+ * Create a TidStore. The returned object is allocated in backend-local memory.
+ * The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
+ */
+TidStore *
+tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
+{
+ TidStore *ts;
+
+ ts = palloc0(sizeof(TidStore));
+
+ /*
+ * Create the radix tree for the main storage.
+ *
+ * Memory consumption depends on the number of Tids stored, but also on the
+ * distribution of them, how the radix tree stores, and the memory management
+ * that backed the radix tree. The maximum bytes that a TidStore can
+ * use is specified by the max_bytes in tidstore_create(). We want the total
+ * amount of memory consumption not to exceed the max_bytes.
+ *
+ * In non-shared cases, the radix tree uses slab allocators for each kind of
+ * node class. The most memory consuming case while adding Tids associated
+ * with one page (i.e. during tidstore_add_tids()) is that we allocate the
+ * largest radix tree node in a new slab block, which is approximately 70kB.
+ * Therefore, we deduct 70kB from the maximum bytes.
+ *
+ * In shared cases, DSA allocates the memory segments big enough to follow
+ * a geometric series that approximately doubles the total DSA size (see
+ * make_new_segment() in dsa.c). We simulated the how DSA increases segment
+ * size and the simulation revealed, the 75% threshold for the maximum bytes
+ * perfectly works in case where it is a power-of-2, and the 60% threshold
+ * works for other cases.
+ */
+ if (area != NULL)
+ {
+ dsa_pointer dp;
+ float ratio = ((max_bytes & (max_bytes - 1)) == 0) ? 0.75 : 0.6;
+
+ ts->tree.shared = shared_rt_create(CurrentMemoryContext, area,
+ LWTRANCHE_SHARED_TIDSTORE);
+
+ dp = dsa_allocate0(area, sizeof(TidStoreControl));
+ ts->control = (TidStoreControl *) dsa_get_address(area, dp);
+ ts->control->max_bytes =(uint64) (max_bytes * ratio);
+ ts->area = area;
+
+ ts->control->magic = TIDSTORE_MAGIC;
+ LWLockInitialize(&ts->control->lock, LWTRANCHE_SHARED_TIDSTORE);
+ ts->control->handle = dp;
+ ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+ }
+ else
+ {
+ ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+ ts->control = (TidStoreControl *) palloc0(sizeof(TidStoreControl));
+ ts->control->max_bytes = max_bytes - (1024 * 70);
+ }
+
+ ts->control->max_offset = max_offset;
+ ts->control->offset_nbits = pg_ceil_log2_32(max_offset);
+
+ if (ts->control->offset_nbits > TIDSTORE_VALUE_NBITS)
+ {
+ ts->control->encode_tids = true;
+ ts->control->offset_key_nbits =
+ ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
+ }
+ else
+ {
+ ts->control->encode_tids = false;
+ ts->control->offset_key_nbits = 0;
+ }
+
+ return ts;
+}
+
+/*
+ * Attach to the shared TidStore using a handle. The returned object is
+ * allocated in backend-local memory using the CurrentMemoryContext.
+ */
+TidStore *
+tidstore_attach(dsa_area *area, tidstore_handle handle)
+{
+ TidStore *ts;
+ dsa_pointer control;
+
+ Assert(area != NULL);
+ Assert(DsaPointerIsValid(handle));
+
+ /* create per-backend state */
+ ts = palloc0(sizeof(TidStore));
+
+ /* Find the control object in shared memory */
+ control = handle;
+
+ /* Set up the TidStore */
+ ts->control = (TidStoreControl *) dsa_get_address(area, control);
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ ts->tree.shared = shared_rt_attach(area, ts->control->tree_handle);
+ ts->area = area;
+
+ return ts;
+}
+
+/*
+ * Detach from a TidStore. This detaches from radix tree and frees the
+ * backend-local resources. The radix tree will continue to exist until
+ * it is either explicitly destroyed, or the area that backs it is returned
+ * to the operating system.
+ */
+void
+tidstore_detach(TidStore *ts)
+{
+ Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+ shared_rt_detach(ts->tree.shared);
+ pfree(ts);
+}
+
+/*
+ * Destroy a TidStore, returning all memory.
+ *
+ * TODO: The caller must be certain that no other backend will attempt to
+ * access the TidStore before calling this function. Other backend must
+ * explicitly call tidstore_detach to free up backend-local memory associated
+ * with the TidStore. The backend that calls tidstore_destroy must not call
+ * tidstore_detach.
+ */
+void
+tidstore_destroy(TidStore *ts)
+{
+ if (TidStoreIsShared(ts))
+ {
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ /*
+ * Vandalize the control block to help catch programming error where
+ * other backends access the memory formerly occupied by this radix
+ * tree.
+ */
+ ts->control->magic = 0;
+ dsa_free(ts->area, ts->control->handle);
+ shared_rt_free(ts->tree.shared);
+ }
+ else
+ {
+ pfree(ts->control);
+ local_rt_free(ts->tree.local);
+ }
+
+ pfree(ts);
+}
+
+/* Forget all collected Tids */
+void
+tidstore_reset(TidStore *ts)
+{
+ if (TidStoreIsShared(ts))
+ {
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+ /*
+ * Free the radix tree and return allocated DSA segments to
+ * the operating system.
+ */
+ shared_rt_free(ts->tree.shared);
+ dsa_trim(ts->area);
+
+ /* Recreate the radix tree */
+ ts->tree.shared = shared_rt_create(CurrentMemoryContext, ts->area,
+ LWTRANCHE_SHARED_TIDSTORE);
+
+ /* update the radix tree handle as we recreated it */
+ ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
+
+ LWLockRelease(&ts->control->lock);
+ }
+ else
+ {
+ local_rt_free(ts->tree.local);
+ ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
+ }
+}
+
+static inline void
+tidstore_insert_kv(TidStore *ts, uint64 key, uint64 val)
+{
+ if (TidStoreIsShared(ts))
+ shared_rt_set(ts->tree.shared, key, val);
+ else
+ local_rt_set(ts->tree.local, key, val);
+}
+
+/* Add Tids on a block to TidStore */
+void
+tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets)
+{
+ ItemPointerData tid;
+ uint64 key_base;
+ uint64 *values;
+ int nkeys;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ ItemPointerSetBlockNumber(&tid, blkno);
+
+ if (ts->control->encode_tids)
+ {
+ key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
+ nkeys = UINT64CONST(1) << ts->control->offset_key_nbits;
+ }
+ else
+ {
+ key_base = (uint64) blkno;
+ nkeys = 1;
+ }
+
+ values = palloc0(sizeof(uint64) * nkeys);
+
+ for (int i = 0; i < num_offsets; i++)
+ {
+ uint64 key;
+ uint32 off;
+ int idx;
+
+ ItemPointerSetOffsetNumber(&tid, offsets[i]);
+
+ /* encode the tid to key and val */
+ key = tid_to_key_off(ts, &tid, &off);
+
+ idx = key - key_base;
+ Assert(idx >= 0 && idx < nkeys);
+
+ values[idx] |= UINT64CONST(1) << off;
+ }
+
+ if (TidStoreIsShared(ts))
+ LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+ /* insert the calculated key-values to the tree */
+ for (int i = 0; i < nkeys; i++)
+ {
+ if (values[i])
+ {
+ uint64 key = key_base + i;
+
+ tidstore_insert_kv(ts, key, values[i]);
+ }
+ }
+
+ /* update statistics */
+ ts->control->num_tids += num_offsets;
+
+ if (TidStoreIsShared(ts))
+ LWLockRelease(&ts->control->lock);
+
+ pfree(values);
+}
+
+/* Return true if the given tid is present in the TidStore */
+bool
+tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
+{
+ uint64 key;
+ uint64 val = 0;
+ uint32 off;
+ bool found;
+
+ key = tid_to_key_off(ts, tid, &off);
+
+ if (TidStoreIsShared(ts))
+ found = shared_rt_search(ts->tree.shared, key, &val);
+ else
+ found = local_rt_search(ts->tree.local, key, &val);
+
+ if (!found)
+ return false;
+
+ return (val & (UINT64CONST(1) << off)) != 0;
+}
+
+/*
+ * Prepare to iterate through a TidStore. The caller must be certain that
+ * no other backend will attempt to update the TidStore during the iteration.
+ */
+TidStoreIter *
+tidstore_begin_iterate(TidStore *ts)
+{
+ TidStoreIter *iter;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ iter = palloc0(sizeof(TidStoreIter));
+ iter->ts = ts;
+
+ iter->result.blkno = InvalidBlockNumber;
+ iter->result.offsets = palloc(sizeof(OffsetNumber) * ts->control->max_offset);
+
+ if (TidStoreIsShared(ts))
+ iter->tree_iter.shared = shared_rt_begin_iterate(ts->tree.shared);
+ else
+ iter->tree_iter.local = local_rt_begin_iterate(ts->tree.local);
+
+ /* If the TidStore is empty, there is no business */
+ if (tidstore_num_tids(ts) == 0)
+ iter->finished = true;
+
+ return iter;
+}
+
+static inline bool
+tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
+{
+ if (TidStoreIsShared(iter->ts))
+ return shared_rt_iterate_next(iter->tree_iter.shared, key, val);
+ else
+ return local_rt_iterate_next(iter->tree_iter.local, key, val);
+}
+
+/*
+ * Scan the TidStore and return a TidStoreIterResult representing Tids
+ * in one page. Offset numbers in the result is sorted.
+ */
+TidStoreIterResult *
+tidstore_iterate_next(TidStoreIter *iter)
+{
+ uint64 key;
+ uint64 val;
+ TidStoreIterResult *result = &(iter->result);
+
+ if (iter->finished)
+ return NULL;
+
+ if (BlockNumberIsValid(result->blkno))
+ {
+ result->num_offsets = 0;
+ tidstore_iter_extract_tids(iter, iter->next_key, iter->next_val);
+ }
+
+ while (tidstore_iter_kv(iter, &key, &val))
+ {
+ BlockNumber blkno;
+
+ blkno = key_get_blkno(iter->ts, key);
+
+ if (BlockNumberIsValid(result->blkno) && result->blkno != blkno)
+ {
+ /*
+ * Remember the key-value pair for the next block for the
+ * next iteration.
+ */
+ iter->next_key = key;
+ iter->next_val = val;
+ return result;
+ }
+
+ /* Collect tids extracted from the key-value pair */
+ tidstore_iter_extract_tids(iter, key, val);
+ }
+
+ iter->finished = true;
+ return result;
+}
+
+/* Finish an iteration over TidStore */
+void
+tidstore_end_iterate(TidStoreIter *iter)
+{
+ if (TidStoreIsShared(iter->ts))
+ shared_rt_end_iterate(iter->tree_iter.shared);
+ else
+ local_rt_end_iterate(iter->tree_iter.local);
+
+ pfree(iter->result.offsets);
+ pfree(iter);
+}
+
+/* Return the number of Tids we collected so far */
+int64
+tidstore_num_tids(TidStore *ts)
+{
+ uint64 num_tids;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ if (TidStoreIsShared(ts))
+ return ts->control->num_tids;
+
+ LWLockAcquire(&ts->control->lock, LW_SHARED);
+ num_tids = ts->control->num_tids;
+ LWLockRelease(&ts->control->lock);
+
+ return num_tids;
+}
+
+/* Return true if the current memory usage of TidStore exceeds the limit */
+bool
+tidstore_is_full(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ return (tidstore_memory_usage(ts) > ts->control->max_bytes);
+}
+
+/* Return the maximum memory TidStore can use */
+size_t
+tidstore_max_memory(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ return ts->control->max_bytes;
+}
+
+/* Return the memory usage of TidStore */
+size_t
+tidstore_memory_usage(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ /*
+ * In the shared case, TidStoreControl and radix_tree are backed by the
+ * same DSA area and rt_memory_usage() returns the value including both.
+ * So we don't need to add the size of TidStoreControl separately.
+ */
+ if (TidStoreIsShared(ts))
+ return sizeof(TidStore) + shared_rt_memory_usage(ts->tree.shared);
+ else
+ return sizeof(TidStore) + sizeof(TidStore) +
+ local_rt_memory_usage(ts->tree.local);
+}
+
+/*
+ * Get a handle that can be used by other processes to attach to this TidStore
+ */
+tidstore_handle
+tidstore_get_handle(TidStore *ts)
+{
+ Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+ return ts->control->handle;
+}
+
+/* Extract Tids from the given key-value pair */
+static void
+tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
+{
+ TidStoreIterResult *result = (&iter->result);
+
+ for (int i = 0; i < sizeof(uint64) * BITS_PER_BYTE; i++)
+ {
+ uint64 tid_i;
+ OffsetNumber off;
+
+ if (i > iter->ts->control->max_offset)
+ break;
+
+ if ((val & (UINT64CONST(1) << i)) == 0)
+ continue;
+
+ tid_i = key << TIDSTORE_VALUE_NBITS;
+ tid_i |= i;
+
+ off = tid_i & ((UINT64CONST(1) << iter->ts->control->offset_nbits) - 1);
+
+ Assert(result->num_offsets < iter->ts->control->max_offset);
+ result->offsets[result->num_offsets++] = off;
+ }
+
+ result->blkno = key_get_blkno(iter->ts, key);
+}
+
+/* Get block number from the given key */
+static inline BlockNumber
+key_get_blkno(TidStore *ts, uint64 key)
+{
+ if (ts->control->encode_tids)
+ return (BlockNumber) (key >> ts->control->offset_key_nbits);
+ else
+ return (BlockNumber) key;
+}
+
+/* Encode a tid to key and offset */
+static inline uint64
+tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off)
+{
+ uint64 key;
+ uint64 tid_i;
+
+ if (!ts->control->encode_tids)
+ {
+ *off = ItemPointerGetOffsetNumber(tid);
+
+ /* Use the block number as the key */
+ return (int64) ItemPointerGetBlockNumber(tid);
+ }
+
+ tid_i = ItemPointerGetOffsetNumber(tid);
+ tid_i |= (uint64) ItemPointerGetBlockNumber(tid) << ts->control->offset_nbits;
+
+ *off = tid_i & ((UINT64CONST(1) << TIDSTORE_VALUE_NBITS) - 1);
+ key = tid_i >> TIDSTORE_VALUE_NBITS;
+ Assert(*off < (sizeof(uint64) * BITS_PER_BYTE));
+
+ return key;
+}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index d2ec396045..55b3a04097 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -176,6 +176,8 @@ static const char *const BuiltinTrancheNames[] = {
"SharedTupleStore",
/* LWTRANCHE_SHARED_TIDBITMAP: */
"SharedTidBitmap",
+ /* LWTRANCHE_SHARED_TIDSTORE: */
+ "SharedTidStore",
/* LWTRANCHE_PARALLEL_APPEND: */
"ParallelAppend",
/* LWTRANCHE_PER_XACT_PREDICATE_LIST: */
diff --git a/src/include/access/tidstore.h b/src/include/access/tidstore.h
new file mode 100644
index 0000000000..a35a52124a
--- /dev/null
+++ b/src/include/access/tidstore.h
@@ -0,0 +1,49 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.h
+ * Tid storage.
+ *
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/tidstore.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef TIDSTORE_H
+#define TIDSTORE_H
+
+#include "storage/itemptr.h"
+#include "utils/dsa.h"
+
+typedef dsa_pointer tidstore_handle;
+
+typedef struct TidStore TidStore;
+typedef struct TidStoreIter TidStoreIter;
+
+typedef struct TidStoreIterResult
+{
+ BlockNumber blkno;
+ OffsetNumber *offsets;
+ int num_offsets;
+} TidStoreIterResult;
+
+extern TidStore *tidstore_create(size_t max_bytes, int max_offset, dsa_area *dsa);
+extern TidStore *tidstore_attach(dsa_area *dsa, dsa_pointer handle);
+extern void tidstore_detach(TidStore *ts);
+extern void tidstore_destroy(TidStore *ts);
+extern void tidstore_reset(TidStore *ts);
+extern void tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets);
+extern bool tidstore_lookup_tid(TidStore *ts, ItemPointer tid);
+extern TidStoreIter * tidstore_begin_iterate(TidStore *ts);
+extern TidStoreIterResult *tidstore_iterate_next(TidStoreIter *iter);
+extern void tidstore_end_iterate(TidStoreIter *iter);
+extern int64 tidstore_num_tids(TidStore *ts);
+extern bool tidstore_is_full(TidStore *ts);
+extern size_t tidstore_max_memory(TidStore *ts);
+extern size_t tidstore_memory_usage(TidStore *ts);
+extern tidstore_handle tidstore_get_handle(TidStore *ts);
+
+#endif /* TIDSTORE_H */
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index d2c7afb8f4..07002fdfbe 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -199,6 +199,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_PER_SESSION_RECORD_TYPMOD,
LWTRANCHE_SHARED_TUPLESTORE,
LWTRANCHE_SHARED_TIDBITMAP,
+ LWTRANCHE_SHARED_TIDSTORE,
LWTRANCHE_PARALLEL_APPEND,
LWTRANCHE_PER_XACT_PREDICATE_LIST,
LWTRANCHE_PGSTATS_DSA,
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 9659eb85d7..bddc16ada7 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
test_rls_hooks \
test_shm_mq \
test_slru \
+ test_tidstore \
unsafe_tests \
worker_spi
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 232cbdac80..c0d5645ad8 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -30,5 +30,6 @@ subdir('test_regex')
subdir('test_rls_hooks')
subdir('test_shm_mq')
subdir('test_slru')
+subdir('test_tidstore')
subdir('unsafe_tests')
subdir('worker_spi')
diff --git a/src/test/modules/test_tidstore/Makefile b/src/test/modules/test_tidstore/Makefile
new file mode 100644
index 0000000000..dab107d70c
--- /dev/null
+++ b/src/test/modules/test_tidstore/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_tidstore/Makefile
+
+MODULE_big = test_tidstore
+OBJS = \
+ $(WIN32RES) \
+ test_tidstore.o
+PGFILEDESC = "test_tidstore - test code for src/backend/access/common/tidstore.c"
+
+EXTENSION = test_tidstore
+DATA = test_tidstore--1.0.sql
+
+REGRESS = test_tidstore
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_tidstore
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_tidstore/expected/test_tidstore.out b/src/test/modules/test_tidstore/expected/test_tidstore.out
new file mode 100644
index 0000000000..7ff2f9af87
--- /dev/null
+++ b/src/test/modules/test_tidstore/expected/test_tidstore.out
@@ -0,0 +1,13 @@
+CREATE EXTENSION test_tidstore;
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
+NOTICE: testing empty tidstore
+NOTICE: testing basic operations
+ test_tidstore
+---------------
+
+(1 row)
+
diff --git a/src/test/modules/test_tidstore/meson.build b/src/test/modules/test_tidstore/meson.build
new file mode 100644
index 0000000000..31f2da7b61
--- /dev/null
+++ b/src/test/modules/test_tidstore/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_tidstore_sources = files(
+ 'test_tidstore.c',
+)
+
+if host_system == 'windows'
+ test_tidstore_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_tidstore',
+ '--FILEDESC', 'test_tidstore - test code for src/backend/access/common/tidstore.c',])
+endif
+
+test_tidstore = shared_module('test_tidstore',
+ test_tidstore_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += test_tidstore
+
+install_data(
+ 'test_tidstore.control',
+ 'test_tidstore--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'test_tidstore',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'test_tidstore',
+ ],
+ },
+}
diff --git a/src/test/modules/test_tidstore/sql/test_tidstore.sql b/src/test/modules/test_tidstore/sql/test_tidstore.sql
new file mode 100644
index 0000000000..03aea31815
--- /dev/null
+++ b/src/test/modules/test_tidstore/sql/test_tidstore.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_tidstore;
+
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
diff --git a/src/test/modules/test_tidstore/test_tidstore--1.0.sql b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
new file mode 100644
index 0000000000..47e9149900
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_tidstore/test_tidstore--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_tidstore" to load this file. \quit
+
+CREATE FUNCTION test_tidstore()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_tidstore/test_tidstore.c b/src/test/modules/test_tidstore/test_tidstore.c
new file mode 100644
index 0000000000..9b849ae8e8
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.c
@@ -0,0 +1,195 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_tidstore.c
+ * Test TidStore data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_tidstore/test_tidstore.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/tidstore.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "utils/memutils.h"
+
+PG_MODULE_MAGIC;
+
+#define TEST_TIDSTORE_MAX_BYTES (2 * 1024 * 1024L) /* 2MB */
+
+PG_FUNCTION_INFO_V1(test_tidstore);
+
+static void
+check_tid(TidStore *ts, BlockNumber blkno, OffsetNumber off, bool expect)
+{
+ ItemPointerData tid;
+ bool found;
+
+ ItemPointerSet(&tid, blkno, off);
+
+ found = tidstore_lookup_tid(ts, &tid);
+
+ if (found != expect)
+ elog(ERROR, "lookup TID (%u, %u) returned %d, expected %d",
+ blkno, off, found, expect);
+}
+
+static void
+test_basic(int max_offset)
+{
+#define TEST_TIDSTORE_NUM_BLOCKS 5
+#define TEST_TIDSTORE_NUM_OFFSETS 5
+
+ TidStore *ts;
+ TidStoreIter *iter;
+ TidStoreIterResult *iter_result;
+ BlockNumber blks[TEST_TIDSTORE_NUM_BLOCKS] = {
+ 0, MaxBlockNumber, MaxBlockNumber - 1, 1, MaxBlockNumber / 2,
+ };
+ BlockNumber blks_sorted[TEST_TIDSTORE_NUM_BLOCKS] = {
+ 0, 1, MaxBlockNumber / 2, MaxBlockNumber - 1, MaxBlockNumber
+ };
+ OffsetNumber offs[TEST_TIDSTORE_NUM_OFFSETS];
+ int blk_idx;
+
+ /* prepare the offset array */
+ offs[0] = FirstOffsetNumber;
+ offs[1] = FirstOffsetNumber + 1;
+ offs[2] = max_offset / 2;
+ offs[3] = max_offset - 1;
+ offs[4] = max_offset;
+
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
+
+ /* add tids */
+ for (int i = 0; i < TEST_TIDSTORE_NUM_BLOCKS; i++)
+ tidstore_add_tids(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
+
+ /* lookup test */
+ for (OffsetNumber off = FirstOffsetNumber ; off < max_offset; off++)
+ {
+ bool expect = false;
+ for (int i = 0; i < TEST_TIDSTORE_NUM_OFFSETS; i++)
+ {
+ if (offs[i] == off)
+ {
+ expect = true;
+ break;
+ }
+ }
+
+ check_tid(ts, 0, off, expect);
+ check_tid(ts, 2, off, false);
+ check_tid(ts, MaxBlockNumber - 2, off, false);
+ check_tid(ts, MaxBlockNumber, off, expect);
+ }
+
+ /* test the number of tids */
+ if (tidstore_num_tids(ts) != (TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS))
+ elog(ERROR, "tidstore_num_tids returned " UINT64_FORMAT ", expected %d",
+ tidstore_num_tids(ts),
+ TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS);
+
+ /* iteration test */
+ iter = tidstore_begin_iterate(ts);
+ blk_idx = 0;
+ while ((iter_result = tidstore_iterate_next(iter)) != NULL)
+ {
+ /* check the returned block number */
+ if (blks_sorted[blk_idx] != iter_result->blkno)
+ elog(ERROR, "tidstore_iterate_next returned block number %u, expected %u",
+ iter_result->blkno, blks_sorted[blk_idx]);
+
+ /* check the returned offset numbers */
+ if (TEST_TIDSTORE_NUM_OFFSETS != iter_result->num_offsets)
+ elog(ERROR, "tidstore_iterate_next returned %u offsets, expected %u",
+ iter_result->num_offsets, TEST_TIDSTORE_NUM_OFFSETS);
+
+ for (int i = 0; i < iter_result->num_offsets; i++)
+ {
+ if (offs[i] != iter_result->offsets[i])
+ elog(ERROR, "tidstore_iterate_next returned offset number %u on block %u, expected %u",
+ iter_result->offsets[i], iter_result->blkno, offs[i]);
+ }
+
+ blk_idx++;
+ }
+
+ if (blk_idx != TEST_TIDSTORE_NUM_BLOCKS)
+ elog(ERROR, "tidstore_iterate_next returned %d blocks, expected %d",
+ blk_idx, TEST_TIDSTORE_NUM_BLOCKS);
+
+ /* remove all tids */
+ tidstore_reset(ts);
+
+ /* test the number of tids */
+ if (tidstore_num_tids(ts) != 0)
+ elog(ERROR, "tidstore_num_tids on empty store returned non-zero");
+
+ /* lookup test for empty store */
+ for (OffsetNumber off = FirstOffsetNumber ; off < MaxHeapTuplesPerPage;
+ off++)
+ {
+ check_tid(ts, 0, off, false);
+ check_tid(ts, 2, off, false);
+ check_tid(ts, MaxBlockNumber - 2, off, false);
+ check_tid(ts, MaxBlockNumber, off, false);
+ }
+
+ tidstore_destroy(ts);
+}
+
+static void
+test_empty(void)
+{
+ TidStore *ts;
+ TidStoreIter *iter;
+ ItemPointerData tid;
+
+ elog(NOTICE, "testing empty tidstore");
+
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
+
+ ItemPointerSet(&tid, 0, FirstOffsetNumber);
+ if (tidstore_lookup_tid(ts, &tid))
+ elog(ERROR, "tidstore_lookup_tid for (0,1) on empty store returned true");
+
+ ItemPointerSet(&tid, MaxBlockNumber, MaxOffsetNumber);
+ if (tidstore_lookup_tid(ts, &tid))
+ elog(ERROR, "tidstore_lookup_tid for (%u,%u) on empty store returned true",
+ MaxBlockNumber, MaxOffsetNumber);
+
+ if (tidstore_num_tids(ts) != 0)
+ elog(ERROR, "tidstore_num_entries on empty store returned non-zero");
+
+ if (tidstore_is_full(ts))
+ elog(ERROR, "tidstore_is_full on empty store returned true");
+
+ iter = tidstore_begin_iterate(ts);
+
+ if (tidstore_iterate_next(iter) != NULL)
+ elog(ERROR, "tidstore_iterate_next on empty store returned TIDs");
+
+ tidstore_end_iterate(iter);
+
+ tidstore_destroy(ts);
+}
+
+Datum
+test_tidstore(PG_FUNCTION_ARGS)
+{
+ test_empty();
+
+ elog(NOTICE, "testing basic operations");
+ test_basic(MaxHeapTuplesPerPage);
+ test_basic(10);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_tidstore/test_tidstore.control b/src/test/modules/test_tidstore/test_tidstore.control
new file mode 100644
index 0000000000..9b6bd4638f
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.control
@@ -0,0 +1,4 @@
+comment = 'Test code for tidstore'
+default_version = '1.0'
+module_pathname = '$libdir/test_tidstore'
+relocatable = true
--
2.31.1
[application/octet-stream] v24-0002-Move-some-bitmap-logic-out-of-bitmapset.c.patch (6.1K, ../../CAD21AoCSkEDYAqW3+JtSQQY+R2tDCrXA=PgNAUXUdUQ5gBOp0Q@mail.gmail.com/7-v24-0002-Move-some-bitmap-logic-out-of-bitmapset.c.patch)
download | inline diff:
From f4ae4a7c957b5e9351607ffbd85cd044ed09c339 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 6 Dec 2022 13:39:41 +0700
Subject: [PATCH v24 2/9] Move some bitmap logic out of bitmapset.c
Add pg_rightmost_one32/64 functions. This functionality was previously
private to bitmapset.c as the RIGHTMOST_ONE macro. It has practical
use in other contexts, so move to pg_bitutils.h.
Also move the logic for selecting appropriate pg_bitutils functions
based on word size to bitmapset.h for wider visibility and add
appropriate selection for bmw_rightmost_one().
Since the previous macro relied on casting to signedbitmapword,
and the new functions do not, remove that typedef.
Design input and review by Tom Lane
Discussion: https://www.postgresql.org/message-id/CAFBsxsFW2JjTo58jtDB%2B3sZhxMx3t-3evew8%3DAcr%2BGGhC%2BkFaA%40mail.gmail.com
---
src/backend/nodes/bitmapset.c | 36 ++------------------------------
src/include/nodes/bitmapset.h | 16 ++++++++++++--
src/include/port/pg_bitutils.h | 31 +++++++++++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 -
4 files changed, 47 insertions(+), 37 deletions(-)
diff --git a/src/backend/nodes/bitmapset.c b/src/backend/nodes/bitmapset.c
index 98660524ad..fcd8e2ccbc 100644
--- a/src/backend/nodes/bitmapset.c
+++ b/src/backend/nodes/bitmapset.c
@@ -32,39 +32,7 @@
#define BITMAPSET_SIZE(nwords) \
(offsetof(Bitmapset, words) + (nwords) * sizeof(bitmapword))
-/*----------
- * This is a well-known cute trick for isolating the rightmost one-bit
- * in a word. It assumes two's complement arithmetic. Consider any
- * nonzero value, and focus attention on the rightmost one. The value is
- * then something like
- * xxxxxx10000
- * where x's are unspecified bits. The two's complement negative is formed
- * by inverting all the bits and adding one. Inversion gives
- * yyyyyy01111
- * where each y is the inverse of the corresponding x. Incrementing gives
- * yyyyyy10000
- * and then ANDing with the original value gives
- * 00000010000
- * This works for all cases except original value = zero, where of course
- * we get zero.
- *----------
- */
-#define RIGHTMOST_ONE(x) ((signedbitmapword) (x) & -((signedbitmapword) (x)))
-
-#define HAS_MULTIPLE_ONES(x) ((bitmapword) RIGHTMOST_ONE(x) != (x))
-
-/* Select appropriate bit-twiddling functions for bitmap word size */
-#if BITS_PER_BITMAPWORD == 32
-#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w)
-#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w)
-#define bmw_popcount(w) pg_popcount32(w)
-#elif BITS_PER_BITMAPWORD == 64
-#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w)
-#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w)
-#define bmw_popcount(w) pg_popcount64(w)
-#else
-#error "invalid BITS_PER_BITMAPWORD"
-#endif
+#define HAS_MULTIPLE_ONES(x) (bmw_rightmost_one(x) != (x))
/*
@@ -1013,7 +981,7 @@ bms_first_member(Bitmapset *a)
{
int result;
- w = RIGHTMOST_ONE(w);
+ w = bmw_rightmost_one(w);
a->words[wordnum] &= ~w;
result = wordnum * BITS_PER_BITMAPWORD;
diff --git a/src/include/nodes/bitmapset.h b/src/include/nodes/bitmapset.h
index 0dca6bc5fa..80e91fac0f 100644
--- a/src/include/nodes/bitmapset.h
+++ b/src/include/nodes/bitmapset.h
@@ -38,13 +38,11 @@ struct List;
#define BITS_PER_BITMAPWORD 64
typedef uint64 bitmapword; /* must be an unsigned type */
-typedef int64 signedbitmapword; /* must be the matching signed type */
#else
#define BITS_PER_BITMAPWORD 32
typedef uint32 bitmapword; /* must be an unsigned type */
-typedef int32 signedbitmapword; /* must be the matching signed type */
#endif
@@ -75,6 +73,20 @@ typedef enum
BMS_MULTIPLE /* >1 member */
} BMS_Membership;
+/* Select appropriate bit-twiddling functions for bitmap word size */
+#if BITS_PER_BITMAPWORD == 32
+#define bmw_rightmost_one(w) pg_rightmost_one32(w)
+#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w)
+#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w)
+#define bmw_popcount(w) pg_popcount32(w)
+#elif BITS_PER_BITMAPWORD == 64
+#define bmw_rightmost_one(w) pg_rightmost_one64(w)
+#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w)
+#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w)
+#define bmw_popcount(w) pg_popcount64(w)
+#else
+#error "invalid BITS_PER_BITMAPWORD"
+#endif
/*
* function prototypes in nodes/bitmapset.c
diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h
index a49a9c03d9..7235ad25ee 100644
--- a/src/include/port/pg_bitutils.h
+++ b/src/include/port/pg_bitutils.h
@@ -17,6 +17,37 @@ extern PGDLLIMPORT const uint8 pg_leftmost_one_pos[256];
extern PGDLLIMPORT const uint8 pg_rightmost_one_pos[256];
extern PGDLLIMPORT const uint8 pg_number_of_ones[256];
+/*----------
+ * This is a well-known cute trick for isolating the rightmost one-bit
+ * in a word. It assumes two's complement arithmetic. Consider any
+ * nonzero value, and focus attention on the rightmost one. The value is
+ * then something like
+ * xxxxxx10000
+ * where x's are unspecified bits. The two's complement negative is formed
+ * by inverting all the bits and adding one. Inversion gives
+ * yyyyyy01111
+ * where each y is the inverse of the corresponding x. Incrementing gives
+ * yyyyyy10000
+ * and then ANDing with the original value gives
+ * 00000010000
+ * This works for all cases except original value = zero, where of course
+ * we get zero.
+ *----------
+ */
+static inline uint32
+pg_rightmost_one32(uint32 word)
+{
+ int32 result = (int32) word & -((int32) word);
+ return (uint32) result;
+}
+
+static inline uint64
+pg_rightmost_one64(uint64 word)
+{
+ int64 result = (int64) word & -((int64) word);
+ return (uint64) result;
+}
+
/*
* pg_leftmost_one_pos32
* Returns the position of the most significant set bit in "word",
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 07fbb7ccf6..f4d1d60cd2 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3662,7 +3662,6 @@ shmem_request_hook_type
shmem_startup_hook_type
sig_atomic_t
sigjmp_buf
-signedbitmapword
sigset_t
size_t
slist_head
--
2.31.1
[application/octet-stream] v24-0001-introduce-vector8_min-and-vector8_highbit_mask.patch (2.7K, ../../CAD21AoCSkEDYAqW3+JtSQQY+R2tDCrXA=PgNAUXUdUQ5gBOp0Q@mail.gmail.com/8-v24-0001-introduce-vector8_min-and-vector8_highbit_mask.patch)
download | inline diff:
From a42eb01c87675698ae5972421f8896f85f048f2b Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 24 Oct 2022 14:07:09 +0900
Subject: [PATCH v24 1/9] introduce vector8_min and vector8_highbit_mask
TODO: commit message
TODO: Remove uint64 case.
separate-commit TODO: move non-SIMD fallbacks to own header
to clean up the #ifdef maze.
---
src/include/port/simd.h | 47 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)
diff --git a/src/include/port/simd.h b/src/include/port/simd.h
index c836360d4b..f0bba33c53 100644
--- a/src/include/port/simd.h
+++ b/src/include/port/simd.h
@@ -77,6 +77,7 @@ static inline bool vector8_has(const Vector8 v, const uint8 c);
static inline bool vector8_has_zero(const Vector8 v);
static inline bool vector8_has_le(const Vector8 v, const uint8 c);
static inline bool vector8_is_highbit_set(const Vector8 v);
+static inline uint32 vector8_highbit_mask(const Vector8 v);
#ifndef USE_NO_SIMD
static inline bool vector32_is_highbit_set(const Vector32 v);
#endif
@@ -96,6 +97,7 @@ static inline Vector8 vector8_ssub(const Vector8 v1, const Vector8 v2);
*/
#ifndef USE_NO_SIMD
static inline Vector8 vector8_eq(const Vector8 v1, const Vector8 v2);
+static inline Vector8 vector8_min(const Vector8 v1, const Vector8 v2);
static inline Vector32 vector32_eq(const Vector32 v1, const Vector32 v2);
#endif
@@ -277,6 +279,36 @@ vector8_is_highbit_set(const Vector8 v)
#endif
}
+/*
+ * Return the bitmask of the high-bit of each element.
+ */
+static inline uint32
+vector8_highbit_mask(const Vector8 v)
+{
+#ifdef USE_SSE2
+ return (uint32) _mm_movemask_epi8(v);
+#elif defined(USE_NEON)
+ static const uint8 mask[16] = {
+ 1 << 0, 1 << 1, 1 << 2, 1 << 3,
+ 1 << 4, 1 << 5, 1 << 6, 1 << 7,
+ 1 << 0, 1 << 1, 1 << 2, 1 << 3,
+ 1 << 4, 1 << 5, 1 << 6, 1 << 7,
+ };
+
+ uint8x16_t masked = vandq_u8(vld1q_u8(mask), (uint8x16_t) vshrq_n_s8(v, 7));
+ uint8x16_t maskedhi = vextq_u8(masked, masked, 8);
+
+ return (uint32) vaddvq_u16((uint16x8_t) vzip1q_u8(masked, maskedhi));
+#else
+ uint32 mask = 0;
+
+ for (Size i = 0; i < sizeof(Vector8); i++)
+ mask |= (((const uint8 *) &v)[i] >> 7) << i;
+
+ return mask;
+#endif
+}
+
/*
* Exactly like vector8_is_highbit_set except for the input type, so it
* looks at each byte separately.
@@ -372,4 +404,19 @@ vector32_eq(const Vector32 v1, const Vector32 v2)
}
#endif /* ! USE_NO_SIMD */
+/*
+ * Compare the given vectors and return the vector of minimum elements.
+ */
+#ifndef USE_NO_SIMD
+static inline Vector8
+vector8_min(const Vector8 v1, const Vector8 v2)
+{
+#ifdef USE_SSE2
+ return _mm_min_epu8(v1, v2);
+#elif defined(USE_NEON)
+ return vminq_u8(v1, v2);
+#endif
+}
+#endif /* ! USE_NO_SIMD */
+
#endif /* SIMD_H */
--
2.31.1
[application/octet-stream] v24-0003-Add-radixtree-template.patch (113.1K, ../../CAD21AoCSkEDYAqW3+JtSQQY+R2tDCrXA=PgNAUXUdUQ5gBOp0Q@mail.gmail.com/9-v24-0003-Add-radixtree-template.patch)
download | inline diff:
From 3d16fe0d216f4efb093dd880da02a6e54651d109 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 14 Sep 2022 12:38:51 +0000
Subject: [PATCH v24 3/9] Add radixtree template
The only thing configurable in this commit is function scope,
prefix, and local/shared memory.
The key and value type are still hard-coded to uint64.
(A later commit in v21 will make value type configurable)
It might be good at some point to offer a different tree type,
e.g. "single-value leaves" to allow for variable length keys
and values, giving full flexibility to developers.
TODO: Much broader commit message
---
src/backend/utils/mmgr/dsa.c | 12 +
src/include/lib/radixtree.h | 2426 +++++++++++++++++
src/include/lib/radixtree_delete_impl.h | 106 +
src/include/lib/radixtree_insert_impl.h | 317 +++
src/include/lib/radixtree_iter_impl.h | 138 +
src/include/lib/radixtree_search_impl.h | 122 +
src/include/utils/dsa.h | 1 +
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_radixtree/.gitignore | 4 +
src/test/modules/test_radixtree/Makefile | 23 +
src/test/modules/test_radixtree/README | 7 +
.../expected/test_radixtree.out | 36 +
src/test/modules/test_radixtree/meson.build | 35 +
.../test_radixtree/sql/test_radixtree.sql | 7 +
.../test_radixtree/test_radixtree--1.0.sql | 8 +
.../modules/test_radixtree/test_radixtree.c | 673 +++++
.../test_radixtree/test_radixtree.control | 4 +
src/tools/pginclude/cpluspluscheck | 6 +
src/tools/pginclude/headerscheck | 6 +
20 files changed, 3933 insertions(+)
create mode 100644 src/include/lib/radixtree.h
create mode 100644 src/include/lib/radixtree_delete_impl.h
create mode 100644 src/include/lib/radixtree_insert_impl.h
create mode 100644 src/include/lib/radixtree_iter_impl.h
create mode 100644 src/include/lib/radixtree_search_impl.h
create mode 100644 src/test/modules/test_radixtree/.gitignore
create mode 100644 src/test/modules/test_radixtree/Makefile
create mode 100644 src/test/modules/test_radixtree/README
create mode 100644 src/test/modules/test_radixtree/expected/test_radixtree.out
create mode 100644 src/test/modules/test_radixtree/meson.build
create mode 100644 src/test/modules/test_radixtree/sql/test_radixtree.sql
create mode 100644 src/test/modules/test_radixtree/test_radixtree--1.0.sql
create mode 100644 src/test/modules/test_radixtree/test_radixtree.c
create mode 100644 src/test/modules/test_radixtree/test_radixtree.control
diff --git a/src/backend/utils/mmgr/dsa.c b/src/backend/utils/mmgr/dsa.c
index f5a62061a3..80555aefff 100644
--- a/src/backend/utils/mmgr/dsa.c
+++ b/src/backend/utils/mmgr/dsa.c
@@ -1024,6 +1024,18 @@ dsa_set_size_limit(dsa_area *area, size_t limit)
LWLockRelease(DSA_AREA_LOCK(area));
}
+size_t
+dsa_get_total_size(dsa_area *area)
+{
+ size_t size;
+
+ LWLockAcquire(DSA_AREA_LOCK(area), LW_SHARED);
+ size = area->control->total_segment_size;
+ LWLockRelease(DSA_AREA_LOCK(area));
+
+ return size;
+}
+
/*
* Aggressively free all spare memory in the hope of returning DSM segments to
* the operating system.
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
new file mode 100644
index 0000000000..f591d903fc
--- /dev/null
+++ b/src/include/lib/radixtree.h
@@ -0,0 +1,2426 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree.h
+ * Implementation for adaptive radix tree.
+ *
+ * This module employs the idea from the paper "The Adaptive Radix Tree: ARTful
+ * Indexing for Main-Memory Databases" by Viktor Leis, Alfons Kemper, and Thomas
+ * Neumann, 2013. The radix tree uses adaptive node sizes, a small number of node
+ * types, each with a different numbers of elements. Depending on the number of
+ * children, the appropriate node type is used.
+ *
+ * WIP: notes about traditional radix tree trading off span vs height...
+ *
+ * There are two kinds of nodes, inner nodes and leaves. Inner nodes
+ * map partial keys to child pointers.
+ *
+ * The ART paper mentions three ways to implement leaves:
+ *
+ * "- Single-value leaves: The values are stored using an addi-
+ * tional leaf node type which stores one value.
+ * - Multi-value leaves: The values are stored in one of four
+ * different leaf node types, which mirror the structure of
+ * inner nodes, but contain values instead of pointers.
+ * - Combined pointer/value slots: If values fit into point-
+ * ers, no separate node types are necessary. Instead, each
+ * pointer storage location in an inner node can either
+ * store a pointer or a value."
+ *
+ * We chose "multi-value leaves" to avoid the additional pointer traversal
+ * required by "single-value leaves"
+ *
+ * For simplicity, the key is assumed to be 64-bit unsigned integer. The
+ * tree doesn't need to contain paths where the highest bytes of all keys
+ * are zero. That way, the tree's height adapts to the distribution of keys.
+ *
+ * TODO: In the future it might be worthwhile to offer configurability of
+ * leaf implementation for different use cases. Single-values leaves would
+ * give more flexibility in key type, including variable-length keys.
+ *
+ * There are some optimizations not yet implemented, particularly path
+ * compression and lazy path expansion.
+ *
+ * WIP: the radix tree nodes don't shrink.
+ *
+ * To generate a radix tree and associated functions for a use case several
+ * macros have to be #define'ed before this file is included. Including
+ * the file #undef's all those, so a new radix tree can be generated
+ * afterwards.
+ * The relevant parameters are:
+ * - RT_PREFIX - prefix for all symbol names generated. A prefix of 'foo'
+ * will result in radix tree type 'foo_radix_tree' and functions like
+ * 'foo_create'/'foo_free' and so forth.
+ * - RT_DECLARE - if defined function prototypes and type declarations are
+ * generated
+ * - RT_DEFINE - if defined function definitions are generated
+ * - RT_SCOPE - in which scope (e.g. extern, static inline) do function
+ * declarations reside
+ * - RT_VALUE_TYPE - the type of the value.
+ *
+ * Optional parameters:
+ * - RT_SHMEM - if defined, the radix tree is created in the DSA area
+ * so that multiple processes can access it simultaneously.
+ * - RT_DEBUG - if defined add stats tracking and debugging functions
+ *
+ * Interface
+ * ---------
+ *
+ * RT_CREATE - Create a new, empty radix tree
+ * RT_FREE - Free the radix tree
+ * RT_SEARCH - Search a key-value pair
+ * RT_SET - Set a key-value pair
+ * RT_BEGIN_ITERATE - Begin iterating through all key-value pairs
+ * RT_ITERATE_NEXT - Return next key-value pair, if any
+ * RT_END_ITER - End iteration
+ * RT_MEMORY_USAGE - Get the memory usage
+ *
+ * Interface for Shared Memory
+ * ---------
+ *
+ * RT_ATTACH - Attach to the radix tree
+ * RT_DETACH - Detach from the radix tree
+ * RT_GET_HANDLE - Return the handle of the radix tree
+ *
+ * Optional Interface
+ * ---------
+ *
+ * RT_DELETE - Delete a key-value pair. Declared/define if RT_USE_DELETE is defined
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/include/lib/radixtree.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/stringinfo.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "port/pg_bitutils.h"
+#include "port/simd.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/* helpers */
+#define RT_MAKE_PREFIX(a) CppConcat(a,_)
+#define RT_MAKE_NAME(name) RT_MAKE_NAME_(RT_MAKE_PREFIX(RT_PREFIX),name)
+#define RT_MAKE_NAME_(a,b) CppConcat(a,b)
+
+/* function declarations */
+#define RT_CREATE RT_MAKE_NAME(create)
+#define RT_FREE RT_MAKE_NAME(free)
+#define RT_SEARCH RT_MAKE_NAME(search)
+#ifdef RT_SHMEM
+#define RT_ATTACH RT_MAKE_NAME(attach)
+#define RT_DETACH RT_MAKE_NAME(detach)
+#define RT_GET_HANDLE RT_MAKE_NAME(get_handle)
+#endif
+#define RT_SET RT_MAKE_NAME(set)
+#define RT_BEGIN_ITERATE RT_MAKE_NAME(begin_iterate)
+#define RT_ITERATE_NEXT RT_MAKE_NAME(iterate_next)
+#define RT_END_ITERATE RT_MAKE_NAME(end_iterate)
+#ifdef RT_USE_DELETE
+#define RT_DELETE RT_MAKE_NAME(delete)
+#endif
+#define RT_MEMORY_USAGE RT_MAKE_NAME(memory_usage)
+#ifdef RT_DEBUG
+#define RT_DUMP RT_MAKE_NAME(dump)
+#define RT_DUMP_NODE RT_MAKE_NAME(dump_node)
+#define RT_DUMP_SEARCH RT_MAKE_NAME(dump_search)
+#define RT_STATS RT_MAKE_NAME(stats)
+#endif
+
+/* internal helper functions (no externally visible prototypes) */
+#define RT_NEW_ROOT RT_MAKE_NAME(new_root)
+#define RT_ALLOC_NODE RT_MAKE_NAME(alloc_node)
+#define RT_INIT_NODE RT_MAKE_NAME(init_node)
+#define RT_FREE_NODE RT_MAKE_NAME(free_node)
+#define RT_FREE_RECURSE RT_MAKE_NAME(free_recurse)
+#define RT_EXTEND RT_MAKE_NAME(extend)
+#define RT_SET_EXTEND RT_MAKE_NAME(set_extend)
+#define RT_SWITCH_NODE_KIND RT_MAKE_NAME(grow_node_kind)
+#define RT_COPY_NODE RT_MAKE_NAME(copy_node)
+#define RT_REPLACE_NODE RT_MAKE_NAME(replace_node)
+#define RT_PTR_GET_LOCAL RT_MAKE_NAME(ptr_get_local)
+#define RT_PTR_ALLOC_IS_VALID RT_MAKE_NAME(ptr_stored_is_valid)
+#define RT_NODE_3_SEARCH_EQ RT_MAKE_NAME(node_3_search_eq)
+#define RT_NODE_32_SEARCH_EQ RT_MAKE_NAME(node_32_search_eq)
+#define RT_NODE_3_GET_INSERTPOS RT_MAKE_NAME(node_3_get_insertpos)
+#define RT_NODE_32_GET_INSERTPOS RT_MAKE_NAME(node_32_get_insertpos)
+#define RT_CHUNK_CHILDREN_ARRAY_SHIFT RT_MAKE_NAME(chunk_children_array_shift)
+#define RT_CHUNK_VALUES_ARRAY_SHIFT RT_MAKE_NAME(chunk_values_array_shift)
+#define RT_CHUNK_CHILDREN_ARRAY_DELETE RT_MAKE_NAME(chunk_children_array_delete)
+#define RT_CHUNK_VALUES_ARRAY_DELETE RT_MAKE_NAME(chunk_values_array_delete)
+#define RT_CHUNK_CHILDREN_ARRAY_COPY RT_MAKE_NAME(chunk_children_array_copy)
+#define RT_CHUNK_VALUES_ARRAY_COPY RT_MAKE_NAME(chunk_values_array_copy)
+#define RT_NODE_125_IS_CHUNK_USED RT_MAKE_NAME(node_125_is_chunk_used)
+#define RT_NODE_INNER_125_GET_CHILD RT_MAKE_NAME(node_inner_125_get_child)
+#define RT_NODE_LEAF_125_GET_VALUE RT_MAKE_NAME(node_leaf_125_get_value)
+#define RT_NODE_INNER_256_IS_CHUNK_USED RT_MAKE_NAME(node_inner_256_is_chunk_used)
+#define RT_NODE_LEAF_256_IS_CHUNK_USED RT_MAKE_NAME(node_leaf_256_is_chunk_used)
+#define RT_NODE_INNER_256_GET_CHILD RT_MAKE_NAME(node_inner_256_get_child)
+#define RT_NODE_LEAF_256_GET_VALUE RT_MAKE_NAME(node_leaf_256_get_value)
+#define RT_NODE_INNER_256_SET RT_MAKE_NAME(node_inner_256_set)
+#define RT_NODE_LEAF_256_SET RT_MAKE_NAME(node_leaf_256_set)
+#define RT_NODE_INNER_256_DELETE RT_MAKE_NAME(node_inner_256_delete)
+#define RT_NODE_LEAF_256_DELETE RT_MAKE_NAME(node_leaf_256_delete)
+#define RT_KEY_GET_SHIFT RT_MAKE_NAME(key_get_shift)
+#define RT_SHIFT_GET_MAX_VAL RT_MAKE_NAME(shift_get_max_val)
+#define RT_NODE_SEARCH_INNER RT_MAKE_NAME(node_search_inner)
+#define RT_NODE_SEARCH_LEAF RT_MAKE_NAME(node_search_leaf)
+#define RT_NODE_UPDATE_INNER RT_MAKE_NAME(node_update_inner)
+#define RT_NODE_DELETE_INNER RT_MAKE_NAME(node_delete_inner)
+#define RT_NODE_DELETE_LEAF RT_MAKE_NAME(node_delete_leaf)
+#define RT_NODE_INSERT_INNER RT_MAKE_NAME(node_insert_inner)
+#define RT_NODE_INSERT_LEAF RT_MAKE_NAME(node_insert_leaf)
+#define RT_NODE_INNER_ITERATE_NEXT RT_MAKE_NAME(node_inner_iterate_next)
+#define RT_NODE_LEAF_ITERATE_NEXT RT_MAKE_NAME(node_leaf_iterate_next)
+#define RT_UPDATE_ITER_STACK RT_MAKE_NAME(update_iter_stack)
+#define RT_ITER_UPDATE_KEY RT_MAKE_NAME(iter_update_key)
+#define RT_VERIFY_NODE RT_MAKE_NAME(verify_node)
+
+/* type declarations */
+#define RT_RADIX_TREE RT_MAKE_NAME(radix_tree)
+#define RT_RADIX_TREE_CONTROL RT_MAKE_NAME(radix_tree_control)
+#define RT_ITER RT_MAKE_NAME(iter)
+#ifdef RT_SHMEM
+#define RT_HANDLE RT_MAKE_NAME(handle)
+#endif
+#define RT_NODE RT_MAKE_NAME(node)
+#define RT_NODE_ITER RT_MAKE_NAME(node_iter)
+#define RT_NODE_BASE_3 RT_MAKE_NAME(node_base_3)
+#define RT_NODE_BASE_32 RT_MAKE_NAME(node_base_32)
+#define RT_NODE_BASE_125 RT_MAKE_NAME(node_base_125)
+#define RT_NODE_BASE_256 RT_MAKE_NAME(node_base_256)
+#define RT_NODE_INNER_3 RT_MAKE_NAME(node_inner_3)
+#define RT_NODE_INNER_32 RT_MAKE_NAME(node_inner_32)
+#define RT_NODE_INNER_125 RT_MAKE_NAME(node_inner_125)
+#define RT_NODE_INNER_256 RT_MAKE_NAME(node_inner_256)
+#define RT_NODE_LEAF_3 RT_MAKE_NAME(node_leaf_3)
+#define RT_NODE_LEAF_32 RT_MAKE_NAME(node_leaf_32)
+#define RT_NODE_LEAF_125 RT_MAKE_NAME(node_leaf_125)
+#define RT_NODE_LEAF_256 RT_MAKE_NAME(node_leaf_256)
+#define RT_SIZE_CLASS RT_MAKE_NAME(size_class)
+#define RT_SIZE_CLASS_ELEM RT_MAKE_NAME(size_class_elem)
+#define RT_SIZE_CLASS_INFO RT_MAKE_NAME(size_class_info)
+#define RT_CLASS_3 RT_MAKE_NAME(class_3)
+#define RT_CLASS_32_MIN RT_MAKE_NAME(class_32_min)
+#define RT_CLASS_32_MAX RT_MAKE_NAME(class_32_max)
+#define RT_CLASS_125 RT_MAKE_NAME(class_125)
+#define RT_CLASS_256 RT_MAKE_NAME(class_256)
+
+/* generate forward declarations necessary to use the radix tree */
+#ifdef RT_DECLARE
+
+typedef struct RT_RADIX_TREE RT_RADIX_TREE;
+typedef struct RT_ITER RT_ITER;
+
+#ifdef RT_SHMEM
+typedef dsa_pointer RT_HANDLE;
+#endif
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx, dsa_area *dsa);
+RT_SCOPE RT_RADIX_TREE * RT_ATTACH(dsa_area *dsa, dsa_pointer dp);
+RT_SCOPE void RT_DETACH(RT_RADIX_TREE *tree);
+RT_SCOPE RT_HANDLE RT_GET_HANDLE(RT_RADIX_TREE *tree);
+#else
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx);
+#endif
+RT_SCOPE void RT_FREE(RT_RADIX_TREE *tree);
+
+RT_SCOPE bool RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *val_p);
+RT_SCOPE bool RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE val);
+#ifdef RT_USE_DELETE
+RT_SCOPE bool RT_DELETE(RT_RADIX_TREE *tree, uint64 key);
+#endif
+
+RT_SCOPE RT_ITER * RT_BEGIN_ITERATE(RT_RADIX_TREE *tree);
+RT_SCOPE bool RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p);
+RT_SCOPE void RT_END_ITERATE(RT_ITER *iter);
+
+RT_SCOPE uint64 RT_MEMORY_USAGE(RT_RADIX_TREE *tree);
+
+#ifdef RT_DEBUG
+RT_SCOPE void RT_DUMP(RT_RADIX_TREE *tree);
+RT_SCOPE void RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key);
+RT_SCOPE void RT_STATS(RT_RADIX_TREE *tree);
+#endif
+
+#endif /* RT_DECLARE */
+
+
+/* generate implementation of the radix tree */
+#ifdef RT_DEFINE
+
+/* The number of bits encoded in one tree level */
+#define RT_NODE_SPAN BITS_PER_BYTE
+
+/* The number of maximum slots in the node */
+#define RT_NODE_MAX_SLOTS (1 << RT_NODE_SPAN)
+
+/* Mask for extracting a chunk from the key */
+#define RT_CHUNK_MASK ((1 << RT_NODE_SPAN) - 1)
+
+/* Maximum shift the radix tree uses */
+#define RT_MAX_SHIFT RT_KEY_GET_SHIFT(UINT64_MAX)
+
+/* Tree level the radix tree uses */
+#define RT_MAX_LEVEL ((sizeof(uint64) * BITS_PER_BYTE) / RT_NODE_SPAN)
+
+/*
+ * Number of bits necessary for isset array in the slot-index node.
+ * Since bitmapword can be 64 bits, the only values that make sense
+ * here are 64 and 128.
+ */
+#define RT_SLOT_IDX_LIMIT (RT_NODE_MAX_SLOTS / 2)
+
+/* Invalid index used in node-125 */
+#define RT_INVALID_SLOT_IDX 0xFF
+
+/* Get a chunk from the key */
+#define RT_GET_KEY_CHUNK(key, shift) ((uint8) (((key) >> (shift)) & RT_CHUNK_MASK))
+
+/* For accessing bitmaps */
+#define BM_IDX(x) ((x) / BITS_PER_BITMAPWORD)
+#define BM_BIT(x) ((x) % BITS_PER_BITMAPWORD)
+
+/*
+ * Node kinds
+ *
+ * The different node kinds are what make the tree "adaptive".
+ *
+ * Each node kind is associated with a different datatype and different
+ * search/set/delete/iterate algorithms adapted for its size. The largest
+ * kind, node256 is basically the same as a traditional radix tree,
+ * and would be most wasteful of memory when sparsely populated. The
+ * smaller nodes expend some additional CPU time to enable a smaller
+ * memory footprint.
+ *
+ * XXX There are 4 node kinds, and this should never be increased,
+ * for several reasons:
+ * 1. With 5 or more kinds, gcc tends to use a jump table for switch
+ * statements.
+ * 2. The 4 kinds can be represented with 2 bits, so we have the option
+ * in the future to tag the node pointer with the kind, even on
+ * platforms with 32-bit pointers. This might speed up node traversal
+ * in trees with highly random node kinds.
+ * 3. We can have multiple size classes per node kind.
+ */
+#define RT_NODE_KIND_3 0x00
+#define RT_NODE_KIND_32 0x01
+#define RT_NODE_KIND_125 0x02
+#define RT_NODE_KIND_256 0x03
+#define RT_NODE_KIND_COUNT 4
+
+/*
+ * Calculate the slab blocksize so that we can allocate at least 32 chunks
+ * from the block.
+ */
+#define RT_SLAB_BLOCK_SIZE(size) \
+ Max((SLAB_DEFAULT_BLOCK_SIZE / (size)) * (size), (size) * 32)
+
+/* Common type for all nodes types */
+typedef struct RT_NODE
+{
+ /*
+ * Number of children. We use uint16 to be able to indicate 256 children
+ * at the fanout of 8.
+ */
+ uint16 count;
+
+ /*
+ * Max capacity for the current size class. Storing this in the
+ * node enables multiple size classes per node kind.
+ * Technically, kinds with a single size class don't need this, so we could
+ * keep this in the individual base types, but the code is simpler this way.
+ * Note: node256 is unique in that it cannot possibly have more than a
+ * single size class, so for that kind we store zero, and uint8 is
+ * sufficient for other kinds.
+ */
+ uint8 fanout;
+
+ /*
+ * Shift indicates which part of the key space is represented by this
+ * node. That is, the key is shifted by 'shift' and the lowest
+ * RT_NODE_SPAN bits are then represented in chunk.
+ */
+ uint8 shift;
+
+ /* Node kind, one per search/set algorithm */
+ uint8 kind;
+} RT_NODE;
+
+
+#define RT_PTR_LOCAL RT_NODE *
+
+#ifdef RT_SHMEM
+#define RT_PTR_ALLOC dsa_pointer
+#else
+#define RT_PTR_ALLOC RT_PTR_LOCAL
+#endif
+
+
+#ifdef RT_SHMEM
+#define RT_INVALID_PTR_ALLOC InvalidDsaPointer
+#else
+#define RT_INVALID_PTR_ALLOC NULL
+#endif
+
+/*
+ * Inner nodes and leaf nodes have analogous structure. To distinguish
+ * them at runtime, we take advantage of the fact that the key chunk
+ * is accessed by shifting: Inner tree nodes (shift > 0), store the
+ * pointer to its child node in the slot. In leaf nodes (shift == 0),
+ * the slot contains the value corresponding to the key.
+ */
+#define RT_NODE_IS_LEAF(n) (((RT_PTR_LOCAL) (n))->shift == 0)
+
+#define RT_NODE_MUST_GROW(node) \
+ ((node)->base.n.count == (node)->base.n.fanout)
+
+/*
+ * Base type of each node kinds for leaf and inner nodes.
+ * The base types must be a be able to accommodate the largest size
+ * class for variable-sized node kinds.
+ */
+typedef struct RT_NODE_BASE_3
+{
+ RT_NODE n;
+
+ /* 3 children, for key chunks */
+ uint8 chunks[3];
+} RT_NODE_BASE_3;
+
+typedef struct RT_NODE_BASE_32
+{
+ RT_NODE n;
+
+ /* 32 children, for key chunks */
+ uint8 chunks[32];
+} RT_NODE_BASE_32;
+
+/*
+ * node-125 uses slot_idx array, an array of RT_NODE_MAX_SLOTS length
+ * to store indexes into a second array that contains the values (or
+ * child pointers).
+ */
+typedef struct RT_NODE_BASE_125
+{
+ RT_NODE n;
+
+ /* The index of slots for each fanout */
+ uint8 slot_idxs[RT_NODE_MAX_SLOTS];
+
+ /* isset is a bitmap to track which slot is in use */
+ bitmapword isset[BM_IDX(RT_SLOT_IDX_LIMIT)];
+} RT_NODE_BASE_125;
+
+typedef struct RT_NODE_BASE_256
+{
+ RT_NODE n;
+} RT_NODE_BASE_256;
+
+/*
+ * Inner and leaf nodes.
+ *
+ * Theres are separate because the value type might be different than
+ * something fitting into a pointer-width type.
+ */
+typedef struct RT_NODE_INNER_3
+{
+ RT_NODE_BASE_3 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_3;
+
+typedef struct RT_NODE_LEAF_3
+{
+ RT_NODE_BASE_3 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_3;
+
+typedef struct RT_NODE_INNER_32
+{
+ RT_NODE_BASE_32 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_32;
+
+typedef struct RT_NODE_LEAF_32
+{
+ RT_NODE_BASE_32 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_32;
+
+typedef struct RT_NODE_INNER_125
+{
+ RT_NODE_BASE_125 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_125;
+
+typedef struct RT_NODE_LEAF_125
+{
+ RT_NODE_BASE_125 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_125;
+
+/*
+ * node-256 is the largest node type. This node has an array
+ * for directly storing values (or child pointers in inner nodes).
+ * Unlike other node kinds, it's array size is by definition
+ * fixed.
+ */
+typedef struct RT_NODE_INNER_256
+{
+ RT_NODE_BASE_256 base;
+
+ /* Slots for 256 children */
+ RT_PTR_ALLOC children[RT_NODE_MAX_SLOTS];
+} RT_NODE_INNER_256;
+
+typedef struct RT_NODE_LEAF_256
+{
+ RT_NODE_BASE_256 base;
+
+ /*
+ * Unlike with inner256, zero is a valid value here, so we use a
+ * bitmap to track which slot is in use.
+ */
+ bitmapword isset[BM_IDX(RT_NODE_MAX_SLOTS)];
+
+ /* Slots for 256 values */
+ RT_VALUE_TYPE values[RT_NODE_MAX_SLOTS];
+} RT_NODE_LEAF_256;
+
+/*
+ * Node size classes
+ *
+ * Nodes of different kinds necessarily belong to different size classes.
+ * The main innovation in our implementation compared to the ART paper
+ * is decoupling the notion of size class from kind.
+ *
+ * The size classes within a given node kind have the same underlying
+ * type, but a variable number of children/values. This is possible
+ * because the base type contains small fixed data structures that
+ * work the same way regardless of how full the node is. We store the
+ * node's allocated capacity in the "fanout" member of RT_NODE, to allow
+ * runtime introspection.
+ *
+ * Growing from one node kind to another requires special code for each
+ * case, but growing from one size class to another within the same kind
+ * is basically just allocate + memcpy.
+ *
+ * The size classes have been chosen so that inner nodes on platforms
+ * with 64-bit pointers (and leaf nodes when using a 64-bit key) are
+ * equal to or slightly smaller than some DSA size class.
+ */
+typedef enum RT_SIZE_CLASS
+{
+ RT_CLASS_3 = 0,
+ RT_CLASS_32_MIN,
+ RT_CLASS_32_MAX,
+ RT_CLASS_125,
+ RT_CLASS_256
+} RT_SIZE_CLASS;
+
+/* Information for each size class */
+typedef struct RT_SIZE_CLASS_ELEM
+{
+ const char *name;
+ int fanout;
+
+ /* slab chunk size */
+ Size inner_size;
+ Size leaf_size;
+} RT_SIZE_CLASS_ELEM;
+
+static const RT_SIZE_CLASS_ELEM RT_SIZE_CLASS_INFO[] = {
+ [RT_CLASS_3] = {
+ .name = "radix tree node 3",
+ .fanout = 3,
+ .inner_size = sizeof(RT_NODE_INNER_3) + 3 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_3) + 3 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_32_MIN] = {
+ .name = "radix tree node 15",
+ .fanout = 15,
+ .inner_size = sizeof(RT_NODE_INNER_32) + 15 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_32) + 15 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_32_MAX] = {
+ .name = "radix tree node 32",
+ .fanout = 32,
+ .inner_size = sizeof(RT_NODE_INNER_32) + 32 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_32) + 32 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_125] = {
+ .name = "radix tree node 125",
+ .fanout = 125,
+ .inner_size = sizeof(RT_NODE_INNER_125) + 125 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_125) + 125 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_256] = {
+ .name = "radix tree node 256",
+ .fanout = 256,
+ .inner_size = sizeof(RT_NODE_INNER_256),
+ .leaf_size = sizeof(RT_NODE_LEAF_256),
+ },
+};
+
+#define RT_SIZE_CLASS_COUNT lengthof(RT_SIZE_CLASS_INFO)
+
+#ifdef RT_SHMEM
+/* A magic value used to identify our radix tree */
+#define RT_RADIX_TREE_MAGIC 0x54A48167
+#endif
+
+/* Contains the actual tree and ancillary info */
+// WIP: this name is a bit strange
+typedef struct RT_RADIX_TREE_CONTROL
+{
+#ifdef RT_SHMEM
+ RT_HANDLE handle;
+ uint32 magic;
+#endif
+
+ RT_PTR_ALLOC root;
+ uint64 max_val;
+ uint64 num_keys;
+
+ /* statistics */
+#ifdef RT_DEBUG
+ int32 cnt[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE_CONTROL;
+
+/* Entry point for allocating and accessing the tree */
+typedef struct RT_RADIX_TREE
+{
+ MemoryContext context;
+
+ /* pointing to either local memory or DSA */
+ RT_RADIX_TREE_CONTROL *ctl;
+
+#ifdef RT_SHMEM
+ dsa_area *dsa;
+#else
+ MemoryContextData *inner_slabs[RT_SIZE_CLASS_COUNT];
+ MemoryContextData *leaf_slabs[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE;
+
+/*
+ * Iteration support.
+ *
+ * Iterating the radix tree returns each pair of key and value in the ascending
+ * order of the key. To support this, the we iterate nodes of each level.
+ *
+ * RT_NODE_ITER struct is used to track the iteration within a node.
+ *
+ * RT_ITER is the struct for iteration of the radix tree, and uses RT_NODE_ITER
+ * in order to track the iteration of each level. During iteration, we also
+ * construct the key whenever updating the node iteration information, e.g., when
+ * advancing the current index within the node or when moving to the next node
+ * at the same level.
+ *
+ * XXX: Currently we allow only one process to do iteration. Therefore, rt_node_iter
+ * has the local pointers to nodes, rather than RT_PTR_ALLOC.
+ * We need either a safeguard to disallow other processes to begin the iteration
+ * while one process is doing or to allow multiple processes to do the iteration.
+ */
+typedef struct RT_NODE_ITER
+{
+ RT_PTR_LOCAL node; /* current node being iterated */
+ int current_idx; /* current position. -1 for initial value */
+} RT_NODE_ITER;
+
+typedef struct RT_ITER
+{
+ RT_RADIX_TREE *tree;
+
+ /* Track the iteration on nodes of each level */
+ RT_NODE_ITER stack[RT_MAX_LEVEL];
+ int stack_len;
+
+ /* The key is constructed during iteration */
+ uint64 key;
+} RT_ITER;
+
+
+static bool RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_PTR_ALLOC child);
+static bool RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_VALUE_TYPE value);
+
+/* verification (available only with assertion) */
+static void RT_VERIFY_NODE(RT_PTR_LOCAL node);
+
+/* Get the local address of an allocated node */
+static inline RT_PTR_LOCAL
+RT_PTR_GET_LOCAL(RT_RADIX_TREE *tree, RT_PTR_ALLOC node)
+{
+#ifdef RT_SHMEM
+ return dsa_get_address(tree->dsa, (dsa_pointer) node);
+#else
+ return node;
+#endif
+}
+
+static inline bool
+RT_PTR_ALLOC_IS_VALID(RT_PTR_ALLOC ptr)
+{
+#ifdef RT_SHMEM
+ return DsaPointerIsValid(ptr);
+#else
+ return PointerIsValid(ptr);
+#endif
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_3_SEARCH_EQ(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+ int idx = -1;
+
+ for (int i = 0; i < node->n.count; i++)
+ {
+ if (node->chunks[i] == chunk)
+ {
+ idx = i;
+ break;
+ }
+ }
+
+ return idx;
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_3_GET_INSERTPOS(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+ int idx;
+
+ for (idx = 0; idx < node->n.count; idx++)
+ {
+ if (node->chunks[idx] >= chunk)
+ break;
+ }
+
+ return idx;
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_32_SEARCH_EQ(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+ int count = node->n.count;
+#ifndef USE_NO_SIMD
+ Vector8 spread_chunk;
+ Vector8 haystack1;
+ Vector8 haystack2;
+ Vector8 cmp1;
+ Vector8 cmp2;
+ uint32 bitfield;
+ int index_simd = -1;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+ int index = -1;
+
+ for (int i = 0; i < count; i++)
+ {
+ if (node->chunks[i] == chunk)
+ {
+ index = i;
+ break;
+ }
+ }
+#endif
+
+#ifndef USE_NO_SIMD
+ /* replicate the search key */
+ spread_chunk = vector8_broadcast(chunk);
+
+ /* compare to all 32 keys stored in the node */
+ vector8_load(&haystack1, &node->chunks[0]);
+ vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+ cmp1 = vector8_eq(spread_chunk, haystack1);
+ cmp2 = vector8_eq(spread_chunk, haystack2);
+
+ /* convert comparison to a bitfield */
+ bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+
+ /* mask off invalid entries */
+ bitfield &= ((UINT64CONST(1) << count) - 1);
+
+ /* convert bitfield to index by counting trailing zeros */
+ if (bitfield)
+ index_simd = pg_rightmost_one_pos32(bitfield);
+
+ Assert(index_simd == index);
+ return index_simd;
+#else
+ return index;
+#endif
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_32_GET_INSERTPOS(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+ int count = node->n.count;
+#ifndef USE_NO_SIMD
+ Vector8 spread_chunk;
+ Vector8 haystack1;
+ Vector8 haystack2;
+ Vector8 cmp1;
+ Vector8 cmp2;
+ Vector8 min1;
+ Vector8 min2;
+ uint32 bitfield;
+ int index_simd;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+ int index;
+
+ for (index = 0; index < count; index++)
+ {
+ /*
+ * This is coded with '>=' to match what we can do with SIMD,
+ * with an assert to keep us honest.
+ */
+ if (node->chunks[index] >= chunk)
+ {
+ Assert(node->chunks[index] != chunk);
+ break;
+ }
+ }
+#endif
+
+#ifndef USE_NO_SIMD
+ /*
+ * This is a bit more complicated than RT_NODE_32_SEARCH_EQ(), because
+ * no unsigned uint8 comparison instruction exists, at least for SSE2. So
+ * we need to play some trickery using vector8_min() to effectively get
+ * <=. There'll never be any equal elements in urrent uses, but that's
+ * what we get here...
+ */
+ spread_chunk = vector8_broadcast(chunk);
+ vector8_load(&haystack1, &node->chunks[0]);
+ vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+ min1 = vector8_min(spread_chunk, haystack1);
+ min2 = vector8_min(spread_chunk, haystack2);
+ cmp1 = vector8_eq(spread_chunk, min1);
+ cmp2 = vector8_eq(spread_chunk, min2);
+ bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+ bitfield &= ((UINT64CONST(1) << count) - 1);
+
+ if (bitfield)
+ index_simd = pg_rightmost_one_pos32(bitfield);
+ else
+ index_simd = count;
+
+ Assert(index_simd == index);
+ return index_simd;
+#else
+ return index;
+#endif
+}
+
+
+/*
+ * Functions to manipulate both chunks array and children/values array.
+ * These are used for node-3 and node-32.
+ */
+
+/* Shift the elements right at 'idx' by one */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_SHIFT(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+ memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+ memmove(&(children[idx + 1]), &(children[idx]), sizeof(RT_PTR_ALLOC) * (count - idx));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_SHIFT(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+ memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+ memmove(&(values[idx + 1]), &(values[idx]), sizeof(RT_VALUE_TYPE) * (count - idx));
+}
+
+/* Delete the element at 'idx' */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_DELETE(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+ memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+ memmove(&(children[idx]), &(children[idx + 1]), sizeof(RT_PTR_ALLOC) * (count - idx - 1));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_DELETE(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+ memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+ memmove(&(values[idx]), &(values[idx + 1]), sizeof(RT_VALUE_TYPE) * (count - idx - 1));
+}
+
+/* Copy both chunks and children/values arrays */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_COPY(uint8 *src_chunks, RT_PTR_ALLOC *src_children,
+ uint8 *dst_chunks, RT_PTR_ALLOC *dst_children)
+{
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const Size chunk_size = sizeof(uint8) * fanout;
+ const Size children_size = sizeof(RT_PTR_ALLOC) * fanout;
+
+ memcpy(dst_chunks, src_chunks, chunk_size);
+ memcpy(dst_children, src_children, children_size);
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_COPY(uint8 *src_chunks, RT_VALUE_TYPE *src_values,
+ uint8 *dst_chunks, RT_VALUE_TYPE *dst_values)
+{
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const Size chunk_size = sizeof(uint8) * fanout;
+ const Size values_size = sizeof(RT_VALUE_TYPE) * fanout;
+
+ memcpy(dst_chunks, src_chunks, chunk_size);
+ memcpy(dst_values, src_values, values_size);
+}
+
+/* Functions to manipulate inner and leaf node-125 */
+
+/* Does the given chunk in the node has the value? */
+static inline bool
+RT_NODE_125_IS_CHUNK_USED(RT_NODE_BASE_125 *node, uint8 chunk)
+{
+ return node->slot_idxs[chunk] != RT_INVALID_SLOT_IDX;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_125_GET_CHILD(RT_NODE_INNER_125 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ return node->children[node->base.slot_idxs[chunk]];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_125_GET_VALUE(RT_NODE_LEAF_125 *node, uint8 chunk)
+{
+ Assert(RT_NODE_IS_LEAF(node));
+ Assert(((RT_NODE_BASE_125 *) node)->slot_idxs[chunk] != RT_INVALID_SLOT_IDX);
+ return node->values[node->base.slot_idxs[chunk]];
+}
+
+/* Functions to manipulate inner and leaf node-256 */
+
+/* Return true if the slot corresponding to the given chunk is in use */
+static inline bool
+RT_NODE_INNER_256_IS_CHUNK_USED(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ return node->children[chunk] != RT_INVALID_PTR_ALLOC;
+}
+
+static inline bool
+RT_NODE_LEAF_256_IS_CHUNK_USED(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ int idx = BM_IDX(chunk);
+ int bitnum = BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ return (node->isset[idx] & ((bitmapword) 1 << bitnum)) != 0;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_256_GET_CHILD(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ Assert(RT_NODE_INNER_256_IS_CHUNK_USED(node, chunk));
+ return node->children[chunk];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_256_GET_VALUE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ Assert(RT_NODE_IS_LEAF(node));
+ Assert(RT_NODE_LEAF_256_IS_CHUNK_USED(node, chunk));
+ return node->values[chunk];
+}
+
+/* Set the child in the node-256 */
+static inline void
+RT_NODE_INNER_256_SET(RT_NODE_INNER_256 *node, uint8 chunk, RT_PTR_ALLOC child)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ node->children[chunk] = child;
+}
+
+/* Set the value in the node-256 */
+static inline void
+RT_NODE_LEAF_256_SET(RT_NODE_LEAF_256 *node, uint8 chunk, RT_VALUE_TYPE value)
+{
+ int idx = BM_IDX(chunk);
+ int bitnum = BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ node->isset[idx] |= ((bitmapword) 1 << bitnum);
+ node->values[chunk] = value;
+}
+
+/* Set the slot at the given chunk position */
+static inline void
+RT_NODE_INNER_256_DELETE(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ node->children[chunk] = RT_INVALID_PTR_ALLOC;
+}
+
+static inline void
+RT_NODE_LEAF_256_DELETE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ int idx = BM_IDX(chunk);
+ int bitnum = BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ node->isset[idx] &= ~((bitmapword) 1 << bitnum);
+}
+
+/*
+ * Return the largest shift that will allowing storing the given key.
+ */
+static inline int
+RT_KEY_GET_SHIFT(uint64 key)
+{
+ if (key == 0)
+ return 0;
+ else
+ return (pg_leftmost_one_pos64(key) / RT_NODE_SPAN) * RT_NODE_SPAN;
+}
+
+/*
+ * Return the max value that can be stored in the tree with the given shift.
+ */
+static uint64
+RT_SHIFT_GET_MAX_VAL(int shift)
+{
+ if (shift == RT_MAX_SHIFT)
+ return UINT64_MAX;
+
+ return (UINT64CONST(1) << (shift + RT_NODE_SPAN)) - 1;
+}
+
+/*
+ * Allocate a new node with the given node kind.
+ */
+static RT_PTR_ALLOC
+RT_ALLOC_NODE(RT_RADIX_TREE *tree, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+ RT_PTR_ALLOC allocnode;
+ size_t allocsize;
+
+ if (is_leaf)
+ allocsize = RT_SIZE_CLASS_INFO[size_class].leaf_size;
+ else
+ allocsize = RT_SIZE_CLASS_INFO[size_class].inner_size;
+
+#ifdef RT_SHMEM
+ allocnode = dsa_allocate(tree->dsa, allocsize);
+#else
+ if (is_leaf)
+ allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->leaf_slabs[size_class],
+ allocsize);
+ else
+ allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->inner_slabs[size_class],
+ allocsize);
+#endif
+
+#ifdef RT_DEBUG
+ /* update the statistics */
+ tree->ctl->cnt[size_class]++;
+#endif
+
+ return allocnode;
+}
+
+/* Initialize the node contents */
+static inline void
+RT_INIT_NODE(RT_PTR_LOCAL node, uint8 kind, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+ if (is_leaf)
+ MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].leaf_size);
+ else
+ MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].inner_size);
+
+ node->kind = kind;
+
+ if (kind == RT_NODE_KIND_256)
+ /* See comment for the RT_NODE type */
+ Assert(node->fanout == 0);
+ else
+ node->fanout = RT_SIZE_CLASS_INFO[size_class].fanout;
+
+ /* Initialize slot_idxs to invalid values */
+ if (kind == RT_NODE_KIND_125)
+ {
+ RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+
+ memset(n125->slot_idxs, RT_INVALID_SLOT_IDX, sizeof(n125->slot_idxs));
+ }
+}
+
+/*
+ * Create a new node as the root. Subordinate nodes will be created during
+ * the insertion.
+ */
+static void
+RT_NEW_ROOT(RT_RADIX_TREE *tree, uint64 key)
+{
+ int shift = RT_KEY_GET_SHIFT(key);
+ bool is_leaf = shift == 0;
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(newnode, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ newnode->shift = shift;
+ tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(shift);
+ tree->ctl->root = allocnode;
+}
+
+static inline void
+RT_COPY_NODE(RT_PTR_LOCAL newnode, RT_PTR_LOCAL oldnode)
+{
+ newnode->shift = oldnode->shift;
+ newnode->count = oldnode->count;
+}
+
+/*
+ * Given a new allocated node and an old node, initalize the new
+ * node with the necessary fields and return its local pointer.
+ */
+static inline RT_PTR_LOCAL
+RT_SWITCH_NODE_KIND(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, RT_PTR_LOCAL node,
+ uint8 new_kind, uint8 new_class, bool is_leaf)
+{
+ RT_PTR_LOCAL newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(newnode, new_kind, new_class, is_leaf);
+ RT_COPY_NODE(newnode, node);
+
+ return newnode;
+}
+
+/* Free the given node */
+static void
+RT_FREE_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode)
+{
+ /* If we're deleting the root node, make the tree empty */
+ if (tree->ctl->root == allocnode)
+ {
+ tree->ctl->root = RT_INVALID_PTR_ALLOC;
+ tree->ctl->max_val = 0;
+ }
+
+#ifdef RT_DEBUG
+ {
+ int i;
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+ /* update the statistics */
+ for (i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ if (node->fanout == RT_SIZE_CLASS_INFO[i].fanout)
+ break;
+ }
+
+ /* fanout of node256 is intentionally 0 */
+ if (i == RT_SIZE_CLASS_COUNT)
+ i = RT_CLASS_256;
+
+ tree->ctl->cnt[i]--;
+ Assert(tree->ctl->cnt[i] >= 0);
+ }
+#endif
+
+#ifdef RT_SHMEM
+ dsa_free(tree->dsa, allocnode);
+#else
+ pfree(allocnode);
+#endif
+}
+
+/* Update the parent's pointer when growing a node */
+static inline void
+RT_NODE_UPDATE_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC new_child)
+{
+#define RT_ACTION_UPDATE
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+#undef RT_ACTION_UPDATE
+}
+
+/*
+ * Replace old_child with new_child, and free the old one.
+ */
+static void
+RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
+ RT_PTR_ALLOC stored_old_child, RT_PTR_LOCAL old_child,
+ RT_PTR_ALLOC new_child, uint64 key)
+{
+#ifdef USE_ASSERT_CHECKING
+ RT_PTR_LOCAL new = RT_PTR_GET_LOCAL(tree, new_child);
+
+ Assert(old_child->shift == new->shift);
+ Assert(old_child->count == new->count);
+#endif
+
+ if (parent == old_child)
+ {
+ /* Replace the root node with the new larger node */
+ tree->ctl->root = new_child;
+ }
+ else
+ RT_NODE_UPDATE_INNER(parent, key, new_child);
+
+ RT_FREE_NODE(tree, stored_old_child);
+}
+
+/*
+ * The radix tree doesn't have sufficient height. Extend the radix tree so
+ * it can store the key.
+ */
+static void
+RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
+{
+ int target_shift;
+ RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+ int shift = root->shift + RT_NODE_SPAN;
+
+ target_shift = RT_KEY_GET_SHIFT(key);
+
+ /* Grow tree from 'shift' to 'target_shift' */
+ while (shift <= target_shift)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL node;
+ RT_NODE_INNER_3 *n3;
+
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, true);
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(node, RT_NODE_KIND_3, RT_CLASS_3, true);
+ node->shift = shift;
+ node->count = 1;
+
+ n3 = (RT_NODE_INNER_3 *) node;
+ n3->base.chunks[0] = 0;
+ n3->children[0] = tree->ctl->root;
+
+ /* Update the root */
+ tree->ctl->root = allocnode;
+
+ shift += RT_NODE_SPAN;
+ }
+
+ tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(target_shift);
+}
+
+/*
+ * The radix tree doesn't have inner and leaf nodes for given key-value pair.
+ * Insert inner and leaf nodes from 'node' to bottom.
+ */
+static inline void
+RT_SET_EXTEND(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE value, RT_PTR_LOCAL parent,
+ RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node)
+{
+ int shift = node->shift;
+
+ Assert(RT_PTR_GET_LOCAL(tree, stored_node) == node);
+
+ while (shift >= RT_NODE_SPAN)
+ {
+ RT_PTR_ALLOC allocchild;
+ RT_PTR_LOCAL newchild;
+ int newshift = shift - RT_NODE_SPAN;
+ bool is_leaf = newshift == 0;
+
+ allocchild = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ newchild = RT_PTR_GET_LOCAL(tree, allocchild);
+ RT_INIT_NODE(newchild, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ newchild->shift = newshift;
+ RT_NODE_INSERT_INNER(tree, parent, stored_node, node, key, allocchild);
+
+ parent = node;
+ node = newchild;
+ stored_node = allocchild;
+ shift -= RT_NODE_SPAN;
+ }
+
+ RT_NODE_INSERT_LEAF(tree, parent, stored_node, node, key, value);
+ tree->ctl->num_keys++;
+}
+
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the child
+ * pointer is set to child_p.
+ */
+static inline bool
+RT_NODE_SEARCH_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC *child_p)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the pointer
+ * to the value is set to value_p.
+ */
+static inline bool
+RT_NODE_SEARCH_LEAF(RT_PTR_LOCAL node, uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_INNER(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_LEAF(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+#endif
+
+/*
+ * Insert "child" into "node".
+ *
+ * "parent" is the parent of "node", so the grandparent of the child.
+ * If the node we're inserting into needs to grow, we update the parent's
+ * child pointer with the pointer to the new larger node.
+ */
+static bool
+RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_PTR_ALLOC child)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/* Like RT_NODE_INSERT_INNER, but for leaf nodes */
+static bool
+RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_VALUE_TYPE value)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Create the radix tree in the given memory context and return it.
+ */
+RT_SCOPE RT_RADIX_TREE *
+#ifdef RT_SHMEM
+RT_CREATE(MemoryContext ctx, dsa_area *dsa)
+#else
+RT_CREATE(MemoryContext ctx)
+#endif
+{
+ RT_RADIX_TREE *tree;
+ MemoryContext old_ctx;
+#ifdef RT_SHMEM
+ dsa_pointer dp;
+#endif
+
+ old_ctx = MemoryContextSwitchTo(ctx);
+
+ tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+ tree->context = ctx;
+
+#ifdef RT_SHMEM
+ tree->dsa = dsa;
+ dp = dsa_allocate0(dsa, sizeof(RT_RADIX_TREE_CONTROL));
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, dp);
+ tree->ctl->handle = dp;
+ tree->ctl->magic = RT_RADIX_TREE_MAGIC;
+#else
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) palloc0(sizeof(RT_RADIX_TREE_CONTROL));
+
+ /* Create a slab context for each size class */
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ RT_SIZE_CLASS_ELEM size_class = RT_SIZE_CLASS_INFO[i];
+ size_t inner_blocksize = RT_SLAB_BLOCK_SIZE(size_class.inner_size);
+ size_t leaf_blocksize = RT_SLAB_BLOCK_SIZE(size_class.leaf_size);
+
+ tree->inner_slabs[i] = SlabContextCreate(ctx,
+ size_class.name,
+ inner_blocksize,
+ size_class.inner_size);
+ tree->leaf_slabs[i] = SlabContextCreate(ctx,
+ size_class.name,
+ leaf_blocksize,
+ size_class.leaf_size);
+ }
+#endif
+
+ tree->ctl->root = RT_INVALID_PTR_ALLOC;
+
+ MemoryContextSwitchTo(old_ctx);
+
+ return tree;
+}
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE *
+RT_ATTACH(dsa_area *dsa, RT_HANDLE handle)
+{
+ RT_RADIX_TREE *tree;
+ dsa_pointer control;
+
+ tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+
+ /* Find the control object in shard memory */
+ control = handle;
+
+ tree->dsa = dsa;
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, control);
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+ return tree;
+}
+
+RT_SCOPE void
+RT_DETACH(RT_RADIX_TREE *tree)
+{
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ pfree(tree);
+}
+
+RT_SCOPE RT_HANDLE
+RT_GET_HANDLE(RT_RADIX_TREE *tree)
+{
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ return tree->ctl->handle;
+}
+
+/*
+ * Recursively free all nodes allocated to the DSA area.
+ */
+static inline void
+RT_FREE_RECURSE(RT_RADIX_TREE *tree, RT_PTR_ALLOC ptr)
+{
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, ptr);
+
+ check_stack_depth();
+ CHECK_FOR_INTERRUPTS();
+
+ /* The leaf node doesn't have child pointers */
+ if (RT_NODE_IS_LEAF(node))
+ {
+ dsa_free(tree->dsa, ptr);
+ return;
+ }
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+ for (int i = 0; i < n3->base.n.count; i++)
+ RT_FREE_RECURSE(tree, n3->children[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+ for (int i = 0; i < n32->base.n.count; i++)
+ RT_FREE_RECURSE(tree, n32->children[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) node;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+ continue;
+
+ RT_FREE_RECURSE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+ }
+
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ RT_FREE_RECURSE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+ }
+
+ break;
+ }
+ }
+
+ /* Free the inner node */
+ dsa_free(tree->dsa, ptr);
+}
+#endif
+
+/*
+ * Free the given radix tree.
+ */
+RT_SCOPE void
+RT_FREE(RT_RADIX_TREE *tree)
+{
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+ /* Free all memory used for radix tree nodes */
+ if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ RT_FREE_RECURSE(tree, tree->ctl->root);
+
+ /*
+ * Vandalize the control block to help catch programming error where
+ * other backends access the memory formerly occupied by this radix tree.
+ */
+ tree->ctl->magic = 0;
+ dsa_free(tree->dsa, tree->ctl->handle);
+#else
+ pfree(tree->ctl);
+
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ MemoryContextDelete(tree->inner_slabs[i]);
+ MemoryContextDelete(tree->leaf_slabs[i]);
+ }
+#endif
+
+ pfree(tree);
+}
+
+/*
+ * Set key to value. If the entry already exists, we update its value to 'value'
+ * and return true. Returns false if entry doesn't yet exist.
+ */
+RT_SCOPE bool
+RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE value)
+{
+ int shift;
+ bool updated;
+ RT_PTR_LOCAL parent;
+ RT_PTR_ALLOC stored_child;
+ RT_PTR_LOCAL child;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ /* Empty tree, create the root */
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ RT_NEW_ROOT(tree, key);
+
+ /* Extend the tree if necessary */
+ if (key > tree->ctl->max_val)
+ RT_EXTEND(tree, key);
+
+ stored_child = tree->ctl->root;
+ parent = RT_PTR_GET_LOCAL(tree, stored_child);
+ shift = parent->shift;
+
+ /* Descend the tree until we reach a leaf node */
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC new_child = RT_INVALID_PTR_ALLOC;;
+
+ child = RT_PTR_GET_LOCAL(tree, stored_child);
+
+ if (RT_NODE_IS_LEAF(child))
+ break;
+
+ if (!RT_NODE_SEARCH_INNER(child, key, &new_child))
+ {
+ RT_SET_EXTEND(tree, key, value, parent, stored_child, child);
+ return false;
+ }
+
+ parent = child;
+ stored_child = new_child;
+ shift -= RT_NODE_SPAN;
+ }
+
+ updated = RT_NODE_INSERT_LEAF(tree, parent, stored_child, child, key, value);
+
+ /* Update the statistics */
+ if (!updated)
+ tree->ctl->num_keys++;
+
+ return updated;
+}
+
+/*
+ * Search the given key in the radix tree. Return true if there is the key,
+ * otherwise return false. On success, we set the value to *val_p so it must
+ * not be NULL.
+ */
+RT_SCOPE bool
+RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+ RT_PTR_LOCAL node;
+ int shift;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+ Assert(value_p != NULL);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ return false;
+
+ node = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+ shift = node->shift;
+
+ /* Descend the tree until a leaf node */
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+ if (RT_NODE_IS_LEAF(node))
+ break;
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ return false;
+
+ node = RT_PTR_GET_LOCAL(tree, child);
+ shift -= RT_NODE_SPAN;
+ }
+
+ return RT_NODE_SEARCH_LEAF(node, key, value_p);
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Delete the given key from the radix tree. Return true if the key is found (and
+ * deleted), otherwise do nothing and return false.
+ */
+RT_SCOPE bool
+RT_DELETE(RT_RADIX_TREE *tree, uint64 key)
+{
+ RT_PTR_LOCAL node;
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_ALLOC stack[RT_MAX_LEVEL] = {0};
+ int shift;
+ int level;
+ bool deleted;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ return false;
+
+ /*
+ * Descend the tree to search the key while building a stack of nodes we
+ * visited.
+ */
+ allocnode = tree->ctl->root;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift = node->shift;
+ level = -1;
+ while (shift > 0)
+ {
+ RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+ /* Push the current node to the stack */
+ stack[++level] = allocnode;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ return false;
+
+ allocnode = child;
+ shift -= RT_NODE_SPAN;
+ }
+
+ /* Delete the key from the leaf node if exists */
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ deleted = RT_NODE_DELETE_LEAF(node, key);
+
+ if (!deleted)
+ {
+ /* no key is found in the leaf node */
+ return false;
+ }
+
+ /* Found the key to delete. Update the statistics */
+ tree->ctl->num_keys--;
+
+ /*
+ * Return if the leaf node still has keys and we don't need to delete the
+ * node.
+ */
+ if (node->count > 0)
+ return true;
+
+ /* Free the empty leaf node */
+ RT_FREE_NODE(tree, allocnode);
+
+ /* Delete the key in inner nodes recursively */
+ while (level >= 0)
+ {
+ allocnode = stack[level--];
+
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ deleted = RT_NODE_DELETE_INNER(node, key);
+ Assert(deleted);
+
+ /* If the node didn't become empty, we stop deleting the key */
+ if (node->count > 0)
+ break;
+
+ /* The node became empty */
+ RT_FREE_NODE(tree, allocnode);
+ }
+
+ return true;
+}
+#endif
+
+static inline void
+RT_ITER_UPDATE_KEY(RT_ITER *iter, uint8 chunk, uint8 shift)
+{
+ iter->key &= ~(((uint64) RT_CHUNK_MASK) << shift);
+ iter->key |= (((uint64) chunk) << shift);
+}
+
+/*
+ * Advance the slot in the inner node. Return the child if exists, otherwise
+ * null.
+ */
+static inline RT_PTR_LOCAL
+RT_NODE_INNER_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Advance the slot in the leaf node. On success, return true and the value
+ * is set to value_p, otherwise return false.
+ */
+static inline bool
+RT_NODE_LEAF_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter,
+ RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Update each node_iter for inner nodes in the iterator node stack.
+ */
+static void
+RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
+{
+ int level = from;
+ RT_PTR_LOCAL node = from_node;
+
+ for (;;)
+ {
+ RT_NODE_ITER *node_iter = &(iter->stack[level--]);
+
+ node_iter->node = node;
+ node_iter->current_idx = -1;
+
+ /* We don't advance the leaf node iterator here */
+ if (RT_NODE_IS_LEAF(node))
+ return;
+
+ /* Advance to the next slot in the inner node */
+ node = RT_NODE_INNER_ITERATE_NEXT(iter, node_iter);
+
+ /* We must find the first children in the node */
+ Assert(node);
+ }
+}
+
+/* Create and return the iterator for the given radix tree */
+RT_SCOPE RT_ITER *
+RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
+{
+ MemoryContext old_ctx;
+ RT_ITER *iter;
+ RT_PTR_LOCAL root;
+ int top_level;
+
+ old_ctx = MemoryContextSwitchTo(tree->context);
+
+ iter = (RT_ITER *) palloc0(sizeof(RT_ITER));
+ iter->tree = tree;
+
+ /* empty tree */
+ if (!iter->tree->ctl->root)
+ return iter;
+
+ root = RT_PTR_GET_LOCAL(tree, iter->tree->ctl->root);
+ top_level = root->shift / RT_NODE_SPAN;
+ iter->stack_len = top_level;
+
+ /*
+ * Descend to the left most leaf node from the root. The key is being
+ * constructed while descending to the leaf.
+ */
+ RT_UPDATE_ITER_STACK(iter, root, top_level);
+
+ MemoryContextSwitchTo(old_ctx);
+
+ return iter;
+}
+
+/*
+ * Return true with setting key_p and value_p if there is next key. Otherwise,
+ * return false.
+ */
+RT_SCOPE bool
+RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p)
+{
+ /* Empty tree */
+ if (!iter->tree->ctl->root)
+ return false;
+
+ for (;;)
+ {
+ RT_PTR_LOCAL child = NULL;
+ RT_VALUE_TYPE value;
+ int level;
+ bool found;
+
+ /* Advance the leaf node iterator to get next key-value pair */
+ found = RT_NODE_LEAF_ITERATE_NEXT(iter, &(iter->stack[0]), &value);
+
+ if (found)
+ {
+ *key_p = iter->key;
+ *value_p = value;
+ return true;
+ }
+
+ /*
+ * We've visited all values in the leaf node, so advance inner node
+ * iterators from the level=1 until we find the next child node.
+ */
+ for (level = 1; level <= iter->stack_len; level++)
+ {
+ child = RT_NODE_INNER_ITERATE_NEXT(iter, &(iter->stack[level]));
+
+ if (child)
+ break;
+ }
+
+ /* the iteration finished */
+ if (!child)
+ return false;
+
+ /*
+ * Set the node to the node iterator and update the iterator stack
+ * from this node.
+ */
+ RT_UPDATE_ITER_STACK(iter, child, level - 1);
+
+ /* Node iterators are updated, so try again from the leaf */
+ }
+
+ return false;
+}
+
+RT_SCOPE void
+RT_END_ITERATE(RT_ITER *iter)
+{
+ pfree(iter);
+}
+
+/*
+ * Return the statistics of the amount of memory used by the radix tree.
+ */
+RT_SCOPE uint64
+RT_MEMORY_USAGE(RT_RADIX_TREE *tree)
+{
+ Size total = 0;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ total = dsa_get_total_size(tree->dsa);
+#else
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ total += MemoryContextMemAllocated(tree->inner_slabs[i], true);
+ total += MemoryContextMemAllocated(tree->leaf_slabs[i], true);
+ }
+#endif
+
+ return total;
+}
+
+/*
+ * Verify the radix tree node.
+ */
+static void
+RT_VERIFY_NODE(RT_PTR_LOCAL node)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(node->count >= 0);
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE_BASE_3 *n3 = (RT_NODE_BASE_3 *) node;
+
+ for (int i = 1; i < n3->n.count; i++)
+ Assert(n3->chunks[i - 1] < n3->chunks[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE_BASE_32 *n32 = (RT_NODE_BASE_32 *) node;
+
+ for (int i = 1; i < n32->n.count; i++)
+ Assert(n32->chunks[i - 1] < n32->chunks[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+ int cnt = 0;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ uint8 slot = n125->slot_idxs[i];
+ int idx = BM_IDX(slot);
+ int bitnum = BM_BIT(slot);
+
+ if (!RT_NODE_125_IS_CHUNK_USED(n125, i))
+ continue;
+
+ /* Check if the corresponding slot is used */
+ Assert(slot < node->fanout);
+ Assert((n125->isset[idx] & ((bitmapword) 1 << bitnum)) != 0);
+
+ cnt++;
+ }
+
+ Assert(n125->n.count == cnt);
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+ int cnt = 0;
+
+ for (int i = 0; i < BM_IDX(RT_NODE_MAX_SLOTS); i++)
+ cnt += bmw_popcount(n256->isset[i]);
+
+ /* Check if the number of used chunk matches */
+ Assert(n256->base.n.count == cnt);
+
+ break;
+ }
+ }
+ }
+#endif
+}
+
+/***************** DEBUG FUNCTIONS *****************/
+#ifdef RT_DEBUG
+
+#define RT_UINT64_FORMAT_HEX "%" INT64_MODIFIER "X"
+
+RT_SCOPE void
+RT_STATS(RT_RADIX_TREE *tree)
+{
+ fprintf(stderr, "max_val = " UINT64_FORMAT "\n", tree->ctl->max_val);
+ fprintf(stderr, "num_keys = " UINT64_FORMAT "\n", tree->ctl->num_keys);
+
+#ifdef RT_SHMEM
+ fprintf(stderr, "handle = " UINT64_FORMAT "\n", tree->ctl->handle);
+#endif
+
+ if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+
+ fprintf(stderr, "height = %d, n3 = %u, n15 = %u, n32 = %u, n125 = %u, n256 = %u\n",
+ root->shift / RT_NODE_SPAN,
+ tree->ctl->cnt[RT_CLASS_3],
+ tree->ctl->cnt[RT_CLASS_32_MIN],
+ tree->ctl->cnt[RT_CLASS_32_MAX],
+ tree->ctl->cnt[RT_CLASS_125],
+ tree->ctl->cnt[RT_CLASS_256]);
+ }
+}
+
+static void
+RT_DUMP_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, int level,
+ bool recurse, StringInfo buf)
+{
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+ StringInfoData spaces;
+
+ initStringInfo(&spaces);
+ appendStringInfoSpaces(&spaces, (level * 4) + 1);
+
+ appendStringInfo(buf, "%s%s[%s] kind %d, fanout %d, count %u, shift %u:\n",
+ spaces.data,
+ level == 0 ? "" : "-> ",
+ RT_NODE_IS_LEAF(node) ? "LEAF" : "INNR",
+ (node->kind == RT_NODE_KIND_3) ? 3 :
+ (node->kind == RT_NODE_KIND_32) ? 32 :
+ (node->kind == RT_NODE_KIND_125) ? 125 : 256,
+ node->fanout == 0 ? 256 : node->fanout,
+ node->count, node->shift);
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ for (int i = 0; i < node->count; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_3 *n3 = (RT_NODE_LEAF_3 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+ spaces.data, i, n3->base.chunks[i]);
+ }
+ else
+ {
+ RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X",
+ spaces.data, i, n3->base.chunks[i]);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, n3->children[i], level + 1,
+ recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ for (int i = 0; i < node->count; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_32 *n32 = (RT_NODE_LEAF_32 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+ spaces.data, i, n32->base.chunks[i]);
+ }
+ else
+ {
+ RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X",
+ spaces.data, i, n32->base.chunks[i]);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, n32->children[i], level + 1,
+ recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_BASE_125 *b125 = (RT_NODE_BASE_125 *) node;
+ char *sep = "";
+
+ appendStringInfo(buf, "%sslot_idxs: ", spaces.data);
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+ continue;
+
+ appendStringInfo(buf, "%s[%d]=%d ",
+ sep, i, b125->slot_idxs[i]);
+ sep = ",";
+ }
+
+ appendStringInfo(buf, "\n%sisset-bitmap: ", spaces.data);
+ for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+ appendStringInfo(buf, "%X ", ((uint8 *) b125->isset)[i]);
+ appendStringInfo(buf, "\n");
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+ continue;
+
+ if (RT_NODE_IS_LEAF(node))
+ appendStringInfo(buf, "%schunk 0x%X\n",
+ spaces.data, i);
+ else
+ {
+ RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) b125;
+
+ appendStringInfo(buf, "%schunk 0x%X",
+ spaces.data, i);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i),
+ level + 1, recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+ appendStringInfo(buf, "%sisset-bitmap: ", spaces.data);
+ for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+ appendStringInfo(buf, "%X ", ((uint8 *) n256->isset)[i]);
+ appendStringInfo(buf, "\n");
+ }
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ appendStringInfo(buf, "%schunk 0x%X\n",
+ spaces.data, i);
+ }
+ else
+ {
+ RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ appendStringInfo(buf, "%schunk 0x%X",
+ spaces.data, i);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i),
+ level + 1, recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ }
+}
+
+RT_SCOPE void
+RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key)
+{
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL node;
+ StringInfoData buf;
+ int shift;
+ int level = 0;
+
+ RT_STATS(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ fprintf(stderr, "empty tree\n");
+ return;
+ }
+
+ if (key > tree->ctl->max_val)
+ {
+ fprintf(stderr, "key " UINT64_FORMAT "(0x" RT_UINT64_FORMAT_HEX ") is larger than max val\n",
+ key, key);
+ return;
+ }
+
+ initStringInfo(&buf);
+ allocnode = tree->ctl->root;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift = node->shift;
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC child;
+
+ RT_DUMP_NODE(tree, allocnode, level, false, &buf);
+
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_VALUE_TYPE dummy;
+
+ /* We reached at a leaf node, find the corresponding slot */
+ RT_NODE_SEARCH_LEAF(node, key, &dummy);
+
+ break;
+ }
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ break;
+
+ allocnode = child;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift -= RT_NODE_SPAN;
+ level++;
+ }
+
+ fprintf(stderr, "%s", buf.data);
+}
+
+RT_SCOPE void
+RT_DUMP(RT_RADIX_TREE *tree)
+{
+ StringInfoData buf;
+
+ RT_STATS(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ fprintf(stderr, "empty tree\n");
+ return;
+ }
+
+ initStringInfo(&buf);
+
+ RT_DUMP_NODE(tree, tree->ctl->root, 0, true, &buf);
+
+ fprintf(stderr, "%s",buf.data);
+}
+#endif
+
+#endif /* RT_DEFINE */
+
+
+/* undefine external parameters, so next radix tree can be defined */
+#undef RT_PREFIX
+#undef RT_SCOPE
+#undef RT_DECLARE
+#undef RT_DEFINE
+#undef RT_VALUE_TYPE
+
+/* locally declared macros */
+#undef RT_MAKE_PREFIX
+#undef RT_MAKE_NAME
+#undef RT_MAKE_NAME_
+#undef RT_NODE_SPAN
+#undef RT_NODE_MAX_SLOTS
+#undef RT_CHUNK_MASK
+#undef RT_MAX_SHIFT
+#undef RT_MAX_LEVEL
+#undef RT_GET_KEY_CHUNK
+#undef BM_IDX
+#undef BM_BIT
+#undef RT_NODE_IS_LEAF
+#undef RT_NODE_MUST_GROW
+#undef RT_NODE_KIND_COUNT
+#undef RT_SIZE_CLASS_COUNT
+#undef RT_SLOT_IDX_LIMIT
+#undef RT_INVALID_SLOT_IDX
+#undef RT_SLAB_BLOCK_SIZE
+#undef RT_RADIX_TREE_MAGIC
+#undef RT_UINT64_FORMAT_HEX
+
+/* type declarations */
+#undef RT_RADIX_TREE
+#undef RT_RADIX_TREE_CONTROL
+#undef RT_PTR_LOCAL
+#undef RT_PTR_ALLOC
+#undef RT_INVALID_PTR_ALLOC
+#undef RT_HANDLE
+#undef RT_ITER
+#undef RT_NODE
+#undef RT_NODE_ITER
+#undef RT_NODE_KIND_3
+#undef RT_NODE_KIND_32
+#undef RT_NODE_KIND_125
+#undef RT_NODE_KIND_256
+#undef RT_NODE_BASE_3
+#undef RT_NODE_BASE_32
+#undef RT_NODE_BASE_125
+#undef RT_NODE_BASE_256
+#undef RT_NODE_INNER_3
+#undef RT_NODE_INNER_32
+#undef RT_NODE_INNER_125
+#undef RT_NODE_INNER_256
+#undef RT_NODE_LEAF_3
+#undef RT_NODE_LEAF_32
+#undef RT_NODE_LEAF_125
+#undef RT_NODE_LEAF_256
+#undef RT_SIZE_CLASS
+#undef RT_SIZE_CLASS_ELEM
+#undef RT_SIZE_CLASS_INFO
+#undef RT_CLASS_3
+#undef RT_CLASS_32_MIN
+#undef RT_CLASS_32_MAX
+#undef RT_CLASS_125
+#undef RT_CLASS_256
+
+/* function declarations */
+#undef RT_CREATE
+#undef RT_FREE
+#undef RT_ATTACH
+#undef RT_DETACH
+#undef RT_GET_HANDLE
+#undef RT_SEARCH
+#undef RT_SET
+#undef RT_BEGIN_ITERATE
+#undef RT_ITERATE_NEXT
+#undef RT_END_ITERATE
+#undef RT_USE_DELETE
+#undef RT_DELETE
+#undef RT_MEMORY_USAGE
+#undef RT_DUMP
+#undef RT_DUMP_NODE
+#undef RT_DUMP_SEARCH
+#undef RT_STATS
+
+/* internal helper functions */
+#undef RT_NEW_ROOT
+#undef RT_ALLOC_NODE
+#undef RT_INIT_NODE
+#undef RT_FREE_NODE
+#undef RT_FREE_RECURSE
+#undef RT_EXTEND
+#undef RT_SET_EXTEND
+#undef RT_SWITCH_NODE_KIND
+#undef RT_COPY_NODE
+#undef RT_REPLACE_NODE
+#undef RT_PTR_GET_LOCAL
+#undef RT_PTR_ALLOC_IS_VALID
+#undef RT_NODE_3_SEARCH_EQ
+#undef RT_NODE_32_SEARCH_EQ
+#undef RT_NODE_3_GET_INSERTPOS
+#undef RT_NODE_32_GET_INSERTPOS
+#undef RT_CHUNK_CHILDREN_ARRAY_SHIFT
+#undef RT_CHUNK_VALUES_ARRAY_SHIFT
+#undef RT_CHUNK_CHILDREN_ARRAY_DELETE
+#undef RT_CHUNK_VALUES_ARRAY_DELETE
+#undef RT_CHUNK_CHILDREN_ARRAY_COPY
+#undef RT_CHUNK_VALUES_ARRAY_COPY
+#undef RT_NODE_125_IS_CHUNK_USED
+#undef RT_NODE_INNER_125_GET_CHILD
+#undef RT_NODE_LEAF_125_GET_VALUE
+#undef RT_NODE_INNER_256_IS_CHUNK_USED
+#undef RT_NODE_LEAF_256_IS_CHUNK_USED
+#undef RT_NODE_INNER_256_GET_CHILD
+#undef RT_NODE_LEAF_256_GET_VALUE
+#undef RT_NODE_INNER_256_SET
+#undef RT_NODE_LEAF_256_SET
+#undef RT_NODE_INNER_256_DELETE
+#undef RT_NODE_LEAF_256_DELETE
+#undef RT_KEY_GET_SHIFT
+#undef RT_SHIFT_GET_MAX_VAL
+#undef RT_NODE_SEARCH_INNER
+#undef RT_NODE_SEARCH_LEAF
+#undef RT_NODE_UPDATE_INNER
+#undef RT_NODE_DELETE_INNER
+#undef RT_NODE_DELETE_LEAF
+#undef RT_NODE_INSERT_INNER
+#undef RT_NODE_INSERT_LEAF
+#undef RT_NODE_INNER_ITERATE_NEXT
+#undef RT_NODE_LEAF_ITERATE_NEXT
+#undef RT_UPDATE_ITER_STACK
+#undef RT_ITER_UPDATE_KEY
+#undef RT_VERIFY_NODE
+
+#undef RT_DEBUG
diff --git a/src/include/lib/radixtree_delete_impl.h b/src/include/lib/radixtree_delete_impl.h
new file mode 100644
index 0000000000..99c90771b9
--- /dev/null
+++ b/src/include/lib/radixtree_delete_impl.h
@@ -0,0 +1,106 @@
+/* TODO: shrink nodes */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_DELETE(n3->base.chunks, n3->values,
+ n3->base.n.count, idx);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_DELETE(n3->base.chunks, n3->children,
+ n3->base.n.count, idx);
+#endif
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_DELETE(n32->base.chunks, n32->values,
+ n32->base.n.count, idx);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_DELETE(n32->base.chunks, n32->children,
+ n32->base.n.count, idx);
+#endif
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+ int idx;
+ int bitnum;
+
+ if (slotpos == RT_INVALID_SLOT_IDX)
+ return false;
+
+ idx = BM_IDX(slotpos);
+ bitnum = BM_BIT(slotpos);
+ n125->base.isset[idx] &= ~((bitmapword) 1 << bitnum);
+ n125->base.slot_idxs[chunk] = RT_INVALID_SLOT_IDX;
+
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_DELETE(n256, chunk);
+#else
+ RT_NODE_INNER_256_DELETE(n256, chunk);
+#endif
+ break;
+ }
+ }
+
+ /* update statistics */
+ node->count--;
+
+ return true;
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
new file mode 100644
index 0000000000..22aca0e6cc
--- /dev/null
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -0,0 +1,317 @@
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+ bool chunk_exists = false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ const bool is_leaf = true;
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+ const bool is_leaf = false;
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx;
+
+ idx = RT_NODE_3_SEARCH_EQ(&n3->base, chunk);
+ if (idx != -1)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+#ifdef RT_NODE_LEVEL_LEAF
+ n3->values[idx] = value;
+#else
+ n3->children[idx] = child;
+#endif
+ break;
+ }
+
+ if (unlikely(RT_NODE_MUST_GROW(n3)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE32_TYPE *new32;
+ const uint8 new_kind = RT_NODE_KIND_32;
+ const RT_SIZE_CLASS new_class = RT_CLASS_32_MIN;
+
+ /* grow node from 3 to 32 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_COPY(n3->base.chunks, n3->values,
+ new32->base.chunks, new32->values);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_COPY(n3->base.chunks, n3->children,
+ new32->base.chunks, new32->children);
+#endif
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
+ int count = n3->base.n.count;
+
+ /* shift chunks and children */
+ if (insertpos < count)
+ {
+ Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_SHIFT(n3->base.chunks, n3->values,
+ count, insertpos);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_SHIFT(n3->base.chunks, n3->children,
+ count, insertpos);
+#endif
+ }
+
+ n3->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+ n3->values[insertpos] = value;
+#else
+ n3->children[insertpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_32:
+ {
+ const RT_SIZE_CLASS_ELEM class32_max = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX];
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx;
+
+ idx = RT_NODE_32_SEARCH_EQ(&n32->base, chunk);
+ if (idx != -1)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+#ifdef RT_NODE_LEVEL_LEAF
+ n32->values[idx] = value;
+#else
+ n32->children[idx] = child;
+#endif
+ break;
+ }
+
+ if (unlikely(RT_NODE_MUST_GROW(n32)) &&
+ n32->base.n.fanout < class32_max.fanout)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ const RT_SIZE_CLASS_ELEM class32_min = RT_SIZE_CLASS_INFO[RT_CLASS_32_MIN];
+ const RT_SIZE_CLASS new_class = RT_CLASS_32_MAX;
+
+ Assert(n32->base.n.fanout == class32_min.fanout);
+
+ /* grow to the next size class of this kind */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ n32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ memcpy(newnode, node, class32_min.leaf_size);
+#else
+ memcpy(newnode, node, class32_min.inner_size);
+#endif
+ newnode->fanout = class32_max.fanout;
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+
+ if (unlikely(RT_NODE_MUST_GROW(n32)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE125_TYPE *new125;
+ const uint8 new_kind = RT_NODE_KIND_125;
+ const RT_SIZE_CLASS new_class = RT_CLASS_125;
+
+ Assert(n32->base.n.fanout == class32_max.fanout);
+
+ /* grow node from 32 to 125 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new125 = (RT_NODE125_TYPE *) newnode;
+
+ for (int i = 0; i < class32_max.fanout; i++)
+ {
+ new125->base.slot_idxs[n32->base.chunks[i]] = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ new125->values[i] = n32->values[i];
+#else
+ new125->children[i] = n32->children[i];
+#endif
+ }
+
+ /*
+ * Since we just copied a dense array, we can set the bits
+ * using a single store, provided the length of that array
+ * is at most the number of bits in a bitmapword.
+ */
+ Assert(class32_max.fanout <= sizeof(bitmapword) * BITS_PER_BYTE);
+ new125->base.isset[0] = (bitmapword) (((uint64) 1 << class32_max.fanout) - 1);
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
+ int count = n32->base.n.count;
+
+ if (insertpos < count)
+ {
+ Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_SHIFT(n32->base.chunks, n32->values,
+ count, insertpos);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_SHIFT(n32->base.chunks, n32->children,
+ count, insertpos);
+#endif
+ }
+
+ n32->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+ n32->values[insertpos] = value;
+#else
+ n32->children[insertpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+ int cnt = 0;
+
+ if (slotpos != RT_INVALID_SLOT_IDX)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+#ifdef RT_NODE_LEVEL_LEAF
+ n125->values[slotpos] = value;
+#else
+ n125->children[slotpos] = child;
+#endif
+ break;
+ }
+
+ if (unlikely(RT_NODE_MUST_GROW(n125)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE256_TYPE *new256;
+ const uint8 new_kind = RT_NODE_KIND_256;
+ const RT_SIZE_CLASS new_class = RT_CLASS_256;
+
+ /* grow node from 125 to 256 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new256 = (RT_NODE256_TYPE *) newnode;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS && cnt < n125->base.n.count; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+ continue;
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_SET(new256, i, RT_NODE_LEAF_125_GET_VALUE(n125, i));
+#else
+ RT_NODE_INNER_256_SET(new256, i, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+ cnt++;
+ }
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int idx;
+ bitmapword inverse;
+
+ /* get the first word with at least one bit not set */
+ for (idx = 0; idx < BM_IDX(RT_SLOT_IDX_LIMIT); idx++)
+ {
+ if (n125->base.isset[idx] < ~((bitmapword) 0))
+ break;
+ }
+
+ /* To get the first unset bit in X, get the first set bit in ~X */
+ inverse = ~(n125->base.isset[idx]);
+ slotpos = idx * BITS_PER_BITMAPWORD;
+ slotpos += bmw_rightmost_one_pos(inverse);
+ Assert(slotpos < node->fanout);
+
+ /* mark the slot used */
+ n125->base.isset[idx] |= bmw_rightmost_one(inverse);
+ n125->base.slot_idxs[chunk] = slotpos;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ n125->values[slotpos] = value;
+#else
+ n125->children[slotpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ chunk_exists = RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk);
+#else
+ chunk_exists = RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk);
+#endif
+ Assert(chunk_exists || node->count < RT_NODE_MAX_SLOTS);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_SET(n256, chunk, value);
+#else
+ RT_NODE_INNER_256_SET(n256, chunk, child);
+#endif
+ break;
+ }
+ }
+
+ /* Update statistics */
+ if (!chunk_exists)
+ node->count++;
+
+ /*
+ * Done. Finally, verify the chunk and value is inserted or replaced
+ * properly in the node.
+ */
+ RT_VERIFY_NODE(node);
+
+ return chunk_exists;
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_iter_impl.h b/src/include/lib/radixtree_iter_impl.h
new file mode 100644
index 0000000000..823d7107c4
--- /dev/null
+++ b/src/include/lib/radixtree_iter_impl.h
@@ -0,0 +1,138 @@
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ bool found = false;
+ uint8 key_chunk;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_VALUE_TYPE value;
+
+ Assert(RT_NODE_IS_LEAF(node_iter->node));
+#else
+ RT_PTR_LOCAL child = NULL;
+
+ Assert(!RT_NODE_IS_LEAF(node_iter->node));
+#endif
+
+#ifdef RT_SHMEM
+ Assert(iter->tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ switch (node_iter->node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node_iter->node;
+
+ node_iter->current_idx++;
+ if (node_iter->current_idx >= n3->base.n.count)
+ break;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = n3->values[node_iter->current_idx];
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, n3->children[node_iter->current_idx]);
+#endif
+ key_chunk = n3->base.chunks[node_iter->current_idx];
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node_iter->node;
+
+ node_iter->current_idx++;
+ if (node_iter->current_idx >= n32->base.n.count)
+ break;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ value = n32->values[node_iter->current_idx];
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, n32->children[node_iter->current_idx]);
+#endif
+ key_chunk = n32->base.chunks[node_iter->current_idx];
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node_iter->node;
+ int i;
+
+ for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (RT_NODE_125_IS_CHUNK_USED((RT_NODE_BASE_125 *) n125, i))
+ break;
+ }
+
+ if (i >= RT_NODE_MAX_SLOTS)
+ break;
+
+ node_iter->current_idx = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = RT_NODE_LEAF_125_GET_VALUE(n125, i);
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+ key_chunk = i;
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node_iter->node;
+ int i;
+
+ for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ {
+#ifdef RT_NODE_LEVEL_LEAF
+ if (RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+#else
+ if (RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+#endif
+ break;
+ }
+
+ if (i >= RT_NODE_MAX_SLOTS)
+ break;
+
+ node_iter->current_idx = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = RT_NODE_LEAF_256_GET_VALUE(n256, i);
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+#endif
+ key_chunk = i;
+ found = true;
+ break;
+ }
+ }
+
+ if (found)
+ {
+ RT_ITER_UPDATE_KEY(iter, key_chunk, node_iter->node->shift);
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = value;
+#endif
+ }
+
+#ifdef RT_NODE_LEVEL_LEAF
+ return found;
+#else
+ return child;
+#endif
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_search_impl.h b/src/include/lib/radixtree_search_impl.h
new file mode 100644
index 0000000000..c8410e9a5c
--- /dev/null
+++ b/src/include/lib/radixtree_search_impl.h
@@ -0,0 +1,122 @@
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(value_p != NULL);
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+#ifndef RT_ACTION_UPDATE
+ Assert(child_p != NULL);
+#endif
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+#ifdef RT_ACTION_UPDATE
+ Assert(idx >= 0);
+ n3->children[idx] = new_child;
+#else
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n3->values[idx];
+#else
+ *child_p = n3->children[idx];
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+#ifdef RT_ACTION_UPDATE
+ Assert(idx >= 0);
+ n32->children[idx] = new_child;
+#else
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n32->values[idx];
+#else
+ *child_p = n32->children[idx];
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+
+#ifdef RT_ACTION_UPDATE
+ Assert(slotpos != RT_INVALID_SLOT_IDX);
+ n125->children[slotpos] = new_child;
+#else
+ if (slotpos == RT_INVALID_SLOT_IDX)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_125_GET_VALUE(n125, chunk);
+#else
+ *child_p = RT_NODE_INNER_125_GET_CHILD(n125, chunk);
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_ACTION_UPDATE
+ RT_NODE_INNER_256_SET(n256, chunk, new_child);
+#else
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_256_GET_VALUE(n256, chunk);
+#else
+ *child_p = RT_NODE_INNER_256_GET_CHILD(n256, chunk);
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ }
+
+#ifdef RT_ACTION_UPDATE
+ return;
+#else
+ return true;
+#endif /* RT_ACTION_UPDATE */
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/utils/dsa.h b/src/include/utils/dsa.h
index 3ce4ee300a..2af215484f 100644
--- a/src/include/utils/dsa.h
+++ b/src/include/utils/dsa.h
@@ -121,6 +121,7 @@ extern dsa_handle dsa_get_handle(dsa_area *area);
extern dsa_pointer dsa_allocate_extended(dsa_area *area, size_t size, int flags);
extern void dsa_free(dsa_area *area, dsa_pointer dp);
extern void *dsa_get_address(dsa_area *area, dsa_pointer dp);
+extern size_t dsa_get_total_size(dsa_area *area);
extern void dsa_trim(dsa_area *area);
extern void dsa_dump(dsa_area *area);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index c629cbe383..9659eb85d7 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -28,6 +28,7 @@ SUBDIRS = \
test_pg_db_role_setting \
test_pg_dump \
test_predtest \
+ test_radixtree \
test_rbtree \
test_regex \
test_rls_hooks \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 1baa6b558d..232cbdac80 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -24,6 +24,7 @@ subdir('test_parser')
subdir('test_pg_db_role_setting')
subdir('test_pg_dump')
subdir('test_predtest')
+subdir('test_radixtree')
subdir('test_rbtree')
subdir('test_regex')
subdir('test_rls_hooks')
diff --git a/src/test/modules/test_radixtree/.gitignore b/src/test/modules/test_radixtree/.gitignore
new file mode 100644
index 0000000000..5dcb3ff972
--- /dev/null
+++ b/src/test/modules/test_radixtree/.gitignore
@@ -0,0 +1,4 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
diff --git a/src/test/modules/test_radixtree/Makefile b/src/test/modules/test_radixtree/Makefile
new file mode 100644
index 0000000000..da06b93da3
--- /dev/null
+++ b/src/test/modules/test_radixtree/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_radixtree/Makefile
+
+MODULE_big = test_radixtree
+OBJS = \
+ $(WIN32RES) \
+ test_radixtree.o
+PGFILEDESC = "test_radixtree - test code for src/backend/lib/radixtree.c"
+
+EXTENSION = test_radixtree
+DATA = test_radixtree--1.0.sql
+
+REGRESS = test_radixtree
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_radixtree
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_radixtree/README b/src/test/modules/test_radixtree/README
new file mode 100644
index 0000000000..a8b271869a
--- /dev/null
+++ b/src/test/modules/test_radixtree/README
@@ -0,0 +1,7 @@
+test_integerset contains unit tests for testing the integer set implementation
+in src/backend/lib/integerset.c.
+
+The tests verify the correctness of the implementation, but they can also be
+used as a micro-benchmark. If you set the 'intset_test_stats' flag in
+test_integerset.c, the tests will print extra information about execution time
+and memory usage.
diff --git a/src/test/modules/test_radixtree/expected/test_radixtree.out b/src/test/modules/test_radixtree/expected/test_radixtree.out
new file mode 100644
index 0000000000..ce645cb8b5
--- /dev/null
+++ b/src/test/modules/test_radixtree/expected/test_radixtree.out
@@ -0,0 +1,36 @@
+CREATE EXTENSION test_radixtree;
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
+NOTICE: testing basic operations with leaf node 4
+NOTICE: testing basic operations with inner node 4
+NOTICE: testing basic operations with leaf node 32
+NOTICE: testing basic operations with inner node 32
+NOTICE: testing basic operations with leaf node 125
+NOTICE: testing basic operations with inner node 125
+NOTICE: testing basic operations with leaf node 256
+NOTICE: testing basic operations with inner node 256
+NOTICE: testing radix tree node types with shift "0"
+NOTICE: testing radix tree node types with shift "8"
+NOTICE: testing radix tree node types with shift "16"
+NOTICE: testing radix tree node types with shift "24"
+NOTICE: testing radix tree node types with shift "32"
+NOTICE: testing radix tree node types with shift "40"
+NOTICE: testing radix tree node types with shift "48"
+NOTICE: testing radix tree node types with shift "56"
+NOTICE: testing radix tree with pattern "all ones"
+NOTICE: testing radix tree with pattern "alternating bits"
+NOTICE: testing radix tree with pattern "clusters of ten"
+NOTICE: testing radix tree with pattern "clusters of hundred"
+NOTICE: testing radix tree with pattern "one-every-64k"
+NOTICE: testing radix tree with pattern "sparse"
+NOTICE: testing radix tree with pattern "single values, distance > 2^32"
+NOTICE: testing radix tree with pattern "clusters, distance > 2^32"
+NOTICE: testing radix tree with pattern "clusters, distance > 2^60"
+ test_radixtree
+----------------
+
+(1 row)
+
diff --git a/src/test/modules/test_radixtree/meson.build b/src/test/modules/test_radixtree/meson.build
new file mode 100644
index 0000000000..6add06bbdb
--- /dev/null
+++ b/src/test/modules/test_radixtree/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_radixtree_sources = files(
+ 'test_radixtree.c',
+)
+
+if host_system == 'windows'
+ test_radixtree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_radixtree',
+ '--FILEDESC', 'test_radixtree - test code for src/include//lib/radixtree.h',])
+endif
+
+test_radixtree = shared_module('test_radixtree',
+ test_radixtree_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += test_radixtree
+
+install_data(
+ 'test_radixtree.control',
+ 'test_radixtree--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'test_radixtree',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'test_radixtree',
+ ],
+ },
+}
diff --git a/src/test/modules/test_radixtree/sql/test_radixtree.sql b/src/test/modules/test_radixtree/sql/test_radixtree.sql
new file mode 100644
index 0000000000..41ece5e9f5
--- /dev/null
+++ b/src/test/modules/test_radixtree/sql/test_radixtree.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_radixtree;
+
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
diff --git a/src/test/modules/test_radixtree/test_radixtree--1.0.sql b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
new file mode 100644
index 0000000000..074a5a7ea7
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_radixtree/test_radixtree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_radixtree" to load this file. \quit
+
+CREATE FUNCTION test_radixtree()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
new file mode 100644
index 0000000000..2a93e731ae
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -0,0 +1,673 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_radixtree.c
+ * Test radixtree set data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_radixtree/test_radixtree.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "storage/lwlock.h"
+#include "utils/memutils.h"
+#include "utils/timestamp.h"
+
+#define UINT64_HEX_FORMAT "%" INT64_MODIFIER "X"
+
+/*
+ * The tests pass with uint32, but build with warnings because the string
+ * format expects uint64.
+ */
+typedef uint64 TestValueType;
+
+/*
+ * If you enable this, the "pattern" tests will print information about
+ * how long populating, probing, and iterating the test set takes, and
+ * how much memory the test set consumed. That can be used as
+ * micro-benchmark of various operations and input patterns (you might
+ * want to increase the number of values used in each of the test, if
+ * you do that, to reduce noise).
+ *
+ * The information is printed to the server's stderr, mostly because
+ * that's where MemoryContextStats() output goes.
+ */
+static const bool rt_test_stats = false;
+
+static int rt_node_kind_fanouts[] = {
+ 0,
+ 4, /* RT_NODE_KIND_4 */
+ 32, /* RT_NODE_KIND_32 */
+ 125, /* RT_NODE_KIND_125 */
+ 256 /* RT_NODE_KIND_256 */
+};
+/*
+ * A struct to define a pattern of integers, for use with the test_pattern()
+ * function.
+ */
+typedef struct
+{
+ char *test_name; /* short name of the test, for humans */
+ char *pattern_str; /* a bit pattern */
+ uint64 spacing; /* pattern repeats at this interval */
+ uint64 num_values; /* number of integers to set in total */
+} test_spec;
+
+/* Test patterns borrowed from test_integerset.c */
+static const test_spec test_specs[] = {
+ {
+ "all ones", "1111111111",
+ 10, 1000000
+ },
+ {
+ "alternating bits", "0101010101",
+ 10, 1000000
+ },
+ {
+ "clusters of ten", "1111111111",
+ 10000, 1000000
+ },
+ {
+ "clusters of hundred",
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111",
+ 10000, 1000000
+ },
+ {
+ "one-every-64k", "1",
+ 65536, 1000000
+ },
+ {
+ "sparse", "100000000000000000000000000000001",
+ 10000000, 1000000
+ },
+ {
+ "single values, distance > 2^32", "1",
+ UINT64CONST(10000000000), 100000
+ },
+ {
+ "clusters, distance > 2^32", "10101010",
+ UINT64CONST(10000000000), 1000000
+ },
+ {
+ "clusters, distance > 2^60", "10101010",
+ UINT64CONST(2000000000000000000),
+ 23 /* can't be much higher than this, or we
+ * overflow uint64 */
+ }
+};
+
+/* define the radix tree implementation to test */
+#define RT_PREFIX rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE TestValueType
+// WIP: compiles with warnings because rt_attach is defined but not used
+// #define RT_SHMEM
+#include "lib/radixtree.h"
+
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+ return tree->ctl->num_keys;
+}
+
+PG_MODULE_MAGIC;
+
+PG_FUNCTION_INFO_V1(test_radixtree);
+
+static void
+test_empty(void)
+{
+ rt_radix_tree *radixtree;
+ rt_iter *iter;
+ TestValueType dummy;
+ uint64 key;
+ TestValueType val;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+ dsa = dsa_create(tranche_id);
+
+ radixtree = rt_create(CurrentMemoryContext, dsa);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ if (rt_search(radixtree, 0, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_search(radixtree, 1, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_search(radixtree, PG_UINT64_MAX, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_delete(radixtree, 0))
+ elog(ERROR, "rt_delete on empty tree returned true");
+
+ if (rt_num_entries(radixtree) != 0)
+ elog(ERROR, "rt_num_entries on empty tree return non-zero");
+
+ iter = rt_begin_iterate(radixtree);
+
+ if (rt_iterate_next(iter, &key, &val))
+ elog(ERROR, "rt_itereate_next on empty tree returned true");
+
+ rt_end_iterate(iter);
+
+ rt_free(radixtree);
+
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+static void
+test_basic(int children, bool test_inner)
+{
+ rt_radix_tree *radixtree;
+ uint64 *keys;
+ int shift = test_inner ? 8 : 0;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing basic operations with %s node %d",
+ test_inner ? "inner" : "leaf", children);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(CurrentMemoryContext, dsa);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ /* prepare keys in order like 1, 32, 2, 31, 2, ... */
+ keys = palloc(sizeof(uint64) * children);
+ for (int i = 0; i < children; i++)
+ {
+ if (i % 2 == 0)
+ keys[i] = (uint64) ((i / 2) + 1) << shift;
+ else
+ keys[i] = (uint64) (children - (i / 2)) << shift;
+ }
+
+ /* insert keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (rt_set(radixtree, keys[i], (TestValueType) keys[i]))
+ elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+ }
+
+ /* look up keys */
+ for (int i = 0; i < children; i++)
+ {
+ TestValueType value;
+
+ if (!rt_search(radixtree, keys[i], &value))
+ elog(ERROR, "could not find key 0x" UINT64_HEX_FORMAT, keys[i]);
+ if (value != (TestValueType) keys[i])
+ elog(ERROR, "rt_search returned 0x" UINT64_HEX_FORMAT ", expected " UINT64_HEX_FORMAT,
+ value, (TestValueType) keys[i]);
+ }
+
+ /* update keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (!rt_set(radixtree, keys[i], (TestValueType) (keys[i] + 1)))
+ elog(ERROR, "could not update key 0x" UINT64_HEX_FORMAT, keys[i]);
+ }
+
+ /* repeat deleting and inserting keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (!rt_delete(radixtree, keys[i]))
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, keys[i]);
+ if (rt_set(radixtree, keys[i], (TestValueType) keys[i]))
+ elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+ }
+
+ pfree(keys);
+ rt_free(radixtree);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Check if keys from start to end with the shift exist in the tree.
+ */
+static void
+check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end,
+ int incr)
+{
+ for (int i = start; i < end; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ TestValueType val;
+
+ if (!rt_search(radixtree, key, &val))
+ elog(ERROR, "key 0x" UINT64_HEX_FORMAT " is not found on node-%d",
+ key, end);
+ if (val != (TestValueType) key)
+ elog(ERROR, "rt_search with key 0x" UINT64_HEX_FORMAT " returns 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+ key, val, key);
+ }
+}
+
+static void
+test_node_types_insert(rt_radix_tree *radixtree, uint8 shift, bool insert_asc)
+{
+ uint64 num_entries;
+ int ninserted = 0;
+ int start = insert_asc ? 0 : 256;
+ int incr = insert_asc ? 1 : -1;
+ int end = insert_asc ? 256 : 0;
+ int node_kind_idx = 1;
+
+ for (int i = start; i != end; i += incr)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_set(radixtree, key, (TestValueType) key);
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " is found", key);
+
+ /*
+ * After filling all slots in each node type, check if the values
+ * are stored properly.
+ */
+ if (ninserted == rt_node_kind_fanouts[node_kind_idx] - 1)
+ {
+ int check_start = insert_asc
+ ? rt_node_kind_fanouts[node_kind_idx - 1]
+ : rt_node_kind_fanouts[node_kind_idx];
+ int check_end = insert_asc
+ ? rt_node_kind_fanouts[node_kind_idx]
+ : rt_node_kind_fanouts[node_kind_idx - 1];
+
+ check_search_on_node(radixtree, shift, check_start, check_end, incr);
+ node_kind_idx++;
+ }
+
+ ninserted++;
+ }
+
+ num_entries = rt_num_entries(radixtree);
+
+ if (num_entries != 256)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+static void
+test_node_types_delete(rt_radix_tree *radixtree, uint8 shift)
+{
+ uint64 num_entries;
+
+ for (int i = 0; i < 256; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_delete(radixtree, key);
+
+ if (!found)
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, key);
+ }
+
+ num_entries = rt_num_entries(radixtree);
+
+ /* The tree must be empty */
+ if (num_entries != 0)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+/*
+ * Test for inserting and deleting key-value pairs to each node type at the given shift
+ * level.
+ */
+static void
+test_node_types(uint8 shift)
+{
+ rt_radix_tree *radixtree;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing radix tree node types with shift \"%d\"", shift);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(CurrentMemoryContext, dsa);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ /*
+ * Insert and search entries for every node type at the 'shift' level,
+ * then delete all entries to make it empty, and insert and search entries
+ * again.
+ */
+ test_node_types_insert(radixtree, shift, true);
+ test_node_types_delete(radixtree, shift);
+ test_node_types_insert(radixtree, shift, false);
+
+ rt_free(radixtree);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Test with a repeating pattern, defined by the 'spec'.
+ */
+static void
+test_pattern(const test_spec * spec)
+{
+ rt_radix_tree *radixtree;
+ rt_iter *iter;
+ MemoryContext radixtree_ctx;
+ TimestampTz starttime;
+ TimestampTz endtime;
+ uint64 n;
+ uint64 last_int;
+ uint64 ndeleted;
+ uint64 nbefore;
+ uint64 nafter;
+ int patternlen;
+ uint64 *pattern_values;
+ uint64 pattern_num_values;
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing radix tree with pattern \"%s\"", spec->test_name);
+ if (rt_test_stats)
+ fprintf(stderr, "-----\ntesting radix tree with pattern \"%s\"\n", spec->test_name);
+
+ /* Pre-process the pattern, creating an array of integers from it. */
+ patternlen = strlen(spec->pattern_str);
+ pattern_values = palloc(patternlen * sizeof(uint64));
+ pattern_num_values = 0;
+ for (int i = 0; i < patternlen; i++)
+ {
+ if (spec->pattern_str[i] == '1')
+ pattern_values[pattern_num_values++] = i;
+ }
+
+ /*
+ * Allocate the radix tree.
+ *
+ * Allocate it in a separate memory context, so that we can print its
+ * memory usage easily.
+ */
+ radixtree_ctx = AllocSetContextCreate(CurrentMemoryContext,
+ "radixtree test",
+ ALLOCSET_SMALL_SIZES);
+ MemoryContextSetIdentifier(radixtree_ctx, spec->test_name);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(radixtree_ctx, dsa);
+#else
+ radixtree = rt_create(radixtree_ctx);
+#endif
+
+
+ /*
+ * Add values to the set.
+ */
+ starttime = GetCurrentTimestamp();
+
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ uint64 x = 0;
+
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ bool found;
+
+ x = last_int + pattern_values[i];
+
+ found = rt_set(radixtree, x, (TestValueType) x);
+
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " found", x);
+
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+
+ endtime = GetCurrentTimestamp();
+
+ if (rt_test_stats)
+ fprintf(stderr, "added " UINT64_FORMAT " values in %d ms\n",
+ spec->num_values, (int) (endtime - starttime) / 1000);
+
+ /*
+ * Print stats on the amount of memory used.
+ *
+ * We print the usage reported by rt_memory_usage(), as well as the stats
+ * from the memory context. They should be in the same ballpark, but it's
+ * hard to automate testing that, so if you're making changes to the
+ * implementation, just observe that manually.
+ */
+ if (rt_test_stats)
+ {
+ uint64 mem_usage;
+
+ /*
+ * Also print memory usage as reported by rt_memory_usage(). It
+ * should be in the same ballpark as the usage reported by
+ * MemoryContextStats().
+ */
+ mem_usage = rt_memory_usage(radixtree);
+ fprintf(stderr, "rt_memory_usage() reported " UINT64_FORMAT " (%0.2f bytes / integer)\n",
+ mem_usage, (double) mem_usage / spec->num_values);
+
+ MemoryContextStats(radixtree_ctx);
+ }
+
+ /* Check that rt_num_entries works */
+ n = rt_num_entries(radixtree);
+ if (n != spec->num_values)
+ elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT, n, spec->num_values);
+
+ /*
+ * Test random-access probes with rt_search()
+ */
+ starttime = GetCurrentTimestamp();
+
+ for (n = 0; n < 100000; n++)
+ {
+ bool found;
+ bool expected;
+ uint64 x;
+ TestValueType v;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit values that are actually in the set.
+ */
+ x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+ /* Do we expect this value to be present in the set? */
+ if (x >= last_int)
+ expected = false;
+ else
+ {
+ uint64 idx = x % spec->spacing;
+
+ if (idx >= patternlen)
+ expected = false;
+ else if (spec->pattern_str[idx] == '1')
+ expected = true;
+ else
+ expected = false;
+ }
+
+ /* Is it present according to rt_search() ? */
+ found = rt_search(radixtree, x, &v);
+
+ if (found != expected)
+ elog(ERROR, "mismatch at 0x" UINT64_HEX_FORMAT ": %d vs %d", x, found, expected);
+ if (found && (v != (TestValueType) x))
+ elog(ERROR, "found 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+ v, x);
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "probed " UINT64_FORMAT " values in %d ms\n",
+ n, (int) (endtime - starttime) / 1000);
+
+ /*
+ * Test iterator
+ */
+ starttime = GetCurrentTimestamp();
+
+ iter = rt_begin_iterate(radixtree);
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ uint64 expected = last_int + pattern_values[i];
+ uint64 x;
+ TestValueType val;
+
+ if (!rt_iterate_next(iter, &x, &val))
+ break;
+
+ if (x != expected)
+ elog(ERROR,
+ "iterate returned wrong key; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d",
+ x, expected, i);
+ if (val != (TestValueType) expected)
+ elog(ERROR,
+ "iterate returned wrong value; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d", x, expected, i);
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "iterated " UINT64_FORMAT " values in %d ms\n",
+ n, (int) (endtime - starttime) / 1000);
+
+ rt_end_iterate(iter);
+
+ if (n < spec->num_values)
+ elog(ERROR, "iterator stopped short after " UINT64_FORMAT " entries, expected " UINT64_FORMAT, n, spec->num_values);
+ if (n > spec->num_values)
+ elog(ERROR, "iterator returned " UINT64_FORMAT " entries, " UINT64_FORMAT " was expected", n, spec->num_values);
+
+ /*
+ * Test random-access probes with rt_delete()
+ */
+ starttime = GetCurrentTimestamp();
+
+ nbefore = rt_num_entries(radixtree);
+ ndeleted = 0;
+ for (n = 0; n < 1; n++)
+ {
+ bool found;
+ uint64 x;
+ TestValueType v;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit values that are actually in the set.
+ */
+ x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+ /* Is it present according to rt_search() ? */
+ found = rt_search(radixtree, x, &v);
+
+ if (!found)
+ continue;
+
+ /* If the key is found, delete it and check again */
+ if (!rt_delete(radixtree, x))
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, x);
+ if (rt_search(radixtree, x, &v))
+ elog(ERROR, "found deleted key 0x" UINT64_HEX_FORMAT, x);
+ if (rt_delete(radixtree, x))
+ elog(ERROR, "deleted already-deleted key 0x" UINT64_HEX_FORMAT, x);
+
+ ndeleted++;
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "deleted " UINT64_FORMAT " values in %d ms\n",
+ ndeleted, (int) (endtime - starttime) / 1000);
+
+ nafter = rt_num_entries(radixtree);
+
+ /* Check that rt_num_entries works */
+ if ((nbefore - ndeleted) != nafter)
+ elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT "after " UINT64_FORMAT " deletion",
+ nafter, (nbefore - ndeleted), ndeleted);
+
+ rt_free(radixtree);
+ MemoryContextDelete(radixtree_ctx);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+Datum
+test_radixtree(PG_FUNCTION_ARGS)
+{
+ test_empty();
+
+ for (int i = 1; i < lengthof(rt_node_kind_fanouts); i++)
+ {
+ test_basic(rt_node_kind_fanouts[i], false);
+ test_basic(rt_node_kind_fanouts[i], true);
+ }
+
+ for (int shift = 0; shift <= (64 - 8); shift += 8)
+ test_node_types(shift);
+
+ /* Test different test patterns, with lots of entries */
+ for (int i = 0; i < lengthof(test_specs); i++)
+ test_pattern(&test_specs[i]);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_radixtree/test_radixtree.control b/src/test/modules/test_radixtree/test_radixtree.control
new file mode 100644
index 0000000000..e53f2a3e0c
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.control
@@ -0,0 +1,4 @@
+comment = 'Test code for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/test_radixtree'
+relocatable = true
diff --git a/src/tools/pginclude/cpluspluscheck b/src/tools/pginclude/cpluspluscheck
index e52fe9f509..09fa6e7432 100755
--- a/src/tools/pginclude/cpluspluscheck
+++ b/src/tools/pginclude/cpluspluscheck
@@ -101,6 +101,12 @@ do
test "$f" = src/include/nodes/nodetags.h && continue
test "$f" = src/backend/nodes/nodetags.h && continue
+ # radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+ test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+ test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+ test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+ test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
# These files are not meant to be included standalone, because
# they contain lists that might have multiple use-cases.
test "$f" = src/include/access/rmgrlist.h && continue
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index abbba7aa63..d4d2f1da03 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -96,6 +96,12 @@ do
test "$f" = src/include/nodes/nodetags.h && continue
test "$f" = src/backend/nodes/nodetags.h && continue
+ # radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+ test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+ test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+ test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+ test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
# These files are not meant to be included standalone, because
# they contain lists that might have multiple use-cases.
test "$f" = src/include/access/rmgrlist.h && continue
--
2.31.1
[application/octet-stream] v24-0004-Tool-for-measuring-radix-tree-performance.patch (22.0K, ../../CAD21AoCSkEDYAqW3+JtSQQY+R2tDCrXA=PgNAUXUdUQ5gBOp0Q@mail.gmail.com/10-v24-0004-Tool-for-measuring-radix-tree-performance.patch)
download | inline diff:
From aa1bb230f2760dbc9185b3237bbd4aba735b20c0 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 16 Sep 2022 11:57:03 +0900
Subject: [PATCH v24 4/9] Tool for measuring radix tree performance
Includes Meson support, but commented out to avoid warnings
XXX: Not for commit
---
contrib/bench_radix_tree/Makefile | 21 +
.../bench_radix_tree--1.0.sql | 76 ++
contrib/bench_radix_tree/bench_radix_tree.c | 656 ++++++++++++++++++
.../bench_radix_tree/bench_radix_tree.control | 6 +
contrib/bench_radix_tree/expected/bench.out | 13 +
contrib/bench_radix_tree/meson.build | 33 +
contrib/bench_radix_tree/sql/bench.sql | 16 +
contrib/meson.build | 1 +
8 files changed, 822 insertions(+)
create mode 100644 contrib/bench_radix_tree/Makefile
create mode 100644 contrib/bench_radix_tree/bench_radix_tree--1.0.sql
create mode 100644 contrib/bench_radix_tree/bench_radix_tree.c
create mode 100644 contrib/bench_radix_tree/bench_radix_tree.control
create mode 100644 contrib/bench_radix_tree/expected/bench.out
create mode 100644 contrib/bench_radix_tree/meson.build
create mode 100644 contrib/bench_radix_tree/sql/bench.sql
diff --git a/contrib/bench_radix_tree/Makefile b/contrib/bench_radix_tree/Makefile
new file mode 100644
index 0000000000..952bb0ceae
--- /dev/null
+++ b/contrib/bench_radix_tree/Makefile
@@ -0,0 +1,21 @@
+# contrib/bench_radix_tree/Makefile
+
+MODULE_big = bench_radix_tree
+OBJS = \
+ bench_radix_tree.o
+
+EXTENSION = bench_radix_tree
+DATA = bench_radix_tree--1.0.sql
+
+REGRESS = bench_fixed_height
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/bench_radix_tree
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
new file mode 100644
index 0000000000..2fd689aa91
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
@@ -0,0 +1,76 @@
+/* contrib/bench_radix_tree/bench_radix_tree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION bench_radix_tree" to load this file. \quit
+
+create function bench_shuffle_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_seq_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_load_random_int(
+cnt int8,
+OUT mem_allocated int8,
+OUT load_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_search_random_nodes(
+cnt int8,
+filter_str text DEFAULT NULL,
+OUT mem_allocated int8,
+OUT search_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C VOLATILE PARALLEL UNSAFE;
+
+create function bench_fixed_height_search(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT rt_search_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_node128_load(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_sparseload_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
new file mode 100644
index 0000000000..4c785c7336
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -0,0 +1,656 @@
+/*-------------------------------------------------------------------------
+ *
+ * bench_radix_tree.c
+ *
+ * Copyright (c) 2016-2022, PostgreSQL Global Development Group
+ *
+ * contrib/bench_radix_tree/bench_radix_tree.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "funcapi.h"
+#include "lib/radixtree.h"
+#include <math.h>
+#include "miscadmin.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/timestamp.h"
+
+PG_MODULE_MAGIC;
+
+/* run benchmark also for binary-search case? */
+/* #define MEASURE_BINARY_SEARCH 1 */
+
+#define TIDS_PER_BLOCK_FOR_LOAD 30
+#define TIDS_PER_BLOCK_FOR_LOOKUP 50
+
+#define RT_DEBUG
+#define RT_PREFIX rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE uint64
+// WIP: compiles with warnings because rt_attach is defined but not used
+// #define RT_SHMEM
+#include "lib/radixtree.h"
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+ return tree->ctl->num_keys;
+}
+
+
+PG_FUNCTION_INFO_V1(bench_seq_search);
+PG_FUNCTION_INFO_V1(bench_shuffle_search);
+PG_FUNCTION_INFO_V1(bench_load_random_int);
+PG_FUNCTION_INFO_V1(bench_fixed_height_search);
+PG_FUNCTION_INFO_V1(bench_search_random_nodes);
+PG_FUNCTION_INFO_V1(bench_node128_load);
+
+static uint64
+tid_to_key_off(ItemPointer tid, uint32 *off)
+{
+ uint64 upper;
+ uint32 shift = pg_ceil_log2_32(MaxHeapTuplesPerPage);
+ int64 tid_i;
+
+ Assert(ItemPointerGetOffsetNumber(tid) < MaxHeapTuplesPerPage);
+
+ tid_i = ItemPointerGetOffsetNumber(tid);
+ tid_i |= (uint64) ItemPointerGetBlockNumber(tid) << shift;
+
+ /* log(sizeof(uint64) * BITS_PER_BYTE, 2) = log(64, 2) = 6 */
+ *off = tid_i & ((1 << 6) - 1);
+ upper = tid_i >> 6;
+ Assert(*off < (sizeof(uint64) * BITS_PER_BYTE));
+
+ Assert(*off < 64);
+
+ return upper;
+}
+
+static int
+shuffle_randrange(pg_prng_state *state, int lower, int upper)
+{
+ return (int) floor(pg_prng_double(state) * ((upper - lower) + 0.999999)) + lower;
+}
+
+/* Naive Fisher-Yates implementation*/
+static void
+shuffle_itemptrs(ItemPointer itemptr, uint64 nitems)
+{
+ /* reproducability */
+ pg_prng_state state;
+
+ pg_prng_seed(&state, 0);
+
+ for (int i = 0; i < nitems - 1; i++)
+ {
+ int j = shuffle_randrange(&state, i, nitems - 1);
+ ItemPointerData t = itemptr[j];
+
+ itemptr[j] = itemptr[i];
+ itemptr[i] = t;
+ }
+}
+
+static ItemPointer
+generate_tids(BlockNumber minblk, BlockNumber maxblk, int ntids_per_blk, uint64 *ntids_p,
+ bool random_block)
+{
+ ItemPointer tids;
+ uint64 maxitems;
+ uint64 ntids = 0;
+ pg_prng_state state;
+
+ maxitems = (maxblk - minblk + 1) * ntids_per_blk;
+ tids = MemoryContextAllocHuge(TopTransactionContext,
+ sizeof(ItemPointerData) * maxitems);
+
+ if (random_block)
+ pg_prng_seed(&state, 0x9E3779B185EBCA87);
+
+ for (BlockNumber blk = minblk; blk < maxblk; blk++)
+ {
+ if (random_block && !pg_prng_bool(&state))
+ continue;
+
+ for (OffsetNumber off = FirstOffsetNumber;
+ off <= ntids_per_blk; off++)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ ItemPointerSetBlockNumber(&(tids[ntids]), blk);
+ ItemPointerSetOffsetNumber(&(tids[ntids]), off);
+
+ ntids++;
+ }
+ }
+
+ *ntids_p = ntids;
+ return tids;
+}
+
+#ifdef MEASURE_BINARY_SEARCH
+static int
+vac_cmp_itemptr(const void *left, const void *right)
+{
+ BlockNumber lblk,
+ rblk;
+ OffsetNumber loff,
+ roff;
+
+ lblk = ItemPointerGetBlockNumber((ItemPointer) left);
+ rblk = ItemPointerGetBlockNumber((ItemPointer) right);
+
+ if (lblk < rblk)
+ return -1;
+ if (lblk > rblk)
+ return 1;
+
+ loff = ItemPointerGetOffsetNumber((ItemPointer) left);
+ roff = ItemPointerGetOffsetNumber((ItemPointer) right);
+
+ if (loff < roff)
+ return -1;
+ if (loff > roff)
+ return 1;
+
+ return 0;
+}
+#endif
+
+static Datum
+bench_search(FunctionCallInfo fcinfo, bool shuffle)
+{
+ BlockNumber minblk = PG_GETARG_INT32(0);
+ BlockNumber maxblk = PG_GETARG_INT32(1);
+ bool random_block = PG_GETARG_BOOL(2);
+ rt_radix_tree *rt = NULL;
+ uint64 ntids;
+ uint64 key;
+ uint64 last_key = PG_UINT64_MAX;
+ uint64 val = 0;
+ ItemPointer tids;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_load_ms,
+ rt_search_ms;
+ Datum values[7];
+ bool nulls[7];
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ tids = generate_tids(minblk, maxblk, TIDS_PER_BLOCK_FOR_LOAD, &ntids, random_block);
+
+ /* measure the load time of the radix tree */
+ rt = rt_create(CurrentMemoryContext);
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ uint32 off;
+
+ CHECK_FOR_INTERRUPTS();
+
+ key = tid_to_key_off(tid, &off);
+
+ if (last_key != PG_UINT64_MAX && last_key != key)
+ {
+ rt_set(rt, last_key, val);
+ val = 0;
+ }
+
+ last_key = key;
+ val |= (uint64) 1 << off;
+ }
+ if (last_key != PG_UINT64_MAX)
+ rt_set(rt, last_key, val);
+
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_load_ms = secs * 1000 + usecs / 1000;
+
+ rt_stats(rt);
+
+ if (shuffle)
+ shuffle_itemptrs(tids, ntids);
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* meaure the serach time of the radix tree */
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ uint32 off;
+ volatile bool ret; /* prevent calling rt_search from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ key = tid_to_key_off(tid, &off);
+
+ ret = rt_search(rt, key, &val);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_search_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int64GetDatum(rt_num_entries(rt));
+ values[1] = Int64GetDatum(rt_memory_usage(rt));
+ values[2] = Int64GetDatum(sizeof(ItemPointerData) * ntids);
+ values[3] = Int64GetDatum(rt_load_ms);
+ nulls[4] = true; /* ar_load_ms */
+ values[5] = Int64GetDatum(rt_search_ms);
+ nulls[6] = true; /* ar_search_ms */
+
+#ifdef MEASURE_BINARY_SEARCH
+ {
+ ItemPointer itemptrs = NULL;
+
+ int64 ar_load_ms,
+ ar_search_ms;
+
+ /* measure the load time of the array */
+ itemptrs = MemoryContextAllocHuge(CurrentMemoryContext,
+ sizeof(ItemPointerData) * ntids);
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointerSetBlockNumber(&(itemptrs[i]),
+ ItemPointerGetBlockNumber(&(tids[i])));
+ ItemPointerSetOffsetNumber(&(itemptrs[i]),
+ ItemPointerGetOffsetNumber(&(tids[i])));
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ ar_load_ms = secs * 1000 + usecs / 1000;
+
+ /* next, measure the serach time of the array */
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ volatile bool ret; /* prevent calling bsearch from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ ret = bsearch((void *) tid,
+ (void *) itemptrs,
+ ntids,
+ sizeof(ItemPointerData),
+ vac_cmp_itemptr);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ ar_search_ms = secs * 1000 + usecs / 1000;
+
+ /* set the result */
+ nulls[4] = false;
+ values[4] = Int64GetDatum(ar_load_ms);
+ nulls[6] = false;
+ values[6] = Int64GetDatum(ar_search_ms);
+ }
+#endif
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_seq_search(PG_FUNCTION_ARGS)
+{
+ return bench_search(fcinfo, false);
+}
+
+Datum
+bench_shuffle_search(PG_FUNCTION_ARGS)
+{
+ return bench_search(fcinfo, true);
+}
+
+Datum
+bench_load_random_int(PG_FUNCTION_ARGS)
+{
+ uint64 cnt = (uint64) PG_GETARG_INT64(0);
+ rt_radix_tree *rt;
+ pg_prng_state state;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_time_ms;
+ Datum values[2];
+ bool nulls[2];
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ pg_prng_seed(&state, 0);
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 key = pg_prng_uint64(&state);
+
+ rt_set(rt, key, key);
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_time_ms = secs * 1000 + usecs / 1000;
+
+ rt_stats(rt);
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int64GetDatum(rt_memory_usage(rt));
+ values[1] = Int64GetDatum(load_time_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+/* copy of splitmix64() */
+static uint64
+hash64(uint64 x)
+{
+ x ^= x >> 30;
+ x *= UINT64CONST(0xbf58476d1ce4e5b9);
+ x ^= x >> 27;
+ x *= UINT64CONST(0x94d049bb133111eb);
+ x ^= x >> 31;
+ return x;
+}
+
+/* attempts to have a relatively even population of node kinds */
+Datum
+bench_search_random_nodes(PG_FUNCTION_ARGS)
+{
+ uint64 cnt = (uint64) PG_GETARG_INT64(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 search_time_ms;
+ Datum values[2] = {0};
+ bool nulls[2] = {0};
+ /* from trial and error */
+ uint64 filter = (((uint64) 0x7F<<32) | (0x07<<24) | (0xFF<<16) | 0xFF);
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (!PG_ARGISNULL(1))
+ {
+ char *filter_str = text_to_cstring(PG_GETARG_TEXT_P(1));
+
+ if (sscanf(filter_str, "0x%lX", &filter) == 0)
+ elog(ERROR, "invalid filter string %s", filter_str);
+ }
+ elog(NOTICE, "bench with filter 0x%lX", filter);
+
+ rt = rt_create(CurrentMemoryContext);
+
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ const uint64 hash = hash64(i);
+ const uint64 key = hash & filter;
+
+ rt_set(rt, key, key);
+ }
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ const uint64 hash = hash64(i);
+ const uint64 key = hash & filter;
+ uint64 val;
+ volatile bool ret; /* prevent calling rt_search from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ ret = rt_search(rt, key, &val);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ search_time_ms = secs * 1000 + usecs / 1000;
+
+ rt_stats(rt);
+
+ values[0] = Int64GetDatum(rt_memory_usage(rt));
+ values[1] = Int64GetDatum(search_time_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_fixed_height_search(PG_FUNCTION_ARGS)
+{
+ int fanout = PG_GETARG_INT32(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_load_ms,
+ rt_search_ms;
+ Datum values[5];
+ bool nulls[5];
+
+ /* test boundary between vector and iteration */
+ const int n_keys = 5 * 16 * 16 * 16 * 16;
+ uint64 r,
+ h,
+ i,
+ j,
+ k;
+ int key_id;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+
+ key_id = 0;
+
+ /*
+ * lower nodes have limited fanout, the top is only limited by
+ * bits-per-byte
+ */
+ for (r = 1;; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ for (i = 1; i <= fanout; i++)
+ {
+ for (j = 1; j <= fanout; j++)
+ {
+ for (k = 1; k <= fanout; k++)
+ {
+ uint64 key;
+
+ key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+ CHECK_FOR_INTERRUPTS();
+
+ key_id++;
+ if (key_id > n_keys)
+ goto finish_set;
+
+ rt_set(rt, key, key_id);
+ }
+ }
+ }
+ }
+ }
+finish_set:
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_load_ms = secs * 1000 + usecs / 1000;
+
+ rt_stats(rt);
+
+ /* meaure the search time of the radix tree */
+ start_time = GetCurrentTimestamp();
+
+ key_id = 0;
+ for (r = 1;; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ for (i = 1; i <= fanout; i++)
+ {
+ for (j = 1; j <= fanout; j++)
+ {
+ for (k = 1; k <= fanout; k++)
+ {
+ uint64 key,
+ val;
+
+ key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+ CHECK_FOR_INTERRUPTS();
+
+ key_id++;
+ if (key_id > n_keys)
+ goto finish_search;
+
+ rt_search(rt, key, &val);
+ }
+ }
+ }
+ }
+ }
+finish_search:
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_search_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int32GetDatum(fanout);
+ values[1] = Int64GetDatum(rt_num_entries(rt));
+ values[2] = Int64GetDatum(rt_memory_usage(rt));
+ values[3] = Int64GetDatum(rt_load_ms);
+ values[4] = Int64GetDatum(rt_search_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_node128_load(PG_FUNCTION_ARGS)
+{
+ int fanout = PG_GETARG_INT32(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_sparseload_ms;
+ Datum values[5];
+ bool nulls[5];
+
+ uint64 r,
+ h;
+ int key_id;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ rt = rt_create(CurrentMemoryContext);
+
+ key_id = 0;
+
+ for (r = 1; r <= fanout; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (h);
+
+ key_id++;
+ rt_set(rt, key, key_id);
+ }
+ }
+
+ rt_stats(rt);
+
+ /* measure sparse deletion and re-loading */
+ start_time = GetCurrentTimestamp();
+
+ for (int t = 0; t<10000; t++)
+ {
+ /* delete one key in each leaf */
+ for (r = 1; r <= fanout; r++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (fanout);
+
+ rt_delete(rt, key);
+ }
+
+ /* add them all back */
+ key_id = 0;
+ for (r = 1; r <= fanout; r++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (fanout);
+
+ key_id++;
+ rt_set(rt, key, key_id);
+ }
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_sparseload_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int32GetDatum(fanout);
+ values[1] = Int64GetDatum(rt_num_entries(rt));
+ values[2] = Int64GetDatum(rt_memory_usage(rt));
+ values[3] = Int64GetDatum(rt_sparseload_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
diff --git a/contrib/bench_radix_tree/bench_radix_tree.control b/contrib/bench_radix_tree/bench_radix_tree.control
new file mode 100644
index 0000000000..1d988e6c9a
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.control
@@ -0,0 +1,6 @@
+# bench_radix_tree extension
+comment = 'benchmark suits for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/bench_radix_tree'
+relocatable = true
+trusted = true
diff --git a/contrib/bench_radix_tree/expected/bench.out b/contrib/bench_radix_tree/expected/bench.out
new file mode 100644
index 0000000000..60c303892e
--- /dev/null
+++ b/contrib/bench_radix_tree/expected/bench.out
@@ -0,0 +1,13 @@
+create extension bench_radix_tree;
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/bench_radix_tree/meson.build b/contrib/bench_radix_tree/meson.build
new file mode 100644
index 0000000000..332c1ae7df
--- /dev/null
+++ b/contrib/bench_radix_tree/meson.build
@@ -0,0 +1,33 @@
+bench_radix_tree_sources = files(
+ 'bench_radix_tree.c',
+)
+
+if host_system == 'windows'
+ bench_radix_tree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'bench_radix_tree',
+ '--FILEDESC', 'bench_radix_tree - performance test code for radix tree',])
+endif
+
+bench_radix_tree = shared_module('bench_radix_tree',
+ bench_radix_tree_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += bench_radix_tree
+
+install_data(
+ 'bench_radix_tree.control',
+ 'bench_radix_tree--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'bench_radix_tree',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'bench_radix_tree',
+ ],
+ },
+}
diff --git a/contrib/bench_radix_tree/sql/bench.sql b/contrib/bench_radix_tree/sql/bench.sql
new file mode 100644
index 0000000000..a46018c9d4
--- /dev/null
+++ b/contrib/bench_radix_tree/sql/bench.sql
@@ -0,0 +1,16 @@
+create extension bench_radix_tree;
+
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/meson.build b/contrib/meson.build
index bd4a57c43c..52253de793 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,6 +12,7 @@ subdir('amcheck')
subdir('auth_delay')
subdir('auto_explain')
subdir('basic_archive')
+#subdir('bench_radix_tree')
subdir('bloom')
subdir('basebackup_to_shell')
subdir('bool_plperl')
--
2.31.1
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-02-07 09:25 ` John Naylor <[email protected]>
2023-02-07 10:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
0 siblings, 2 replies; 78+ messages in thread
From: John Naylor @ 2023-02-07 09:25 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Tue, Jan 31, 2023 at 9:43 PM Masahiko Sawada <[email protected]>
wrote:
> I've attached v24 patches. The locking support patch is separated
> (0005 patch). Also I kept the updates for TidStore and the vacuum
> integration from v23 separate.
Okay, that's a lot more simple, and closer to what I imagined. For v25, I
squashed v24's additions and added a couple of my own. I've kept the CF
status at "needs review" because no specific action is required at the
moment.
I did start to review the TID store some more, but that's on hold because
something else came up: On a lark I decided to re-run some benchmarks to
see if anything got lost in converting to a template, and that led me down
a rabbit hole -- some good and bad news on that below.
0001:
I removed the uint64 case, as discussed. There is now a brief commit
message, but needs to be fleshed out a bit. I took another look at the Arm
optimization that Nathan found some month ago, for forming the highbit
mask, but that doesn't play nicely with how node32 uses it, so I decided
against it. I added a comment to describe the reasoning in case someone
else gets a similar idea.
I briefly looked into "separate-commit TODO: move non-SIMD fallbacks to
their own header to clean up the #ifdef maze.", but decided it wasn't such
a clear win to justify starting the work now. It's still in the back of my
mind, but I removed the reminder from the commit message.
0003:
The template now requires the value to be passed as a pointer. That was a
pretty trivial change, but affected multiple other patches, so not sent
separately. Also adds a forgotten RT_ prefix to the bitmap macros and adds
a top comment to the *_impl.h headers. There are some comment fixes. The
changes were either trivial or discussed earlier, so also not sent
separately.
0004/5: I wanted to measure the load time as well as search time in
bench_search_random_nodes(). That's kept separate to make it easier to test
other patch versions.
The bad news is that the speed of loading TIDs in
bench_seq/shuffle_search() has regressed noticeably. I can't reproduce this
in any other bench function and was the reason for writing 0005 to begin
with. More confusingly, my efforts to fix this improved *other* functions,
but the former didn't budge at all. First the patches:
0006 adds and removes some "inline" declarations (where it made sense), and
added some for "pg_noinline" based on Andres' advice some months ago.
0007 removes some dead code. RT_NODE_INSERT_INNER is only called during
RT_SET_EXTEND, so it can't possibly find an existing key. This kind of
change is much easier with the inner/node cases handled together in a
template, as far as being sure of how those cases are different. I thought
about trying the search in assert builds and verifying it doesn't exist,
but thought yet another #ifdef would be too messy.
v25-addendum-try-no-maintain-order.txt -- It makes optional keeping the key
chunks in order for the linear-search nodes. I believe the TID store no
longer cares about the ordering, but this is a text file for now because I
don't want to clutter the CI with a behavior change. Also, the second ART
paper (on concurrency) mentioned that some locking schemes don't allow
these arrays to be shifted. So it might make sense to give up entirely on
guaranteeing ordered iteration, or at least make it optional as in the
patch.
Now for some numbers:
========================================
psql -c "select * from bench_search_random_nodes(10*1000*1000)"
(min load time of three)
v15:
mem_allocated | load_ms | search_ms
---------------+---------+-----------
334182184 | 3352 | 2073
v25-0005:
mem_allocated | load_ms | search_ms
---------------+---------+-----------
331987008 | 3426 | 2126
v25-0006 (inlining or not):
mem_allocated | load_ms | search_ms
---------------+---------+-----------
331987008 | 3327 | 2035
v25-0007 (remove dead code):
mem_allocated | load_ms | search_ms
---------------+---------+-----------
331987008 | 3313 | 2037
v25-addendum...txt (no ordering):
mem_allocated | load_ms | search_ms
---------------+---------+-----------
331987008 | 2762 | 2042
Allowing unordered inserts helps a lot here in loading. That's expected
because there are a lot of inserts into the linear nodes. 0006 might help a
little.
========================================
psql -c "select avg(load_ms) from generate_series(1,30) x(x), lateral
(select * from bench_load_random_int(500 * 1000 * (1+x-x))) a"
v15:
avg
----------------------
207.3000000000000000
v25-0005:
avg
----------------------
190.6000000000000000
v25-0006 (inlining or not):
avg
----------------------
189.3333333333333333
v25-0007 (remove dead code):
avg
----------------------
186.4666666666666667
v25-addendum...txt (no ordering):
avg
----------------------
179.7000000000000000
Most of the improvement from v15 to v25 probably comes from the change from
node4 to node3, and this test stresses that node the most. That shows in
the total memory used: it goes from 152MB to 132MB. Allowing unordered
inserts helps some, the others are not convincing.
========================================
psql -c "select rt_load_ms, rt_search_ms from bench_seq_search(0, 1 * 1000
* 1000)"
(min load time of three)
v15:
rt_load_ms | rt_search_ms
------------+--------------
113 | 455
v25-0005:
rt_load_ms | rt_search_ms
------------+--------------
135 | 456
v25-0006 (inlining or not):
rt_load_ms | rt_search_ms
------------+--------------
136 | 455
v25-0007 (remove dead code):
rt_load_ms | rt_search_ms
------------+--------------
135 | 455
v25-addendum...txt (no ordering):
rt_load_ms | rt_search_ms
------------+--------------
134 | 455
Note: The regression seems to have started in v17, which is the first with
a full template.
Nothing so far has helped here, and previous experience has shown that
trying to profile 100ms will not be useful. Instead of putting more effort
into diving deeper, it seems a better use of time to write a benchmark that
calls the tid store itself. That's more realistic, since this function was
intended to test load and search of tids, but the tid store doesn't quite
operate so simply anymore. What do you think, Masahiko?
I'm inclined to keep 0006, because it might give a slight boost, and 0007
because it's never a bad idea to remove dead code.
--
John Naylor
EDB: http://www.enterprisedb.com
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
index 4e00b46d9b..3f831227c9 100644
--- a/src/include/lib/radixtree_insert_impl.h
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -80,9 +80,10 @@
}
else
{
- int insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
+ int insertpos;// = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
int count = n3->base.n.count;
-
+#ifdef RT_MAINTAIN_ORDERING
+ insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
/* shift chunks and children */
if (insertpos < count)
{
@@ -95,6 +96,9 @@
count, insertpos);
#endif
}
+#else
+ insertpos = count;
+#endif /* order */
n3->base.chunks[insertpos] = chunk;
#ifdef RT_NODE_LEVEL_LEAF
@@ -186,8 +190,10 @@
}
else
{
- int insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
+ int insertpos;
int count = n32->base.n.count;
+#ifdef RT_MAINTAIN_ORDERING
+ insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
if (insertpos < count)
{
@@ -200,6 +206,9 @@
count, insertpos);
#endif
}
+#else
+ insertpos = count;
+#endif
n32->base.chunks[insertpos] = chunk;
#ifdef RT_NODE_LEVEL_LEAF
Attachments:
[text/plain] v25-addendum-try-no-maintain-order.txt (1.3K, ../../CAFBsxsHbBm_M22gLBO+AZT4mfMq3L_oX3wdKZxjeNnT7fHsYMQ@mail.gmail.com/3-v25-addendum-try-no-maintain-order.txt)
download | inline diff:
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
index 4e00b46d9b..3f831227c9 100644
--- a/src/include/lib/radixtree_insert_impl.h
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -80,9 +80,10 @@
}
else
{
- int insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
+ int insertpos;// = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
int count = n3->base.n.count;
-
+#ifdef RT_MAINTAIN_ORDERING
+ insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
/* shift chunks and children */
if (insertpos < count)
{
@@ -95,6 +96,9 @@
count, insertpos);
#endif
}
+#else
+ insertpos = count;
+#endif /* order */
n3->base.chunks[insertpos] = chunk;
#ifdef RT_NODE_LEVEL_LEAF
@@ -186,8 +190,10 @@
}
else
{
- int insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
+ int insertpos;
int count = n32->base.n.count;
+#ifdef RT_MAINTAIN_ORDERING
+ insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
if (insertpos < count)
{
@@ -200,6 +206,9 @@
count, insertpos);
#endif
}
+#else
+ insertpos = count;
+#endif
n32->base.chunks[insertpos] = chunk;
#ifdef RT_NODE_LEVEL_LEAF
[text/x-patch] v25-0002-Move-some-bitmap-logic-out-of-bitmapset.c.patch (6.1K, ../../CAFBsxsHbBm_M22gLBO+AZT4mfMq3L_oX3wdKZxjeNnT7fHsYMQ@mail.gmail.com/4-v25-0002-Move-some-bitmap-logic-out-of-bitmapset.c.patch)
download | inline diff:
From 86c2d232a0ea193a856cb0348e0825b5e4b7a4b7 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 6 Dec 2022 13:39:41 +0700
Subject: [PATCH v25 2/9] Move some bitmap logic out of bitmapset.c
Add pg_rightmost_one32/64 functions. This functionality was previously
private to bitmapset.c as the RIGHTMOST_ONE macro. It has practical
use in other contexts, so move to pg_bitutils.h.
Also move the logic for selecting appropriate pg_bitutils functions
based on word size to bitmapset.h for wider visibility and add
appropriate selection for bmw_rightmost_one().
Since the previous macro relied on casting to signedbitmapword,
and the new functions do not, remove that typedef.
Design input and review by Tom Lane
Discussion: https://www.postgresql.org/message-id/CAFBsxsFW2JjTo58jtDB%2B3sZhxMx3t-3evew8%3DAcr%2BGGhC%2BkFaA%40mail.gmail.com
---
src/backend/nodes/bitmapset.c | 36 ++------------------------------
src/include/nodes/bitmapset.h | 16 ++++++++++++--
src/include/port/pg_bitutils.h | 31 +++++++++++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 -
4 files changed, 47 insertions(+), 37 deletions(-)
diff --git a/src/backend/nodes/bitmapset.c b/src/backend/nodes/bitmapset.c
index 98660524ad..fcd8e2ccbc 100644
--- a/src/backend/nodes/bitmapset.c
+++ b/src/backend/nodes/bitmapset.c
@@ -32,39 +32,7 @@
#define BITMAPSET_SIZE(nwords) \
(offsetof(Bitmapset, words) + (nwords) * sizeof(bitmapword))
-/*----------
- * This is a well-known cute trick for isolating the rightmost one-bit
- * in a word. It assumes two's complement arithmetic. Consider any
- * nonzero value, and focus attention on the rightmost one. The value is
- * then something like
- * xxxxxx10000
- * where x's are unspecified bits. The two's complement negative is formed
- * by inverting all the bits and adding one. Inversion gives
- * yyyyyy01111
- * where each y is the inverse of the corresponding x. Incrementing gives
- * yyyyyy10000
- * and then ANDing with the original value gives
- * 00000010000
- * This works for all cases except original value = zero, where of course
- * we get zero.
- *----------
- */
-#define RIGHTMOST_ONE(x) ((signedbitmapword) (x) & -((signedbitmapword) (x)))
-
-#define HAS_MULTIPLE_ONES(x) ((bitmapword) RIGHTMOST_ONE(x) != (x))
-
-/* Select appropriate bit-twiddling functions for bitmap word size */
-#if BITS_PER_BITMAPWORD == 32
-#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w)
-#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w)
-#define bmw_popcount(w) pg_popcount32(w)
-#elif BITS_PER_BITMAPWORD == 64
-#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w)
-#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w)
-#define bmw_popcount(w) pg_popcount64(w)
-#else
-#error "invalid BITS_PER_BITMAPWORD"
-#endif
+#define HAS_MULTIPLE_ONES(x) (bmw_rightmost_one(x) != (x))
/*
@@ -1013,7 +981,7 @@ bms_first_member(Bitmapset *a)
{
int result;
- w = RIGHTMOST_ONE(w);
+ w = bmw_rightmost_one(w);
a->words[wordnum] &= ~w;
result = wordnum * BITS_PER_BITMAPWORD;
diff --git a/src/include/nodes/bitmapset.h b/src/include/nodes/bitmapset.h
index 3d2225e1ae..5f9a511b4a 100644
--- a/src/include/nodes/bitmapset.h
+++ b/src/include/nodes/bitmapset.h
@@ -38,13 +38,11 @@ struct List;
#define BITS_PER_BITMAPWORD 64
typedef uint64 bitmapword; /* must be an unsigned type */
-typedef int64 signedbitmapword; /* must be the matching signed type */
#else
#define BITS_PER_BITMAPWORD 32
typedef uint32 bitmapword; /* must be an unsigned type */
-typedef int32 signedbitmapword; /* must be the matching signed type */
#endif
@@ -75,6 +73,20 @@ typedef enum
BMS_MULTIPLE /* >1 member */
} BMS_Membership;
+/* Select appropriate bit-twiddling functions for bitmap word size */
+#if BITS_PER_BITMAPWORD == 32
+#define bmw_rightmost_one(w) pg_rightmost_one32(w)
+#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w)
+#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w)
+#define bmw_popcount(w) pg_popcount32(w)
+#elif BITS_PER_BITMAPWORD == 64
+#define bmw_rightmost_one(w) pg_rightmost_one64(w)
+#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w)
+#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w)
+#define bmw_popcount(w) pg_popcount64(w)
+#else
+#error "invalid BITS_PER_BITMAPWORD"
+#endif
/*
* function prototypes in nodes/bitmapset.c
diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h
index a49a9c03d9..7235ad25ee 100644
--- a/src/include/port/pg_bitutils.h
+++ b/src/include/port/pg_bitutils.h
@@ -17,6 +17,37 @@ extern PGDLLIMPORT const uint8 pg_leftmost_one_pos[256];
extern PGDLLIMPORT const uint8 pg_rightmost_one_pos[256];
extern PGDLLIMPORT const uint8 pg_number_of_ones[256];
+/*----------
+ * This is a well-known cute trick for isolating the rightmost one-bit
+ * in a word. It assumes two's complement arithmetic. Consider any
+ * nonzero value, and focus attention on the rightmost one. The value is
+ * then something like
+ * xxxxxx10000
+ * where x's are unspecified bits. The two's complement negative is formed
+ * by inverting all the bits and adding one. Inversion gives
+ * yyyyyy01111
+ * where each y is the inverse of the corresponding x. Incrementing gives
+ * yyyyyy10000
+ * and then ANDing with the original value gives
+ * 00000010000
+ * This works for all cases except original value = zero, where of course
+ * we get zero.
+ *----------
+ */
+static inline uint32
+pg_rightmost_one32(uint32 word)
+{
+ int32 result = (int32) word & -((int32) word);
+ return (uint32) result;
+}
+
+static inline uint64
+pg_rightmost_one64(uint64 word)
+{
+ int64 result = (int64) word & -((int64) word);
+ return (uint64) result;
+}
+
/*
* pg_leftmost_one_pos32
* Returns the position of the most significant set bit in "word",
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 07fbb7ccf6..f4d1d60cd2 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3662,7 +3662,6 @@ shmem_request_hook_type
shmem_startup_hook_type
sig_atomic_t
sigjmp_buf
-signedbitmapword
sigset_t
size_t
slist_head
--
2.39.1
[text/x-patch] v25-0001-Introduce-helper-SIMD-functions-for-small-byte-a.patch (2.9K, ../../CAFBsxsHbBm_M22gLBO+AZT4mfMq3L_oX3wdKZxjeNnT7fHsYMQ@mail.gmail.com/5-v25-0001-Introduce-helper-SIMD-functions-for-small-byte-a.patch)
download | inline diff:
From 949c6eef5ff7cc4f8ef2673f9aa63142a1d913ae Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 24 Oct 2022 14:07:09 +0900
Subject: [PATCH v25 1/9] Introduce helper SIMD functions for small byte arrays
vector8_min - helper for emulating ">=" semantics
vector8_highbit_mask - used to turn the result of a vector
comparison into a bitmask
Masahiko Sawada
Reviewed by Nathan Bossart, additional adjustments by me
Discussion: https://www.postgresql.org/message-id/CAD21AoDap240WDDdUDE0JMpCmuMMnGajrKrkCRxM7zn9Xk3JRA%40mail.gmail.com
---
src/include/port/simd.h | 47 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)
diff --git a/src/include/port/simd.h b/src/include/port/simd.h
index c836360d4b..350e2caaea 100644
--- a/src/include/port/simd.h
+++ b/src/include/port/simd.h
@@ -79,6 +79,7 @@ static inline bool vector8_has_le(const Vector8 v, const uint8 c);
static inline bool vector8_is_highbit_set(const Vector8 v);
#ifndef USE_NO_SIMD
static inline bool vector32_is_highbit_set(const Vector32 v);
+static inline uint32 vector8_highbit_mask(const Vector8 v);
#endif
/* arithmetic operations */
@@ -96,6 +97,7 @@ static inline Vector8 vector8_ssub(const Vector8 v1, const Vector8 v2);
*/
#ifndef USE_NO_SIMD
static inline Vector8 vector8_eq(const Vector8 v1, const Vector8 v2);
+static inline Vector8 vector8_min(const Vector8 v1, const Vector8 v2);
static inline Vector32 vector32_eq(const Vector32 v1, const Vector32 v2);
#endif
@@ -299,6 +301,36 @@ vector32_is_highbit_set(const Vector32 v)
}
#endif /* ! USE_NO_SIMD */
+/*
+ * Return a bitmask formed from the high-bit of each element.
+ */
+#ifndef USE_NO_SIMD
+static inline uint32
+vector8_highbit_mask(const Vector8 v)
+{
+#ifdef USE_SSE2
+ return (uint32) _mm_movemask_epi8(v);
+#elif defined(USE_NEON)
+ /*
+ * Note: There is a faster way to do this, but it returns a uint64 and
+ * and if the caller wanted to extract the bit position using CTZ,
+ * it would have to divide that result by 4.
+ */
+ static const uint8 mask[16] = {
+ 1 << 0, 1 << 1, 1 << 2, 1 << 3,
+ 1 << 4, 1 << 5, 1 << 6, 1 << 7,
+ 1 << 0, 1 << 1, 1 << 2, 1 << 3,
+ 1 << 4, 1 << 5, 1 << 6, 1 << 7,
+ };
+
+ uint8x16_t masked = vandq_u8(vld1q_u8(mask), (uint8x16_t) vshrq_n_s8(v, 7));
+ uint8x16_t maskedhi = vextq_u8(masked, masked, 8);
+
+ return (uint32) vaddvq_u16((uint16x8_t) vzip1q_u8(masked, maskedhi));
+#endif
+}
+#endif /* ! USE_NO_SIMD */
+
/*
* Return the bitwise OR of the inputs
*/
@@ -372,4 +404,19 @@ vector32_eq(const Vector32 v1, const Vector32 v2)
}
#endif /* ! USE_NO_SIMD */
+/*
+ * Given two vectors, return a vector with the minimum element of each.
+ */
+#ifndef USE_NO_SIMD
+static inline Vector8
+vector8_min(const Vector8 v1, const Vector8 v2)
+{
+#ifdef USE_SSE2
+ return _mm_min_epu8(v1, v2);
+#elif defined(USE_NEON)
+ return vminq_u8(v1, v2);
+#endif
+}
+#endif /* ! USE_NO_SIMD */
+
#endif /* SIMD_H */
--
2.39.1
[text/x-patch] v25-0005-Measure-load-time-of-bench_search_random_nodes.patch (2.4K, ../../CAFBsxsHbBm_M22gLBO+AZT4mfMq3L_oX3wdKZxjeNnT7fHsYMQ@mail.gmail.com/6-v25-0005-Measure-load-time-of-bench_search_random_nodes.patch)
download | inline diff:
From 8edd5b4c0fcbf7681c5388faaf85a96ae451c99e Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 7 Feb 2023 13:06:00 +0700
Subject: [PATCH v25 5/9] Measure load time of bench_search_random_nodes
---
.../bench_radix_tree/bench_radix_tree--1.0.sql | 1 +
contrib/bench_radix_tree/bench_radix_tree.c | 17 ++++++++++++-----
2 files changed, 13 insertions(+), 5 deletions(-)
diff --git a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
index 2fd689aa91..95eedbbe10 100644
--- a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
+++ b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
@@ -47,6 +47,7 @@ create function bench_search_random_nodes(
cnt int8,
filter_str text DEFAULT NULL,
OUT mem_allocated int8,
+OUT load_ms int8,
OUT search_ms int8)
returns record
as 'MODULE_PATHNAME'
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
index 73ddee32de..7d1e2eee57 100644
--- a/contrib/bench_radix_tree/bench_radix_tree.c
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -395,9 +395,10 @@ bench_search_random_nodes(PG_FUNCTION_ARGS)
end_time;
long secs;
int usecs;
+ int64 load_time_ms;
int64 search_time_ms;
- Datum values[2] = {0};
- bool nulls[2] = {0};
+ Datum values[3] = {0};
+ bool nulls[3] = {0};
/* from trial and error */
uint64 filter = (((uint64) 0x7F<<32) | (0x07<<24) | (0xFF<<16) | 0xFF);
@@ -416,13 +417,18 @@ bench_search_random_nodes(PG_FUNCTION_ARGS)
rt = rt_create(CurrentMemoryContext);
+ start_time = GetCurrentTimestamp();
for (uint64 i = 0; i < cnt; i++)
{
- const uint64 hash = hash64(i);
- const uint64 key = hash & filter;
+ uint64 hash = hash64(i);
+ uint64 key = hash & filter;
rt_set(rt, key, &key);
}
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_time_ms = secs * 1000 + usecs / 1000;
elog(NOTICE, "sleeping for 2 seconds...");
pg_usleep(2 * 1000000L);
@@ -449,7 +455,8 @@ bench_search_random_nodes(PG_FUNCTION_ARGS)
rt_stats(rt);
values[0] = Int64GetDatum(rt_memory_usage(rt));
- values[1] = Int64GetDatum(search_time_ms);
+ values[1] = Int64GetDatum(load_time_ms);
+ values[2] = Int64GetDatum(search_time_ms);
rt_free(rt);
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
--
2.39.1
[text/x-patch] v25-0004-Tool-for-measuring-radix-tree-performance.patch (22.0K, ../../CAFBsxsHbBm_M22gLBO+AZT4mfMq3L_oX3wdKZxjeNnT7fHsYMQ@mail.gmail.com/7-v25-0004-Tool-for-measuring-radix-tree-performance.patch)
download | inline diff:
From 6fb21eb0b44b5923c0b736d82e86b1d4a40a71d6 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 16 Sep 2022 11:57:03 +0900
Subject: [PATCH v25 4/9] Tool for measuring radix tree performance
Includes Meson support, but commented out to avoid warnings
XXX: Not for commit
---
contrib/bench_radix_tree/Makefile | 21 +
.../bench_radix_tree--1.0.sql | 76 ++
contrib/bench_radix_tree/bench_radix_tree.c | 656 ++++++++++++++++++
.../bench_radix_tree/bench_radix_tree.control | 6 +
contrib/bench_radix_tree/expected/bench.out | 13 +
contrib/bench_radix_tree/meson.build | 33 +
contrib/bench_radix_tree/sql/bench.sql | 16 +
contrib/meson.build | 1 +
8 files changed, 822 insertions(+)
create mode 100644 contrib/bench_radix_tree/Makefile
create mode 100644 contrib/bench_radix_tree/bench_radix_tree--1.0.sql
create mode 100644 contrib/bench_radix_tree/bench_radix_tree.c
create mode 100644 contrib/bench_radix_tree/bench_radix_tree.control
create mode 100644 contrib/bench_radix_tree/expected/bench.out
create mode 100644 contrib/bench_radix_tree/meson.build
create mode 100644 contrib/bench_radix_tree/sql/bench.sql
diff --git a/contrib/bench_radix_tree/Makefile b/contrib/bench_radix_tree/Makefile
new file mode 100644
index 0000000000..952bb0ceae
--- /dev/null
+++ b/contrib/bench_radix_tree/Makefile
@@ -0,0 +1,21 @@
+# contrib/bench_radix_tree/Makefile
+
+MODULE_big = bench_radix_tree
+OBJS = \
+ bench_radix_tree.o
+
+EXTENSION = bench_radix_tree
+DATA = bench_radix_tree--1.0.sql
+
+REGRESS = bench_fixed_height
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/bench_radix_tree
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
new file mode 100644
index 0000000000..2fd689aa91
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
@@ -0,0 +1,76 @@
+/* contrib/bench_radix_tree/bench_radix_tree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION bench_radix_tree" to load this file. \quit
+
+create function bench_shuffle_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_seq_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_load_random_int(
+cnt int8,
+OUT mem_allocated int8,
+OUT load_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_search_random_nodes(
+cnt int8,
+filter_str text DEFAULT NULL,
+OUT mem_allocated int8,
+OUT search_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C VOLATILE PARALLEL UNSAFE;
+
+create function bench_fixed_height_search(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT rt_search_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_node128_load(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_sparseload_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
new file mode 100644
index 0000000000..73ddee32de
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -0,0 +1,656 @@
+/*-------------------------------------------------------------------------
+ *
+ * bench_radix_tree.c
+ *
+ * Copyright (c) 2016-2022, PostgreSQL Global Development Group
+ *
+ * contrib/bench_radix_tree/bench_radix_tree.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "funcapi.h"
+#include "lib/radixtree.h"
+#include <math.h>
+#include "miscadmin.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/timestamp.h"
+
+PG_MODULE_MAGIC;
+
+/* run benchmark also for binary-search case? */
+/* #define MEASURE_BINARY_SEARCH 1 */
+
+#define TIDS_PER_BLOCK_FOR_LOAD 30
+#define TIDS_PER_BLOCK_FOR_LOOKUP 50
+
+#define RT_DEBUG
+#define RT_PREFIX rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE uint64
+// WIP: compiles with warnings because rt_attach is defined but not used
+// #define RT_SHMEM
+#include "lib/radixtree.h"
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+ return tree->ctl->num_keys;
+}
+
+
+PG_FUNCTION_INFO_V1(bench_seq_search);
+PG_FUNCTION_INFO_V1(bench_shuffle_search);
+PG_FUNCTION_INFO_V1(bench_load_random_int);
+PG_FUNCTION_INFO_V1(bench_fixed_height_search);
+PG_FUNCTION_INFO_V1(bench_search_random_nodes);
+PG_FUNCTION_INFO_V1(bench_node128_load);
+
+static uint64
+tid_to_key_off(ItemPointer tid, uint32 *off)
+{
+ uint64 upper;
+ uint32 shift = pg_ceil_log2_32(MaxHeapTuplesPerPage);
+ int64 tid_i;
+
+ Assert(ItemPointerGetOffsetNumber(tid) < MaxHeapTuplesPerPage);
+
+ tid_i = ItemPointerGetOffsetNumber(tid);
+ tid_i |= (uint64) ItemPointerGetBlockNumber(tid) << shift;
+
+ /* log(sizeof(uint64) * BITS_PER_BYTE, 2) = log(64, 2) = 6 */
+ *off = tid_i & ((1 << 6) - 1);
+ upper = tid_i >> 6;
+ Assert(*off < (sizeof(uint64) * BITS_PER_BYTE));
+
+ Assert(*off < 64);
+
+ return upper;
+}
+
+static int
+shuffle_randrange(pg_prng_state *state, int lower, int upper)
+{
+ return (int) floor(pg_prng_double(state) * ((upper - lower) + 0.999999)) + lower;
+}
+
+/* Naive Fisher-Yates implementation*/
+static void
+shuffle_itemptrs(ItemPointer itemptr, uint64 nitems)
+{
+ /* reproducability */
+ pg_prng_state state;
+
+ pg_prng_seed(&state, 0);
+
+ for (int i = 0; i < nitems - 1; i++)
+ {
+ int j = shuffle_randrange(&state, i, nitems - 1);
+ ItemPointerData t = itemptr[j];
+
+ itemptr[j] = itemptr[i];
+ itemptr[i] = t;
+ }
+}
+
+static ItemPointer
+generate_tids(BlockNumber minblk, BlockNumber maxblk, int ntids_per_blk, uint64 *ntids_p,
+ bool random_block)
+{
+ ItemPointer tids;
+ uint64 maxitems;
+ uint64 ntids = 0;
+ pg_prng_state state;
+
+ maxitems = (maxblk - minblk + 1) * ntids_per_blk;
+ tids = MemoryContextAllocHuge(TopTransactionContext,
+ sizeof(ItemPointerData) * maxitems);
+
+ if (random_block)
+ pg_prng_seed(&state, 0x9E3779B185EBCA87);
+
+ for (BlockNumber blk = minblk; blk < maxblk; blk++)
+ {
+ if (random_block && !pg_prng_bool(&state))
+ continue;
+
+ for (OffsetNumber off = FirstOffsetNumber;
+ off <= ntids_per_blk; off++)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ ItemPointerSetBlockNumber(&(tids[ntids]), blk);
+ ItemPointerSetOffsetNumber(&(tids[ntids]), off);
+
+ ntids++;
+ }
+ }
+
+ *ntids_p = ntids;
+ return tids;
+}
+
+#ifdef MEASURE_BINARY_SEARCH
+static int
+vac_cmp_itemptr(const void *left, const void *right)
+{
+ BlockNumber lblk,
+ rblk;
+ OffsetNumber loff,
+ roff;
+
+ lblk = ItemPointerGetBlockNumber((ItemPointer) left);
+ rblk = ItemPointerGetBlockNumber((ItemPointer) right);
+
+ if (lblk < rblk)
+ return -1;
+ if (lblk > rblk)
+ return 1;
+
+ loff = ItemPointerGetOffsetNumber((ItemPointer) left);
+ roff = ItemPointerGetOffsetNumber((ItemPointer) right);
+
+ if (loff < roff)
+ return -1;
+ if (loff > roff)
+ return 1;
+
+ return 0;
+}
+#endif
+
+static Datum
+bench_search(FunctionCallInfo fcinfo, bool shuffle)
+{
+ BlockNumber minblk = PG_GETARG_INT32(0);
+ BlockNumber maxblk = PG_GETARG_INT32(1);
+ bool random_block = PG_GETARG_BOOL(2);
+ rt_radix_tree *rt = NULL;
+ uint64 ntids;
+ uint64 key;
+ uint64 last_key = PG_UINT64_MAX;
+ uint64 val = 0;
+ ItemPointer tids;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_load_ms,
+ rt_search_ms;
+ Datum values[7];
+ bool nulls[7];
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ tids = generate_tids(minblk, maxblk, TIDS_PER_BLOCK_FOR_LOAD, &ntids, random_block);
+
+ /* measure the load time of the radix tree */
+ rt = rt_create(CurrentMemoryContext);
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ uint32 off;
+
+ CHECK_FOR_INTERRUPTS();
+
+ key = tid_to_key_off(tid, &off);
+
+ if (last_key != PG_UINT64_MAX && last_key != key)
+ {
+ rt_set(rt, last_key, &val);
+ val = 0;
+ }
+
+ last_key = key;
+ val |= (uint64) 1 << off;
+ }
+ if (last_key != PG_UINT64_MAX)
+ rt_set(rt, last_key, &val);
+
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_load_ms = secs * 1000 + usecs / 1000;
+
+ rt_stats(rt);
+
+ if (shuffle)
+ shuffle_itemptrs(tids, ntids);
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* meaure the serach time of the radix tree */
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ uint32 off;
+ volatile bool ret; /* prevent calling rt_search from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ key = tid_to_key_off(tid, &off);
+
+ ret = rt_search(rt, key, &val);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_search_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int64GetDatum(rt_num_entries(rt));
+ values[1] = Int64GetDatum(rt_memory_usage(rt));
+ values[2] = Int64GetDatum(sizeof(ItemPointerData) * ntids);
+ values[3] = Int64GetDatum(rt_load_ms);
+ nulls[4] = true; /* ar_load_ms */
+ values[5] = Int64GetDatum(rt_search_ms);
+ nulls[6] = true; /* ar_search_ms */
+
+#ifdef MEASURE_BINARY_SEARCH
+ {
+ ItemPointer itemptrs = NULL;
+
+ int64 ar_load_ms,
+ ar_search_ms;
+
+ /* measure the load time of the array */
+ itemptrs = MemoryContextAllocHuge(CurrentMemoryContext,
+ sizeof(ItemPointerData) * ntids);
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointerSetBlockNumber(&(itemptrs[i]),
+ ItemPointerGetBlockNumber(&(tids[i])));
+ ItemPointerSetOffsetNumber(&(itemptrs[i]),
+ ItemPointerGetOffsetNumber(&(tids[i])));
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ ar_load_ms = secs * 1000 + usecs / 1000;
+
+ /* next, measure the serach time of the array */
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ volatile bool ret; /* prevent calling bsearch from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ ret = bsearch((void *) tid,
+ (void *) itemptrs,
+ ntids,
+ sizeof(ItemPointerData),
+ vac_cmp_itemptr);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ ar_search_ms = secs * 1000 + usecs / 1000;
+
+ /* set the result */
+ nulls[4] = false;
+ values[4] = Int64GetDatum(ar_load_ms);
+ nulls[6] = false;
+ values[6] = Int64GetDatum(ar_search_ms);
+ }
+#endif
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_seq_search(PG_FUNCTION_ARGS)
+{
+ return bench_search(fcinfo, false);
+}
+
+Datum
+bench_shuffle_search(PG_FUNCTION_ARGS)
+{
+ return bench_search(fcinfo, true);
+}
+
+Datum
+bench_load_random_int(PG_FUNCTION_ARGS)
+{
+ uint64 cnt = (uint64) PG_GETARG_INT64(0);
+ rt_radix_tree *rt;
+ pg_prng_state state;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_time_ms;
+ Datum values[2];
+ bool nulls[2];
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ pg_prng_seed(&state, 0);
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 key = pg_prng_uint64(&state);
+
+ rt_set(rt, key, &key);
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_time_ms = secs * 1000 + usecs / 1000;
+
+ rt_stats(rt);
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int64GetDatum(rt_memory_usage(rt));
+ values[1] = Int64GetDatum(load_time_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+/* copy of splitmix64() */
+static uint64
+hash64(uint64 x)
+{
+ x ^= x >> 30;
+ x *= UINT64CONST(0xbf58476d1ce4e5b9);
+ x ^= x >> 27;
+ x *= UINT64CONST(0x94d049bb133111eb);
+ x ^= x >> 31;
+ return x;
+}
+
+/* attempts to have a relatively even population of node kinds */
+Datum
+bench_search_random_nodes(PG_FUNCTION_ARGS)
+{
+ uint64 cnt = (uint64) PG_GETARG_INT64(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 search_time_ms;
+ Datum values[2] = {0};
+ bool nulls[2] = {0};
+ /* from trial and error */
+ uint64 filter = (((uint64) 0x7F<<32) | (0x07<<24) | (0xFF<<16) | 0xFF);
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (!PG_ARGISNULL(1))
+ {
+ char *filter_str = text_to_cstring(PG_GETARG_TEXT_P(1));
+
+ if (sscanf(filter_str, "0x%lX", &filter) == 0)
+ elog(ERROR, "invalid filter string %s", filter_str);
+ }
+ elog(NOTICE, "bench with filter 0x%lX", filter);
+
+ rt = rt_create(CurrentMemoryContext);
+
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ const uint64 hash = hash64(i);
+ const uint64 key = hash & filter;
+
+ rt_set(rt, key, &key);
+ }
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 hash = hash64(i);
+ uint64 key = hash & filter;
+ uint64 val;
+ volatile bool ret; /* prevent calling rt_search from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ ret = rt_search(rt, key, &val);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ search_time_ms = secs * 1000 + usecs / 1000;
+
+ rt_stats(rt);
+
+ values[0] = Int64GetDatum(rt_memory_usage(rt));
+ values[1] = Int64GetDatum(search_time_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_fixed_height_search(PG_FUNCTION_ARGS)
+{
+ int fanout = PG_GETARG_INT32(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_load_ms,
+ rt_search_ms;
+ Datum values[5];
+ bool nulls[5];
+
+ /* test boundary between vector and iteration */
+ const int n_keys = 5 * 16 * 16 * 16 * 16;
+ uint64 r,
+ h,
+ i,
+ j,
+ k;
+ uint64 key_id;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+
+ key_id = 0;
+
+ /*
+ * lower nodes have limited fanout, the top is only limited by
+ * bits-per-byte
+ */
+ for (r = 1;; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ for (i = 1; i <= fanout; i++)
+ {
+ for (j = 1; j <= fanout; j++)
+ {
+ for (k = 1; k <= fanout; k++)
+ {
+ uint64 key;
+
+ key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+ CHECK_FOR_INTERRUPTS();
+
+ key_id++;
+ if (key_id > n_keys)
+ goto finish_set;
+
+ rt_set(rt, key, &key_id);
+ }
+ }
+ }
+ }
+ }
+finish_set:
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_load_ms = secs * 1000 + usecs / 1000;
+
+ rt_stats(rt);
+
+ /* meaure the search time of the radix tree */
+ start_time = GetCurrentTimestamp();
+
+ key_id = 0;
+ for (r = 1;; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ for (i = 1; i <= fanout; i++)
+ {
+ for (j = 1; j <= fanout; j++)
+ {
+ for (k = 1; k <= fanout; k++)
+ {
+ uint64 key,
+ val;
+
+ key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+ CHECK_FOR_INTERRUPTS();
+
+ key_id++;
+ if (key_id > n_keys)
+ goto finish_search;
+
+ rt_search(rt, key, &val);
+ }
+ }
+ }
+ }
+ }
+finish_search:
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_search_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int32GetDatum(fanout);
+ values[1] = Int64GetDatum(rt_num_entries(rt));
+ values[2] = Int64GetDatum(rt_memory_usage(rt));
+ values[3] = Int64GetDatum(rt_load_ms);
+ values[4] = Int64GetDatum(rt_search_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_node128_load(PG_FUNCTION_ARGS)
+{
+ int fanout = PG_GETARG_INT32(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_sparseload_ms;
+ Datum values[5];
+ bool nulls[5];
+
+ uint64 r,
+ h;
+ uint64 key_id;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ rt = rt_create(CurrentMemoryContext);
+
+ key_id = 0;
+
+ for (r = 1; r <= fanout; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (h);
+
+ key_id++;
+ rt_set(rt, key, &key_id);
+ }
+ }
+
+ rt_stats(rt);
+
+ /* measure sparse deletion and re-loading */
+ start_time = GetCurrentTimestamp();
+
+ for (int t = 0; t<10000; t++)
+ {
+ /* delete one key in each leaf */
+ for (r = 1; r <= fanout; r++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (fanout);
+
+ rt_delete(rt, key);
+ }
+
+ /* add them all back */
+ key_id = 0;
+ for (r = 1; r <= fanout; r++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (fanout);
+
+ key_id++;
+ rt_set(rt, key, &key_id);
+ }
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_sparseload_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int32GetDatum(fanout);
+ values[1] = Int64GetDatum(rt_num_entries(rt));
+ values[2] = Int64GetDatum(rt_memory_usage(rt));
+ values[3] = Int64GetDatum(rt_sparseload_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
diff --git a/contrib/bench_radix_tree/bench_radix_tree.control b/contrib/bench_radix_tree/bench_radix_tree.control
new file mode 100644
index 0000000000..1d988e6c9a
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.control
@@ -0,0 +1,6 @@
+# bench_radix_tree extension
+comment = 'benchmark suits for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/bench_radix_tree'
+relocatable = true
+trusted = true
diff --git a/contrib/bench_radix_tree/expected/bench.out b/contrib/bench_radix_tree/expected/bench.out
new file mode 100644
index 0000000000..60c303892e
--- /dev/null
+++ b/contrib/bench_radix_tree/expected/bench.out
@@ -0,0 +1,13 @@
+create extension bench_radix_tree;
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/bench_radix_tree/meson.build b/contrib/bench_radix_tree/meson.build
new file mode 100644
index 0000000000..332c1ae7df
--- /dev/null
+++ b/contrib/bench_radix_tree/meson.build
@@ -0,0 +1,33 @@
+bench_radix_tree_sources = files(
+ 'bench_radix_tree.c',
+)
+
+if host_system == 'windows'
+ bench_radix_tree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'bench_radix_tree',
+ '--FILEDESC', 'bench_radix_tree - performance test code for radix tree',])
+endif
+
+bench_radix_tree = shared_module('bench_radix_tree',
+ bench_radix_tree_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += bench_radix_tree
+
+install_data(
+ 'bench_radix_tree.control',
+ 'bench_radix_tree--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'bench_radix_tree',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'bench_radix_tree',
+ ],
+ },
+}
diff --git a/contrib/bench_radix_tree/sql/bench.sql b/contrib/bench_radix_tree/sql/bench.sql
new file mode 100644
index 0000000000..a46018c9d4
--- /dev/null
+++ b/contrib/bench_radix_tree/sql/bench.sql
@@ -0,0 +1,16 @@
+create extension bench_radix_tree;
+
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/meson.build b/contrib/meson.build
index bd4a57c43c..52253de793 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,6 +12,7 @@ subdir('amcheck')
subdir('auth_delay')
subdir('auto_explain')
subdir('basic_archive')
+#subdir('bench_radix_tree')
subdir('bloom')
subdir('basebackup_to_shell')
subdir('bool_plperl')
--
2.39.1
[text/x-patch] v25-0003-Add-radixtree-template.patch (116.9K, ../../CAFBsxsHbBm_M22gLBO+AZT4mfMq3L_oX3wdKZxjeNnT7fHsYMQ@mail.gmail.com/8-v25-0003-Add-radixtree-template.patch)
download | inline diff:
From f421579a2e04baa04b258399e01f01485ce6f358 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 14 Sep 2022 12:38:51 +0000
Subject: [PATCH v25 3/9] Add radixtree template
WIP: commit message based on template comments
---
src/backend/utils/mmgr/dsa.c | 12 +
src/include/lib/radixtree.h | 2516 +++++++++++++++++
src/include/lib/radixtree_delete_impl.h | 122 +
src/include/lib/radixtree_insert_impl.h | 332 +++
src/include/lib/radixtree_iter_impl.h | 153 +
src/include/lib/radixtree_search_impl.h | 138 +
src/include/utils/dsa.h | 1 +
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_radixtree/.gitignore | 4 +
src/test/modules/test_radixtree/Makefile | 23 +
src/test/modules/test_radixtree/README | 7 +
.../expected/test_radixtree.out | 36 +
src/test/modules/test_radixtree/meson.build | 35 +
.../test_radixtree/sql/test_radixtree.sql | 7 +
.../test_radixtree/test_radixtree--1.0.sql | 8 +
.../modules/test_radixtree/test_radixtree.c | 674 +++++
.../test_radixtree/test_radixtree.control | 4 +
src/tools/pginclude/cpluspluscheck | 6 +
src/tools/pginclude/headerscheck | 6 +
20 files changed, 4086 insertions(+)
create mode 100644 src/include/lib/radixtree.h
create mode 100644 src/include/lib/radixtree_delete_impl.h
create mode 100644 src/include/lib/radixtree_insert_impl.h
create mode 100644 src/include/lib/radixtree_iter_impl.h
create mode 100644 src/include/lib/radixtree_search_impl.h
create mode 100644 src/test/modules/test_radixtree/.gitignore
create mode 100644 src/test/modules/test_radixtree/Makefile
create mode 100644 src/test/modules/test_radixtree/README
create mode 100644 src/test/modules/test_radixtree/expected/test_radixtree.out
create mode 100644 src/test/modules/test_radixtree/meson.build
create mode 100644 src/test/modules/test_radixtree/sql/test_radixtree.sql
create mode 100644 src/test/modules/test_radixtree/test_radixtree--1.0.sql
create mode 100644 src/test/modules/test_radixtree/test_radixtree.c
create mode 100644 src/test/modules/test_radixtree/test_radixtree.control
diff --git a/src/backend/utils/mmgr/dsa.c b/src/backend/utils/mmgr/dsa.c
index f5a62061a3..80555aefff 100644
--- a/src/backend/utils/mmgr/dsa.c
+++ b/src/backend/utils/mmgr/dsa.c
@@ -1024,6 +1024,18 @@ dsa_set_size_limit(dsa_area *area, size_t limit)
LWLockRelease(DSA_AREA_LOCK(area));
}
+size_t
+dsa_get_total_size(dsa_area *area)
+{
+ size_t size;
+
+ LWLockAcquire(DSA_AREA_LOCK(area), LW_SHARED);
+ size = area->control->total_segment_size;
+ LWLockRelease(DSA_AREA_LOCK(area));
+
+ return size;
+}
+
/*
* Aggressively free all spare memory in the hope of returning DSM segments to
* the operating system.
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
new file mode 100644
index 0000000000..d6919aef08
--- /dev/null
+++ b/src/include/lib/radixtree.h
@@ -0,0 +1,2516 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree.h
+ * Template for adaptive radix tree.
+ *
+ * This module employs the idea from the paper "The Adaptive Radix Tree: ARTful
+ * Indexing for Main-Memory Databases" by Viktor Leis, Alfons Kemper, and Thomas
+ * Neumann, 2013. The radix tree uses adaptive node sizes, a small number of node
+ * types, each with a different numbers of elements. Depending on the number of
+ * children, the appropriate node type is used.
+ *
+ * WIP: notes about traditional radix tree trading off span vs height...
+ *
+ * There are two kinds of nodes, inner nodes and leaves. Inner nodes
+ * map partial keys to child pointers.
+ *
+ * The ART paper mentions three ways to implement leaves:
+ *
+ * "- Single-value leaves: The values are stored using an addi-
+ * tional leaf node type which stores one value.
+ * - Multi-value leaves: The values are stored in one of four
+ * different leaf node types, which mirror the structure of
+ * inner nodes, but contain values instead of pointers.
+ * - Combined pointer/value slots: If values fit into point-
+ * ers, no separate node types are necessary. Instead, each
+ * pointer storage location in an inner node can either
+ * store a pointer or a value."
+ *
+ * We chose "multi-value leaves" to avoid the additional pointer traversal
+ * required by "single-value leaves"
+ *
+ * For simplicity, the key is assumed to be 64-bit unsigned integer. The
+ * tree doesn't need to contain paths where the highest bytes of all keys
+ * are zero. That way, the tree's height adapts to the distribution of keys.
+ *
+ * TODO: In the future it might be worthwhile to offer configurability of
+ * leaf implementation for different use cases. Single-values leaves would
+ * give more flexibility in key type, including variable-length keys.
+ *
+ * There are some optimizations not yet implemented, particularly path
+ * compression and lazy path expansion.
+ *
+ * To handle concurrency, we use a single reader-writer lock for the radix
+ * tree. The radix tree is exclusively locked during write operations such
+ * as RT_SET() and RT_DELETE(), and shared locked during read operations
+ * such as RT_SEARCH(). An iteration also holds the shared lock on the radix
+ * tree until it is completed.
+ *
+ * TODO: The current locking mechanism is not optimized for high concurrency
+ * with mixed read-write workloads. In the future it might be worthwhile
+ * to replace it with the Optimistic Lock Coupling or ROWEX mentioned in
+ * the paper "The ART of Practical Synchronization" by the same authors as
+ * the ART paper, 2016.
+ *
+ * WIP: the radix tree nodes don't shrink.
+ *
+ * To generate a radix tree and associated functions for a use case several
+ * macros have to be #define'ed before this file is included. Including
+ * the file #undef's all those, so a new radix tree can be generated
+ * afterwards.
+ * The relevant parameters are:
+ * - RT_PREFIX - prefix for all symbol names generated. A prefix of 'foo'
+ * will result in radix tree type 'foo_radix_tree' and functions like
+ * 'foo_create'/'foo_free' and so forth.
+ * - RT_DECLARE - if defined function prototypes and type declarations are
+ * generated
+ * - RT_DEFINE - if defined function definitions are generated
+ * - RT_SCOPE - in which scope (e.g. extern, static inline) do function
+ * declarations reside
+ * - RT_VALUE_TYPE - the type of the value.
+ *
+ * Optional parameters:
+ * - RT_SHMEM - if defined, the radix tree is created in the DSA area
+ * so that multiple processes can access it simultaneously.
+ * - RT_DEBUG - if defined add stats tracking and debugging functions
+ *
+ * Interface
+ * ---------
+ *
+ * RT_CREATE - Create a new, empty radix tree
+ * RT_FREE - Free the radix tree
+ * RT_SEARCH - Search a key-value pair
+ * RT_SET - Set a key-value pair
+ * RT_BEGIN_ITERATE - Begin iterating through all key-value pairs
+ * RT_ITERATE_NEXT - Return next key-value pair, if any
+ * RT_END_ITER - End iteration
+ * RT_MEMORY_USAGE - Get the memory usage
+ *
+ * Interface for Shared Memory
+ * ---------
+ *
+ * RT_ATTACH - Attach to the radix tree
+ * RT_DETACH - Detach from the radix tree
+ * RT_GET_HANDLE - Return the handle of the radix tree
+ *
+ * Optional Interface
+ * ---------
+ *
+ * RT_DELETE - Delete a key-value pair. Declared/define if RT_USE_DELETE is defined
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/include/lib/radixtree.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/stringinfo.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "port/pg_bitutils.h"
+#include "port/simd.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/* helpers */
+#define RT_MAKE_PREFIX(a) CppConcat(a,_)
+#define RT_MAKE_NAME(name) RT_MAKE_NAME_(RT_MAKE_PREFIX(RT_PREFIX),name)
+#define RT_MAKE_NAME_(a,b) CppConcat(a,b)
+
+/* function declarations */
+#define RT_CREATE RT_MAKE_NAME(create)
+#define RT_FREE RT_MAKE_NAME(free)
+#define RT_SEARCH RT_MAKE_NAME(search)
+#ifdef RT_SHMEM
+#define RT_ATTACH RT_MAKE_NAME(attach)
+#define RT_DETACH RT_MAKE_NAME(detach)
+#define RT_GET_HANDLE RT_MAKE_NAME(get_handle)
+#endif
+#define RT_SET RT_MAKE_NAME(set)
+#define RT_BEGIN_ITERATE RT_MAKE_NAME(begin_iterate)
+#define RT_ITERATE_NEXT RT_MAKE_NAME(iterate_next)
+#define RT_END_ITERATE RT_MAKE_NAME(end_iterate)
+#ifdef RT_USE_DELETE
+#define RT_DELETE RT_MAKE_NAME(delete)
+#endif
+#define RT_MEMORY_USAGE RT_MAKE_NAME(memory_usage)
+#ifdef RT_DEBUG
+#define RT_DUMP RT_MAKE_NAME(dump)
+#define RT_DUMP_NODE RT_MAKE_NAME(dump_node)
+#define RT_DUMP_SEARCH RT_MAKE_NAME(dump_search)
+#define RT_STATS RT_MAKE_NAME(stats)
+#endif
+
+/* internal helper functions (no externally visible prototypes) */
+#define RT_NEW_ROOT RT_MAKE_NAME(new_root)
+#define RT_ALLOC_NODE RT_MAKE_NAME(alloc_node)
+#define RT_INIT_NODE RT_MAKE_NAME(init_node)
+#define RT_FREE_NODE RT_MAKE_NAME(free_node)
+#define RT_FREE_RECURSE RT_MAKE_NAME(free_recurse)
+#define RT_EXTEND RT_MAKE_NAME(extend)
+#define RT_SET_EXTEND RT_MAKE_NAME(set_extend)
+#define RT_SWITCH_NODE_KIND RT_MAKE_NAME(grow_node_kind)
+#define RT_COPY_NODE RT_MAKE_NAME(copy_node)
+#define RT_REPLACE_NODE RT_MAKE_NAME(replace_node)
+#define RT_PTR_GET_LOCAL RT_MAKE_NAME(ptr_get_local)
+#define RT_PTR_ALLOC_IS_VALID RT_MAKE_NAME(ptr_stored_is_valid)
+#define RT_NODE_3_SEARCH_EQ RT_MAKE_NAME(node_3_search_eq)
+#define RT_NODE_32_SEARCH_EQ RT_MAKE_NAME(node_32_search_eq)
+#define RT_NODE_3_GET_INSERTPOS RT_MAKE_NAME(node_3_get_insertpos)
+#define RT_NODE_32_GET_INSERTPOS RT_MAKE_NAME(node_32_get_insertpos)
+#define RT_CHUNK_CHILDREN_ARRAY_SHIFT RT_MAKE_NAME(chunk_children_array_shift)
+#define RT_CHUNK_VALUES_ARRAY_SHIFT RT_MAKE_NAME(chunk_values_array_shift)
+#define RT_CHUNK_CHILDREN_ARRAY_DELETE RT_MAKE_NAME(chunk_children_array_delete)
+#define RT_CHUNK_VALUES_ARRAY_DELETE RT_MAKE_NAME(chunk_values_array_delete)
+#define RT_CHUNK_CHILDREN_ARRAY_COPY RT_MAKE_NAME(chunk_children_array_copy)
+#define RT_CHUNK_VALUES_ARRAY_COPY RT_MAKE_NAME(chunk_values_array_copy)
+#define RT_NODE_125_IS_CHUNK_USED RT_MAKE_NAME(node_125_is_chunk_used)
+#define RT_NODE_INNER_125_GET_CHILD RT_MAKE_NAME(node_inner_125_get_child)
+#define RT_NODE_LEAF_125_GET_VALUE RT_MAKE_NAME(node_leaf_125_get_value)
+#define RT_NODE_INNER_256_IS_CHUNK_USED RT_MAKE_NAME(node_inner_256_is_chunk_used)
+#define RT_NODE_LEAF_256_IS_CHUNK_USED RT_MAKE_NAME(node_leaf_256_is_chunk_used)
+#define RT_NODE_INNER_256_GET_CHILD RT_MAKE_NAME(node_inner_256_get_child)
+#define RT_NODE_LEAF_256_GET_VALUE RT_MAKE_NAME(node_leaf_256_get_value)
+#define RT_NODE_INNER_256_SET RT_MAKE_NAME(node_inner_256_set)
+#define RT_NODE_LEAF_256_SET RT_MAKE_NAME(node_leaf_256_set)
+#define RT_NODE_INNER_256_DELETE RT_MAKE_NAME(node_inner_256_delete)
+#define RT_NODE_LEAF_256_DELETE RT_MAKE_NAME(node_leaf_256_delete)
+#define RT_KEY_GET_SHIFT RT_MAKE_NAME(key_get_shift)
+#define RT_SHIFT_GET_MAX_VAL RT_MAKE_NAME(shift_get_max_val)
+#define RT_NODE_SEARCH_INNER RT_MAKE_NAME(node_search_inner)
+#define RT_NODE_SEARCH_LEAF RT_MAKE_NAME(node_search_leaf)
+#define RT_NODE_UPDATE_INNER RT_MAKE_NAME(node_update_inner)
+#define RT_NODE_DELETE_INNER RT_MAKE_NAME(node_delete_inner)
+#define RT_NODE_DELETE_LEAF RT_MAKE_NAME(node_delete_leaf)
+#define RT_NODE_INSERT_INNER RT_MAKE_NAME(node_insert_inner)
+#define RT_NODE_INSERT_LEAF RT_MAKE_NAME(node_insert_leaf)
+#define RT_NODE_INNER_ITERATE_NEXT RT_MAKE_NAME(node_inner_iterate_next)
+#define RT_NODE_LEAF_ITERATE_NEXT RT_MAKE_NAME(node_leaf_iterate_next)
+#define RT_UPDATE_ITER_STACK RT_MAKE_NAME(update_iter_stack)
+#define RT_ITER_UPDATE_KEY RT_MAKE_NAME(iter_update_key)
+#define RT_VERIFY_NODE RT_MAKE_NAME(verify_node)
+
+/* type declarations */
+#define RT_RADIX_TREE RT_MAKE_NAME(radix_tree)
+#define RT_RADIX_TREE_CONTROL RT_MAKE_NAME(radix_tree_control)
+#define RT_ITER RT_MAKE_NAME(iter)
+#ifdef RT_SHMEM
+#define RT_HANDLE RT_MAKE_NAME(handle)
+#endif
+#define RT_NODE RT_MAKE_NAME(node)
+#define RT_NODE_ITER RT_MAKE_NAME(node_iter)
+#define RT_NODE_BASE_3 RT_MAKE_NAME(node_base_3)
+#define RT_NODE_BASE_32 RT_MAKE_NAME(node_base_32)
+#define RT_NODE_BASE_125 RT_MAKE_NAME(node_base_125)
+#define RT_NODE_BASE_256 RT_MAKE_NAME(node_base_256)
+#define RT_NODE_INNER_3 RT_MAKE_NAME(node_inner_3)
+#define RT_NODE_INNER_32 RT_MAKE_NAME(node_inner_32)
+#define RT_NODE_INNER_125 RT_MAKE_NAME(node_inner_125)
+#define RT_NODE_INNER_256 RT_MAKE_NAME(node_inner_256)
+#define RT_NODE_LEAF_3 RT_MAKE_NAME(node_leaf_3)
+#define RT_NODE_LEAF_32 RT_MAKE_NAME(node_leaf_32)
+#define RT_NODE_LEAF_125 RT_MAKE_NAME(node_leaf_125)
+#define RT_NODE_LEAF_256 RT_MAKE_NAME(node_leaf_256)
+#define RT_SIZE_CLASS RT_MAKE_NAME(size_class)
+#define RT_SIZE_CLASS_ELEM RT_MAKE_NAME(size_class_elem)
+#define RT_SIZE_CLASS_INFO RT_MAKE_NAME(size_class_info)
+#define RT_CLASS_3 RT_MAKE_NAME(class_3)
+#define RT_CLASS_32_MIN RT_MAKE_NAME(class_32_min)
+#define RT_CLASS_32_MAX RT_MAKE_NAME(class_32_max)
+#define RT_CLASS_125 RT_MAKE_NAME(class_125)
+#define RT_CLASS_256 RT_MAKE_NAME(class_256)
+
+/* generate forward declarations necessary to use the radix tree */
+#ifdef RT_DECLARE
+
+typedef struct RT_RADIX_TREE RT_RADIX_TREE;
+typedef struct RT_ITER RT_ITER;
+
+#ifdef RT_SHMEM
+typedef dsa_pointer RT_HANDLE;
+#endif
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id);
+RT_SCOPE RT_RADIX_TREE * RT_ATTACH(dsa_area *dsa, dsa_pointer dp);
+RT_SCOPE void RT_DETACH(RT_RADIX_TREE *tree);
+RT_SCOPE RT_HANDLE RT_GET_HANDLE(RT_RADIX_TREE *tree);
+#else
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx);
+#endif
+RT_SCOPE void RT_FREE(RT_RADIX_TREE *tree);
+
+RT_SCOPE bool RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p);
+RT_SCOPE bool RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p);
+#ifdef RT_USE_DELETE
+RT_SCOPE bool RT_DELETE(RT_RADIX_TREE *tree, uint64 key);
+#endif
+
+RT_SCOPE RT_ITER * RT_BEGIN_ITERATE(RT_RADIX_TREE *tree);
+RT_SCOPE bool RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p);
+RT_SCOPE void RT_END_ITERATE(RT_ITER *iter);
+
+RT_SCOPE uint64 RT_MEMORY_USAGE(RT_RADIX_TREE *tree);
+
+#ifdef RT_DEBUG
+RT_SCOPE void RT_DUMP(RT_RADIX_TREE *tree);
+RT_SCOPE void RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key);
+RT_SCOPE void RT_STATS(RT_RADIX_TREE *tree);
+#endif
+
+#endif /* RT_DECLARE */
+
+
+/* generate implementation of the radix tree */
+#ifdef RT_DEFINE
+
+/* The number of bits encoded in one tree level */
+#define RT_NODE_SPAN BITS_PER_BYTE
+
+/* The number of maximum slots in the node */
+#define RT_NODE_MAX_SLOTS (1 << RT_NODE_SPAN)
+
+/* Mask for extracting a chunk from the key */
+#define RT_CHUNK_MASK ((1 << RT_NODE_SPAN) - 1)
+
+/* Maximum shift the radix tree uses */
+#define RT_MAX_SHIFT RT_KEY_GET_SHIFT(UINT64_MAX)
+
+/* Tree level the radix tree uses */
+#define RT_MAX_LEVEL ((sizeof(uint64) * BITS_PER_BYTE) / RT_NODE_SPAN)
+
+/*
+ * Number of bits necessary for isset array in the slot-index node.
+ * Since bitmapword can be 64 bits, the only values that make sense
+ * here are 64 and 128.
+ */
+#define RT_SLOT_IDX_LIMIT (RT_NODE_MAX_SLOTS / 2)
+
+/* Invalid index used in node-125 */
+#define RT_INVALID_SLOT_IDX 0xFF
+
+/* Get a chunk from the key */
+#define RT_GET_KEY_CHUNK(key, shift) ((uint8) (((key) >> (shift)) & RT_CHUNK_MASK))
+
+/* For accessing bitmaps */
+#define RT_BM_IDX(x) ((x) / BITS_PER_BITMAPWORD)
+#define RT_BM_BIT(x) ((x) % BITS_PER_BITMAPWORD)
+
+/*
+ * Node kinds
+ *
+ * The different node kinds are what make the tree "adaptive".
+ *
+ * Each node kind is associated with a different datatype and different
+ * search/set/delete/iterate algorithms adapted for its size. The largest
+ * kind, node256 is basically the same as a traditional radix tree,
+ * and would be most wasteful of memory when sparsely populated. The
+ * smaller nodes expend some additional CPU time to enable a smaller
+ * memory footprint.
+ *
+ * XXX There are 4 node kinds, and this should never be increased,
+ * for several reasons:
+ * 1. With 5 or more kinds, gcc tends to use a jump table for switch
+ * statements.
+ * 2. The 4 kinds can be represented with 2 bits, so we have the option
+ * in the future to tag the node pointer with the kind, even on
+ * platforms with 32-bit pointers. This might speed up node traversal
+ * in trees with highly random node kinds.
+ * 3. We can have multiple size classes per node kind.
+ */
+#define RT_NODE_KIND_3 0x00
+#define RT_NODE_KIND_32 0x01
+#define RT_NODE_KIND_125 0x02
+#define RT_NODE_KIND_256 0x03
+#define RT_NODE_KIND_COUNT 4
+
+/*
+ * Calculate the slab blocksize so that we can allocate at least 32 chunks
+ * from the block.
+ */
+#define RT_SLAB_BLOCK_SIZE(size) \
+ Max((SLAB_DEFAULT_BLOCK_SIZE / (size)) * (size), (size) * 32)
+
+/* Common type for all nodes types */
+typedef struct RT_NODE
+{
+ /*
+ * Number of children. We use uint16 to be able to indicate 256 children
+ * at the fanout of 8.
+ */
+ uint16 count;
+
+ /*
+ * Max capacity for the current size class. Storing this in the
+ * node enables multiple size classes per node kind.
+ * Technically, kinds with a single size class don't need this, so we could
+ * keep this in the individual base types, but the code is simpler this way.
+ * Note: node256 is unique in that it cannot possibly have more than a
+ * single size class, so for that kind we store zero, and uint8 is
+ * sufficient for other kinds.
+ */
+ uint8 fanout;
+
+ /*
+ * Shift indicates which part of the key space is represented by this
+ * node. That is, the key is shifted by 'shift' and the lowest
+ * RT_NODE_SPAN bits are then represented in chunk.
+ */
+ uint8 shift;
+
+ /* Node kind, one per search/set algorithm */
+ uint8 kind;
+} RT_NODE;
+
+
+#define RT_PTR_LOCAL RT_NODE *
+
+#ifdef RT_SHMEM
+#define RT_PTR_ALLOC dsa_pointer
+#else
+#define RT_PTR_ALLOC RT_PTR_LOCAL
+#endif
+
+
+#ifdef RT_SHMEM
+#define RT_INVALID_PTR_ALLOC InvalidDsaPointer
+#else
+#define RT_INVALID_PTR_ALLOC NULL
+#endif
+
+#ifdef RT_SHMEM
+#define RT_LOCK_EXCLUSIVE(tree) LWLockAcquire(&tree->ctl->lock, LW_EXCLUSIVE)
+#define RT_LOCK_SHARED(tree) LWLockAcquire(&tree->ctl->lock, LW_SHARED)
+#define RT_UNLOCK(tree) LWLockRelease(&tree->ctl->lock);
+#else
+#define RT_LOCK_EXCLUSIVE(tree) ((void) 0)
+#define RT_LOCK_SHARED(tree) ((void) 0)
+#define RT_UNLOCK(tree) ((void) 0)
+#endif
+
+/*
+ * Inner nodes and leaf nodes have analogous structure. To distinguish
+ * them at runtime, we take advantage of the fact that the key chunk
+ * is accessed by shifting: Inner tree nodes (shift > 0), store the
+ * pointer to its child node in the slot. In leaf nodes (shift == 0),
+ * the slot contains the value corresponding to the key.
+ */
+#define RT_NODE_IS_LEAF(n) (((RT_PTR_LOCAL) (n))->shift == 0)
+
+#define RT_NODE_MUST_GROW(node) \
+ ((node)->base.n.count == (node)->base.n.fanout)
+
+/*
+ * Base type of each node kinds for leaf and inner nodes.
+ * The base types must be a be able to accommodate the largest size
+ * class for variable-sized node kinds.
+ */
+typedef struct RT_NODE_BASE_3
+{
+ RT_NODE n;
+
+ /* 3 children, for key chunks */
+ uint8 chunks[3];
+} RT_NODE_BASE_3;
+
+typedef struct RT_NODE_BASE_32
+{
+ RT_NODE n;
+
+ /* 32 children, for key chunks */
+ uint8 chunks[32];
+} RT_NODE_BASE_32;
+
+/*
+ * node-125 uses slot_idx array, an array of RT_NODE_MAX_SLOTS length
+ * to store indexes into a second array that contains the values (or
+ * child pointers).
+ */
+typedef struct RT_NODE_BASE_125
+{
+ RT_NODE n;
+
+ /* The index of slots for each fanout */
+ uint8 slot_idxs[RT_NODE_MAX_SLOTS];
+
+ /* bitmap to track which slots are in use */
+ bitmapword isset[RT_BM_IDX(RT_SLOT_IDX_LIMIT)];
+} RT_NODE_BASE_125;
+
+typedef struct RT_NODE_BASE_256
+{
+ RT_NODE n;
+} RT_NODE_BASE_256;
+
+/*
+ * Inner and leaf nodes.
+ *
+ * These are separate because the value type might be different than
+ * something fitting into a pointer-width type.
+ */
+typedef struct RT_NODE_INNER_3
+{
+ RT_NODE_BASE_3 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_3;
+
+typedef struct RT_NODE_LEAF_3
+{
+ RT_NODE_BASE_3 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_3;
+
+typedef struct RT_NODE_INNER_32
+{
+ RT_NODE_BASE_32 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_32;
+
+typedef struct RT_NODE_LEAF_32
+{
+ RT_NODE_BASE_32 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_32;
+
+typedef struct RT_NODE_INNER_125
+{
+ RT_NODE_BASE_125 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_125;
+
+typedef struct RT_NODE_LEAF_125
+{
+ RT_NODE_BASE_125 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_125;
+
+/*
+ * node-256 is the largest node type. This node has an array
+ * for directly storing values (or child pointers in inner nodes).
+ * Unlike other node kinds, it's array size is by definition
+ * fixed.
+ */
+typedef struct RT_NODE_INNER_256
+{
+ RT_NODE_BASE_256 base;
+
+ /* Slots for 256 children */
+ RT_PTR_ALLOC children[RT_NODE_MAX_SLOTS];
+} RT_NODE_INNER_256;
+
+typedef struct RT_NODE_LEAF_256
+{
+ RT_NODE_BASE_256 base;
+
+ /*
+ * Unlike with inner256, zero is a valid value here, so we use a
+ * bitmap to track which slots are in use.
+ */
+ bitmapword isset[RT_BM_IDX(RT_NODE_MAX_SLOTS)];
+
+ /* Slots for 256 values */
+ RT_VALUE_TYPE values[RT_NODE_MAX_SLOTS];
+} RT_NODE_LEAF_256;
+
+/*
+ * Node size classes
+ *
+ * Nodes of different kinds necessarily belong to different size classes.
+ * The main innovation in our implementation compared to the ART paper
+ * is decoupling the notion of size class from kind.
+ *
+ * The size classes within a given node kind have the same underlying
+ * type, but a variable number of children/values. This is possible
+ * because the base type contains small fixed data structures that
+ * work the same way regardless of how full the node is. We store the
+ * node's allocated capacity in the "fanout" member of RT_NODE, to allow
+ * runtime introspection.
+ *
+ * Growing from one node kind to another requires special code for each
+ * case, but growing from one size class to another within the same kind
+ * is basically just allocate + memcpy.
+ *
+ * The size classes have been chosen so that inner nodes on platforms
+ * with 64-bit pointers (and leaf nodes when using a 64-bit key) are
+ * equal to or slightly smaller than some DSA size class.
+ */
+typedef enum RT_SIZE_CLASS
+{
+ RT_CLASS_3 = 0,
+ RT_CLASS_32_MIN,
+ RT_CLASS_32_MAX,
+ RT_CLASS_125,
+ RT_CLASS_256
+} RT_SIZE_CLASS;
+
+/* Information for each size class */
+typedef struct RT_SIZE_CLASS_ELEM
+{
+ const char *name;
+ int fanout;
+
+ /* slab chunk size */
+ Size inner_size;
+ Size leaf_size;
+} RT_SIZE_CLASS_ELEM;
+
+static const RT_SIZE_CLASS_ELEM RT_SIZE_CLASS_INFO[] = {
+ [RT_CLASS_3] = {
+ .name = "radix tree node 3",
+ .fanout = 3,
+ .inner_size = sizeof(RT_NODE_INNER_3) + 3 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_3) + 3 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_32_MIN] = {
+ .name = "radix tree node 15",
+ .fanout = 15,
+ .inner_size = sizeof(RT_NODE_INNER_32) + 15 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_32) + 15 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_32_MAX] = {
+ .name = "radix tree node 32",
+ .fanout = 32,
+ .inner_size = sizeof(RT_NODE_INNER_32) + 32 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_32) + 32 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_125] = {
+ .name = "radix tree node 125",
+ .fanout = 125,
+ .inner_size = sizeof(RT_NODE_INNER_125) + 125 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_125) + 125 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_256] = {
+ .name = "radix tree node 256",
+ .fanout = 256,
+ .inner_size = sizeof(RT_NODE_INNER_256),
+ .leaf_size = sizeof(RT_NODE_LEAF_256),
+ },
+};
+
+#define RT_SIZE_CLASS_COUNT lengthof(RT_SIZE_CLASS_INFO)
+
+#ifdef RT_SHMEM
+/* A magic value used to identify our radix tree */
+#define RT_RADIX_TREE_MAGIC 0x54A48167
+#endif
+
+/* Contains the actual tree and ancillary info */
+// WIP: this name is a bit strange
+typedef struct RT_RADIX_TREE_CONTROL
+{
+#ifdef RT_SHMEM
+ RT_HANDLE handle;
+ uint32 magic;
+ LWLock lock;
+#endif
+
+ RT_PTR_ALLOC root;
+ uint64 max_val;
+ uint64 num_keys;
+
+ /* statistics */
+#ifdef RT_DEBUG
+ int32 cnt[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE_CONTROL;
+
+/* Entry point for allocating and accessing the tree */
+typedef struct RT_RADIX_TREE
+{
+ MemoryContext context;
+
+ /* pointing to either local memory or DSA */
+ RT_RADIX_TREE_CONTROL *ctl;
+
+#ifdef RT_SHMEM
+ dsa_area *dsa;
+#else
+ MemoryContextData *inner_slabs[RT_SIZE_CLASS_COUNT];
+ MemoryContextData *leaf_slabs[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE;
+
+/*
+ * Iteration support.
+ *
+ * Iterating the radix tree returns each pair of key and value in the ascending
+ * order of the key. To support this, the we iterate nodes of each level.
+ *
+ * RT_NODE_ITER struct is used to track the iteration within a node.
+ *
+ * RT_ITER is the struct for iteration of the radix tree, and uses RT_NODE_ITER
+ * in order to track the iteration of each level. During iteration, we also
+ * construct the key whenever updating the node iteration information, e.g., when
+ * advancing the current index within the node or when moving to the next node
+ * at the same level.
+ *
+ * XXX: Currently we allow only one process to do iteration. Therefore, rt_node_iter
+ * has the local pointers to nodes, rather than RT_PTR_ALLOC.
+ * We need either a safeguard to disallow other processes to begin the iteration
+ * while one process is doing or to allow multiple processes to do the iteration.
+ */
+typedef struct RT_NODE_ITER
+{
+ RT_PTR_LOCAL node; /* current node being iterated */
+ int current_idx; /* current position. -1 for initial value */
+} RT_NODE_ITER;
+
+typedef struct RT_ITER
+{
+ RT_RADIX_TREE *tree;
+
+ /* Track the iteration on nodes of each level */
+ RT_NODE_ITER stack[RT_MAX_LEVEL];
+ int stack_len;
+
+ /* The key is constructed during iteration */
+ uint64 key;
+} RT_ITER;
+
+
+static bool RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_PTR_ALLOC child);
+static bool RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_VALUE_TYPE *value_p);
+
+/* verification (available only with assertion) */
+static void RT_VERIFY_NODE(RT_PTR_LOCAL node);
+
+/* Get the local address of an allocated node */
+static inline RT_PTR_LOCAL
+RT_PTR_GET_LOCAL(RT_RADIX_TREE *tree, RT_PTR_ALLOC node)
+{
+#ifdef RT_SHMEM
+ return dsa_get_address(tree->dsa, (dsa_pointer) node);
+#else
+ return node;
+#endif
+}
+
+static inline bool
+RT_PTR_ALLOC_IS_VALID(RT_PTR_ALLOC ptr)
+{
+#ifdef RT_SHMEM
+ return DsaPointerIsValid(ptr);
+#else
+ return PointerIsValid(ptr);
+#endif
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_3_SEARCH_EQ(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+ int idx = -1;
+
+ for (int i = 0; i < node->n.count; i++)
+ {
+ if (node->chunks[i] == chunk)
+ {
+ idx = i;
+ break;
+ }
+ }
+
+ return idx;
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_3_GET_INSERTPOS(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+ int idx;
+
+ for (idx = 0; idx < node->n.count; idx++)
+ {
+ if (node->chunks[idx] >= chunk)
+ break;
+ }
+
+ return idx;
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_32_SEARCH_EQ(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+ int count = node->n.count;
+#ifndef USE_NO_SIMD
+ Vector8 spread_chunk;
+ Vector8 haystack1;
+ Vector8 haystack2;
+ Vector8 cmp1;
+ Vector8 cmp2;
+ uint32 bitfield;
+ int index_simd = -1;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+ int index = -1;
+
+ for (int i = 0; i < count; i++)
+ {
+ if (node->chunks[i] == chunk)
+ {
+ index = i;
+ break;
+ }
+ }
+#endif
+
+#ifndef USE_NO_SIMD
+ /* replicate the search key */
+ spread_chunk = vector8_broadcast(chunk);
+
+ /* compare to all 32 keys stored in the node */
+ vector8_load(&haystack1, &node->chunks[0]);
+ vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+ cmp1 = vector8_eq(spread_chunk, haystack1);
+ cmp2 = vector8_eq(spread_chunk, haystack2);
+
+ /* convert comparison to a bitfield */
+ bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+
+ /* mask off invalid entries */
+ bitfield &= ((UINT64CONST(1) << count) - 1);
+
+ /* convert bitfield to index by counting trailing zeros */
+ if (bitfield)
+ index_simd = pg_rightmost_one_pos32(bitfield);
+
+ Assert(index_simd == index);
+ return index_simd;
+#else
+ return index;
+#endif
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_32_GET_INSERTPOS(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+ int count = node->n.count;
+#ifndef USE_NO_SIMD
+ Vector8 spread_chunk;
+ Vector8 haystack1;
+ Vector8 haystack2;
+ Vector8 cmp1;
+ Vector8 cmp2;
+ Vector8 min1;
+ Vector8 min2;
+ uint32 bitfield;
+ int index_simd;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+ int index;
+
+ for (index = 0; index < count; index++)
+ {
+ /*
+ * This is coded with '>=' to match what we can do with SIMD,
+ * with an assert to keep us honest.
+ */
+ if (node->chunks[index] >= chunk)
+ {
+ Assert(node->chunks[index] != chunk);
+ break;
+ }
+ }
+#endif
+
+#ifndef USE_NO_SIMD
+ /*
+ * This is a bit more complicated than RT_NODE_32_SEARCH_EQ(), because
+ * no unsigned uint8 comparison instruction exists, at least for SSE2. So
+ * we need to play some trickery using vector8_min() to effectively get
+ * >=. There'll never be any equal elements in current uses, but that's
+ * what we get here...
+ */
+ spread_chunk = vector8_broadcast(chunk);
+ vector8_load(&haystack1, &node->chunks[0]);
+ vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+ min1 = vector8_min(spread_chunk, haystack1);
+ min2 = vector8_min(spread_chunk, haystack2);
+ cmp1 = vector8_eq(spread_chunk, min1);
+ cmp2 = vector8_eq(spread_chunk, min2);
+ bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+ bitfield &= ((UINT64CONST(1) << count) - 1);
+
+ if (bitfield)
+ index_simd = pg_rightmost_one_pos32(bitfield);
+ else
+ index_simd = count;
+
+ Assert(index_simd == index);
+ return index_simd;
+#else
+ return index;
+#endif
+}
+
+
+/*
+ * Functions to manipulate both chunks array and children/values array.
+ * These are used for node-3 and node-32.
+ */
+
+/* Shift the elements right at 'idx' by one */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_SHIFT(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+ memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+ memmove(&(children[idx + 1]), &(children[idx]), sizeof(RT_PTR_ALLOC) * (count - idx));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_SHIFT(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+ memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+ memmove(&(values[idx + 1]), &(values[idx]), sizeof(RT_VALUE_TYPE) * (count - idx));
+}
+
+/* Delete the element at 'idx' */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_DELETE(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+ memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+ memmove(&(children[idx]), &(children[idx + 1]), sizeof(RT_PTR_ALLOC) * (count - idx - 1));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_DELETE(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+ memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+ memmove(&(values[idx]), &(values[idx + 1]), sizeof(RT_VALUE_TYPE) * (count - idx - 1));
+}
+
+/* Copy both chunks and children/values arrays */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_COPY(uint8 *src_chunks, RT_PTR_ALLOC *src_children,
+ uint8 *dst_chunks, RT_PTR_ALLOC *dst_children)
+{
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const Size chunk_size = sizeof(uint8) * fanout;
+ const Size children_size = sizeof(RT_PTR_ALLOC) * fanout;
+
+ memcpy(dst_chunks, src_chunks, chunk_size);
+ memcpy(dst_children, src_children, children_size);
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_COPY(uint8 *src_chunks, RT_VALUE_TYPE *src_values,
+ uint8 *dst_chunks, RT_VALUE_TYPE *dst_values)
+{
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const Size chunk_size = sizeof(uint8) * fanout;
+ const Size values_size = sizeof(RT_VALUE_TYPE) * fanout;
+
+ memcpy(dst_chunks, src_chunks, chunk_size);
+ memcpy(dst_values, src_values, values_size);
+}
+
+/* Functions to manipulate inner and leaf node-125 */
+
+/* Does the given chunk in the node has the value? */
+static inline bool
+RT_NODE_125_IS_CHUNK_USED(RT_NODE_BASE_125 *node, uint8 chunk)
+{
+ return node->slot_idxs[chunk] != RT_INVALID_SLOT_IDX;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_125_GET_CHILD(RT_NODE_INNER_125 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ return node->children[node->base.slot_idxs[chunk]];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_125_GET_VALUE(RT_NODE_LEAF_125 *node, uint8 chunk)
+{
+ Assert(RT_NODE_IS_LEAF(node));
+ Assert(((RT_NODE_BASE_125 *) node)->slot_idxs[chunk] != RT_INVALID_SLOT_IDX);
+ return node->values[node->base.slot_idxs[chunk]];
+}
+
+/* Functions to manipulate inner and leaf node-256 */
+
+/* Return true if the slot corresponding to the given chunk is in use */
+static inline bool
+RT_NODE_INNER_256_IS_CHUNK_USED(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ return node->children[chunk] != RT_INVALID_PTR_ALLOC;
+}
+
+static inline bool
+RT_NODE_LEAF_256_IS_CHUNK_USED(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ return (node->isset[idx] & ((bitmapword) 1 << bitnum)) != 0;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_256_GET_CHILD(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ Assert(RT_NODE_INNER_256_IS_CHUNK_USED(node, chunk));
+ return node->children[chunk];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_256_GET_VALUE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ Assert(RT_NODE_IS_LEAF(node));
+ Assert(RT_NODE_LEAF_256_IS_CHUNK_USED(node, chunk));
+ return node->values[chunk];
+}
+
+/* Set the child in the node-256 */
+static inline void
+RT_NODE_INNER_256_SET(RT_NODE_INNER_256 *node, uint8 chunk, RT_PTR_ALLOC child)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ node->children[chunk] = child;
+}
+
+/* Set the value in the node-256 */
+static inline void
+RT_NODE_LEAF_256_SET(RT_NODE_LEAF_256 *node, uint8 chunk, RT_VALUE_TYPE value)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ node->isset[idx] |= ((bitmapword) 1 << bitnum);
+ node->values[chunk] = value;
+}
+
+/* Set the slot at the given chunk position */
+static inline void
+RT_NODE_INNER_256_DELETE(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ node->children[chunk] = RT_INVALID_PTR_ALLOC;
+}
+
+static inline void
+RT_NODE_LEAF_256_DELETE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ node->isset[idx] &= ~((bitmapword) 1 << bitnum);
+}
+
+/*
+ * Return the largest shift that will allowing storing the given key.
+ */
+static inline int
+RT_KEY_GET_SHIFT(uint64 key)
+{
+ if (key == 0)
+ return 0;
+ else
+ return (pg_leftmost_one_pos64(key) / RT_NODE_SPAN) * RT_NODE_SPAN;
+}
+
+/*
+ * Return the max value that can be stored in the tree with the given shift.
+ */
+static uint64
+RT_SHIFT_GET_MAX_VAL(int shift)
+{
+ if (shift == RT_MAX_SHIFT)
+ return UINT64_MAX;
+
+ return (UINT64CONST(1) << (shift + RT_NODE_SPAN)) - 1;
+}
+
+/*
+ * Allocate a new node with the given node kind.
+ */
+static RT_PTR_ALLOC
+RT_ALLOC_NODE(RT_RADIX_TREE *tree, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+ RT_PTR_ALLOC allocnode;
+ size_t allocsize;
+
+ if (is_leaf)
+ allocsize = RT_SIZE_CLASS_INFO[size_class].leaf_size;
+ else
+ allocsize = RT_SIZE_CLASS_INFO[size_class].inner_size;
+
+#ifdef RT_SHMEM
+ allocnode = dsa_allocate(tree->dsa, allocsize);
+#else
+ if (is_leaf)
+ allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->leaf_slabs[size_class],
+ allocsize);
+ else
+ allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->inner_slabs[size_class],
+ allocsize);
+#endif
+
+#ifdef RT_DEBUG
+ /* update the statistics */
+ tree->ctl->cnt[size_class]++;
+#endif
+
+ return allocnode;
+}
+
+/* Initialize the node contents */
+static inline void
+RT_INIT_NODE(RT_PTR_LOCAL node, uint8 kind, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+ if (is_leaf)
+ MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].leaf_size);
+ else
+ MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].inner_size);
+
+ node->kind = kind;
+
+ if (kind == RT_NODE_KIND_256)
+ /* See comment for the RT_NODE type */
+ Assert(node->fanout == 0);
+ else
+ node->fanout = RT_SIZE_CLASS_INFO[size_class].fanout;
+
+ /* Initialize slot_idxs to invalid values */
+ if (kind == RT_NODE_KIND_125)
+ {
+ RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+
+ memset(n125->slot_idxs, RT_INVALID_SLOT_IDX, sizeof(n125->slot_idxs));
+ }
+}
+
+/*
+ * Create a new node as the root. Subordinate nodes will be created during
+ * the insertion.
+ */
+static void
+RT_NEW_ROOT(RT_RADIX_TREE *tree, uint64 key)
+{
+ int shift = RT_KEY_GET_SHIFT(key);
+ bool is_leaf = shift == 0;
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(newnode, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ newnode->shift = shift;
+ tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(shift);
+ tree->ctl->root = allocnode;
+}
+
+static inline void
+RT_COPY_NODE(RT_PTR_LOCAL newnode, RT_PTR_LOCAL oldnode)
+{
+ newnode->shift = oldnode->shift;
+ newnode->count = oldnode->count;
+}
+
+/*
+ * Given a new allocated node and an old node, initalize the new
+ * node with the necessary fields and return its local pointer.
+ */
+static inline RT_PTR_LOCAL
+RT_SWITCH_NODE_KIND(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, RT_PTR_LOCAL node,
+ uint8 new_kind, uint8 new_class, bool is_leaf)
+{
+ RT_PTR_LOCAL newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(newnode, new_kind, new_class, is_leaf);
+ RT_COPY_NODE(newnode, node);
+
+ return newnode;
+}
+
+/* Free the given node */
+static void
+RT_FREE_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode)
+{
+ /* If we're deleting the root node, make the tree empty */
+ if (tree->ctl->root == allocnode)
+ {
+ tree->ctl->root = RT_INVALID_PTR_ALLOC;
+ tree->ctl->max_val = 0;
+ }
+
+#ifdef RT_DEBUG
+ {
+ int i;
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+ /* update the statistics */
+ for (i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ if (node->fanout == RT_SIZE_CLASS_INFO[i].fanout)
+ break;
+ }
+
+ /* fanout of node256 is intentionally 0 */
+ if (i == RT_SIZE_CLASS_COUNT)
+ i = RT_CLASS_256;
+
+ tree->ctl->cnt[i]--;
+ Assert(tree->ctl->cnt[i] >= 0);
+ }
+#endif
+
+#ifdef RT_SHMEM
+ dsa_free(tree->dsa, allocnode);
+#else
+ pfree(allocnode);
+#endif
+}
+
+/* Update the parent's pointer when growing a node */
+static inline void
+RT_NODE_UPDATE_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC new_child)
+{
+#define RT_ACTION_UPDATE
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+#undef RT_ACTION_UPDATE
+}
+
+/*
+ * Replace old_child with new_child, and free the old one.
+ */
+static void
+RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
+ RT_PTR_ALLOC stored_old_child, RT_PTR_LOCAL old_child,
+ RT_PTR_ALLOC new_child, uint64 key)
+{
+#ifdef USE_ASSERT_CHECKING
+ RT_PTR_LOCAL new = RT_PTR_GET_LOCAL(tree, new_child);
+
+ Assert(old_child->shift == new->shift);
+ Assert(old_child->count == new->count);
+#endif
+
+ if (parent == old_child)
+ {
+ /* Replace the root node with the new larger node */
+ tree->ctl->root = new_child;
+ }
+ else
+ RT_NODE_UPDATE_INNER(parent, key, new_child);
+
+ RT_FREE_NODE(tree, stored_old_child);
+}
+
+/*
+ * The radix tree doesn't have sufficient height. Extend the radix tree so
+ * it can store the key.
+ */
+static void
+RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
+{
+ int target_shift;
+ RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+ int shift = root->shift + RT_NODE_SPAN;
+
+ target_shift = RT_KEY_GET_SHIFT(key);
+
+ /* Grow tree from 'shift' to 'target_shift' */
+ while (shift <= target_shift)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL node;
+ RT_NODE_INNER_3 *n3;
+
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, true);
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(node, RT_NODE_KIND_3, RT_CLASS_3, true);
+ node->shift = shift;
+ node->count = 1;
+
+ n3 = (RT_NODE_INNER_3 *) node;
+ n3->base.chunks[0] = 0;
+ n3->children[0] = tree->ctl->root;
+
+ /* Update the root */
+ tree->ctl->root = allocnode;
+
+ shift += RT_NODE_SPAN;
+ }
+
+ tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(target_shift);
+}
+
+/*
+ * The radix tree doesn't have inner and leaf nodes for given key-value pair.
+ * Insert inner and leaf nodes from 'node' to bottom.
+ */
+static inline void
+RT_SET_EXTEND(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
+ RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node)
+{
+ int shift = node->shift;
+
+ Assert(RT_PTR_GET_LOCAL(tree, stored_node) == node);
+
+ while (shift >= RT_NODE_SPAN)
+ {
+ RT_PTR_ALLOC allocchild;
+ RT_PTR_LOCAL newchild;
+ int newshift = shift - RT_NODE_SPAN;
+ bool is_leaf = newshift == 0;
+
+ allocchild = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ newchild = RT_PTR_GET_LOCAL(tree, allocchild);
+ RT_INIT_NODE(newchild, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ newchild->shift = newshift;
+ RT_NODE_INSERT_INNER(tree, parent, stored_node, node, key, allocchild);
+
+ parent = node;
+ node = newchild;
+ stored_node = allocchild;
+ shift -= RT_NODE_SPAN;
+ }
+
+ RT_NODE_INSERT_LEAF(tree, parent, stored_node, node, key, value_p);
+ tree->ctl->num_keys++;
+}
+
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the child
+ * pointer is set to child_p.
+ */
+static inline bool
+RT_NODE_SEARCH_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC *child_p)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the pointer
+ * to the value is set to value_p.
+ */
+static inline bool
+RT_NODE_SEARCH_LEAF(RT_PTR_LOCAL node, uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_INNER(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_LEAF(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+#endif
+
+/*
+ * Insert "child" into "node".
+ *
+ * "parent" is the parent of "node", so the grandparent of the child.
+ * If the node we're inserting into needs to grow, we update the parent's
+ * child pointer with the pointer to the new larger node.
+ */
+static bool
+RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_PTR_ALLOC child)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/* Like RT_NODE_INSERT_INNER, but for leaf nodes */
+static bool
+RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Create the radix tree in the given memory context and return it.
+ */
+RT_SCOPE RT_RADIX_TREE *
+#ifdef RT_SHMEM
+RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id)
+#else
+RT_CREATE(MemoryContext ctx)
+#endif
+{
+ RT_RADIX_TREE *tree;
+ MemoryContext old_ctx;
+#ifdef RT_SHMEM
+ dsa_pointer dp;
+#endif
+
+ old_ctx = MemoryContextSwitchTo(ctx);
+
+ tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+ tree->context = ctx;
+
+#ifdef RT_SHMEM
+ tree->dsa = dsa;
+ dp = dsa_allocate0(dsa, sizeof(RT_RADIX_TREE_CONTROL));
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, dp);
+ tree->ctl->handle = dp;
+ tree->ctl->magic = RT_RADIX_TREE_MAGIC;
+ LWLockInitialize(&tree->ctl->lock, tranche_id);
+#else
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) palloc0(sizeof(RT_RADIX_TREE_CONTROL));
+
+ /* Create a slab context for each size class */
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ RT_SIZE_CLASS_ELEM size_class = RT_SIZE_CLASS_INFO[i];
+ size_t inner_blocksize = RT_SLAB_BLOCK_SIZE(size_class.inner_size);
+ size_t leaf_blocksize = RT_SLAB_BLOCK_SIZE(size_class.leaf_size);
+
+ tree->inner_slabs[i] = SlabContextCreate(ctx,
+ size_class.name,
+ inner_blocksize,
+ size_class.inner_size);
+ tree->leaf_slabs[i] = SlabContextCreate(ctx,
+ size_class.name,
+ leaf_blocksize,
+ size_class.leaf_size);
+ }
+#endif
+
+ tree->ctl->root = RT_INVALID_PTR_ALLOC;
+
+ MemoryContextSwitchTo(old_ctx);
+
+ return tree;
+}
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE *
+RT_ATTACH(dsa_area *dsa, RT_HANDLE handle)
+{
+ RT_RADIX_TREE *tree;
+ dsa_pointer control;
+
+ tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+
+ /* Find the control object in shard memory */
+ control = handle;
+
+ tree->dsa = dsa;
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, control);
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+ return tree;
+}
+
+RT_SCOPE void
+RT_DETACH(RT_RADIX_TREE *tree)
+{
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ pfree(tree);
+}
+
+RT_SCOPE RT_HANDLE
+RT_GET_HANDLE(RT_RADIX_TREE *tree)
+{
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ return tree->ctl->handle;
+}
+
+/*
+ * Recursively free all nodes allocated to the DSA area.
+ */
+static inline void
+RT_FREE_RECURSE(RT_RADIX_TREE *tree, RT_PTR_ALLOC ptr)
+{
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, ptr);
+
+ check_stack_depth();
+ CHECK_FOR_INTERRUPTS();
+
+ /* The leaf node doesn't have child pointers */
+ if (RT_NODE_IS_LEAF(node))
+ {
+ dsa_free(tree->dsa, ptr);
+ return;
+ }
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+ for (int i = 0; i < n3->base.n.count; i++)
+ RT_FREE_RECURSE(tree, n3->children[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+ for (int i = 0; i < n32->base.n.count; i++)
+ RT_FREE_RECURSE(tree, n32->children[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) node;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+ continue;
+
+ RT_FREE_RECURSE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+ }
+
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ RT_FREE_RECURSE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+ }
+
+ break;
+ }
+ }
+
+ /* Free the inner node */
+ dsa_free(tree->dsa, ptr);
+}
+#endif
+
+/*
+ * Free the given radix tree.
+ */
+RT_SCOPE void
+RT_FREE(RT_RADIX_TREE *tree)
+{
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+ /* Free all memory used for radix tree nodes */
+ if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ RT_FREE_RECURSE(tree, tree->ctl->root);
+
+ /*
+ * Vandalize the control block to help catch programming error where
+ * other backends access the memory formerly occupied by this radix tree.
+ */
+ tree->ctl->magic = 0;
+ dsa_free(tree->dsa, tree->ctl->handle);
+#else
+ pfree(tree->ctl);
+
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ MemoryContextDelete(tree->inner_slabs[i]);
+ MemoryContextDelete(tree->leaf_slabs[i]);
+ }
+#endif
+
+ pfree(tree);
+}
+
+/*
+ * Set key to value. If the entry already exists, we update its value to 'value'
+ * and return true. Returns false if entry doesn't yet exist.
+ */
+RT_SCOPE bool
+RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+ int shift;
+ bool updated;
+ RT_PTR_LOCAL parent;
+ RT_PTR_ALLOC stored_child;
+ RT_PTR_LOCAL child;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ RT_LOCK_EXCLUSIVE(tree);
+
+ /* Empty tree, create the root */
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ RT_NEW_ROOT(tree, key);
+
+ /* Extend the tree if necessary */
+ if (key > tree->ctl->max_val)
+ RT_EXTEND(tree, key);
+
+ stored_child = tree->ctl->root;
+ parent = RT_PTR_GET_LOCAL(tree, stored_child);
+ shift = parent->shift;
+
+ /* Descend the tree until we reach a leaf node */
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC new_child = RT_INVALID_PTR_ALLOC;;
+
+ child = RT_PTR_GET_LOCAL(tree, stored_child);
+
+ if (RT_NODE_IS_LEAF(child))
+ break;
+
+ if (!RT_NODE_SEARCH_INNER(child, key, &new_child))
+ {
+ RT_SET_EXTEND(tree, key, value_p, parent, stored_child, child);
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ parent = child;
+ stored_child = new_child;
+ shift -= RT_NODE_SPAN;
+ }
+
+ updated = RT_NODE_INSERT_LEAF(tree, parent, stored_child, child, key, value_p);
+
+ /* Update the statistics */
+ if (!updated)
+ tree->ctl->num_keys++;
+
+ RT_UNLOCK(tree);
+ return updated;
+}
+
+/*
+ * Search the given key in the radix tree. Return true if there is the key,
+ * otherwise return false. On success, we set the value to *value_p so it must
+ * not be NULL.
+ */
+RT_SCOPE bool
+RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+ RT_PTR_LOCAL node;
+ int shift;
+ bool found;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+ Assert(value_p != NULL);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ node = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+ shift = node->shift;
+
+ /* Descend the tree until a leaf node */
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+ if (RT_NODE_IS_LEAF(node))
+ break;
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ node = RT_PTR_GET_LOCAL(tree, child);
+ shift -= RT_NODE_SPAN;
+ }
+
+ found = RT_NODE_SEARCH_LEAF(node, key, value_p);
+
+ RT_UNLOCK(tree);
+ return found;
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Delete the given key from the radix tree. Return true if the key is found (and
+ * deleted), otherwise do nothing and return false.
+ */
+RT_SCOPE bool
+RT_DELETE(RT_RADIX_TREE *tree, uint64 key)
+{
+ RT_PTR_LOCAL node;
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_ALLOC stack[RT_MAX_LEVEL] = {0};
+ int shift;
+ int level;
+ bool deleted;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ RT_LOCK_EXCLUSIVE(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ /*
+ * Descend the tree to search the key while building a stack of nodes we
+ * visited.
+ */
+ allocnode = tree->ctl->root;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift = node->shift;
+ level = -1;
+ while (shift > 0)
+ {
+ RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+ /* Push the current node to the stack */
+ stack[++level] = allocnode;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ allocnode = child;
+ shift -= RT_NODE_SPAN;
+ }
+
+ /* Delete the key from the leaf node if exists */
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ deleted = RT_NODE_DELETE_LEAF(node, key);
+
+ if (!deleted)
+ {
+ /* no key is found in the leaf node */
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ /* Found the key to delete. Update the statistics */
+ tree->ctl->num_keys--;
+
+ /*
+ * Return if the leaf node still has keys and we don't need to delete the
+ * node.
+ */
+ if (node->count > 0)
+ {
+ RT_UNLOCK(tree);
+ return true;
+ }
+
+ /* Free the empty leaf node */
+ RT_FREE_NODE(tree, allocnode);
+
+ /* Delete the key in inner nodes recursively */
+ while (level >= 0)
+ {
+ allocnode = stack[level--];
+
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ deleted = RT_NODE_DELETE_INNER(node, key);
+ Assert(deleted);
+
+ /* If the node didn't become empty, we stop deleting the key */
+ if (node->count > 0)
+ break;
+
+ /* The node became empty */
+ RT_FREE_NODE(tree, allocnode);
+ }
+
+ RT_UNLOCK(tree);
+ return true;
+}
+#endif
+
+static inline void
+RT_ITER_UPDATE_KEY(RT_ITER *iter, uint8 chunk, uint8 shift)
+{
+ iter->key &= ~(((uint64) RT_CHUNK_MASK) << shift);
+ iter->key |= (((uint64) chunk) << shift);
+}
+
+/*
+ * Advance the slot in the inner node. Return the child if exists, otherwise
+ * null.
+ */
+static inline RT_PTR_LOCAL
+RT_NODE_INNER_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Advance the slot in the leaf node. On success, return true and the value
+ * is set to value_p, otherwise return false.
+ */
+static inline bool
+RT_NODE_LEAF_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter,
+ RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Update each node_iter for inner nodes in the iterator node stack.
+ */
+static void
+RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
+{
+ int level = from;
+ RT_PTR_LOCAL node = from_node;
+
+ for (;;)
+ {
+ RT_NODE_ITER *node_iter = &(iter->stack[level--]);
+
+ node_iter->node = node;
+ node_iter->current_idx = -1;
+
+ /* We don't advance the leaf node iterator here */
+ if (RT_NODE_IS_LEAF(node))
+ return;
+
+ /* Advance to the next slot in the inner node */
+ node = RT_NODE_INNER_ITERATE_NEXT(iter, node_iter);
+
+ /* We must find the first children in the node */
+ Assert(node);
+ }
+}
+
+/*
+ * Create and return the iterator for the given radix tree.
+ *
+ * The radix tree is locked in shared mode during the iteration, so
+ * RT_END_ITERATE needs to be called when finished to release the lock.
+ */
+RT_SCOPE RT_ITER *
+RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
+{
+ MemoryContext old_ctx;
+ RT_ITER *iter;
+ RT_PTR_LOCAL root;
+ int top_level;
+
+ old_ctx = MemoryContextSwitchTo(tree->context);
+
+ iter = (RT_ITER *) palloc0(sizeof(RT_ITER));
+ iter->tree = tree;
+
+ RT_LOCK_SHARED(tree);
+
+ /* empty tree */
+ if (!iter->tree->ctl->root)
+ return iter;
+
+ root = RT_PTR_GET_LOCAL(tree, iter->tree->ctl->root);
+ top_level = root->shift / RT_NODE_SPAN;
+ iter->stack_len = top_level;
+
+ /*
+ * Descend to the left most leaf node from the root. The key is being
+ * constructed while descending to the leaf.
+ */
+ RT_UPDATE_ITER_STACK(iter, root, top_level);
+
+ MemoryContextSwitchTo(old_ctx);
+
+ return iter;
+}
+
+/*
+ * Return true with setting key_p and value_p if there is next key. Otherwise
+ * return false.
+ */
+RT_SCOPE bool
+RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p)
+{
+ /* Empty tree */
+ if (!iter->tree->ctl->root)
+ return false;
+
+ for (;;)
+ {
+ RT_PTR_LOCAL child = NULL;
+ RT_VALUE_TYPE value;
+ int level;
+ bool found;
+
+ /* Advance the leaf node iterator to get next key-value pair */
+ found = RT_NODE_LEAF_ITERATE_NEXT(iter, &(iter->stack[0]), &value);
+
+ if (found)
+ {
+ *key_p = iter->key;
+ *value_p = value;
+ return true;
+ }
+
+ /*
+ * We've visited all values in the leaf node, so advance inner node
+ * iterators from the level=1 until we find the next child node.
+ */
+ for (level = 1; level <= iter->stack_len; level++)
+ {
+ child = RT_NODE_INNER_ITERATE_NEXT(iter, &(iter->stack[level]));
+
+ if (child)
+ break;
+ }
+
+ /* the iteration finished */
+ if (!child)
+ return false;
+
+ /*
+ * Set the node to the node iterator and update the iterator stack
+ * from this node.
+ */
+ RT_UPDATE_ITER_STACK(iter, child, level - 1);
+
+ /* Node iterators are updated, so try again from the leaf */
+ }
+
+ return false;
+}
+
+/*
+ * Terminate the iteration and release the lock.
+ *
+ * This function needs to be called after finishing or when exiting an
+ * iteration.
+ */
+RT_SCOPE void
+RT_END_ITERATE(RT_ITER *iter)
+{
+#ifdef RT_SHMEM
+ Assert(LWLockHeldByMe(&iter->tree->ctl->lock));
+#endif
+
+ RT_UNLOCK(iter->tree);
+ pfree(iter);
+}
+
+/*
+ * Return the statistics of the amount of memory used by the radix tree.
+ */
+RT_SCOPE uint64
+RT_MEMORY_USAGE(RT_RADIX_TREE *tree)
+{
+ Size total = 0;
+
+ RT_LOCK_SHARED(tree);
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ total = dsa_get_total_size(tree->dsa);
+#else
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ total += MemoryContextMemAllocated(tree->inner_slabs[i], true);
+ total += MemoryContextMemAllocated(tree->leaf_slabs[i], true);
+ }
+#endif
+
+ RT_UNLOCK(tree);
+ return total;
+}
+
+/*
+ * Verify the radix tree node.
+ */
+static void
+RT_VERIFY_NODE(RT_PTR_LOCAL node)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(node->count >= 0);
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE_BASE_3 *n3 = (RT_NODE_BASE_3 *) node;
+
+ for (int i = 1; i < n3->n.count; i++)
+ Assert(n3->chunks[i - 1] < n3->chunks[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE_BASE_32 *n32 = (RT_NODE_BASE_32 *) node;
+
+ for (int i = 1; i < n32->n.count; i++)
+ Assert(n32->chunks[i - 1] < n32->chunks[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+ int cnt = 0;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ uint8 slot = n125->slot_idxs[i];
+ int idx = RT_BM_IDX(slot);
+ int bitnum = RT_BM_BIT(slot);
+
+ if (!RT_NODE_125_IS_CHUNK_USED(n125, i))
+ continue;
+
+ /* Check if the corresponding slot is used */
+ Assert(slot < node->fanout);
+ Assert((n125->isset[idx] & ((bitmapword) 1 << bitnum)) != 0);
+
+ cnt++;
+ }
+
+ Assert(n125->n.count == cnt);
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+ int cnt = 0;
+
+ for (int i = 0; i < RT_BM_IDX(RT_NODE_MAX_SLOTS); i++)
+ cnt += bmw_popcount(n256->isset[i]);
+
+ /* Check if the number of used chunk matches */
+ Assert(n256->base.n.count == cnt);
+
+ break;
+ }
+ }
+ }
+#endif
+}
+
+/***************** DEBUG FUNCTIONS *****************/
+#ifdef RT_DEBUG
+
+#define RT_UINT64_FORMAT_HEX "%" INT64_MODIFIER "X"
+
+RT_SCOPE void
+RT_STATS(RT_RADIX_TREE *tree)
+{
+ RT_LOCK_SHARED(tree);
+
+ fprintf(stderr, "max_val = " UINT64_FORMAT "\n", tree->ctl->max_val);
+ fprintf(stderr, "num_keys = " UINT64_FORMAT "\n", tree->ctl->num_keys);
+
+#ifdef RT_SHMEM
+ fprintf(stderr, "handle = " UINT64_FORMAT "\n", tree->ctl->handle);
+#endif
+
+ if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+
+ fprintf(stderr, "height = %d, n3 = %u, n15 = %u, n32 = %u, n125 = %u, n256 = %u\n",
+ root->shift / RT_NODE_SPAN,
+ tree->ctl->cnt[RT_CLASS_3],
+ tree->ctl->cnt[RT_CLASS_32_MIN],
+ tree->ctl->cnt[RT_CLASS_32_MAX],
+ tree->ctl->cnt[RT_CLASS_125],
+ tree->ctl->cnt[RT_CLASS_256]);
+ }
+
+ RT_UNLOCK(tree);
+}
+
+static void
+RT_DUMP_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, int level,
+ bool recurse, StringInfo buf)
+{
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+ StringInfoData spaces;
+
+ initStringInfo(&spaces);
+ appendStringInfoSpaces(&spaces, (level * 4) + 1);
+
+ appendStringInfo(buf, "%s%s[%s] kind %d, fanout %d, count %u, shift %u:\n",
+ spaces.data,
+ level == 0 ? "" : "-> ",
+ RT_NODE_IS_LEAF(node) ? "LEAF" : "INNR",
+ (node->kind == RT_NODE_KIND_3) ? 3 :
+ (node->kind == RT_NODE_KIND_32) ? 32 :
+ (node->kind == RT_NODE_KIND_125) ? 125 : 256,
+ node->fanout == 0 ? 256 : node->fanout,
+ node->count, node->shift);
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ for (int i = 0; i < node->count; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_3 *n3 = (RT_NODE_LEAF_3 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+ spaces.data, i, n3->base.chunks[i]);
+ }
+ else
+ {
+ RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X",
+ spaces.data, i, n3->base.chunks[i]);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, n3->children[i], level + 1,
+ recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ for (int i = 0; i < node->count; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_32 *n32 = (RT_NODE_LEAF_32 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+ spaces.data, i, n32->base.chunks[i]);
+ }
+ else
+ {
+ RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X",
+ spaces.data, i, n32->base.chunks[i]);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, n32->children[i], level + 1,
+ recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_BASE_125 *b125 = (RT_NODE_BASE_125 *) node;
+ char *sep = "";
+
+ appendStringInfo(buf, "%sslot_idxs: ", spaces.data);
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+ continue;
+
+ appendStringInfo(buf, "%s[%d]=%d ",
+ sep, i, b125->slot_idxs[i]);
+ sep = ",";
+ }
+
+ appendStringInfo(buf, "\n%sisset-bitmap: ", spaces.data);
+ for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+ appendStringInfo(buf, "%X ", ((uint8 *) b125->isset)[i]);
+ appendStringInfo(buf, "\n");
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+ continue;
+
+ if (RT_NODE_IS_LEAF(node))
+ appendStringInfo(buf, "%schunk 0x%X\n",
+ spaces.data, i);
+ else
+ {
+ RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) b125;
+
+ appendStringInfo(buf, "%schunk 0x%X",
+ spaces.data, i);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i),
+ level + 1, recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+ appendStringInfo(buf, "%sisset-bitmap: ", spaces.data);
+ for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+ appendStringInfo(buf, "%X ", ((uint8 *) n256->isset)[i]);
+ appendStringInfo(buf, "\n");
+ }
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ appendStringInfo(buf, "%schunk 0x%X\n",
+ spaces.data, i);
+ }
+ else
+ {
+ RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ appendStringInfo(buf, "%schunk 0x%X",
+ spaces.data, i);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i),
+ level + 1, recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ }
+}
+
+RT_SCOPE void
+RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key)
+{
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL node;
+ StringInfoData buf;
+ int shift;
+ int level = 0;
+
+ RT_STATS(tree);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "empty tree\n");
+ return;
+ }
+
+ if (key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "key " UINT64_FORMAT "(0x" RT_UINT64_FORMAT_HEX ") is larger than max val\n",
+ key, key);
+ return;
+ }
+
+ initStringInfo(&buf);
+ allocnode = tree->ctl->root;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift = node->shift;
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC child;
+
+ RT_DUMP_NODE(tree, allocnode, level, false, &buf);
+
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_VALUE_TYPE dummy;
+
+ /* We reached at a leaf node, find the corresponding slot */
+ RT_NODE_SEARCH_LEAF(node, key, &dummy);
+
+ break;
+ }
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ break;
+
+ allocnode = child;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift -= RT_NODE_SPAN;
+ level++;
+ }
+ RT_UNLOCK(tree);
+
+ fprintf(stderr, "%s", buf.data);
+}
+
+RT_SCOPE void
+RT_DUMP(RT_RADIX_TREE *tree)
+{
+ StringInfoData buf;
+
+ RT_STATS(tree);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "empty tree\n");
+ return;
+ }
+
+ initStringInfo(&buf);
+
+ RT_DUMP_NODE(tree, tree->ctl->root, 0, true, &buf);
+ RT_UNLOCK(tree);
+
+ fprintf(stderr, "%s",buf.data);
+}
+#endif
+
+#endif /* RT_DEFINE */
+
+
+/* undefine external parameters, so next radix tree can be defined */
+#undef RT_PREFIX
+#undef RT_SCOPE
+#undef RT_DECLARE
+#undef RT_DEFINE
+#undef RT_VALUE_TYPE
+
+/* locally declared macros */
+#undef RT_MAKE_PREFIX
+#undef RT_MAKE_NAME
+#undef RT_MAKE_NAME_
+#undef RT_NODE_SPAN
+#undef RT_NODE_MAX_SLOTS
+#undef RT_CHUNK_MASK
+#undef RT_MAX_SHIFT
+#undef RT_MAX_LEVEL
+#undef RT_GET_KEY_CHUNK
+#undef RT_BM_IDX
+#undef RT_BM_BIT
+#undef RT_LOCK_EXCLUSIVE
+#undef RT_LOCK_SHARED
+#undef RT_UNLOCK
+#undef RT_NODE_IS_LEAF
+#undef RT_NODE_MUST_GROW
+#undef RT_NODE_KIND_COUNT
+#undef RT_SIZE_CLASS_COUNT
+#undef RT_SLOT_IDX_LIMIT
+#undef RT_INVALID_SLOT_IDX
+#undef RT_SLAB_BLOCK_SIZE
+#undef RT_RADIX_TREE_MAGIC
+#undef RT_UINT64_FORMAT_HEX
+
+/* type declarations */
+#undef RT_RADIX_TREE
+#undef RT_RADIX_TREE_CONTROL
+#undef RT_PTR_LOCAL
+#undef RT_PTR_ALLOC
+#undef RT_INVALID_PTR_ALLOC
+#undef RT_HANDLE
+#undef RT_ITER
+#undef RT_NODE
+#undef RT_NODE_ITER
+#undef RT_NODE_KIND_3
+#undef RT_NODE_KIND_32
+#undef RT_NODE_KIND_125
+#undef RT_NODE_KIND_256
+#undef RT_NODE_BASE_3
+#undef RT_NODE_BASE_32
+#undef RT_NODE_BASE_125
+#undef RT_NODE_BASE_256
+#undef RT_NODE_INNER_3
+#undef RT_NODE_INNER_32
+#undef RT_NODE_INNER_125
+#undef RT_NODE_INNER_256
+#undef RT_NODE_LEAF_3
+#undef RT_NODE_LEAF_32
+#undef RT_NODE_LEAF_125
+#undef RT_NODE_LEAF_256
+#undef RT_SIZE_CLASS
+#undef RT_SIZE_CLASS_ELEM
+#undef RT_SIZE_CLASS_INFO
+#undef RT_CLASS_3
+#undef RT_CLASS_32_MIN
+#undef RT_CLASS_32_MAX
+#undef RT_CLASS_125
+#undef RT_CLASS_256
+
+/* function declarations */
+#undef RT_CREATE
+#undef RT_FREE
+#undef RT_ATTACH
+#undef RT_DETACH
+#undef RT_GET_HANDLE
+#undef RT_SEARCH
+#undef RT_SET
+#undef RT_BEGIN_ITERATE
+#undef RT_ITERATE_NEXT
+#undef RT_END_ITERATE
+#undef RT_USE_DELETE
+#undef RT_DELETE
+#undef RT_MEMORY_USAGE
+#undef RT_DUMP
+#undef RT_DUMP_NODE
+#undef RT_DUMP_SEARCH
+#undef RT_STATS
+
+/* internal helper functions */
+#undef RT_NEW_ROOT
+#undef RT_ALLOC_NODE
+#undef RT_INIT_NODE
+#undef RT_FREE_NODE
+#undef RT_FREE_RECURSE
+#undef RT_EXTEND
+#undef RT_SET_EXTEND
+#undef RT_SWITCH_NODE_KIND
+#undef RT_COPY_NODE
+#undef RT_REPLACE_NODE
+#undef RT_PTR_GET_LOCAL
+#undef RT_PTR_ALLOC_IS_VALID
+#undef RT_NODE_3_SEARCH_EQ
+#undef RT_NODE_32_SEARCH_EQ
+#undef RT_NODE_3_GET_INSERTPOS
+#undef RT_NODE_32_GET_INSERTPOS
+#undef RT_CHUNK_CHILDREN_ARRAY_SHIFT
+#undef RT_CHUNK_VALUES_ARRAY_SHIFT
+#undef RT_CHUNK_CHILDREN_ARRAY_DELETE
+#undef RT_CHUNK_VALUES_ARRAY_DELETE
+#undef RT_CHUNK_CHILDREN_ARRAY_COPY
+#undef RT_CHUNK_VALUES_ARRAY_COPY
+#undef RT_NODE_125_IS_CHUNK_USED
+#undef RT_NODE_INNER_125_GET_CHILD
+#undef RT_NODE_LEAF_125_GET_VALUE
+#undef RT_NODE_INNER_256_IS_CHUNK_USED
+#undef RT_NODE_LEAF_256_IS_CHUNK_USED
+#undef RT_NODE_INNER_256_GET_CHILD
+#undef RT_NODE_LEAF_256_GET_VALUE
+#undef RT_NODE_INNER_256_SET
+#undef RT_NODE_LEAF_256_SET
+#undef RT_NODE_INNER_256_DELETE
+#undef RT_NODE_LEAF_256_DELETE
+#undef RT_KEY_GET_SHIFT
+#undef RT_SHIFT_GET_MAX_VAL
+#undef RT_NODE_SEARCH_INNER
+#undef RT_NODE_SEARCH_LEAF
+#undef RT_NODE_UPDATE_INNER
+#undef RT_NODE_DELETE_INNER
+#undef RT_NODE_DELETE_LEAF
+#undef RT_NODE_INSERT_INNER
+#undef RT_NODE_INSERT_LEAF
+#undef RT_NODE_INNER_ITERATE_NEXT
+#undef RT_NODE_LEAF_ITERATE_NEXT
+#undef RT_UPDATE_ITER_STACK
+#undef RT_ITER_UPDATE_KEY
+#undef RT_VERIFY_NODE
+
+#undef RT_DEBUG
diff --git a/src/include/lib/radixtree_delete_impl.h b/src/include/lib/radixtree_delete_impl.h
new file mode 100644
index 0000000000..5f6dda1f12
--- /dev/null
+++ b/src/include/lib/radixtree_delete_impl.h
@@ -0,0 +1,122 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_delete_impl.h
+ * Common implementation for deletion in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ * TODO: Shrink nodes when deletion would allow them to fit in a smaller
+ * size class.
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_delete_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_DELETE(n3->base.chunks, n3->values,
+ n3->base.n.count, idx);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_DELETE(n3->base.chunks, n3->children,
+ n3->base.n.count, idx);
+#endif
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_DELETE(n32->base.chunks, n32->values,
+ n32->base.n.count, idx);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_DELETE(n32->base.chunks, n32->children,
+ n32->base.n.count, idx);
+#endif
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+ int idx;
+ int bitnum;
+
+ if (slotpos == RT_INVALID_SLOT_IDX)
+ return false;
+
+ idx = RT_BM_IDX(slotpos);
+ bitnum = RT_BM_BIT(slotpos);
+ n125->base.isset[idx] &= ~((bitmapword) 1 << bitnum);
+ n125->base.slot_idxs[chunk] = RT_INVALID_SLOT_IDX;
+
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_DELETE(n256, chunk);
+#else
+ RT_NODE_INNER_256_DELETE(n256, chunk);
+#endif
+ break;
+ }
+ }
+
+ /* update statistics */
+ node->count--;
+
+ return true;
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
new file mode 100644
index 0000000000..c18e26b537
--- /dev/null
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -0,0 +1,332 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_insert_impl.h
+ * Common implementation for insertion in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_insert_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+ bool chunk_exists = false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ const bool is_leaf = true;
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+ const bool is_leaf = false;
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx;
+
+ idx = RT_NODE_3_SEARCH_EQ(&n3->base, chunk);
+ if (idx != -1)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+#ifdef RT_NODE_LEVEL_LEAF
+ n3->values[idx] = *value_p;
+#else
+ n3->children[idx] = child;
+#endif
+ break;
+ }
+
+ if (unlikely(RT_NODE_MUST_GROW(n3)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE32_TYPE *new32;
+ const uint8 new_kind = RT_NODE_KIND_32;
+ const RT_SIZE_CLASS new_class = RT_CLASS_32_MIN;
+
+ /* grow node from 3 to 32 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_COPY(n3->base.chunks, n3->values,
+ new32->base.chunks, new32->values);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_COPY(n3->base.chunks, n3->children,
+ new32->base.chunks, new32->children);
+#endif
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
+ int count = n3->base.n.count;
+
+ /* shift chunks and children */
+ if (insertpos < count)
+ {
+ Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_SHIFT(n3->base.chunks, n3->values,
+ count, insertpos);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_SHIFT(n3->base.chunks, n3->children,
+ count, insertpos);
+#endif
+ }
+
+ n3->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+ n3->values[insertpos] = *value_p;
+#else
+ n3->children[insertpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_32:
+ {
+ const RT_SIZE_CLASS_ELEM class32_max = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX];
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx;
+
+ idx = RT_NODE_32_SEARCH_EQ(&n32->base, chunk);
+ if (idx != -1)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+#ifdef RT_NODE_LEVEL_LEAF
+ n32->values[idx] = *value_p;
+#else
+ n32->children[idx] = child;
+#endif
+ break;
+ }
+
+ if (unlikely(RT_NODE_MUST_GROW(n32)) &&
+ n32->base.n.fanout < class32_max.fanout)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ const RT_SIZE_CLASS_ELEM class32_min = RT_SIZE_CLASS_INFO[RT_CLASS_32_MIN];
+ const RT_SIZE_CLASS new_class = RT_CLASS_32_MAX;
+
+ Assert(n32->base.n.fanout == class32_min.fanout);
+
+ /* grow to the next size class of this kind */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ n32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ memcpy(newnode, node, class32_min.leaf_size);
+#else
+ memcpy(newnode, node, class32_min.inner_size);
+#endif
+ newnode->fanout = class32_max.fanout;
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+
+ if (unlikely(RT_NODE_MUST_GROW(n32)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE125_TYPE *new125;
+ const uint8 new_kind = RT_NODE_KIND_125;
+ const RT_SIZE_CLASS new_class = RT_CLASS_125;
+
+ Assert(n32->base.n.fanout == class32_max.fanout);
+
+ /* grow node from 32 to 125 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new125 = (RT_NODE125_TYPE *) newnode;
+
+ for (int i = 0; i < class32_max.fanout; i++)
+ {
+ new125->base.slot_idxs[n32->base.chunks[i]] = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ new125->values[i] = n32->values[i];
+#else
+ new125->children[i] = n32->children[i];
+#endif
+ }
+
+ /*
+ * Since we just copied a dense array, we can set the bits
+ * using a single store, provided the length of that array
+ * is at most the number of bits in a bitmapword.
+ */
+ Assert(class32_max.fanout <= sizeof(bitmapword) * BITS_PER_BYTE);
+ new125->base.isset[0] = (bitmapword) (((uint64) 1 << class32_max.fanout) - 1);
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
+ int count = n32->base.n.count;
+
+ if (insertpos < count)
+ {
+ Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_SHIFT(n32->base.chunks, n32->values,
+ count, insertpos);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_SHIFT(n32->base.chunks, n32->children,
+ count, insertpos);
+#endif
+ }
+
+ n32->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+ n32->values[insertpos] = *value_p;
+#else
+ n32->children[insertpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+ int cnt = 0;
+
+ if (slotpos != RT_INVALID_SLOT_IDX)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+#ifdef RT_NODE_LEVEL_LEAF
+ n125->values[slotpos] = *value_p;
+#else
+ n125->children[slotpos] = child;
+#endif
+ break;
+ }
+
+ if (unlikely(RT_NODE_MUST_GROW(n125)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE256_TYPE *new256;
+ const uint8 new_kind = RT_NODE_KIND_256;
+ const RT_SIZE_CLASS new_class = RT_CLASS_256;
+
+ /* grow node from 125 to 256 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new256 = (RT_NODE256_TYPE *) newnode;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS && cnt < n125->base.n.count; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+ continue;
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_SET(new256, i, RT_NODE_LEAF_125_GET_VALUE(n125, i));
+#else
+ RT_NODE_INNER_256_SET(new256, i, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+ cnt++;
+ }
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int idx;
+ bitmapword inverse;
+
+ /* get the first word with at least one bit not set */
+ for (idx = 0; idx < RT_BM_IDX(RT_SLOT_IDX_LIMIT); idx++)
+ {
+ if (n125->base.isset[idx] < ~((bitmapword) 0))
+ break;
+ }
+
+ /* To get the first unset bit in X, get the first set bit in ~X */
+ inverse = ~(n125->base.isset[idx]);
+ slotpos = idx * BITS_PER_BITMAPWORD;
+ slotpos += bmw_rightmost_one_pos(inverse);
+ Assert(slotpos < node->fanout);
+
+ /* mark the slot used */
+ n125->base.isset[idx] |= bmw_rightmost_one(inverse);
+ n125->base.slot_idxs[chunk] = slotpos;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ n125->values[slotpos] = *value_p;
+#else
+ n125->children[slotpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ chunk_exists = RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk);
+#else
+ chunk_exists = RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk);
+#endif
+ Assert(chunk_exists || node->count < RT_NODE_MAX_SLOTS);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_SET(n256, chunk, *value_p);
+#else
+ RT_NODE_INNER_256_SET(n256, chunk, child);
+#endif
+ break;
+ }
+ }
+
+ /* Update statistics */
+ if (!chunk_exists)
+ node->count++;
+
+ /*
+ * Done. Finally, verify the chunk and value is inserted or replaced
+ * properly in the node.
+ */
+ RT_VERIFY_NODE(node);
+
+ return chunk_exists;
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_iter_impl.h b/src/include/lib/radixtree_iter_impl.h
new file mode 100644
index 0000000000..98c78eb237
--- /dev/null
+++ b/src/include/lib/radixtree_iter_impl.h
@@ -0,0 +1,153 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_iter_impl.h
+ * Common implementation for iteration in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_iter_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ bool found = false;
+ uint8 key_chunk;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_VALUE_TYPE value;
+
+ Assert(RT_NODE_IS_LEAF(node_iter->node));
+#else
+ RT_PTR_LOCAL child = NULL;
+
+ Assert(!RT_NODE_IS_LEAF(node_iter->node));
+#endif
+
+#ifdef RT_SHMEM
+ Assert(iter->tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ switch (node_iter->node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node_iter->node;
+
+ node_iter->current_idx++;
+ if (node_iter->current_idx >= n3->base.n.count)
+ break;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = n3->values[node_iter->current_idx];
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, n3->children[node_iter->current_idx]);
+#endif
+ key_chunk = n3->base.chunks[node_iter->current_idx];
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node_iter->node;
+
+ node_iter->current_idx++;
+ if (node_iter->current_idx >= n32->base.n.count)
+ break;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ value = n32->values[node_iter->current_idx];
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, n32->children[node_iter->current_idx]);
+#endif
+ key_chunk = n32->base.chunks[node_iter->current_idx];
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node_iter->node;
+ int i;
+
+ for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (RT_NODE_125_IS_CHUNK_USED((RT_NODE_BASE_125 *) n125, i))
+ break;
+ }
+
+ if (i >= RT_NODE_MAX_SLOTS)
+ break;
+
+ node_iter->current_idx = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = RT_NODE_LEAF_125_GET_VALUE(n125, i);
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+ key_chunk = i;
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node_iter->node;
+ int i;
+
+ for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ {
+#ifdef RT_NODE_LEVEL_LEAF
+ if (RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+#else
+ if (RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+#endif
+ break;
+ }
+
+ if (i >= RT_NODE_MAX_SLOTS)
+ break;
+
+ node_iter->current_idx = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = RT_NODE_LEAF_256_GET_VALUE(n256, i);
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+#endif
+ key_chunk = i;
+ found = true;
+ break;
+ }
+ }
+
+ if (found)
+ {
+ RT_ITER_UPDATE_KEY(iter, key_chunk, node_iter->node->shift);
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = value;
+#endif
+ }
+
+#ifdef RT_NODE_LEVEL_LEAF
+ return found;
+#else
+ return child;
+#endif
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_search_impl.h b/src/include/lib/radixtree_search_impl.h
new file mode 100644
index 0000000000..a8925c75d0
--- /dev/null
+++ b/src/include/lib/radixtree_search_impl.h
@@ -0,0 +1,138 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_search_impl.h
+ * Common implementation for search in leaf and inner nodes, plus
+ * update for inner nodes only.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_search_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(value_p != NULL);
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+#ifndef RT_ACTION_UPDATE
+ Assert(child_p != NULL);
+#endif
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+#ifdef RT_ACTION_UPDATE
+ Assert(idx >= 0);
+ n3->children[idx] = new_child;
+#else
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n3->values[idx];
+#else
+ *child_p = n3->children[idx];
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+#ifdef RT_ACTION_UPDATE
+ Assert(idx >= 0);
+ n32->children[idx] = new_child;
+#else
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n32->values[idx];
+#else
+ *child_p = n32->children[idx];
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+
+#ifdef RT_ACTION_UPDATE
+ Assert(slotpos != RT_INVALID_SLOT_IDX);
+ n125->children[slotpos] = new_child;
+#else
+ if (slotpos == RT_INVALID_SLOT_IDX)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_125_GET_VALUE(n125, chunk);
+#else
+ *child_p = RT_NODE_INNER_125_GET_CHILD(n125, chunk);
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_ACTION_UPDATE
+ RT_NODE_INNER_256_SET(n256, chunk, new_child);
+#else
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_256_GET_VALUE(n256, chunk);
+#else
+ *child_p = RT_NODE_INNER_256_GET_CHILD(n256, chunk);
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ }
+
+#ifdef RT_ACTION_UPDATE
+ return;
+#else
+ return true;
+#endif /* RT_ACTION_UPDATE */
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/utils/dsa.h b/src/include/utils/dsa.h
index 3ce4ee300a..2af215484f 100644
--- a/src/include/utils/dsa.h
+++ b/src/include/utils/dsa.h
@@ -121,6 +121,7 @@ extern dsa_handle dsa_get_handle(dsa_area *area);
extern dsa_pointer dsa_allocate_extended(dsa_area *area, size_t size, int flags);
extern void dsa_free(dsa_area *area, dsa_pointer dp);
extern void *dsa_get_address(dsa_area *area, dsa_pointer dp);
+extern size_t dsa_get_total_size(dsa_area *area);
extern void dsa_trim(dsa_area *area);
extern void dsa_dump(dsa_area *area);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index c629cbe383..9659eb85d7 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -28,6 +28,7 @@ SUBDIRS = \
test_pg_db_role_setting \
test_pg_dump \
test_predtest \
+ test_radixtree \
test_rbtree \
test_regex \
test_rls_hooks \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 1baa6b558d..232cbdac80 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -24,6 +24,7 @@ subdir('test_parser')
subdir('test_pg_db_role_setting')
subdir('test_pg_dump')
subdir('test_predtest')
+subdir('test_radixtree')
subdir('test_rbtree')
subdir('test_regex')
subdir('test_rls_hooks')
diff --git a/src/test/modules/test_radixtree/.gitignore b/src/test/modules/test_radixtree/.gitignore
new file mode 100644
index 0000000000..5dcb3ff972
--- /dev/null
+++ b/src/test/modules/test_radixtree/.gitignore
@@ -0,0 +1,4 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
diff --git a/src/test/modules/test_radixtree/Makefile b/src/test/modules/test_radixtree/Makefile
new file mode 100644
index 0000000000..da06b93da3
--- /dev/null
+++ b/src/test/modules/test_radixtree/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_radixtree/Makefile
+
+MODULE_big = test_radixtree
+OBJS = \
+ $(WIN32RES) \
+ test_radixtree.o
+PGFILEDESC = "test_radixtree - test code for src/backend/lib/radixtree.c"
+
+EXTENSION = test_radixtree
+DATA = test_radixtree--1.0.sql
+
+REGRESS = test_radixtree
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_radixtree
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_radixtree/README b/src/test/modules/test_radixtree/README
new file mode 100644
index 0000000000..a8b271869a
--- /dev/null
+++ b/src/test/modules/test_radixtree/README
@@ -0,0 +1,7 @@
+test_integerset contains unit tests for testing the integer set implementation
+in src/backend/lib/integerset.c.
+
+The tests verify the correctness of the implementation, but they can also be
+used as a micro-benchmark. If you set the 'intset_test_stats' flag in
+test_integerset.c, the tests will print extra information about execution time
+and memory usage.
diff --git a/src/test/modules/test_radixtree/expected/test_radixtree.out b/src/test/modules/test_radixtree/expected/test_radixtree.out
new file mode 100644
index 0000000000..ce645cb8b5
--- /dev/null
+++ b/src/test/modules/test_radixtree/expected/test_radixtree.out
@@ -0,0 +1,36 @@
+CREATE EXTENSION test_radixtree;
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
+NOTICE: testing basic operations with leaf node 4
+NOTICE: testing basic operations with inner node 4
+NOTICE: testing basic operations with leaf node 32
+NOTICE: testing basic operations with inner node 32
+NOTICE: testing basic operations with leaf node 125
+NOTICE: testing basic operations with inner node 125
+NOTICE: testing basic operations with leaf node 256
+NOTICE: testing basic operations with inner node 256
+NOTICE: testing radix tree node types with shift "0"
+NOTICE: testing radix tree node types with shift "8"
+NOTICE: testing radix tree node types with shift "16"
+NOTICE: testing radix tree node types with shift "24"
+NOTICE: testing radix tree node types with shift "32"
+NOTICE: testing radix tree node types with shift "40"
+NOTICE: testing radix tree node types with shift "48"
+NOTICE: testing radix tree node types with shift "56"
+NOTICE: testing radix tree with pattern "all ones"
+NOTICE: testing radix tree with pattern "alternating bits"
+NOTICE: testing radix tree with pattern "clusters of ten"
+NOTICE: testing radix tree with pattern "clusters of hundred"
+NOTICE: testing radix tree with pattern "one-every-64k"
+NOTICE: testing radix tree with pattern "sparse"
+NOTICE: testing radix tree with pattern "single values, distance > 2^32"
+NOTICE: testing radix tree with pattern "clusters, distance > 2^32"
+NOTICE: testing radix tree with pattern "clusters, distance > 2^60"
+ test_radixtree
+----------------
+
+(1 row)
+
diff --git a/src/test/modules/test_radixtree/meson.build b/src/test/modules/test_radixtree/meson.build
new file mode 100644
index 0000000000..6add06bbdb
--- /dev/null
+++ b/src/test/modules/test_radixtree/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_radixtree_sources = files(
+ 'test_radixtree.c',
+)
+
+if host_system == 'windows'
+ test_radixtree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_radixtree',
+ '--FILEDESC', 'test_radixtree - test code for src/include//lib/radixtree.h',])
+endif
+
+test_radixtree = shared_module('test_radixtree',
+ test_radixtree_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += test_radixtree
+
+install_data(
+ 'test_radixtree.control',
+ 'test_radixtree--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'test_radixtree',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'test_radixtree',
+ ],
+ },
+}
diff --git a/src/test/modules/test_radixtree/sql/test_radixtree.sql b/src/test/modules/test_radixtree/sql/test_radixtree.sql
new file mode 100644
index 0000000000..41ece5e9f5
--- /dev/null
+++ b/src/test/modules/test_radixtree/sql/test_radixtree.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_radixtree;
+
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
diff --git a/src/test/modules/test_radixtree/test_radixtree--1.0.sql b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
new file mode 100644
index 0000000000..074a5a7ea7
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_radixtree/test_radixtree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_radixtree" to load this file. \quit
+
+CREATE FUNCTION test_radixtree()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
new file mode 100644
index 0000000000..f944945db9
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -0,0 +1,674 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_radixtree.c
+ * Test radixtree set data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_radixtree/test_radixtree.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "storage/lwlock.h"
+#include "utils/memutils.h"
+#include "utils/timestamp.h"
+
+#define UINT64_HEX_FORMAT "%" INT64_MODIFIER "X"
+
+/*
+ * The tests pass with uint32, but build with warnings because the string
+ * format expects uint64.
+ */
+typedef uint64 TestValueType;
+
+/*
+ * If you enable this, the "pattern" tests will print information about
+ * how long populating, probing, and iterating the test set takes, and
+ * how much memory the test set consumed. That can be used as
+ * micro-benchmark of various operations and input patterns (you might
+ * want to increase the number of values used in each of the test, if
+ * you do that, to reduce noise).
+ *
+ * The information is printed to the server's stderr, mostly because
+ * that's where MemoryContextStats() output goes.
+ */
+static const bool rt_test_stats = false;
+
+static int rt_node_kind_fanouts[] = {
+ 0,
+ 4, /* RT_NODE_KIND_4 */
+ 32, /* RT_NODE_KIND_32 */
+ 125, /* RT_NODE_KIND_125 */
+ 256 /* RT_NODE_KIND_256 */
+};
+/*
+ * A struct to define a pattern of integers, for use with the test_pattern()
+ * function.
+ */
+typedef struct
+{
+ char *test_name; /* short name of the test, for humans */
+ char *pattern_str; /* a bit pattern */
+ uint64 spacing; /* pattern repeats at this interval */
+ uint64 num_values; /* number of integers to set in total */
+} test_spec;
+
+/* Test patterns borrowed from test_integerset.c */
+static const test_spec test_specs[] = {
+ {
+ "all ones", "1111111111",
+ 10, 1000000
+ },
+ {
+ "alternating bits", "0101010101",
+ 10, 1000000
+ },
+ {
+ "clusters of ten", "1111111111",
+ 10000, 1000000
+ },
+ {
+ "clusters of hundred",
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111",
+ 10000, 1000000
+ },
+ {
+ "one-every-64k", "1",
+ 65536, 1000000
+ },
+ {
+ "sparse", "100000000000000000000000000000001",
+ 10000000, 1000000
+ },
+ {
+ "single values, distance > 2^32", "1",
+ UINT64CONST(10000000000), 100000
+ },
+ {
+ "clusters, distance > 2^32", "10101010",
+ UINT64CONST(10000000000), 1000000
+ },
+ {
+ "clusters, distance > 2^60", "10101010",
+ UINT64CONST(2000000000000000000),
+ 23 /* can't be much higher than this, or we
+ * overflow uint64 */
+ }
+};
+
+/* define the radix tree implementation to test */
+#define RT_PREFIX rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE TestValueType
+// WIP: compiles with warnings because rt_attach is defined but not used
+// #define RT_SHMEM
+#include "lib/radixtree.h"
+
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+ return tree->ctl->num_keys;
+}
+
+PG_MODULE_MAGIC;
+
+PG_FUNCTION_INFO_V1(test_radixtree);
+
+static void
+test_empty(void)
+{
+ rt_radix_tree *radixtree;
+ rt_iter *iter;
+ TestValueType dummy;
+ uint64 key;
+ TestValueType val;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+ dsa = dsa_create(tranche_id);
+
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ if (rt_search(radixtree, 0, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_search(radixtree, 1, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_search(radixtree, PG_UINT64_MAX, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_delete(radixtree, 0))
+ elog(ERROR, "rt_delete on empty tree returned true");
+
+ if (rt_num_entries(radixtree) != 0)
+ elog(ERROR, "rt_num_entries on empty tree return non-zero");
+
+ iter = rt_begin_iterate(radixtree);
+
+ if (rt_iterate_next(iter, &key, &val))
+ elog(ERROR, "rt_itereate_next on empty tree returned true");
+
+ rt_end_iterate(iter);
+
+ rt_free(radixtree);
+
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+static void
+test_basic(int children, bool test_inner)
+{
+ rt_radix_tree *radixtree;
+ uint64 *keys;
+ int shift = test_inner ? 8 : 0;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing basic operations with %s node %d",
+ test_inner ? "inner" : "leaf", children);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ /* prepare keys in order like 1, 32, 2, 31, 2, ... */
+ keys = palloc(sizeof(uint64) * children);
+ for (int i = 0; i < children; i++)
+ {
+ if (i % 2 == 0)
+ keys[i] = (uint64) ((i / 2) + 1) << shift;
+ else
+ keys[i] = (uint64) (children - (i / 2)) << shift;
+ }
+
+ /* insert keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (rt_set(radixtree, keys[i], (TestValueType*) &keys[i]))
+ elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+ }
+
+ /* look up keys */
+ for (int i = 0; i < children; i++)
+ {
+ TestValueType value;
+
+ if (!rt_search(radixtree, keys[i], &value))
+ elog(ERROR, "could not find key 0x" UINT64_HEX_FORMAT, keys[i]);
+ if (value != (TestValueType) keys[i])
+ elog(ERROR, "rt_search returned 0x" UINT64_HEX_FORMAT ", expected " UINT64_HEX_FORMAT,
+ value, (TestValueType) keys[i]);
+ }
+
+ /* update keys */
+ for (int i = 0; i < children; i++)
+ {
+ TestValueType update = keys[i] + 1;
+ if (!rt_set(radixtree, keys[i], (TestValueType*) &update))
+ elog(ERROR, "could not update key 0x" UINT64_HEX_FORMAT, keys[i]);
+ }
+
+ /* repeat deleting and inserting keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (!rt_delete(radixtree, keys[i]))
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, keys[i]);
+ if (rt_set(radixtree, keys[i], (TestValueType*) &keys[i]))
+ elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+ }
+
+ pfree(keys);
+ rt_free(radixtree);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Check if keys from start to end with the shift exist in the tree.
+ */
+static void
+check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end,
+ int incr)
+{
+ for (int i = start; i < end; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ TestValueType val;
+
+ if (!rt_search(radixtree, key, &val))
+ elog(ERROR, "key 0x" UINT64_HEX_FORMAT " is not found on node-%d",
+ key, end);
+ if (val != (TestValueType) key)
+ elog(ERROR, "rt_search with key 0x" UINT64_HEX_FORMAT " returns 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+ key, val, key);
+ }
+}
+
+static void
+test_node_types_insert(rt_radix_tree *radixtree, uint8 shift, bool insert_asc)
+{
+ uint64 num_entries;
+ int ninserted = 0;
+ int start = insert_asc ? 0 : 256;
+ int incr = insert_asc ? 1 : -1;
+ int end = insert_asc ? 256 : 0;
+ int node_kind_idx = 1;
+
+ for (int i = start; i != end; i += incr)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_set(radixtree, key, (TestValueType*) &key);
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " is found", key);
+
+ /*
+ * After filling all slots in each node type, check if the values
+ * are stored properly.
+ */
+ if (ninserted == rt_node_kind_fanouts[node_kind_idx] - 1)
+ {
+ int check_start = insert_asc
+ ? rt_node_kind_fanouts[node_kind_idx - 1]
+ : rt_node_kind_fanouts[node_kind_idx];
+ int check_end = insert_asc
+ ? rt_node_kind_fanouts[node_kind_idx]
+ : rt_node_kind_fanouts[node_kind_idx - 1];
+
+ check_search_on_node(radixtree, shift, check_start, check_end, incr);
+ node_kind_idx++;
+ }
+
+ ninserted++;
+ }
+
+ num_entries = rt_num_entries(radixtree);
+
+ if (num_entries != 256)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+static void
+test_node_types_delete(rt_radix_tree *radixtree, uint8 shift)
+{
+ uint64 num_entries;
+
+ for (int i = 0; i < 256; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_delete(radixtree, key);
+
+ if (!found)
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, key);
+ }
+
+ num_entries = rt_num_entries(radixtree);
+
+ /* The tree must be empty */
+ if (num_entries != 0)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+/*
+ * Test for inserting and deleting key-value pairs to each node type at the given shift
+ * level.
+ */
+static void
+test_node_types(uint8 shift)
+{
+ rt_radix_tree *radixtree;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing radix tree node types with shift \"%d\"", shift);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ /*
+ * Insert and search entries for every node type at the 'shift' level,
+ * then delete all entries to make it empty, and insert and search entries
+ * again.
+ */
+ test_node_types_insert(radixtree, shift, true);
+ test_node_types_delete(radixtree, shift);
+ test_node_types_insert(radixtree, shift, false);
+
+ rt_free(radixtree);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Test with a repeating pattern, defined by the 'spec'.
+ */
+static void
+test_pattern(const test_spec * spec)
+{
+ rt_radix_tree *radixtree;
+ rt_iter *iter;
+ MemoryContext radixtree_ctx;
+ TimestampTz starttime;
+ TimestampTz endtime;
+ uint64 n;
+ uint64 last_int;
+ uint64 ndeleted;
+ uint64 nbefore;
+ uint64 nafter;
+ int patternlen;
+ uint64 *pattern_values;
+ uint64 pattern_num_values;
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing radix tree with pattern \"%s\"", spec->test_name);
+ if (rt_test_stats)
+ fprintf(stderr, "-----\ntesting radix tree with pattern \"%s\"\n", spec->test_name);
+
+ /* Pre-process the pattern, creating an array of integers from it. */
+ patternlen = strlen(spec->pattern_str);
+ pattern_values = palloc(patternlen * sizeof(uint64));
+ pattern_num_values = 0;
+ for (int i = 0; i < patternlen; i++)
+ {
+ if (spec->pattern_str[i] == '1')
+ pattern_values[pattern_num_values++] = i;
+ }
+
+ /*
+ * Allocate the radix tree.
+ *
+ * Allocate it in a separate memory context, so that we can print its
+ * memory usage easily.
+ */
+ radixtree_ctx = AllocSetContextCreate(CurrentMemoryContext,
+ "radixtree test",
+ ALLOCSET_SMALL_SIZES);
+ MemoryContextSetIdentifier(radixtree_ctx, spec->test_name);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(radixtree_ctx, dsa, tranche_id);
+#else
+ radixtree = rt_create(radixtree_ctx);
+#endif
+
+
+ /*
+ * Add values to the set.
+ */
+ starttime = GetCurrentTimestamp();
+
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ uint64 x = 0;
+
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ bool found;
+
+ x = last_int + pattern_values[i];
+
+ found = rt_set(radixtree, x, (TestValueType*) &x);
+
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " found", x);
+
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+
+ endtime = GetCurrentTimestamp();
+
+ if (rt_test_stats)
+ fprintf(stderr, "added " UINT64_FORMAT " values in %d ms\n",
+ spec->num_values, (int) (endtime - starttime) / 1000);
+
+ /*
+ * Print stats on the amount of memory used.
+ *
+ * We print the usage reported by rt_memory_usage(), as well as the stats
+ * from the memory context. They should be in the same ballpark, but it's
+ * hard to automate testing that, so if you're making changes to the
+ * implementation, just observe that manually.
+ */
+ if (rt_test_stats)
+ {
+ uint64 mem_usage;
+
+ /*
+ * Also print memory usage as reported by rt_memory_usage(). It
+ * should be in the same ballpark as the usage reported by
+ * MemoryContextStats().
+ */
+ mem_usage = rt_memory_usage(radixtree);
+ fprintf(stderr, "rt_memory_usage() reported " UINT64_FORMAT " (%0.2f bytes / integer)\n",
+ mem_usage, (double) mem_usage / spec->num_values);
+
+ MemoryContextStats(radixtree_ctx);
+ }
+
+ /* Check that rt_num_entries works */
+ n = rt_num_entries(radixtree);
+ if (n != spec->num_values)
+ elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT, n, spec->num_values);
+
+ /*
+ * Test random-access probes with rt_search()
+ */
+ starttime = GetCurrentTimestamp();
+
+ for (n = 0; n < 100000; n++)
+ {
+ bool found;
+ bool expected;
+ uint64 x;
+ TestValueType v;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit values that are actually in the set.
+ */
+ x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+ /* Do we expect this value to be present in the set? */
+ if (x >= last_int)
+ expected = false;
+ else
+ {
+ uint64 idx = x % spec->spacing;
+
+ if (idx >= patternlen)
+ expected = false;
+ else if (spec->pattern_str[idx] == '1')
+ expected = true;
+ else
+ expected = false;
+ }
+
+ /* Is it present according to rt_search() ? */
+ found = rt_search(radixtree, x, &v);
+
+ if (found != expected)
+ elog(ERROR, "mismatch at 0x" UINT64_HEX_FORMAT ": %d vs %d", x, found, expected);
+ if (found && (v != (TestValueType) x))
+ elog(ERROR, "found 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+ v, x);
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "probed " UINT64_FORMAT " values in %d ms\n",
+ n, (int) (endtime - starttime) / 1000);
+
+ /*
+ * Test iterator
+ */
+ starttime = GetCurrentTimestamp();
+
+ iter = rt_begin_iterate(radixtree);
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ uint64 expected = last_int + pattern_values[i];
+ uint64 x;
+ TestValueType val;
+
+ if (!rt_iterate_next(iter, &x, &val))
+ break;
+
+ if (x != expected)
+ elog(ERROR,
+ "iterate returned wrong key; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d",
+ x, expected, i);
+ if (val != (TestValueType) expected)
+ elog(ERROR,
+ "iterate returned wrong value; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d", x, expected, i);
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "iterated " UINT64_FORMAT " values in %d ms\n",
+ n, (int) (endtime - starttime) / 1000);
+
+ rt_end_iterate(iter);
+
+ if (n < spec->num_values)
+ elog(ERROR, "iterator stopped short after " UINT64_FORMAT " entries, expected " UINT64_FORMAT, n, spec->num_values);
+ if (n > spec->num_values)
+ elog(ERROR, "iterator returned " UINT64_FORMAT " entries, " UINT64_FORMAT " was expected", n, spec->num_values);
+
+ /*
+ * Test random-access probes with rt_delete()
+ */
+ starttime = GetCurrentTimestamp();
+
+ nbefore = rt_num_entries(radixtree);
+ ndeleted = 0;
+ for (n = 0; n < 1; n++)
+ {
+ bool found;
+ uint64 x;
+ TestValueType v;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit values that are actually in the set.
+ */
+ x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+ /* Is it present according to rt_search() ? */
+ found = rt_search(radixtree, x, &v);
+
+ if (!found)
+ continue;
+
+ /* If the key is found, delete it and check again */
+ if (!rt_delete(radixtree, x))
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, x);
+ if (rt_search(radixtree, x, &v))
+ elog(ERROR, "found deleted key 0x" UINT64_HEX_FORMAT, x);
+ if (rt_delete(radixtree, x))
+ elog(ERROR, "deleted already-deleted key 0x" UINT64_HEX_FORMAT, x);
+
+ ndeleted++;
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "deleted " UINT64_FORMAT " values in %d ms\n",
+ ndeleted, (int) (endtime - starttime) / 1000);
+
+ nafter = rt_num_entries(radixtree);
+
+ /* Check that rt_num_entries works */
+ if ((nbefore - ndeleted) != nafter)
+ elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT "after " UINT64_FORMAT " deletion",
+ nafter, (nbefore - ndeleted), ndeleted);
+
+ rt_free(radixtree);
+ MemoryContextDelete(radixtree_ctx);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+Datum
+test_radixtree(PG_FUNCTION_ARGS)
+{
+ test_empty();
+
+ for (int i = 1; i < lengthof(rt_node_kind_fanouts); i++)
+ {
+ test_basic(rt_node_kind_fanouts[i], false);
+ test_basic(rt_node_kind_fanouts[i], true);
+ }
+
+ for (int shift = 0; shift <= (64 - 8); shift += 8)
+ test_node_types(shift);
+
+ /* Test different test patterns, with lots of entries */
+ for (int i = 0; i < lengthof(test_specs); i++)
+ test_pattern(&test_specs[i]);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_radixtree/test_radixtree.control b/src/test/modules/test_radixtree/test_radixtree.control
new file mode 100644
index 0000000000..e53f2a3e0c
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.control
@@ -0,0 +1,4 @@
+comment = 'Test code for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/test_radixtree'
+relocatable = true
diff --git a/src/tools/pginclude/cpluspluscheck b/src/tools/pginclude/cpluspluscheck
index e52fe9f509..09fa6e7432 100755
--- a/src/tools/pginclude/cpluspluscheck
+++ b/src/tools/pginclude/cpluspluscheck
@@ -101,6 +101,12 @@ do
test "$f" = src/include/nodes/nodetags.h && continue
test "$f" = src/backend/nodes/nodetags.h && continue
+ # radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+ test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+ test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+ test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+ test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
# These files are not meant to be included standalone, because
# they contain lists that might have multiple use-cases.
test "$f" = src/include/access/rmgrlist.h && continue
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index abbba7aa63..d4d2f1da03 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -96,6 +96,12 @@ do
test "$f" = src/include/nodes/nodetags.h && continue
test "$f" = src/backend/nodes/nodetags.h && continue
+ # radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+ test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+ test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+ test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+ test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
# These files are not meant to be included standalone, because
# they contain lists that might have multiple use-cases.
test "$f" = src/include/access/rmgrlist.h && continue
--
2.39.1
[text/x-patch] v25-0006-Adjust-some-inlining-declarations.patch (2.1K, ../../CAFBsxsHbBm_M22gLBO+AZT4mfMq3L_oX3wdKZxjeNnT7fHsYMQ@mail.gmail.com/9-v25-0006-Adjust-some-inlining-declarations.patch)
download | inline diff:
From 77541d3f48e6fef39645df5b3c535ac431e12194 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Mon, 6 Feb 2023 21:04:14 +0700
Subject: [PATCH v25 6/9] Adjust some inlining declarations
---
src/include/lib/radixtree.h | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index d6919aef08..4bd0aaa810 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -1124,7 +1124,7 @@ RT_INIT_NODE(RT_PTR_LOCAL node, uint8 kind, RT_SIZE_CLASS size_class, bool is_le
* Create a new node as the root. Subordinate nodes will be created during
* the insertion.
*/
-static void
+static pg_noinline void
RT_NEW_ROOT(RT_RADIX_TREE *tree, uint64 key)
{
int shift = RT_KEY_GET_SHIFT(key);
@@ -1215,7 +1215,7 @@ RT_NODE_UPDATE_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC new_child)
/*
* Replace old_child with new_child, and free the old one.
*/
-static void
+static inline void
RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
RT_PTR_ALLOC stored_old_child, RT_PTR_LOCAL old_child,
RT_PTR_ALLOC new_child, uint64 key)
@@ -1242,7 +1242,7 @@ RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
* The radix tree doesn't have sufficient height. Extend the radix tree so
* it can store the key.
*/
-static void
+static pg_noinline void
RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
{
int target_shift;
@@ -1281,7 +1281,7 @@ RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
* The radix tree doesn't have inner and leaf nodes for given key-value pair.
* Insert inner and leaf nodes from 'node' to bottom.
*/
-static inline void
+static pg_noinline void
RT_SET_EXTEND(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node)
{
@@ -1486,7 +1486,7 @@ RT_GET_HANDLE(RT_RADIX_TREE *tree)
/*
* Recursively free all nodes allocated to the DSA area.
*/
-static inline void
+static void
RT_FREE_RECURSE(RT_RADIX_TREE *tree, RT_PTR_ALLOC ptr)
{
RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, ptr);
--
2.39.1
[text/x-patch] v25-0007-Skip-unnecessary-searches-in-RT_NODE_INSERT_INNE.patch (4.7K, ../../CAFBsxsHbBm_M22gLBO+AZT4mfMq3L_oX3wdKZxjeNnT7fHsYMQ@mail.gmail.com/10-v25-0007-Skip-unnecessary-searches-in-RT_NODE_INSERT_INNE.patch)
download | inline diff:
From f2a3340200ea26c17de5c5261adbeaada64ae4b6 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Mon, 6 Feb 2023 22:04:50 +0700
Subject: [PATCH v25 7/9] Skip unnecessary searches in RT_NODE_INSERT_INNER
For inner nodes, we know the key chunk doesn't exist already,
otherwise we would have found it while descending the tree.
To reinforce this fact, declare this function to return void.
---
src/include/lib/radixtree.h | 4 +--
src/include/lib/radixtree_insert_impl.h | 48 ++++++++++++-------------
2 files changed, 24 insertions(+), 28 deletions(-)
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index 4bd0aaa810..1cdb995e54 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -685,7 +685,7 @@ typedef struct RT_ITER
} RT_ITER;
-static bool RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+static void RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
uint64 key, RT_PTR_ALLOC child);
static bool RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
uint64 key, RT_VALUE_TYPE *value_p);
@@ -1375,7 +1375,7 @@ RT_NODE_DELETE_LEAF(RT_PTR_LOCAL node, uint64 key)
* If the node we're inserting into needs to grow, we update the parent's
* child pointer with the pointer to the new larger node.
*/
-static bool
+static void
RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
uint64 key, RT_PTR_ALLOC child)
{
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
index c18e26b537..d56e58dcac 100644
--- a/src/include/lib/radixtree_insert_impl.h
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -28,10 +28,10 @@
#endif
uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
- bool chunk_exists = false;
#ifdef RT_NODE_LEVEL_LEAF
const bool is_leaf = true;
+ bool chunk_exists = false;
Assert(RT_NODE_IS_LEAF(node));
#else
const bool is_leaf = false;
@@ -43,21 +43,18 @@
case RT_NODE_KIND_3:
{
RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
- int idx;
- idx = RT_NODE_3_SEARCH_EQ(&n3->base, chunk);
+#ifdef RT_NODE_LEVEL_LEAF
+ int idx = RT_NODE_3_SEARCH_EQ(&n3->base, chunk);
+
if (idx != -1)
{
/* found the existing chunk */
chunk_exists = true;
-#ifdef RT_NODE_LEVEL_LEAF
n3->values[idx] = *value_p;
-#else
- n3->children[idx] = child;
-#endif
break;
}
-
+#endif
if (unlikely(RT_NODE_MUST_GROW(n3)))
{
RT_PTR_ALLOC allocnode;
@@ -113,21 +110,18 @@
{
const RT_SIZE_CLASS_ELEM class32_max = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX];
RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
- int idx;
- idx = RT_NODE_32_SEARCH_EQ(&n32->base, chunk);
+#ifdef RT_NODE_LEVEL_LEAF
+ int idx = RT_NODE_32_SEARCH_EQ(&n32->base, chunk);
+
if (idx != -1)
{
/* found the existing chunk */
chunk_exists = true;
-#ifdef RT_NODE_LEVEL_LEAF
n32->values[idx] = *value_p;
-#else
- n32->children[idx] = child;
-#endif
break;
}
-
+#endif
if (unlikely(RT_NODE_MUST_GROW(n32)) &&
n32->base.n.fanout < class32_max.fanout)
{
@@ -220,21 +214,19 @@
case RT_NODE_KIND_125:
{
RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
- int slotpos = n125->base.slot_idxs[chunk];
+ int slotpos;
int cnt = 0;
+#ifdef RT_NODE_LEVEL_LEAF
+ slotpos = n125->base.slot_idxs[chunk];
if (slotpos != RT_INVALID_SLOT_IDX)
{
/* found the existing chunk */
chunk_exists = true;
-#ifdef RT_NODE_LEVEL_LEAF
n125->values[slotpos] = *value_p;
-#else
- n125->children[slotpos] = child;
-#endif
break;
}
-
+#endif
if (unlikely(RT_NODE_MUST_GROW(n125)))
{
RT_PTR_ALLOC allocnode;
@@ -300,14 +292,10 @@
#ifdef RT_NODE_LEVEL_LEAF
chunk_exists = RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk);
-#else
- chunk_exists = RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk);
-#endif
Assert(chunk_exists || node->count < RT_NODE_MAX_SLOTS);
-
-#ifdef RT_NODE_LEVEL_LEAF
RT_NODE_LEAF_256_SET(n256, chunk, *value_p);
#else
+ Assert(node->count < RT_NODE_MAX_SLOTS);
RT_NODE_INNER_256_SET(n256, chunk, child);
#endif
break;
@@ -315,8 +303,12 @@
}
/* Update statistics */
+#ifdef RT_NODE_LEVEL_LEAF
if (!chunk_exists)
node->count++;
+#else
+ node->count++;
+#endif
/*
* Done. Finally, verify the chunk and value is inserted or replaced
@@ -324,7 +316,11 @@
*/
RT_VERIFY_NODE(node);
+#ifdef RT_NODE_LEVEL_LEAF
return chunk_exists;
+#else
+ return;
+#endif
#undef RT_NODE3_TYPE
#undef RT_NODE32_TYPE
--
2.39.1
[text/x-patch] v25-0008-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch (35.2K, ../../CAFBsxsHbBm_M22gLBO+AZT4mfMq3L_oX3wdKZxjeNnT7fHsYMQ@mail.gmail.com/11-v25-0008-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch)
download | inline diff:
From 693e335f77211e9947cd356d9287c9af96e78815 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 23 Dec 2022 23:50:39 +0900
Subject: [PATCH v25 8/9] Add TIDStore, to store sets of TIDs (ItemPointerData)
efficiently.
The TIDStore is designed to store large sets of TIDs efficiently, and
is backed by the radix tree. A TID is encoded into 64-bit key and
value and inserted to the radix tree.
The TIDStore is not used for anything yet, aside from the test code,
but the follow up patch integrates the TIDStore with lazy vacuum,
reducing lazy vacuum memory usage and lifting the 1GB limit on its
size, by storing the list of dead TIDs more efficiently.
This includes a unit test module, in src/test/modules/test_tidstore.
---
doc/src/sgml/monitoring.sgml | 4 +
src/backend/access/common/Makefile | 1 +
src/backend/access/common/meson.build | 1 +
src/backend/access/common/tidstore.c | 688 ++++++++++++++++++
src/backend/storage/lmgr/lwlock.c | 2 +
src/include/access/tidstore.h | 49 ++
src/include/storage/lwlock.h | 1 +
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_tidstore/Makefile | 23 +
.../test_tidstore/expected/test_tidstore.out | 13 +
src/test/modules/test_tidstore/meson.build | 35 +
.../test_tidstore/sql/test_tidstore.sql | 7 +
.../test_tidstore/test_tidstore--1.0.sql | 8 +
.../modules/test_tidstore/test_tidstore.c | 195 +++++
.../test_tidstore/test_tidstore.control | 4 +
16 files changed, 1033 insertions(+)
create mode 100644 src/backend/access/common/tidstore.c
create mode 100644 src/include/access/tidstore.h
create mode 100644 src/test/modules/test_tidstore/Makefile
create mode 100644 src/test/modules/test_tidstore/expected/test_tidstore.out
create mode 100644 src/test/modules/test_tidstore/meson.build
create mode 100644 src/test/modules/test_tidstore/sql/test_tidstore.sql
create mode 100644 src/test/modules/test_tidstore/test_tidstore--1.0.sql
create mode 100644 src/test/modules/test_tidstore/test_tidstore.c
create mode 100644 src/test/modules/test_tidstore/test_tidstore.control
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1756f1a4b6..d936aa3da3 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2192,6 +2192,10 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<entry>Waiting to access a shared TID bitmap during a parallel bitmap
index scan.</entry>
</row>
+ <row>
+ <entry><literal>SharedTidStore</literal></entry>
+ <entry>Waiting to access a shared TID store.</entry>
+ </row>
<row>
<entry><literal>SharedTupleStore</literal></entry>
<entry>Waiting to access a shared tuple store during parallel
diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile
index b9aff0ccfd..67b8cc6108 100644
--- a/src/backend/access/common/Makefile
+++ b/src/backend/access/common/Makefile
@@ -27,6 +27,7 @@ OBJS = \
syncscan.o \
toast_compression.o \
toast_internals.o \
+ tidstore.o \
tupconvert.o \
tupdesc.o
diff --git a/src/backend/access/common/meson.build b/src/backend/access/common/meson.build
index f5ac17b498..fce19c09ce 100644
--- a/src/backend/access/common/meson.build
+++ b/src/backend/access/common/meson.build
@@ -15,6 +15,7 @@ backend_sources += files(
'syncscan.c',
'toast_compression.c',
'toast_internals.c',
+ 'tidstore.c',
'tupconvert.c',
'tupdesc.c',
)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
new file mode 100644
index 0000000000..4c72673ce9
--- /dev/null
+++ b/src/backend/access/common/tidstore.c
@@ -0,0 +1,688 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.c
+ * Tid (ItemPointerData) storage implementation.
+ *
+ * This module provides a in-memory data structure to store Tids (ItemPointer).
+ * Internally, a tid is encoded as a pair of 64-bit key and 64-bit value, and
+ * stored in the radix tree.
+ *
+ * A TidStore can be shared among parallel worker processes by passing DSA area
+ * to tidstore_create(). Other backends can attach to the shared TidStore by
+ * tidstore_attach().
+ *
+ * Regarding the concurrency, it basically relies on the concurrency support in
+ * the radix tree, but we acquires the lock on a TidStore in some cases, for
+ * example, when to reset the store and when to access the number tids in the
+ * store (num_tids).
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/access/common/tidstore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/tidstore.h"
+#include "miscadmin.h"
+#include "port/pg_bitutils.h"
+#include "storage/lwlock.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/*
+ * For encoding purposes, tids are represented as a pair of 64-bit key and
+ * 64-bit value. First, we construct 64-bit unsigned integer by combining
+ * the block number and the offset number. The number of bits used for the
+ * offset number is specified by max_offsets in tidstore_create(). We are
+ * frugal with the bits, because smaller keys could help keeping the radix
+ * tree shallow.
+ *
+ * For example, a tid of heap with 8kB blocks uses the lowest 9 bits for
+ * the offset number and uses the next 32 bits for the block number. That
+ * is, only 41 bits are used:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ *
+ * X = bits used for offset number
+ * Y = bits used for block number
+ * u = unused bit
+ * (high on the left, low on the right)
+ *
+ * 9 bits are enough for the offset number, because MaxHeapTuplesPerPage < 2^9
+ * on 8kB blocks.
+ *
+ * The 64-bit value is the bitmap representation of the lowest 6 bits
+ * (TIDSTORE_VALUE_NBITS) of the integer, and the rest 35 bits are used
+ * as the key:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ * |----| value
+ * |---------------------------------------------| key
+ *
+ * The maximum height of the radix tree is 5 in this case.
+ *
+ * If the number of bits for offset number fits in a 64-bit value, we don't
+ * encode tids but directly use the block number and the offset number as key
+ * and value, respectively.
+ */
+#define TIDSTORE_VALUE_NBITS 6 /* log(64, 2) */
+
+/* A magic value used to identify our TidStores. */
+#define TIDSTORE_MAGIC 0x826f6a10
+
+#define RT_PREFIX local_rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE uint64
+#include "lib/radixtree.h"
+
+#define RT_PREFIX shared_rt
+#define RT_SHMEM
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE uint64
+#include "lib/radixtree.h"
+
+/* The control object for a TidStore */
+typedef struct TidStoreControl
+{
+ /* the number of tids in the store */
+ int64 num_tids;
+
+ /* These values are never changed after creation */
+ size_t max_bytes; /* the maximum bytes a TidStore can use */
+ int max_offset; /* the maximum offset number */
+ int offset_nbits; /* the number of bits required for max_offset */
+ bool encode_tids; /* do we use tid encoding? */
+ int offset_key_nbits; /* the number of bits of a offset number
+ * used for the key */
+
+ /* The below fields are used only in shared case */
+
+ uint32 magic;
+ LWLock lock;
+
+ /* handles for TidStore and radix tree */
+ tidstore_handle handle;
+ shared_rt_handle tree_handle;
+} TidStoreControl;
+
+/* Per-backend state for a TidStore */
+struct TidStore
+{
+ /*
+ * Control object. This is allocated in DSA area 'area' in the shared
+ * case, otherwise in backend-local memory.
+ */
+ TidStoreControl *control;
+
+ /* Storage for Tids. Use either one depending on TidStoreIsShared() */
+ union
+ {
+ local_rt_radix_tree *local;
+ shared_rt_radix_tree *shared;
+ } tree;
+
+ /* DSA area for TidStore if used */
+ dsa_area *area;
+};
+#define TidStoreIsShared(ts) ((ts)->area != NULL)
+
+/* Iterator for TidStore */
+typedef struct TidStoreIter
+{
+ TidStore *ts;
+
+ /* iterator of radix tree. Use either one depending on TidStoreIsShared() */
+ union
+ {
+ shared_rt_iter *shared;
+ local_rt_iter *local;
+ } tree_iter;
+
+ /* we returned all tids? */
+ bool finished;
+
+ /* save for the next iteration */
+ uint64 next_key;
+ uint64 next_val;
+
+ /* output for the caller */
+ TidStoreIterResult result;
+} TidStoreIter;
+
+static void tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val);
+static inline BlockNumber key_get_blkno(TidStore *ts, uint64 key);
+static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off);
+
+/*
+ * Create a TidStore. The returned object is allocated in backend-local memory.
+ * The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
+ */
+TidStore *
+tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
+{
+ TidStore *ts;
+
+ ts = palloc0(sizeof(TidStore));
+
+ /*
+ * Create the radix tree for the main storage.
+ *
+ * Memory consumption depends on the number of stored tids, but also on the
+ * distribution of them, how the radix tree stores, and the memory management
+ * that backed the radix tree. The maximum bytes that a TidStore can
+ * use is specified by the max_bytes in tidstore_create(). We want the total
+ * amount of memory consumption by a TidStore not to exceed the max_bytes.
+ *
+ * In local TidStore cases, the radix tree uses slab allocators for each kind
+ * of node class. The most memory consuming case while adding Tids associated
+ * with one page (i.e. during tidstore_add_tids()) is that we allocate a new
+ * slab block for a new radix tree node, which is approximately 70kB. Therefore,
+ * we deduct 70kB from the max_bytes.
+ *
+ * In shared cases, DSA allocates the memory segments big enough to follow
+ * a geometric series that approximately doubles the total DSA size (see
+ * make_new_segment() in dsa.c). We simulated the how DSA increases segment
+ * size and the simulation revealed, the 75% threshold for the maximum bytes
+ * perfectly works in case where the max_bytes is a power-of-2, and the 60%
+ * threshold works for other cases.
+ */
+ if (area != NULL)
+ {
+ dsa_pointer dp;
+ float ratio = ((max_bytes & (max_bytes - 1)) == 0) ? 0.75 : 0.6;
+
+ ts->tree.shared = shared_rt_create(CurrentMemoryContext, area,
+ LWTRANCHE_SHARED_TIDSTORE);
+
+ dp = dsa_allocate0(area, sizeof(TidStoreControl));
+ ts->control = (TidStoreControl *) dsa_get_address(area, dp);
+ ts->control->max_bytes = (uint64) (max_bytes * ratio);
+ ts->area = area;
+
+ ts->control->magic = TIDSTORE_MAGIC;
+ LWLockInitialize(&ts->control->lock, LWTRANCHE_SHARED_TIDSTORE);
+ ts->control->handle = dp;
+ ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+ }
+ else
+ {
+ ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+ ts->control = (TidStoreControl *) palloc0(sizeof(TidStoreControl));
+ ts->control->max_bytes = max_bytes - (70 * 1024);
+ }
+
+ ts->control->max_offset = max_offset;
+ ts->control->offset_nbits = pg_ceil_log2_32(max_offset);
+
+ /*
+ * We use tid encoding if the number of bits for the offset number doesn't
+ * fix in a value, uint64.
+ */
+ if (ts->control->offset_nbits > TIDSTORE_VALUE_NBITS)
+ {
+ ts->control->encode_tids = true;
+ ts->control->offset_key_nbits =
+ ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
+ }
+ else
+ {
+ ts->control->encode_tids = false;
+ ts->control->offset_key_nbits = 0;
+ }
+
+ return ts;
+}
+
+/*
+ * Attach to the shared TidStore using a handle. The returned object is
+ * allocated in backend-local memory using the CurrentMemoryContext.
+ */
+TidStore *
+tidstore_attach(dsa_area *area, tidstore_handle handle)
+{
+ TidStore *ts;
+ dsa_pointer control;
+
+ Assert(area != NULL);
+ Assert(DsaPointerIsValid(handle));
+
+ /* create per-backend state */
+ ts = palloc0(sizeof(TidStore));
+
+ /* Find the control object in shared memory */
+ control = handle;
+
+ /* Set up the TidStore */
+ ts->control = (TidStoreControl *) dsa_get_address(area, control);
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ ts->tree.shared = shared_rt_attach(area, ts->control->tree_handle);
+ ts->area = area;
+
+ return ts;
+}
+
+/*
+ * Detach from a TidStore. This detaches from radix tree and frees the
+ * backend-local resources. The radix tree will continue to exist until
+ * it is either explicitly destroyed, or the area that backs it is returned
+ * to the operating system.
+ */
+void
+tidstore_detach(TidStore *ts)
+{
+ Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+ shared_rt_detach(ts->tree.shared);
+ pfree(ts);
+}
+
+/*
+ * Destroy a TidStore, returning all memory.
+ *
+ * TODO: The caller must be certain that no other backend will attempt to
+ * access the TidStore before calling this function. Other backend must
+ * explicitly call tidstore_detach to free up backend-local memory associated
+ * with the TidStore. The backend that calls tidstore_destroy must not call
+ * tidstore_detach.
+ */
+void
+tidstore_destroy(TidStore *ts)
+{
+ if (TidStoreIsShared(ts))
+ {
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ /*
+ * Vandalize the control block to help catch programming error where
+ * other backends access the memory formerly occupied by this radix
+ * tree.
+ */
+ ts->control->magic = 0;
+ dsa_free(ts->area, ts->control->handle);
+ shared_rt_free(ts->tree.shared);
+ }
+ else
+ {
+ pfree(ts->control);
+ local_rt_free(ts->tree.local);
+ }
+
+ pfree(ts);
+}
+
+/*
+ * Forget all collected Tids. It's similar to tidstore_destroy but we don't free
+ * entire TidStore but recreate only the radix tree storage.
+ */
+void
+tidstore_reset(TidStore *ts)
+{
+ if (TidStoreIsShared(ts))
+ {
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+ /*
+ * Free the radix tree and return allocated DSA segments to
+ * the operating system.
+ */
+ shared_rt_free(ts->tree.shared);
+ dsa_trim(ts->area);
+
+ /* Recreate the radix tree */
+ ts->tree.shared = shared_rt_create(CurrentMemoryContext, ts->area,
+ LWTRANCHE_SHARED_TIDSTORE);
+
+ /* update the radix tree handle as we recreated it */
+ ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
+
+ LWLockRelease(&ts->control->lock);
+ }
+ else
+ {
+ local_rt_free(ts->tree.local);
+ ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
+ }
+}
+
+/* Add Tids on a block to TidStore */
+void
+tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets)
+{
+ ItemPointerData tid;
+ uint64 key_base;
+ uint64 *values;
+ int nkeys;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ if (ts->control->encode_tids)
+ {
+ key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
+ nkeys = UINT64CONST(1) << ts->control->offset_key_nbits;
+ }
+ else
+ {
+ key_base = (uint64) blkno;
+ nkeys = 1;
+ }
+ values = palloc0(sizeof(uint64) * nkeys);
+
+ ItemPointerSetBlockNumber(&tid, blkno);
+ for (int i = 0; i < num_offsets; i++)
+ {
+ uint64 key;
+ uint32 off;
+ int idx;
+
+ ItemPointerSetOffsetNumber(&tid, offsets[i]);
+
+ /* encode the tid to key and val */
+ key = tid_to_key_off(ts, &tid, &off);
+
+ idx = key - key_base;
+ Assert(idx >= 0 && idx < nkeys);
+
+ values[idx] |= UINT64CONST(1) << off;
+ }
+
+ if (TidStoreIsShared(ts))
+ LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+ /* insert the calculated key-values to the tree */
+ for (int i = 0; i < nkeys; i++)
+ {
+ if (values[i])
+ {
+ uint64 key = key_base + i;
+
+ if (TidStoreIsShared(ts))
+ shared_rt_set(ts->tree.shared, key, &values[i]);
+ else
+ local_rt_set(ts->tree.local, key, &values[i]);
+ }
+ }
+
+ /* update statistics */
+ ts->control->num_tids += num_offsets;
+
+ if (TidStoreIsShared(ts))
+ LWLockRelease(&ts->control->lock);
+
+ pfree(values);
+}
+
+/* Return true if the given tid is present in the TidStore */
+bool
+tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
+{
+ uint64 key;
+ uint64 val = 0;
+ uint32 off;
+ bool found;
+
+ key = tid_to_key_off(ts, tid, &off);
+
+ if (TidStoreIsShared(ts))
+ found = shared_rt_search(ts->tree.shared, key, &val);
+ else
+ found = local_rt_search(ts->tree.local, key, &val);
+
+ if (!found)
+ return false;
+
+ return (val & (UINT64CONST(1) << off)) != 0;
+}
+
+/*
+ * Prepare to iterate through a TidStore. Since the radix tree is locked during the
+ * iteration, so tidstore_end_iterate() needs to called when finished.
+ *
+ * Concurrent updates during the iteration will be blocked when inserting a
+ * key-value to the radix tree.
+ */
+TidStoreIter *
+tidstore_begin_iterate(TidStore *ts)
+{
+ TidStoreIter *iter;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ iter = palloc0(sizeof(TidStoreIter));
+ iter->ts = ts;
+
+ iter->result.blkno = InvalidBlockNumber;
+ iter->result.offsets = palloc(sizeof(OffsetNumber) * ts->control->max_offset);
+
+ if (TidStoreIsShared(ts))
+ iter->tree_iter.shared = shared_rt_begin_iterate(ts->tree.shared);
+ else
+ iter->tree_iter.local = local_rt_begin_iterate(ts->tree.local);
+
+ /* If the TidStore is empty, there is no business */
+ if (tidstore_num_tids(ts) == 0)
+ iter->finished = true;
+
+ return iter;
+}
+
+static inline bool
+tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
+{
+ if (TidStoreIsShared(iter->ts))
+ return shared_rt_iterate_next(iter->tree_iter.shared, key, val);
+
+ return local_rt_iterate_next(iter->tree_iter.local, key, val);
+}
+
+/*
+ * Scan the TidStore and return a pointer to TidStoreIterResult that has tids
+ * in one block. We return the block numbers in ascending order and the offset
+ * numbers in each result is also sorted in ascending order.
+ */
+TidStoreIterResult *
+tidstore_iterate_next(TidStoreIter *iter)
+{
+ uint64 key;
+ uint64 val;
+ TidStoreIterResult *result = &(iter->result);
+
+ if (iter->finished)
+ return NULL;
+
+ if (BlockNumberIsValid(result->blkno))
+ {
+ /* Process the previously collected key-value */
+ result->num_offsets = 0;
+ tidstore_iter_extract_tids(iter, iter->next_key, iter->next_val);
+ }
+
+ while (tidstore_iter_kv(iter, &key, &val))
+ {
+ BlockNumber blkno;
+
+ blkno = key_get_blkno(iter->ts, key);
+
+ if (BlockNumberIsValid(result->blkno) && result->blkno != blkno)
+ {
+ /*
+ * We got a key-value pair for a different block. So return the
+ * collected tids, and remember the key-value for the next iteration.
+ */
+ iter->next_key = key;
+ iter->next_val = val;
+ return result;
+ }
+
+ /* Collect tids extracted from the key-value pair */
+ tidstore_iter_extract_tids(iter, key, val);
+ }
+
+ iter->finished = true;
+ return result;
+}
+
+/*
+ * Finish an iteration over TidStore. This needs to be called after finishing
+ * or when existing an iteration.
+ */
+void
+tidstore_end_iterate(TidStoreIter *iter)
+{
+ if (TidStoreIsShared(iter->ts))
+ shared_rt_end_iterate(iter->tree_iter.shared);
+ else
+ local_rt_end_iterate(iter->tree_iter.local);
+
+ pfree(iter->result.offsets);
+ pfree(iter);
+}
+
+/* Return the number of tids we collected so far */
+int64
+tidstore_num_tids(TidStore *ts)
+{
+ uint64 num_tids;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ if (!TidStoreIsShared(ts))
+ return ts->control->num_tids;
+
+ LWLockAcquire(&ts->control->lock, LW_SHARED);
+ num_tids = ts->control->num_tids;
+ LWLockRelease(&ts->control->lock);
+
+ return num_tids;
+}
+
+/* Return true if the current memory usage of TidStore exceeds the limit */
+bool
+tidstore_is_full(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ return (tidstore_memory_usage(ts) > ts->control->max_bytes);
+}
+
+/* Return the maximum memory TidStore can use */
+size_t
+tidstore_max_memory(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ return ts->control->max_bytes;
+}
+
+/* Return the memory usage of TidStore */
+size_t
+tidstore_memory_usage(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ /*
+ * In the shared case, TidStoreControl and radix_tree are backed by the
+ * same DSA area and rt_memory_usage() returns the value including both.
+ * So we don't need to add the size of TidStoreControl separately.
+ */
+ if (TidStoreIsShared(ts))
+ return sizeof(TidStore) + shared_rt_memory_usage(ts->tree.shared);
+
+ return sizeof(TidStore) + sizeof(TidStore) + local_rt_memory_usage(ts->tree.local);
+}
+
+/*
+ * Get a handle that can be used by other processes to attach to this TidStore
+ */
+tidstore_handle
+tidstore_get_handle(TidStore *ts)
+{
+ Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+ return ts->control->handle;
+}
+
+/* Extract tids from the given key-value pair */
+static void
+tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
+{
+ TidStoreIterResult *result = (&iter->result);
+
+ for (int i = 0; i < sizeof(uint64) * BITS_PER_BYTE; i++)
+ {
+ uint64 tid_i;
+ OffsetNumber off;
+
+ if (i > iter->ts->control->max_offset)
+ {
+ Assert(!iter->ts->control->encode_tids);
+ break;
+ }
+
+ if ((val & (UINT64CONST(1) << i)) == 0)
+ continue;
+
+ tid_i = key << TIDSTORE_VALUE_NBITS;
+ tid_i |= i;
+
+ off = tid_i & ((UINT64CONST(1) << iter->ts->control->offset_nbits) - 1);
+
+ Assert(result->num_offsets < iter->ts->control->max_offset);
+ result->offsets[result->num_offsets++] = off;
+ }
+
+ result->blkno = key_get_blkno(iter->ts, key);
+}
+
+/* Get block number from the given key */
+static inline BlockNumber
+key_get_blkno(TidStore *ts, uint64 key)
+{
+ if (ts->control->encode_tids)
+ return (BlockNumber) (key >> ts->control->offset_key_nbits);
+
+ return (BlockNumber) key;
+}
+
+/* Encode a tid to key and offset */
+static inline uint64
+tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off)
+{
+ uint64 key;
+ uint64 tid_i;
+
+ if (!ts->control->encode_tids)
+ {
+ *off = ItemPointerGetOffsetNumber(tid);
+
+ /* Use the block number as the key */
+ return (int64) ItemPointerGetBlockNumber(tid);
+ }
+
+ tid_i = ItemPointerGetOffsetNumber(tid);
+ tid_i |= (uint64) ItemPointerGetBlockNumber(tid) << ts->control->offset_nbits;
+
+ *off = tid_i & ((UINT64CONST(1) << TIDSTORE_VALUE_NBITS) - 1);
+ key = tid_i >> TIDSTORE_VALUE_NBITS;
+ Assert(*off < (sizeof(uint64) * BITS_PER_BYTE));
+
+ return key;
+}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index d2ec396045..55b3a04097 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -176,6 +176,8 @@ static const char *const BuiltinTrancheNames[] = {
"SharedTupleStore",
/* LWTRANCHE_SHARED_TIDBITMAP: */
"SharedTidBitmap",
+ /* LWTRANCHE_SHARED_TIDSTORE: */
+ "SharedTidStore",
/* LWTRANCHE_PARALLEL_APPEND: */
"ParallelAppend",
/* LWTRANCHE_PER_XACT_PREDICATE_LIST: */
diff --git a/src/include/access/tidstore.h b/src/include/access/tidstore.h
new file mode 100644
index 0000000000..a35a52124a
--- /dev/null
+++ b/src/include/access/tidstore.h
@@ -0,0 +1,49 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.h
+ * Tid storage.
+ *
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/tidstore.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef TIDSTORE_H
+#define TIDSTORE_H
+
+#include "storage/itemptr.h"
+#include "utils/dsa.h"
+
+typedef dsa_pointer tidstore_handle;
+
+typedef struct TidStore TidStore;
+typedef struct TidStoreIter TidStoreIter;
+
+typedef struct TidStoreIterResult
+{
+ BlockNumber blkno;
+ OffsetNumber *offsets;
+ int num_offsets;
+} TidStoreIterResult;
+
+extern TidStore *tidstore_create(size_t max_bytes, int max_offset, dsa_area *dsa);
+extern TidStore *tidstore_attach(dsa_area *dsa, dsa_pointer handle);
+extern void tidstore_detach(TidStore *ts);
+extern void tidstore_destroy(TidStore *ts);
+extern void tidstore_reset(TidStore *ts);
+extern void tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets);
+extern bool tidstore_lookup_tid(TidStore *ts, ItemPointer tid);
+extern TidStoreIter * tidstore_begin_iterate(TidStore *ts);
+extern TidStoreIterResult *tidstore_iterate_next(TidStoreIter *iter);
+extern void tidstore_end_iterate(TidStoreIter *iter);
+extern int64 tidstore_num_tids(TidStore *ts);
+extern bool tidstore_is_full(TidStore *ts);
+extern size_t tidstore_max_memory(TidStore *ts);
+extern size_t tidstore_memory_usage(TidStore *ts);
+extern tidstore_handle tidstore_get_handle(TidStore *ts);
+
+#endif /* TIDSTORE_H */
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index d2c7afb8f4..07002fdfbe 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -199,6 +199,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_PER_SESSION_RECORD_TYPMOD,
LWTRANCHE_SHARED_TUPLESTORE,
LWTRANCHE_SHARED_TIDBITMAP,
+ LWTRANCHE_SHARED_TIDSTORE,
LWTRANCHE_PARALLEL_APPEND,
LWTRANCHE_PER_XACT_PREDICATE_LIST,
LWTRANCHE_PGSTATS_DSA,
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 9659eb85d7..bddc16ada7 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
test_rls_hooks \
test_shm_mq \
test_slru \
+ test_tidstore \
unsafe_tests \
worker_spi
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 232cbdac80..c0d5645ad8 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -30,5 +30,6 @@ subdir('test_regex')
subdir('test_rls_hooks')
subdir('test_shm_mq')
subdir('test_slru')
+subdir('test_tidstore')
subdir('unsafe_tests')
subdir('worker_spi')
diff --git a/src/test/modules/test_tidstore/Makefile b/src/test/modules/test_tidstore/Makefile
new file mode 100644
index 0000000000..dab107d70c
--- /dev/null
+++ b/src/test/modules/test_tidstore/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_tidstore/Makefile
+
+MODULE_big = test_tidstore
+OBJS = \
+ $(WIN32RES) \
+ test_tidstore.o
+PGFILEDESC = "test_tidstore - test code for src/backend/access/common/tidstore.c"
+
+EXTENSION = test_tidstore
+DATA = test_tidstore--1.0.sql
+
+REGRESS = test_tidstore
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_tidstore
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_tidstore/expected/test_tidstore.out b/src/test/modules/test_tidstore/expected/test_tidstore.out
new file mode 100644
index 0000000000..7ff2f9af87
--- /dev/null
+++ b/src/test/modules/test_tidstore/expected/test_tidstore.out
@@ -0,0 +1,13 @@
+CREATE EXTENSION test_tidstore;
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
+NOTICE: testing empty tidstore
+NOTICE: testing basic operations
+ test_tidstore
+---------------
+
+(1 row)
+
diff --git a/src/test/modules/test_tidstore/meson.build b/src/test/modules/test_tidstore/meson.build
new file mode 100644
index 0000000000..31f2da7b61
--- /dev/null
+++ b/src/test/modules/test_tidstore/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_tidstore_sources = files(
+ 'test_tidstore.c',
+)
+
+if host_system == 'windows'
+ test_tidstore_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_tidstore',
+ '--FILEDESC', 'test_tidstore - test code for src/backend/access/common/tidstore.c',])
+endif
+
+test_tidstore = shared_module('test_tidstore',
+ test_tidstore_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += test_tidstore
+
+install_data(
+ 'test_tidstore.control',
+ 'test_tidstore--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'test_tidstore',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'test_tidstore',
+ ],
+ },
+}
diff --git a/src/test/modules/test_tidstore/sql/test_tidstore.sql b/src/test/modules/test_tidstore/sql/test_tidstore.sql
new file mode 100644
index 0000000000..03aea31815
--- /dev/null
+++ b/src/test/modules/test_tidstore/sql/test_tidstore.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_tidstore;
+
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
diff --git a/src/test/modules/test_tidstore/test_tidstore--1.0.sql b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
new file mode 100644
index 0000000000..47e9149900
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_tidstore/test_tidstore--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_tidstore" to load this file. \quit
+
+CREATE FUNCTION test_tidstore()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_tidstore/test_tidstore.c b/src/test/modules/test_tidstore/test_tidstore.c
new file mode 100644
index 0000000000..9b849ae8e8
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.c
@@ -0,0 +1,195 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_tidstore.c
+ * Test TidStore data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_tidstore/test_tidstore.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/tidstore.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "utils/memutils.h"
+
+PG_MODULE_MAGIC;
+
+#define TEST_TIDSTORE_MAX_BYTES (2 * 1024 * 1024L) /* 2MB */
+
+PG_FUNCTION_INFO_V1(test_tidstore);
+
+static void
+check_tid(TidStore *ts, BlockNumber blkno, OffsetNumber off, bool expect)
+{
+ ItemPointerData tid;
+ bool found;
+
+ ItemPointerSet(&tid, blkno, off);
+
+ found = tidstore_lookup_tid(ts, &tid);
+
+ if (found != expect)
+ elog(ERROR, "lookup TID (%u, %u) returned %d, expected %d",
+ blkno, off, found, expect);
+}
+
+static void
+test_basic(int max_offset)
+{
+#define TEST_TIDSTORE_NUM_BLOCKS 5
+#define TEST_TIDSTORE_NUM_OFFSETS 5
+
+ TidStore *ts;
+ TidStoreIter *iter;
+ TidStoreIterResult *iter_result;
+ BlockNumber blks[TEST_TIDSTORE_NUM_BLOCKS] = {
+ 0, MaxBlockNumber, MaxBlockNumber - 1, 1, MaxBlockNumber / 2,
+ };
+ BlockNumber blks_sorted[TEST_TIDSTORE_NUM_BLOCKS] = {
+ 0, 1, MaxBlockNumber / 2, MaxBlockNumber - 1, MaxBlockNumber
+ };
+ OffsetNumber offs[TEST_TIDSTORE_NUM_OFFSETS];
+ int blk_idx;
+
+ /* prepare the offset array */
+ offs[0] = FirstOffsetNumber;
+ offs[1] = FirstOffsetNumber + 1;
+ offs[2] = max_offset / 2;
+ offs[3] = max_offset - 1;
+ offs[4] = max_offset;
+
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
+
+ /* add tids */
+ for (int i = 0; i < TEST_TIDSTORE_NUM_BLOCKS; i++)
+ tidstore_add_tids(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
+
+ /* lookup test */
+ for (OffsetNumber off = FirstOffsetNumber ; off < max_offset; off++)
+ {
+ bool expect = false;
+ for (int i = 0; i < TEST_TIDSTORE_NUM_OFFSETS; i++)
+ {
+ if (offs[i] == off)
+ {
+ expect = true;
+ break;
+ }
+ }
+
+ check_tid(ts, 0, off, expect);
+ check_tid(ts, 2, off, false);
+ check_tid(ts, MaxBlockNumber - 2, off, false);
+ check_tid(ts, MaxBlockNumber, off, expect);
+ }
+
+ /* test the number of tids */
+ if (tidstore_num_tids(ts) != (TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS))
+ elog(ERROR, "tidstore_num_tids returned " UINT64_FORMAT ", expected %d",
+ tidstore_num_tids(ts),
+ TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS);
+
+ /* iteration test */
+ iter = tidstore_begin_iterate(ts);
+ blk_idx = 0;
+ while ((iter_result = tidstore_iterate_next(iter)) != NULL)
+ {
+ /* check the returned block number */
+ if (blks_sorted[blk_idx] != iter_result->blkno)
+ elog(ERROR, "tidstore_iterate_next returned block number %u, expected %u",
+ iter_result->blkno, blks_sorted[blk_idx]);
+
+ /* check the returned offset numbers */
+ if (TEST_TIDSTORE_NUM_OFFSETS != iter_result->num_offsets)
+ elog(ERROR, "tidstore_iterate_next returned %u offsets, expected %u",
+ iter_result->num_offsets, TEST_TIDSTORE_NUM_OFFSETS);
+
+ for (int i = 0; i < iter_result->num_offsets; i++)
+ {
+ if (offs[i] != iter_result->offsets[i])
+ elog(ERROR, "tidstore_iterate_next returned offset number %u on block %u, expected %u",
+ iter_result->offsets[i], iter_result->blkno, offs[i]);
+ }
+
+ blk_idx++;
+ }
+
+ if (blk_idx != TEST_TIDSTORE_NUM_BLOCKS)
+ elog(ERROR, "tidstore_iterate_next returned %d blocks, expected %d",
+ blk_idx, TEST_TIDSTORE_NUM_BLOCKS);
+
+ /* remove all tids */
+ tidstore_reset(ts);
+
+ /* test the number of tids */
+ if (tidstore_num_tids(ts) != 0)
+ elog(ERROR, "tidstore_num_tids on empty store returned non-zero");
+
+ /* lookup test for empty store */
+ for (OffsetNumber off = FirstOffsetNumber ; off < MaxHeapTuplesPerPage;
+ off++)
+ {
+ check_tid(ts, 0, off, false);
+ check_tid(ts, 2, off, false);
+ check_tid(ts, MaxBlockNumber - 2, off, false);
+ check_tid(ts, MaxBlockNumber, off, false);
+ }
+
+ tidstore_destroy(ts);
+}
+
+static void
+test_empty(void)
+{
+ TidStore *ts;
+ TidStoreIter *iter;
+ ItemPointerData tid;
+
+ elog(NOTICE, "testing empty tidstore");
+
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
+
+ ItemPointerSet(&tid, 0, FirstOffsetNumber);
+ if (tidstore_lookup_tid(ts, &tid))
+ elog(ERROR, "tidstore_lookup_tid for (0,1) on empty store returned true");
+
+ ItemPointerSet(&tid, MaxBlockNumber, MaxOffsetNumber);
+ if (tidstore_lookup_tid(ts, &tid))
+ elog(ERROR, "tidstore_lookup_tid for (%u,%u) on empty store returned true",
+ MaxBlockNumber, MaxOffsetNumber);
+
+ if (tidstore_num_tids(ts) != 0)
+ elog(ERROR, "tidstore_num_entries on empty store returned non-zero");
+
+ if (tidstore_is_full(ts))
+ elog(ERROR, "tidstore_is_full on empty store returned true");
+
+ iter = tidstore_begin_iterate(ts);
+
+ if (tidstore_iterate_next(iter) != NULL)
+ elog(ERROR, "tidstore_iterate_next on empty store returned TIDs");
+
+ tidstore_end_iterate(iter);
+
+ tidstore_destroy(ts);
+}
+
+Datum
+test_tidstore(PG_FUNCTION_ARGS)
+{
+ test_empty();
+
+ elog(NOTICE, "testing basic operations");
+ test_basic(MaxHeapTuplesPerPage);
+ test_basic(10);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_tidstore/test_tidstore.control b/src/test/modules/test_tidstore/test_tidstore.control
new file mode 100644
index 0000000000..9b6bd4638f
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.control
@@ -0,0 +1,4 @@
+comment = 'Test code for tidstore'
+default_version = '1.0'
+module_pathname = '$libdir/test_tidstore'
+relocatable = true
--
2.39.1
[text/x-patch] v25-0009-Use-TIDStore-for-storing-dead-tuple-TID-during-l.patch (48.1K, ../../CAFBsxsHbBm_M22gLBO+AZT4mfMq3L_oX3wdKZxjeNnT7fHsYMQ@mail.gmail.com/12-v25-0009-Use-TIDStore-for-storing-dead-tuple-TID-during-l.patch)
download | inline diff:
From dcbcf6cdd786f9debf1536ac73093107debfafe8 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Tue, 17 Jan 2023 17:20:37 +0700
Subject: [PATCH v25 9/9] Use TIDStore for storing dead tuple TID during lazy
vacuum
Previously, we used an array of ItemPointerData to store dead tuple
TIDs, which was not space efficient and slow to lookup. Also, we had
the 1GB limit on its size.
Now we use TIDStore to store dead tuple TIDs. Since the TIDStore,
backed by the radix tree, incrementaly allocates the memory, we get
rid of the 1GB limit.
Since we are no longer able to exactly estimate the maximum number of
TIDs can be stored the pg_stat_progress_vacuum shows the progress
information based on the amount of memory in bytes. The column names
are also changed to max_dead_tuple_bytes and num_dead_tuple_bytes.
In addition, since the TIDStore use the radix tree internally, the
minimum amount of memory required by TIDStore is 1MB, the inital DSA
segment size. Due to that, we increase the minimum value of
maintenance_work_mem (also autovacuum_work_mem) from 1MB to 2MB.
XXX: needs to bump catalog version
---
doc/src/sgml/monitoring.sgml | 8 +-
src/backend/access/heap/vacuumlazy.c | 278 ++++++++-------------
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/vacuum.c | 78 +-----
src/backend/commands/vacuumparallel.c | 73 +++---
src/backend/postmaster/autovacuum.c | 6 +-
src/backend/storage/lmgr/lwlock.c | 2 +
src/backend/utils/misc/guc_tables.c | 2 +-
src/include/commands/progress.h | 4 +-
src/include/commands/vacuum.h | 25 +-
src/include/storage/lwlock.h | 1 +
src/test/regress/expected/cluster.out | 2 +-
src/test/regress/expected/create_index.out | 2 +-
src/test/regress/expected/rules.out | 4 +-
src/test/regress/sql/cluster.sql | 2 +-
src/test/regress/sql/create_index.sql | 2 +-
16 files changed, 177 insertions(+), 314 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index d936aa3da3..0230c74e3d 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6870,10 +6870,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>max_dead_tuples</structfield> <type>bigint</type>
+ <structfield>max_dead_tuple_bytes</structfield> <type>bigint</type>
</para>
<para>
- Number of dead tuples that we can store before needing to perform
+ Amount of dead tuple data that we can store before needing to perform
an index vacuum cycle, based on
<xref linkend="guc-maintenance-work-mem"/>.
</para></entry>
@@ -6881,10 +6881,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>num_dead_tuples</structfield> <type>bigint</type>
+ <structfield>num_dead_tuple_bytes</structfield> <type>bigint</type>
</para>
<para>
- Number of dead tuples collected since the last index vacuum cycle.
+ Amount of dead tuple data collected since the last index vacuum cycle.
</para></entry>
</row>
</tbody>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 8f14cf85f3..b4e40423a8 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3,18 +3,18 @@
* vacuumlazy.c
* Concurrent ("lazy") vacuuming.
*
- * The major space usage for vacuuming is storage for the array of dead TIDs
+ * The major space usage for vacuuming is TidStore, a storage for dead TIDs
* that are to be removed from indexes. We want to ensure we can vacuum even
* the very largest relations with finite memory space usage. To do that, we
- * set upper bounds on the number of TIDs we can keep track of at once.
+ * set upper bounds on the maximum memory that can be used for keeping track
+ * of dead TIDs at once.
*
* We are willing to use at most maintenance_work_mem (or perhaps
* autovacuum_work_mem) memory space to keep track of dead TIDs. We initially
- * allocate an array of TIDs of that size, with an upper limit that depends on
- * table size (this limit ensures we don't allocate a huge area uselessly for
- * vacuuming small tables). If the array threatens to overflow, we must call
- * lazy_vacuum to vacuum indexes (and to vacuum the pages that we've pruned).
- * This frees up the memory space dedicated to storing dead TIDs.
+ * create a TidStore with the maximum bytes that can be used by the TidStore.
+ * If the TidStore is full, we must call lazy_vacuum to vacuum indexes (and to
+ * vacuum the pages that we've pruned). This frees up the memory space dedicated
+ * to storing dead TIDs.
*
* In practice VACUUM will often complete its initial pass over the target
* heap relation without ever running out of space to store TIDs. This means
@@ -40,6 +40,7 @@
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
#include "access/multixact.h"
+#include "access/tidstore.h"
#include "access/transam.h"
#include "access/visibilitymap.h"
#include "access/xact.h"
@@ -188,7 +189,7 @@ typedef struct LVRelState
* lazy_vacuum_heap_rel, which marks the same LP_DEAD line pointers as
* LP_UNUSED during second heap pass.
*/
- VacDeadItems *dead_items; /* TIDs whose index tuples we'll delete */
+ TidStore *dead_items; /* TIDs whose index tuples we'll delete */
BlockNumber rel_pages; /* total number of pages */
BlockNumber scanned_pages; /* # pages examined (not skipped via VM) */
BlockNumber removed_pages; /* # pages removed by relation truncation */
@@ -220,11 +221,14 @@ typedef struct LVRelState
typedef struct LVPagePruneState
{
bool hastup; /* Page prevents rel truncation? */
- bool has_lpdead_items; /* includes existing LP_DEAD items */
+
+ /* collected offsets of LP_DEAD items including existing ones */
+ OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
+ int num_offsets;
/*
* State describes the proper VM bit states to set for the page following
- * pruning and freezing. all_visible implies !has_lpdead_items, but don't
+ * pruning and freezing. all_visible implies num_offsets == 0, but don't
* trust all_frozen result unless all_visible is also set to true.
*/
bool all_visible; /* Every item visible to all? */
@@ -259,8 +263,9 @@ static bool lazy_scan_noprune(LVRelState *vacrel, Buffer buf,
static void lazy_vacuum(LVRelState *vacrel);
static bool lazy_vacuum_all_indexes(LVRelState *vacrel);
static void lazy_vacuum_heap_rel(LVRelState *vacrel);
-static int lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
- Buffer buffer, int index, Buffer vmbuffer);
+static void lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
+ OffsetNumber *offsets, int num_offsets,
+ Buffer buffer, Buffer vmbuffer);
static bool lazy_check_wraparound_failsafe(LVRelState *vacrel);
static void lazy_cleanup_all_indexes(LVRelState *vacrel);
static IndexBulkDeleteResult *lazy_vacuum_one_index(Relation indrel,
@@ -487,11 +492,11 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
}
/*
- * Allocate dead_items array memory using dead_items_alloc. This handles
- * parallel VACUUM initialization as part of allocating shared memory
- * space used for dead_items. (But do a failsafe precheck first, to
- * ensure that parallel VACUUM won't be attempted at all when relfrozenxid
- * is already dangerously old.)
+ * Allocate dead_items memory using dead_items_alloc. This handles parallel
+ * VACUUM initialization as part of allocating shared memory space used for
+ * dead_items. (But do a failsafe precheck first, to ensure that parallel
+ * VACUUM won't be attempted at all when relfrozenxid is already dangerously
+ * old.)
*/
lazy_check_wraparound_failsafe(vacrel);
dead_items_alloc(vacrel, params->nworkers);
@@ -797,7 +802,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* have collected the TIDs whose index tuples need to be removed.
*
* Finally, invokes lazy_vacuum_heap_rel to vacuum heap pages, which
- * largely consists of marking LP_DEAD items (from collected TID array)
+ * largely consists of marking LP_DEAD items (from vacrel->dead_items)
* as LP_UNUSED. This has to happen in a second, final pass over the
* heap, to preserve a basic invariant that all index AMs rely on: no
* extant index tuple can ever be allowed to contain a TID that points to
@@ -825,21 +830,21 @@ lazy_scan_heap(LVRelState *vacrel)
blkno,
next_unskippable_block,
next_fsm_block_to_vacuum = 0;
- VacDeadItems *dead_items = vacrel->dead_items;
+ TidStore *dead_items = vacrel->dead_items;
Buffer vmbuffer = InvalidBuffer;
bool next_unskippable_allvis,
skipping_current_range;
const int initprog_index[] = {
PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_TOTAL_HEAP_BLKS,
- PROGRESS_VACUUM_MAX_DEAD_TUPLES
+ PROGRESS_VACUUM_MAX_DEAD_TUPLE_BYTES
};
int64 initprog_val[3];
/* Report that we're scanning the heap, advertising total # of blocks */
initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
initprog_val[1] = rel_pages;
- initprog_val[2] = dead_items->max_items;
+ initprog_val[2] = tidstore_max_memory(vacrel->dead_items);
pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
/* Set up an initial range of skippable blocks using the visibility map */
@@ -906,8 +911,7 @@ lazy_scan_heap(LVRelState *vacrel)
* dead_items TIDs, pause and do a cycle of vacuuming before we tackle
* this page.
*/
- Assert(dead_items->max_items >= MaxHeapTuplesPerPage);
- if (dead_items->max_items - dead_items->num_items < MaxHeapTuplesPerPage)
+ if (tidstore_is_full(vacrel->dead_items))
{
/*
* Before beginning index vacuuming, we release any pin we may
@@ -969,7 +973,7 @@ lazy_scan_heap(LVRelState *vacrel)
continue;
}
- /* Collect LP_DEAD items in dead_items array, count tuples */
+ /* Collect LP_DEAD items in dead_items, count tuples */
if (lazy_scan_noprune(vacrel, buf, blkno, page, &hastup,
&recordfreespace))
{
@@ -1011,14 +1015,14 @@ lazy_scan_heap(LVRelState *vacrel)
* Prune, freeze, and count tuples.
*
* Accumulates details of remaining LP_DEAD line pointers on page in
- * dead_items array. This includes LP_DEAD line pointers that we
- * pruned ourselves, as well as existing LP_DEAD line pointers that
- * were pruned some time earlier. Also considers freezing XIDs in the
- * tuple headers of remaining items with storage.
+ * dead_items. This includes LP_DEAD line pointers that we pruned
+ * ourselves, as well as existing LP_DEAD line pointers that were pruned
+ * some time earlier. Also considers freezing XIDs in the tuple headers
+ * of remaining items with storage.
*/
lazy_scan_prune(vacrel, buf, blkno, page, &prunestate);
- Assert(!prunestate.all_visible || !prunestate.has_lpdead_items);
+ Assert(!prunestate.all_visible || (prunestate.num_offsets == 0));
/* Remember the location of the last page with nonremovable tuples */
if (prunestate.hastup)
@@ -1034,14 +1038,12 @@ lazy_scan_heap(LVRelState *vacrel)
* performed here can be thought of as the one-pass equivalent of
* a call to lazy_vacuum().
*/
- if (prunestate.has_lpdead_items)
+ if (prunestate.num_offsets > 0)
{
Size freespace;
- lazy_vacuum_heap_page(vacrel, blkno, buf, 0, vmbuffer);
-
- /* Forget the LP_DEAD items that we just vacuumed */
- dead_items->num_items = 0;
+ lazy_vacuum_heap_page(vacrel, blkno, prunestate.deadoffsets,
+ prunestate.num_offsets, buf, vmbuffer);
/*
* Periodically perform FSM vacuuming to make newly-freed
@@ -1078,7 +1080,16 @@ lazy_scan_heap(LVRelState *vacrel)
* with prunestate-driven visibility map and FSM steps (just like
* the two-pass strategy).
*/
- Assert(dead_items->num_items == 0);
+ Assert(tidstore_num_tids(dead_items) == 0);
+ }
+ else if (prunestate.num_offsets > 0)
+ {
+ /* Save details of the LP_DEAD items from the page in dead_items */
+ tidstore_add_tids(dead_items, blkno, prunestate.deadoffsets,
+ prunestate.num_offsets);
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
+ tidstore_memory_usage(dead_items));
}
/*
@@ -1145,7 +1156,7 @@ lazy_scan_heap(LVRelState *vacrel)
* There should never be LP_DEAD items on a page with PD_ALL_VISIBLE
* set, however.
*/
- else if (prunestate.has_lpdead_items && PageIsAllVisible(page))
+ else if ((prunestate.num_offsets > 0) && PageIsAllVisible(page))
{
elog(WARNING, "page containing LP_DEAD items is marked as all-visible in relation \"%s\" page %u",
vacrel->relname, blkno);
@@ -1193,7 +1204,7 @@ lazy_scan_heap(LVRelState *vacrel)
* Final steps for block: drop cleanup lock, record free space in the
* FSM
*/
- if (prunestate.has_lpdead_items && vacrel->do_index_vacuuming)
+ if ((prunestate.num_offsets > 0) && vacrel->do_index_vacuuming)
{
/*
* Wait until lazy_vacuum_heap_rel() to save free space. This
@@ -1249,7 +1260,7 @@ lazy_scan_heap(LVRelState *vacrel)
* Do index vacuuming (call each index's ambulkdelete routine), then do
* related heap vacuuming
*/
- if (dead_items->num_items > 0)
+ if (tidstore_num_tids(dead_items) > 0)
lazy_vacuum(vacrel);
/*
@@ -1524,9 +1535,9 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* The approach we take now is to restart pruning when the race condition is
* detected. This allows heap_page_prune() to prune the tuples inserted by
* the now-aborted transaction. This is a little crude, but it guarantees
- * that any items that make it into the dead_items array are simple LP_DEAD
- * line pointers, and that every remaining item with tuple storage is
- * considered as a candidate for freezing.
+ * that any items that make it into the dead_items are simple LP_DEAD line
+ * pointers, and that every remaining item with tuple storage is considered
+ * as a candidate for freezing.
*/
static void
lazy_scan_prune(LVRelState *vacrel,
@@ -1543,13 +1554,11 @@ lazy_scan_prune(LVRelState *vacrel,
HTSV_Result res;
int tuples_deleted,
tuples_frozen,
- lpdead_items,
live_tuples,
recently_dead_tuples;
int nnewlpdead;
HeapPageFreeze pagefrz;
int64 fpi_before = pgWalUsage.wal_fpi;
- OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
@@ -1571,7 +1580,6 @@ retry:
pagefrz.NoFreezePageRelminMxid = vacrel->NewRelminMxid;
tuples_deleted = 0;
tuples_frozen = 0;
- lpdead_items = 0;
live_tuples = 0;
recently_dead_tuples = 0;
@@ -1580,9 +1588,9 @@ retry:
*
* We count tuples removed by the pruning step as tuples_deleted. Its
* final value can be thought of as the number of tuples that have been
- * deleted from the table. It should not be confused with lpdead_items;
- * lpdead_items's final value can be thought of as the number of tuples
- * that were deleted from indexes.
+ * deleted from the table. It should not be confused with
+ * prunestate->deadoffsets; prunestate->deadoffsets's final value can
+ * be thought of as the number of tuples that were deleted from indexes.
*/
tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest,
InvalidTransactionId, 0, &nnewlpdead,
@@ -1593,7 +1601,7 @@ retry:
* requiring freezing among remaining tuples with storage
*/
prunestate->hastup = false;
- prunestate->has_lpdead_items = false;
+ prunestate->num_offsets = 0;
prunestate->all_visible = true;
prunestate->all_frozen = true;
prunestate->visibility_cutoff_xid = InvalidTransactionId;
@@ -1638,7 +1646,7 @@ retry:
* (This is another case where it's useful to anticipate that any
* LP_DEAD items will become LP_UNUSED during the ongoing VACUUM.)
*/
- deadoffsets[lpdead_items++] = offnum;
+ prunestate->deadoffsets[prunestate->num_offsets++] = offnum;
continue;
}
@@ -1875,7 +1883,7 @@ retry:
*/
#ifdef USE_ASSERT_CHECKING
/* Note that all_frozen value does not matter when !all_visible */
- if (prunestate->all_visible && lpdead_items == 0)
+ if (prunestate->all_visible && prunestate->num_offsets == 0)
{
TransactionId cutoff;
bool all_frozen;
@@ -1888,28 +1896,9 @@ retry:
}
#endif
- /*
- * Now save details of the LP_DEAD items from the page in vacrel
- */
- if (lpdead_items > 0)
+ if (prunestate->num_offsets > 0)
{
- VacDeadItems *dead_items = vacrel->dead_items;
- ItemPointerData tmp;
-
vacrel->lpdead_item_pages++;
- prunestate->has_lpdead_items = true;
-
- ItemPointerSetBlockNumber(&tmp, blkno);
-
- for (int i = 0; i < lpdead_items; i++)
- {
- ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
- dead_items->items[dead_items->num_items++] = tmp;
- }
-
- Assert(dead_items->num_items <= dead_items->max_items);
- pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
- dead_items->num_items);
/*
* It was convenient to ignore LP_DEAD items in all_visible earlier on
@@ -1928,7 +1917,7 @@ retry:
/* Finally, add page-local counts to whole-VACUUM counts */
vacrel->tuples_deleted += tuples_deleted;
vacrel->tuples_frozen += tuples_frozen;
- vacrel->lpdead_items += lpdead_items;
+ vacrel->lpdead_items += prunestate->num_offsets;
vacrel->live_tuples += live_tuples;
vacrel->recently_dead_tuples += recently_dead_tuples;
}
@@ -1940,7 +1929,7 @@ retry:
* lazy_scan_prune, which requires a full cleanup lock. While pruning isn't
* performed here, it's quite possible that an earlier opportunistic pruning
* operation left LP_DEAD items behind. We'll at least collect any such items
- * in the dead_items array for removal from indexes.
+ * in the dead_items for removal from indexes.
*
* For aggressive VACUUM callers, we may return false to indicate that a full
* cleanup lock is required for processing by lazy_scan_prune. This is only
@@ -2099,7 +2088,7 @@ lazy_scan_noprune(LVRelState *vacrel,
vacrel->NewRelfrozenXid = NoFreezePageRelfrozenXid;
vacrel->NewRelminMxid = NoFreezePageRelminMxid;
- /* Save any LP_DEAD items found on the page in dead_items array */
+ /* Save any LP_DEAD items found on the page in dead_items */
if (vacrel->nindexes == 0)
{
/* Using one-pass strategy (since table has no indexes) */
@@ -2129,8 +2118,7 @@ lazy_scan_noprune(LVRelState *vacrel,
}
else
{
- VacDeadItems *dead_items = vacrel->dead_items;
- ItemPointerData tmp;
+ TidStore *dead_items = vacrel->dead_items;
/*
* Page has LP_DEAD items, and so any references/TIDs that remain in
@@ -2139,17 +2127,10 @@ lazy_scan_noprune(LVRelState *vacrel,
*/
vacrel->lpdead_item_pages++;
- ItemPointerSetBlockNumber(&tmp, blkno);
+ tidstore_add_tids(dead_items, blkno, deadoffsets, lpdead_items);
- for (int i = 0; i < lpdead_items; i++)
- {
- ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
- dead_items->items[dead_items->num_items++] = tmp;
- }
-
- Assert(dead_items->num_items <= dead_items->max_items);
- pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
- dead_items->num_items);
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
+ tidstore_memory_usage(dead_items));
vacrel->lpdead_items += lpdead_items;
@@ -2198,7 +2179,7 @@ lazy_vacuum(LVRelState *vacrel)
if (!vacrel->do_index_vacuuming)
{
Assert(!vacrel->do_index_cleanup);
- vacrel->dead_items->num_items = 0;
+ tidstore_reset(vacrel->dead_items);
return;
}
@@ -2227,7 +2208,7 @@ lazy_vacuum(LVRelState *vacrel)
BlockNumber threshold;
Assert(vacrel->num_index_scans == 0);
- Assert(vacrel->lpdead_items == vacrel->dead_items->num_items);
+ Assert(vacrel->lpdead_items == tidstore_num_tids(vacrel->dead_items));
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2254,8 +2235,8 @@ lazy_vacuum(LVRelState *vacrel)
* cases then this may need to be reconsidered.
*/
threshold = (double) vacrel->rel_pages * BYPASS_THRESHOLD_PAGES;
- bypass = (vacrel->lpdead_item_pages < threshold &&
- vacrel->lpdead_items < MAXDEADITEMS(32L * 1024L * 1024L));
+ bypass = (vacrel->lpdead_item_pages < threshold) &&
+ tidstore_memory_usage(vacrel->dead_items) < (32L * 1024L * 1024L);
}
if (bypass)
@@ -2300,7 +2281,7 @@ lazy_vacuum(LVRelState *vacrel)
* Forget the LP_DEAD items that we just vacuumed (or just decided to not
* vacuum)
*/
- vacrel->dead_items->num_items = 0;
+ tidstore_reset(vacrel->dead_items);
}
/*
@@ -2373,7 +2354,7 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
* place).
*/
Assert(vacrel->num_index_scans > 0 ||
- vacrel->dead_items->num_items == vacrel->lpdead_items);
+ tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items);
Assert(allindexes || vacrel->failsafe_active);
/*
@@ -2392,9 +2373,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
/*
* lazy_vacuum_heap_rel() -- second pass over the heap for two pass strategy
*
- * This routine marks LP_DEAD items in vacrel->dead_items array as LP_UNUSED.
- * Pages that never had lazy_scan_prune record LP_DEAD items are not visited
- * at all.
+ * This routine marks LP_DEAD items in vacrel->dead_items as LP_UNUSED. Pages
+ * that never had lazy_scan_prune record LP_DEAD items are not visited at all.
*
* We may also be able to truncate the line pointer array of the heap pages we
* visit. If there is a contiguous group of LP_UNUSED items at the end of the
@@ -2410,10 +2390,11 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
static void
lazy_vacuum_heap_rel(LVRelState *vacrel)
{
- int index = 0;
BlockNumber vacuumed_pages = 0;
Buffer vmbuffer = InvalidBuffer;
LVSavedErrInfo saved_err_info;
+ TidStoreIter *iter;
+ TidStoreIterResult *result;
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2428,7 +2409,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
VACUUM_ERRCB_PHASE_VACUUM_HEAP,
InvalidBlockNumber, InvalidOffsetNumber);
- while (index < vacrel->dead_items->num_items)
+ iter = tidstore_begin_iterate(vacrel->dead_items);
+ while ((result = tidstore_iterate_next(iter)) != NULL)
{
BlockNumber blkno;
Buffer buf;
@@ -2437,7 +2419,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
vacuum_delay_point();
- blkno = ItemPointerGetBlockNumber(&vacrel->dead_items->items[index]);
+ blkno = result->blkno;
vacrel->blkno = blkno;
/*
@@ -2451,7 +2433,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
buf = ReadBufferExtended(vacrel->rel, MAIN_FORKNUM, blkno, RBM_NORMAL,
vacrel->bstrategy);
LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
- index = lazy_vacuum_heap_page(vacrel, blkno, buf, index, vmbuffer);
+ lazy_vacuum_heap_page(vacrel, blkno, result->offsets, result->num_offsets,
+ buf, vmbuffer);
/* Now that we've vacuumed the page, record its available space */
page = BufferGetPage(buf);
@@ -2461,6 +2444,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
vacuumed_pages++;
}
+ tidstore_end_iterate(iter);
vacrel->blkno = InvalidBlockNumber;
if (BufferIsValid(vmbuffer))
@@ -2470,36 +2454,31 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
* We set all LP_DEAD items from the first heap pass to LP_UNUSED during
* the second heap pass. No more, no less.
*/
- Assert(index > 0);
Assert(vacrel->num_index_scans > 1 ||
- (index == vacrel->lpdead_items &&
+ (tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items &&
vacuumed_pages == vacrel->lpdead_item_pages));
ereport(DEBUG2,
- (errmsg("table \"%s\": removed %lld dead item identifiers in %u pages",
- vacrel->relname, (long long) index, vacuumed_pages)));
+ (errmsg("table \"%s\": removed " UINT64_FORMAT "dead item identifiers in %u pages",
+ vacrel->relname, tidstore_num_tids(vacrel->dead_items),
+ vacuumed_pages)));
/* Revert to the previous phase information for error traceback */
restore_vacuum_error_info(vacrel, &saved_err_info);
}
/*
- * lazy_vacuum_heap_page() -- free page's LP_DEAD items listed in the
- * vacrel->dead_items array.
+ * lazy_vacuum_heap_page() -- free page's LP_DEAD items.
*
* Caller must have an exclusive buffer lock on the buffer (though a full
* cleanup lock is also acceptable). vmbuffer must be valid and already have
* a pin on blkno's visibility map page.
- *
- * index is an offset into the vacrel->dead_items array for the first listed
- * LP_DEAD item on the page. The return value is the first index immediately
- * after all LP_DEAD items for the same page in the array.
*/
-static int
-lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
- int index, Buffer vmbuffer)
+static void
+lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
+ OffsetNumber *deadoffsets, int num_offsets, Buffer buffer,
+ Buffer vmbuffer)
{
- VacDeadItems *dead_items = vacrel->dead_items;
Page page = BufferGetPage(buffer);
OffsetNumber unused[MaxHeapTuplesPerPage];
int nunused = 0;
@@ -2518,16 +2497,11 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
START_CRIT_SECTION();
- for (; index < dead_items->num_items; index++)
+ for (int i = 0; i < num_offsets; i++)
{
- BlockNumber tblk;
- OffsetNumber toff;
ItemId itemid;
+ OffsetNumber toff = deadoffsets[i];
- tblk = ItemPointerGetBlockNumber(&dead_items->items[index]);
- if (tblk != blkno)
- break; /* past end of tuples for this block */
- toff = ItemPointerGetOffsetNumber(&dead_items->items[index]);
itemid = PageGetItemId(page, toff);
Assert(ItemIdIsDead(itemid) && !ItemIdHasStorage(itemid));
@@ -2597,7 +2571,6 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
/* Revert to the previous phase information for error traceback */
restore_vacuum_error_info(vacrel, &saved_err_info);
- return index;
}
/*
@@ -2687,8 +2660,8 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
* lazy_vacuum_one_index() -- vacuum index relation.
*
* Delete all the index tuples containing a TID collected in
- * vacrel->dead_items array. Also update running statistics.
- * Exact details depend on index AM's ambulkdelete routine.
+ * vacrel->dead_items. Also update running statistics. Exact
+ * details depend on index AM's ambulkdelete routine.
*
* reltuples is the number of heap tuples to be passed to the
* bulkdelete callback. It's always assumed to be estimated.
@@ -3094,48 +3067,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
}
/*
- * Returns the number of dead TIDs that VACUUM should allocate space to
- * store, given a heap rel of size vacrel->rel_pages, and given current
- * maintenance_work_mem setting (or current autovacuum_work_mem setting,
- * when applicable).
- *
- * See the comments at the head of this file for rationale.
- */
-static int
-dead_items_max_items(LVRelState *vacrel)
-{
- int64 max_items;
- int vac_work_mem = IsAutoVacuumWorkerProcess() &&
- autovacuum_work_mem != -1 ?
- autovacuum_work_mem : maintenance_work_mem;
-
- if (vacrel->nindexes > 0)
- {
- BlockNumber rel_pages = vacrel->rel_pages;
-
- max_items = MAXDEADITEMS(vac_work_mem * 1024L);
- max_items = Min(max_items, INT_MAX);
- max_items = Min(max_items, MAXDEADITEMS(MaxAllocSize));
-
- /* curious coding here to ensure the multiplication can't overflow */
- if ((BlockNumber) (max_items / MaxHeapTuplesPerPage) > rel_pages)
- max_items = rel_pages * MaxHeapTuplesPerPage;
-
- /* stay sane if small maintenance_work_mem */
- max_items = Max(max_items, MaxHeapTuplesPerPage);
- }
- else
- {
- /* One-pass case only stores a single heap page's TIDs at a time */
- max_items = MaxHeapTuplesPerPage;
- }
-
- return (int) max_items;
-}
-
-/*
- * Allocate dead_items (either using palloc, or in dynamic shared memory).
- * Sets dead_items in vacrel for caller.
+ * Allocate a (local or shared) TidStore for storing dead TIDs. Sets dead_items
+ * in vacrel for caller.
*
* Also handles parallel initialization as part of allocating dead_items in
* DSM when required.
@@ -3143,11 +3076,9 @@ dead_items_max_items(LVRelState *vacrel)
static void
dead_items_alloc(LVRelState *vacrel, int nworkers)
{
- VacDeadItems *dead_items;
- int max_items;
-
- max_items = dead_items_max_items(vacrel);
- Assert(max_items >= MaxHeapTuplesPerPage);
+ int vac_work_mem = IsAutoVacuumWorkerProcess() &&
+ autovacuum_work_mem != -1 ?
+ autovacuum_work_mem * 1024L : maintenance_work_mem * 1024L;
/*
* Initialize state for a parallel vacuum. As of now, only one worker can
@@ -3174,7 +3105,7 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
else
vacrel->pvs = parallel_vacuum_init(vacrel->rel, vacrel->indrels,
vacrel->nindexes, nworkers,
- max_items,
+ vac_work_mem, MaxHeapTuplesPerPage,
vacrel->verbose ? INFO : DEBUG2,
vacrel->bstrategy);
@@ -3187,11 +3118,8 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
}
/* Serial VACUUM case */
- dead_items = (VacDeadItems *) palloc(vac_max_items_to_alloc_size(max_items));
- dead_items->max_items = max_items;
- dead_items->num_items = 0;
-
- vacrel->dead_items = dead_items;
+ vacrel->dead_items = tidstore_create(vac_work_mem, MaxHeapTuplesPerPage,
+ NULL);
}
/*
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 8608e3fa5b..a526e607fe 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1165,7 +1165,7 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuple_bytes, S.param7 AS dead_tuple_bytes
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 7b1a4b127e..d8e680ca20 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -97,7 +97,6 @@ static bool vacuum_rel(Oid relid, RangeVar *relation, VacuumParams *params,
static double compute_parallel_delay(void);
static VacOptValue get_vacoptval_from_boolean(DefElem *def);
static bool vac_tid_reaped(ItemPointer itemptr, void *state);
-static int vac_cmp_itemptr(const void *left, const void *right);
/*
* Primary entry point for manual VACUUM and ANALYZE commands
@@ -2303,16 +2302,16 @@ get_vacoptval_from_boolean(DefElem *def)
*/
IndexBulkDeleteResult *
vac_bulkdel_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat,
- VacDeadItems *dead_items)
+ TidStore *dead_items)
{
/* Do bulk deletion */
istat = index_bulk_delete(ivinfo, istat, vac_tid_reaped,
(void *) dead_items);
ereport(ivinfo->message_level,
- (errmsg("scanned index \"%s\" to remove %d row versions",
+ (errmsg("scanned index \"%s\" to remove " UINT64_FORMAT " row versions",
RelationGetRelationName(ivinfo->index),
- dead_items->num_items)));
+ tidstore_num_tids(dead_items))));
return istat;
}
@@ -2343,82 +2342,15 @@ vac_cleanup_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat)
return istat;
}
-/*
- * Returns the total required space for VACUUM's dead_items array given a
- * max_items value.
- */
-Size
-vac_max_items_to_alloc_size(int max_items)
-{
- Assert(max_items <= MAXDEADITEMS(MaxAllocSize));
-
- return offsetof(VacDeadItems, items) + sizeof(ItemPointerData) * max_items;
-}
-
/*
* vac_tid_reaped() -- is a particular tid deletable?
*
* This has the right signature to be an IndexBulkDeleteCallback.
- *
- * Assumes dead_items array is sorted (in ascending TID order).
*/
static bool
vac_tid_reaped(ItemPointer itemptr, void *state)
{
- VacDeadItems *dead_items = (VacDeadItems *) state;
- int64 litem,
- ritem,
- item;
- ItemPointer res;
-
- litem = itemptr_encode(&dead_items->items[0]);
- ritem = itemptr_encode(&dead_items->items[dead_items->num_items - 1]);
- item = itemptr_encode(itemptr);
-
- /*
- * Doing a simple bound check before bsearch() is useful to avoid the
- * extra cost of bsearch(), especially if dead items on the heap are
- * concentrated in a certain range. Since this function is called for
- * every index tuple, it pays to be really fast.
- */
- if (item < litem || item > ritem)
- return false;
-
- res = (ItemPointer) bsearch((void *) itemptr,
- (void *) dead_items->items,
- dead_items->num_items,
- sizeof(ItemPointerData),
- vac_cmp_itemptr);
-
- return (res != NULL);
-}
-
-/*
- * Comparator routines for use with qsort() and bsearch().
- */
-static int
-vac_cmp_itemptr(const void *left, const void *right)
-{
- BlockNumber lblk,
- rblk;
- OffsetNumber loff,
- roff;
-
- lblk = ItemPointerGetBlockNumber((ItemPointer) left);
- rblk = ItemPointerGetBlockNumber((ItemPointer) right);
-
- if (lblk < rblk)
- return -1;
- if (lblk > rblk)
- return 1;
-
- loff = ItemPointerGetOffsetNumber((ItemPointer) left);
- roff = ItemPointerGetOffsetNumber((ItemPointer) right);
-
- if (loff < roff)
- return -1;
- if (loff > roff)
- return 1;
+ TidStore *dead_items = (TidStore *) state;
- return 0;
+ return tidstore_lookup_tid(dead_items, itemptr);
}
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index bcd40c80a1..d653683693 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -9,12 +9,11 @@
* In a parallel vacuum, we perform both index bulk deletion and index cleanup
* with parallel worker processes. Individual indexes are processed by one
* vacuum process. ParalleVacuumState contains shared information as well as
- * the memory space for storing dead items allocated in the DSM segment. We
- * launch parallel worker processes at the start of parallel index
- * bulk-deletion and index cleanup and once all indexes are processed, the
- * parallel worker processes exit. Each time we process indexes in parallel,
- * the parallel context is re-initialized so that the same DSM can be used for
- * multiple passes of index bulk-deletion and index cleanup.
+ * the shared TidStore. We launch parallel worker processes at the start of
+ * parallel index bulk-deletion and index cleanup and once all indexes are
+ * processed, the parallel worker processes exit. Each time we process indexes
+ * in parallel, the parallel context is re-initialized so that the same DSM can
+ * be used for multiple passes of index bulk-deletion and index cleanup.
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -103,6 +102,9 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /* Handle of the shared TidStore */
+ tidstore_handle dead_items_handle;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -166,7 +168,8 @@ struct ParallelVacuumState
PVIndStats *indstats;
/* Shared dead items space among parallel vacuum workers */
- VacDeadItems *dead_items;
+ TidStore *dead_items;
+ dsa_area *dead_items_area;
/* Points to buffer usage area in DSM */
BufferUsage *buffer_usage;
@@ -222,20 +225,23 @@ static void parallel_vacuum_error_callback(void *arg);
*/
ParallelVacuumState *
parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
- int nrequested_workers, int max_items,
- int elevel, BufferAccessStrategy bstrategy)
+ int nrequested_workers, int vac_work_mem,
+ int max_offset, int elevel,
+ BufferAccessStrategy bstrategy)
{
ParallelVacuumState *pvs;
ParallelContext *pcxt;
PVShared *shared;
- VacDeadItems *dead_items;
+ TidStore *dead_items;
PVIndStats *indstats;
BufferUsage *buffer_usage;
WalUsage *wal_usage;
+ void *area_space;
+ dsa_area *dead_items_dsa;
bool *will_parallel_vacuum;
Size est_indstats_len;
Size est_shared_len;
- Size est_dead_items_len;
+ Size dsa_minsize = dsa_minimum_size();
int nindexes_mwm = 0;
int parallel_workers = 0;
int querylen;
@@ -283,9 +289,8 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_estimate_chunk(&pcxt->estimator, est_shared_len);
shm_toc_estimate_keys(&pcxt->estimator, 1);
- /* Estimate size for dead_items -- PARALLEL_VACUUM_KEY_DEAD_ITEMS */
- est_dead_items_len = vac_max_items_to_alloc_size(max_items);
- shm_toc_estimate_chunk(&pcxt->estimator, est_dead_items_len);
+ /* Estimate size for dead tuple DSA -- PARALLEL_VACUUM_KEY_DSA */
+ shm_toc_estimate_chunk(&pcxt->estimator, dsa_minsize);
shm_toc_estimate_keys(&pcxt->estimator, 1);
/*
@@ -351,6 +356,16 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_INDEX_STATS, indstats);
pvs->indstats = indstats;
+ /* Prepare DSA space for dead items */
+ area_space = shm_toc_allocate(pcxt->toc, dsa_minsize);
+ shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, area_space);
+ dead_items_dsa = dsa_create_in_place(area_space, dsa_minsize,
+ LWTRANCHE_PARALLEL_VACUUM_DSA,
+ pcxt->seg);
+ dead_items = tidstore_create(vac_work_mem, max_offset, dead_items_dsa);
+ pvs->dead_items = dead_items;
+ pvs->dead_items_area = dead_items_dsa;
+
/* Prepare shared information */
shared = (PVShared *) shm_toc_allocate(pcxt->toc, est_shared_len);
MemSet(shared, 0, est_shared_len);
@@ -360,6 +375,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
+ shared->dead_items_handle = tidstore_get_handle(dead_items);
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
@@ -368,15 +384,6 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_SHARED, shared);
pvs->shared = shared;
- /* Prepare the dead_items space */
- dead_items = (VacDeadItems *) shm_toc_allocate(pcxt->toc,
- est_dead_items_len);
- dead_items->max_items = max_items;
- dead_items->num_items = 0;
- MemSet(dead_items->items, 0, sizeof(ItemPointerData) * max_items);
- shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, dead_items);
- pvs->dead_items = dead_items;
-
/*
* Allocate space for each worker's BufferUsage and WalUsage; no need to
* initialize
@@ -434,6 +441,9 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
istats[i] = NULL;
}
+ tidstore_destroy(pvs->dead_items);
+ dsa_detach(pvs->dead_items_area);
+
DestroyParallelContext(pvs->pcxt);
ExitParallelMode();
@@ -442,7 +452,7 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
}
/* Returns the dead items space */
-VacDeadItems *
+TidStore *
parallel_vacuum_get_dead_items(ParallelVacuumState *pvs)
{
return pvs->dead_items;
@@ -940,7 +950,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
Relation *indrels;
PVIndStats *indstats;
PVShared *shared;
- VacDeadItems *dead_items;
+ TidStore *dead_items;
+ void *area_space;
+ dsa_area *dead_items_area;
BufferUsage *buffer_usage;
WalUsage *wal_usage;
int nindexes;
@@ -984,10 +996,10 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
PARALLEL_VACUUM_KEY_INDEX_STATS,
false);
- /* Set dead_items space */
- dead_items = (VacDeadItems *) shm_toc_lookup(toc,
- PARALLEL_VACUUM_KEY_DEAD_ITEMS,
- false);
+ /* Set dead items */
+ area_space = shm_toc_lookup(toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, false);
+ dead_items_area = dsa_attach_in_place(area_space, seg);
+ dead_items = tidstore_attach(dead_items_area, shared->dead_items_handle);
/* Set cost-based vacuum delay */
VacuumCostActive = (VacuumCostDelay > 0);
@@ -1033,6 +1045,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
InstrEndParallelQuery(&buffer_usage[ParallelWorkerNumber],
&wal_usage[ParallelWorkerNumber]);
+ tidstore_detach(pvs.dead_items);
+ dsa_detach(dead_items_area);
+
/* Pop the error context stack */
error_context_stack = errcallback.previous;
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index f5ea381c53..d88db3e1f8 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -3397,12 +3397,12 @@ check_autovacuum_work_mem(int *newval, void **extra, GucSource source)
return true;
/*
- * We clamp manually-set values to at least 1MB. Since
+ * We clamp manually-set values to at least 2MB. Since
* maintenance_work_mem is always set to at least this value, do the same
* here.
*/
- if (*newval < 1024)
- *newval = 1024;
+ if (*newval < 2048)
+ *newval = 2048;
return true;
}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 55b3a04097..c223a7dc94 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -192,6 +192,8 @@ static const char *const BuiltinTrancheNames[] = {
"LogicalRepLauncherDSA",
/* LWTRANCHE_LAUNCHER_HASH: */
"LogicalRepLauncherHash",
+ /* LWTRANCHE_PARALLEL_VACUUM_DSA: */
+ "ParallelVacuumDSA",
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index b46e3b8c55..27a88b9369 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2312,7 +2312,7 @@ struct config_int ConfigureNamesInt[] =
GUC_UNIT_KB
},
&maintenance_work_mem,
- 65536, 1024, MAX_KILOBYTES,
+ 65536, 2048, MAX_KILOBYTES,
NULL, NULL, NULL
},
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index e5add41352..b209d3cf84 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -23,8 +23,8 @@
#define PROGRESS_VACUUM_HEAP_BLKS_SCANNED 2
#define PROGRESS_VACUUM_HEAP_BLKS_VACUUMED 3
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
-#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
-#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_MAX_DEAD_TUPLE_BYTES 5
+#define PROGRESS_VACUUM_DEAD_TUPLE_BYTES 6
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 689dbb7702..a3ebb169ef 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -17,6 +17,7 @@
#include "access/htup.h"
#include "access/genam.h"
#include "access/parallel.h"
+#include "access/tidstore.h"
#include "catalog/pg_class.h"
#include "catalog/pg_statistic.h"
#include "catalog/pg_type.h"
@@ -276,21 +277,6 @@ struct VacuumCutoffs
MultiXactId MultiXactCutoff;
};
-/*
- * VacDeadItems stores TIDs whose index tuples are deleted by index vacuuming.
- */
-typedef struct VacDeadItems
-{
- int max_items; /* # slots allocated in array */
- int num_items; /* current # of entries */
-
- /* Sorted array of TIDs to delete from indexes */
- ItemPointerData items[FLEXIBLE_ARRAY_MEMBER];
-} VacDeadItems;
-
-#define MAXDEADITEMS(avail_mem) \
- (((avail_mem) - offsetof(VacDeadItems, items)) / sizeof(ItemPointerData))
-
/* GUC parameters */
extern PGDLLIMPORT int default_statistics_target; /* PGDLLIMPORT for PostGIS */
extern PGDLLIMPORT int vacuum_freeze_min_age;
@@ -339,18 +325,17 @@ extern Relation vacuum_open_relation(Oid relid, RangeVar *relation,
LOCKMODE lmode);
extern IndexBulkDeleteResult *vac_bulkdel_one_index(IndexVacuumInfo *ivinfo,
IndexBulkDeleteResult *istat,
- VacDeadItems *dead_items);
+ TidStore *dead_items);
extern IndexBulkDeleteResult *vac_cleanup_one_index(IndexVacuumInfo *ivinfo,
IndexBulkDeleteResult *istat);
-extern Size vac_max_items_to_alloc_size(int max_items);
/* in commands/vacuumparallel.c */
extern ParallelVacuumState *parallel_vacuum_init(Relation rel, Relation *indrels,
int nindexes, int nrequested_workers,
- int max_items, int elevel,
- BufferAccessStrategy bstrategy);
+ int vac_work_mem, int max_offset,
+ int elevel, BufferAccessStrategy bstrategy);
extern void parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats);
-extern VacDeadItems *parallel_vacuum_get_dead_items(ParallelVacuumState *pvs);
+extern TidStore *parallel_vacuum_get_dead_items(ParallelVacuumState *pvs);
extern void parallel_vacuum_bulkdel_all_indexes(ParallelVacuumState *pvs,
long num_table_tuples,
int num_index_scans);
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index 07002fdfbe..537b34b30c 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -207,6 +207,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_PGSTATS_DATA,
LWTRANCHE_LAUNCHER_DSA,
LWTRANCHE_LAUNCHER_HASH,
+ LWTRANCHE_PARALLEL_VACUUM_DSA,
LWTRANCHE_FIRST_USER_DEFINED
} BuiltinTrancheIds;
diff --git a/src/test/regress/expected/cluster.out b/src/test/regress/expected/cluster.out
index 2eec483eaa..e04f50726f 100644
--- a/src/test/regress/expected/cluster.out
+++ b/src/test/regress/expected/cluster.out
@@ -526,7 +526,7 @@ create index cluster_sort on clstr_4 (hundred, thousand, tenthous);
-- ensure we don't use the index in CLUSTER nor the checking SELECTs
set enable_indexscan = off;
-- Use external sort:
-set maintenance_work_mem = '1MB';
+set maintenance_work_mem = '2MB';
cluster clstr_4 using cluster_sort;
select * from
(select hundred, lag(hundred) over () as lhundred,
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 6cd57e3eaa..d1889b9d10 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -1214,7 +1214,7 @@ DROP TABLE unlogged_hash_table;
-- CREATE INDEX hash_ovfl_index ON hash_ovfl_heap USING hash (x int4_ops);
-- Test hash index build tuplesorting. Force hash tuplesort using low
-- maintenance_work_mem setting and fillfactor:
-SET maintenance_work_mem = '1MB';
+SET maintenance_work_mem = '2MB';
CREATE INDEX hash_tuplesort_idx ON tenk1 USING hash (stringu1 name_ops) WITH (fillfactor = 10);
EXPLAIN (COSTS OFF)
SELECT count(*) FROM tenk1 WHERE stringu1 = 'TVAAAA';
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index e7a2f5856a..f6ae02eb14 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2020,8 +2020,8 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param3 AS heap_blks_scanned,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
- s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param6 AS max_dead_tuple_bytes,
+ s.param7 AS dead_tuple_bytes
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_recovery_prefetch| SELECT stats_reset,
diff --git a/src/test/regress/sql/cluster.sql b/src/test/regress/sql/cluster.sql
index a4cfaae807..a4cb5b98a5 100644
--- a/src/test/regress/sql/cluster.sql
+++ b/src/test/regress/sql/cluster.sql
@@ -258,7 +258,7 @@ create index cluster_sort on clstr_4 (hundred, thousand, tenthous);
set enable_indexscan = off;
-- Use external sort:
-set maintenance_work_mem = '1MB';
+set maintenance_work_mem = '2MB';
cluster clstr_4 using cluster_sort;
select * from
(select hundred, lag(hundred) over () as lhundred,
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index a3738833b2..edb5e4b4f3 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -367,7 +367,7 @@ DROP TABLE unlogged_hash_table;
-- Test hash index build tuplesorting. Force hash tuplesort using low
-- maintenance_work_mem setting and fillfactor:
-SET maintenance_work_mem = '1MB';
+SET maintenance_work_mem = '2MB';
CREATE INDEX hash_tuplesort_idx ON tenk1 USING hash (stringu1 name_ops) WITH (fillfactor = 10);
EXPLAIN (COSTS OFF)
SELECT count(*) FROM tenk1 WHERE stringu1 = 'TVAAAA';
--
2.39.1
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-02-07 10:22 ` John Naylor <[email protected]>
1 sibling, 0 replies; 78+ messages in thread
From: John Naylor @ 2023-02-07 10:22 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Tue, Feb 7, 2023 at 4:25 PM John Naylor <[email protected]>
wrote:
> [v25]
This conflicted with a commit from earlier today, so rebased in v26 with no
further changes.
--
John Naylor
EDB: http://www.enterprisedb.com
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
index 4e00b46d9b..3f831227c9 100644
--- a/src/include/lib/radixtree_insert_impl.h
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -80,9 +80,10 @@
}
else
{
- int insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
+ int insertpos;// = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
int count = n3->base.n.count;
-
+#ifdef RT_MAINTAIN_ORDERING
+ insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
/* shift chunks and children */
if (insertpos < count)
{
@@ -95,6 +96,9 @@
count, insertpos);
#endif
}
+#else
+ insertpos = count;
+#endif /* order */
n3->base.chunks[insertpos] = chunk;
#ifdef RT_NODE_LEVEL_LEAF
@@ -186,8 +190,10 @@
}
else
{
- int insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
+ int insertpos;
int count = n32->base.n.count;
+#ifdef RT_MAINTAIN_ORDERING
+ insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
if (insertpos < count)
{
@@ -200,6 +206,9 @@
count, insertpos);
#endif
}
+#else
+ insertpos = count;
+#endif
n32->base.chunks[insertpos] = chunk;
#ifdef RT_NODE_LEVEL_LEAF
Attachments:
[text/plain] v25-addendum-try-no-maintain-order.txt (1.3K, ../../CAFBsxsE7FLxfc2bfPf-DwdJ6RmDcpUcNRyUnHeuR26LVi6jzGA@mail.gmail.com/3-v25-addendum-try-no-maintain-order.txt)
download | inline diff:
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
index 4e00b46d9b..3f831227c9 100644
--- a/src/include/lib/radixtree_insert_impl.h
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -80,9 +80,10 @@
}
else
{
- int insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
+ int insertpos;// = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
int count = n3->base.n.count;
-
+#ifdef RT_MAINTAIN_ORDERING
+ insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
/* shift chunks and children */
if (insertpos < count)
{
@@ -95,6 +96,9 @@
count, insertpos);
#endif
}
+#else
+ insertpos = count;
+#endif /* order */
n3->base.chunks[insertpos] = chunk;
#ifdef RT_NODE_LEVEL_LEAF
@@ -186,8 +190,10 @@
}
else
{
- int insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
+ int insertpos;
int count = n32->base.n.count;
+#ifdef RT_MAINTAIN_ORDERING
+ insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
if (insertpos < count)
{
@@ -200,6 +206,9 @@
count, insertpos);
#endif
}
+#else
+ insertpos = count;
+#endif
n32->base.chunks[insertpos] = chunk;
#ifdef RT_NODE_LEVEL_LEAF
[text/x-patch] v26-0002-Move-some-bitmap-logic-out-of-bitmapset.c.patch (6.1K, ../../CAFBsxsE7FLxfc2bfPf-DwdJ6RmDcpUcNRyUnHeuR26LVi6jzGA@mail.gmail.com/4-v26-0002-Move-some-bitmap-logic-out-of-bitmapset.c.patch)
download | inline diff:
From f6f476ba71864821cb5144f513165671c64db1b2 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 6 Dec 2022 13:39:41 +0700
Subject: [PATCH v26 2/9] Move some bitmap logic out of bitmapset.c
Add pg_rightmost_one32/64 functions. This functionality was previously
private to bitmapset.c as the RIGHTMOST_ONE macro. It has practical
use in other contexts, so move to pg_bitutils.h.
Also move the logic for selecting appropriate pg_bitutils functions
based on word size to bitmapset.h for wider visibility and add
appropriate selection for bmw_rightmost_one().
Since the previous macro relied on casting to signedbitmapword,
and the new functions do not, remove that typedef.
Design input and review by Tom Lane
Discussion: https://www.postgresql.org/message-id/CAFBsxsFW2JjTo58jtDB%2B3sZhxMx3t-3evew8%3DAcr%2BGGhC%2BkFaA%40mail.gmail.com
---
src/backend/nodes/bitmapset.c | 36 ++------------------------------
src/include/nodes/bitmapset.h | 16 ++++++++++++--
src/include/port/pg_bitutils.h | 31 +++++++++++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 -
4 files changed, 47 insertions(+), 37 deletions(-)
diff --git a/src/backend/nodes/bitmapset.c b/src/backend/nodes/bitmapset.c
index 98660524ad..fcd8e2ccbc 100644
--- a/src/backend/nodes/bitmapset.c
+++ b/src/backend/nodes/bitmapset.c
@@ -32,39 +32,7 @@
#define BITMAPSET_SIZE(nwords) \
(offsetof(Bitmapset, words) + (nwords) * sizeof(bitmapword))
-/*----------
- * This is a well-known cute trick for isolating the rightmost one-bit
- * in a word. It assumes two's complement arithmetic. Consider any
- * nonzero value, and focus attention on the rightmost one. The value is
- * then something like
- * xxxxxx10000
- * where x's are unspecified bits. The two's complement negative is formed
- * by inverting all the bits and adding one. Inversion gives
- * yyyyyy01111
- * where each y is the inverse of the corresponding x. Incrementing gives
- * yyyyyy10000
- * and then ANDing with the original value gives
- * 00000010000
- * This works for all cases except original value = zero, where of course
- * we get zero.
- *----------
- */
-#define RIGHTMOST_ONE(x) ((signedbitmapword) (x) & -((signedbitmapword) (x)))
-
-#define HAS_MULTIPLE_ONES(x) ((bitmapword) RIGHTMOST_ONE(x) != (x))
-
-/* Select appropriate bit-twiddling functions for bitmap word size */
-#if BITS_PER_BITMAPWORD == 32
-#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w)
-#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w)
-#define bmw_popcount(w) pg_popcount32(w)
-#elif BITS_PER_BITMAPWORD == 64
-#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w)
-#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w)
-#define bmw_popcount(w) pg_popcount64(w)
-#else
-#error "invalid BITS_PER_BITMAPWORD"
-#endif
+#define HAS_MULTIPLE_ONES(x) (bmw_rightmost_one(x) != (x))
/*
@@ -1013,7 +981,7 @@ bms_first_member(Bitmapset *a)
{
int result;
- w = RIGHTMOST_ONE(w);
+ w = bmw_rightmost_one(w);
a->words[wordnum] &= ~w;
result = wordnum * BITS_PER_BITMAPWORD;
diff --git a/src/include/nodes/bitmapset.h b/src/include/nodes/bitmapset.h
index 3d2225e1ae..5f9a511b4a 100644
--- a/src/include/nodes/bitmapset.h
+++ b/src/include/nodes/bitmapset.h
@@ -38,13 +38,11 @@ struct List;
#define BITS_PER_BITMAPWORD 64
typedef uint64 bitmapword; /* must be an unsigned type */
-typedef int64 signedbitmapword; /* must be the matching signed type */
#else
#define BITS_PER_BITMAPWORD 32
typedef uint32 bitmapword; /* must be an unsigned type */
-typedef int32 signedbitmapword; /* must be the matching signed type */
#endif
@@ -75,6 +73,20 @@ typedef enum
BMS_MULTIPLE /* >1 member */
} BMS_Membership;
+/* Select appropriate bit-twiddling functions for bitmap word size */
+#if BITS_PER_BITMAPWORD == 32
+#define bmw_rightmost_one(w) pg_rightmost_one32(w)
+#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w)
+#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w)
+#define bmw_popcount(w) pg_popcount32(w)
+#elif BITS_PER_BITMAPWORD == 64
+#define bmw_rightmost_one(w) pg_rightmost_one64(w)
+#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w)
+#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w)
+#define bmw_popcount(w) pg_popcount64(w)
+#else
+#error "invalid BITS_PER_BITMAPWORD"
+#endif
/*
* function prototypes in nodes/bitmapset.c
diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h
index a49a9c03d9..7235ad25ee 100644
--- a/src/include/port/pg_bitutils.h
+++ b/src/include/port/pg_bitutils.h
@@ -17,6 +17,37 @@ extern PGDLLIMPORT const uint8 pg_leftmost_one_pos[256];
extern PGDLLIMPORT const uint8 pg_rightmost_one_pos[256];
extern PGDLLIMPORT const uint8 pg_number_of_ones[256];
+/*----------
+ * This is a well-known cute trick for isolating the rightmost one-bit
+ * in a word. It assumes two's complement arithmetic. Consider any
+ * nonzero value, and focus attention on the rightmost one. The value is
+ * then something like
+ * xxxxxx10000
+ * where x's are unspecified bits. The two's complement negative is formed
+ * by inverting all the bits and adding one. Inversion gives
+ * yyyyyy01111
+ * where each y is the inverse of the corresponding x. Incrementing gives
+ * yyyyyy10000
+ * and then ANDing with the original value gives
+ * 00000010000
+ * This works for all cases except original value = zero, where of course
+ * we get zero.
+ *----------
+ */
+static inline uint32
+pg_rightmost_one32(uint32 word)
+{
+ int32 result = (int32) word & -((int32) word);
+ return (uint32) result;
+}
+
+static inline uint64
+pg_rightmost_one64(uint64 word)
+{
+ int64 result = (int64) word & -((int64) word);
+ return (uint64) result;
+}
+
/*
* pg_leftmost_one_pos32
* Returns the position of the most significant set bit in "word",
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 07fbb7ccf6..f4d1d60cd2 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3662,7 +3662,6 @@ shmem_request_hook_type
shmem_startup_hook_type
sig_atomic_t
sigjmp_buf
-signedbitmapword
sigset_t
size_t
slist_head
--
2.39.1
[text/x-patch] v26-0001-Introduce-helper-SIMD-functions-for-small-byte-a.patch (2.9K, ../../CAFBsxsE7FLxfc2bfPf-DwdJ6RmDcpUcNRyUnHeuR26LVi6jzGA@mail.gmail.com/5-v26-0001-Introduce-helper-SIMD-functions-for-small-byte-a.patch)
download | inline diff:
From cf3e16ed894fc0c6574c48eddad7c587e5dec688 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 24 Oct 2022 14:07:09 +0900
Subject: [PATCH v26 1/9] Introduce helper SIMD functions for small byte arrays
vector8_min - helper for emulating ">=" semantics
vector8_highbit_mask - used to turn the result of a vector
comparison into a bitmask
Masahiko Sawada
Reviewed by Nathan Bossart, additional adjustments by me
Discussion: https://www.postgresql.org/message-id/CAD21AoDap240WDDdUDE0JMpCmuMMnGajrKrkCRxM7zn9Xk3JRA%40mail.gmail.com
---
src/include/port/simd.h | 47 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)
diff --git a/src/include/port/simd.h b/src/include/port/simd.h
index c836360d4b..350e2caaea 100644
--- a/src/include/port/simd.h
+++ b/src/include/port/simd.h
@@ -79,6 +79,7 @@ static inline bool vector8_has_le(const Vector8 v, const uint8 c);
static inline bool vector8_is_highbit_set(const Vector8 v);
#ifndef USE_NO_SIMD
static inline bool vector32_is_highbit_set(const Vector32 v);
+static inline uint32 vector8_highbit_mask(const Vector8 v);
#endif
/* arithmetic operations */
@@ -96,6 +97,7 @@ static inline Vector8 vector8_ssub(const Vector8 v1, const Vector8 v2);
*/
#ifndef USE_NO_SIMD
static inline Vector8 vector8_eq(const Vector8 v1, const Vector8 v2);
+static inline Vector8 vector8_min(const Vector8 v1, const Vector8 v2);
static inline Vector32 vector32_eq(const Vector32 v1, const Vector32 v2);
#endif
@@ -299,6 +301,36 @@ vector32_is_highbit_set(const Vector32 v)
}
#endif /* ! USE_NO_SIMD */
+/*
+ * Return a bitmask formed from the high-bit of each element.
+ */
+#ifndef USE_NO_SIMD
+static inline uint32
+vector8_highbit_mask(const Vector8 v)
+{
+#ifdef USE_SSE2
+ return (uint32) _mm_movemask_epi8(v);
+#elif defined(USE_NEON)
+ /*
+ * Note: There is a faster way to do this, but it returns a uint64 and
+ * and if the caller wanted to extract the bit position using CTZ,
+ * it would have to divide that result by 4.
+ */
+ static const uint8 mask[16] = {
+ 1 << 0, 1 << 1, 1 << 2, 1 << 3,
+ 1 << 4, 1 << 5, 1 << 6, 1 << 7,
+ 1 << 0, 1 << 1, 1 << 2, 1 << 3,
+ 1 << 4, 1 << 5, 1 << 6, 1 << 7,
+ };
+
+ uint8x16_t masked = vandq_u8(vld1q_u8(mask), (uint8x16_t) vshrq_n_s8(v, 7));
+ uint8x16_t maskedhi = vextq_u8(masked, masked, 8);
+
+ return (uint32) vaddvq_u16((uint16x8_t) vzip1q_u8(masked, maskedhi));
+#endif
+}
+#endif /* ! USE_NO_SIMD */
+
/*
* Return the bitwise OR of the inputs
*/
@@ -372,4 +404,19 @@ vector32_eq(const Vector32 v1, const Vector32 v2)
}
#endif /* ! USE_NO_SIMD */
+/*
+ * Given two vectors, return a vector with the minimum element of each.
+ */
+#ifndef USE_NO_SIMD
+static inline Vector8
+vector8_min(const Vector8 v1, const Vector8 v2)
+{
+#ifdef USE_SSE2
+ return _mm_min_epu8(v1, v2);
+#elif defined(USE_NEON)
+ return vminq_u8(v1, v2);
+#endif
+}
+#endif /* ! USE_NO_SIMD */
+
#endif /* SIMD_H */
--
2.39.1
[text/x-patch] v26-0005-Measure-load-time-of-bench_search_random_nodes.patch (2.4K, ../../CAFBsxsE7FLxfc2bfPf-DwdJ6RmDcpUcNRyUnHeuR26LVi6jzGA@mail.gmail.com/6-v26-0005-Measure-load-time-of-bench_search_random_nodes.patch)
download | inline diff:
From 4a0d293937876ec348f30ce4ab94da14b925b020 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 7 Feb 2023 13:06:00 +0700
Subject: [PATCH v26 5/9] Measure load time of bench_search_random_nodes
---
.../bench_radix_tree/bench_radix_tree--1.0.sql | 1 +
contrib/bench_radix_tree/bench_radix_tree.c | 17 ++++++++++++-----
2 files changed, 13 insertions(+), 5 deletions(-)
diff --git a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
index 2fd689aa91..95eedbbe10 100644
--- a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
+++ b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
@@ -47,6 +47,7 @@ create function bench_search_random_nodes(
cnt int8,
filter_str text DEFAULT NULL,
OUT mem_allocated int8,
+OUT load_ms int8,
OUT search_ms int8)
returns record
as 'MODULE_PATHNAME'
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
index 73ddee32de..7d1e2eee57 100644
--- a/contrib/bench_radix_tree/bench_radix_tree.c
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -395,9 +395,10 @@ bench_search_random_nodes(PG_FUNCTION_ARGS)
end_time;
long secs;
int usecs;
+ int64 load_time_ms;
int64 search_time_ms;
- Datum values[2] = {0};
- bool nulls[2] = {0};
+ Datum values[3] = {0};
+ bool nulls[3] = {0};
/* from trial and error */
uint64 filter = (((uint64) 0x7F<<32) | (0x07<<24) | (0xFF<<16) | 0xFF);
@@ -416,13 +417,18 @@ bench_search_random_nodes(PG_FUNCTION_ARGS)
rt = rt_create(CurrentMemoryContext);
+ start_time = GetCurrentTimestamp();
for (uint64 i = 0; i < cnt; i++)
{
- const uint64 hash = hash64(i);
- const uint64 key = hash & filter;
+ uint64 hash = hash64(i);
+ uint64 key = hash & filter;
rt_set(rt, key, &key);
}
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_time_ms = secs * 1000 + usecs / 1000;
elog(NOTICE, "sleeping for 2 seconds...");
pg_usleep(2 * 1000000L);
@@ -449,7 +455,8 @@ bench_search_random_nodes(PG_FUNCTION_ARGS)
rt_stats(rt);
values[0] = Int64GetDatum(rt_memory_usage(rt));
- values[1] = Int64GetDatum(search_time_ms);
+ values[1] = Int64GetDatum(load_time_ms);
+ values[2] = Int64GetDatum(search_time_ms);
rt_free(rt);
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
--
2.39.1
[text/x-patch] v26-0004-Tool-for-measuring-radix-tree-performance.patch (22.0K, ../../CAFBsxsE7FLxfc2bfPf-DwdJ6RmDcpUcNRyUnHeuR26LVi6jzGA@mail.gmail.com/7-v26-0004-Tool-for-measuring-radix-tree-performance.patch)
download | inline diff:
From 0e328f6d85d30797af158f2a4070004fe40d93fe Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 16 Sep 2022 11:57:03 +0900
Subject: [PATCH v26 4/9] Tool for measuring radix tree performance
Includes Meson support, but commented out to avoid warnings
XXX: Not for commit
---
contrib/bench_radix_tree/Makefile | 21 +
.../bench_radix_tree--1.0.sql | 76 ++
contrib/bench_radix_tree/bench_radix_tree.c | 656 ++++++++++++++++++
.../bench_radix_tree/bench_radix_tree.control | 6 +
contrib/bench_radix_tree/expected/bench.out | 13 +
contrib/bench_radix_tree/meson.build | 33 +
contrib/bench_radix_tree/sql/bench.sql | 16 +
contrib/meson.build | 1 +
8 files changed, 822 insertions(+)
create mode 100644 contrib/bench_radix_tree/Makefile
create mode 100644 contrib/bench_radix_tree/bench_radix_tree--1.0.sql
create mode 100644 contrib/bench_radix_tree/bench_radix_tree.c
create mode 100644 contrib/bench_radix_tree/bench_radix_tree.control
create mode 100644 contrib/bench_radix_tree/expected/bench.out
create mode 100644 contrib/bench_radix_tree/meson.build
create mode 100644 contrib/bench_radix_tree/sql/bench.sql
diff --git a/contrib/bench_radix_tree/Makefile b/contrib/bench_radix_tree/Makefile
new file mode 100644
index 0000000000..952bb0ceae
--- /dev/null
+++ b/contrib/bench_radix_tree/Makefile
@@ -0,0 +1,21 @@
+# contrib/bench_radix_tree/Makefile
+
+MODULE_big = bench_radix_tree
+OBJS = \
+ bench_radix_tree.o
+
+EXTENSION = bench_radix_tree
+DATA = bench_radix_tree--1.0.sql
+
+REGRESS = bench_fixed_height
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/bench_radix_tree
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
new file mode 100644
index 0000000000..2fd689aa91
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
@@ -0,0 +1,76 @@
+/* contrib/bench_radix_tree/bench_radix_tree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION bench_radix_tree" to load this file. \quit
+
+create function bench_shuffle_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_seq_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_load_random_int(
+cnt int8,
+OUT mem_allocated int8,
+OUT load_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_search_random_nodes(
+cnt int8,
+filter_str text DEFAULT NULL,
+OUT mem_allocated int8,
+OUT search_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C VOLATILE PARALLEL UNSAFE;
+
+create function bench_fixed_height_search(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT rt_search_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_node128_load(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_sparseload_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
new file mode 100644
index 0000000000..73ddee32de
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -0,0 +1,656 @@
+/*-------------------------------------------------------------------------
+ *
+ * bench_radix_tree.c
+ *
+ * Copyright (c) 2016-2022, PostgreSQL Global Development Group
+ *
+ * contrib/bench_radix_tree/bench_radix_tree.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "funcapi.h"
+#include "lib/radixtree.h"
+#include <math.h>
+#include "miscadmin.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/timestamp.h"
+
+PG_MODULE_MAGIC;
+
+/* run benchmark also for binary-search case? */
+/* #define MEASURE_BINARY_SEARCH 1 */
+
+#define TIDS_PER_BLOCK_FOR_LOAD 30
+#define TIDS_PER_BLOCK_FOR_LOOKUP 50
+
+#define RT_DEBUG
+#define RT_PREFIX rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE uint64
+// WIP: compiles with warnings because rt_attach is defined but not used
+// #define RT_SHMEM
+#include "lib/radixtree.h"
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+ return tree->ctl->num_keys;
+}
+
+
+PG_FUNCTION_INFO_V1(bench_seq_search);
+PG_FUNCTION_INFO_V1(bench_shuffle_search);
+PG_FUNCTION_INFO_V1(bench_load_random_int);
+PG_FUNCTION_INFO_V1(bench_fixed_height_search);
+PG_FUNCTION_INFO_V1(bench_search_random_nodes);
+PG_FUNCTION_INFO_V1(bench_node128_load);
+
+static uint64
+tid_to_key_off(ItemPointer tid, uint32 *off)
+{
+ uint64 upper;
+ uint32 shift = pg_ceil_log2_32(MaxHeapTuplesPerPage);
+ int64 tid_i;
+
+ Assert(ItemPointerGetOffsetNumber(tid) < MaxHeapTuplesPerPage);
+
+ tid_i = ItemPointerGetOffsetNumber(tid);
+ tid_i |= (uint64) ItemPointerGetBlockNumber(tid) << shift;
+
+ /* log(sizeof(uint64) * BITS_PER_BYTE, 2) = log(64, 2) = 6 */
+ *off = tid_i & ((1 << 6) - 1);
+ upper = tid_i >> 6;
+ Assert(*off < (sizeof(uint64) * BITS_PER_BYTE));
+
+ Assert(*off < 64);
+
+ return upper;
+}
+
+static int
+shuffle_randrange(pg_prng_state *state, int lower, int upper)
+{
+ return (int) floor(pg_prng_double(state) * ((upper - lower) + 0.999999)) + lower;
+}
+
+/* Naive Fisher-Yates implementation*/
+static void
+shuffle_itemptrs(ItemPointer itemptr, uint64 nitems)
+{
+ /* reproducability */
+ pg_prng_state state;
+
+ pg_prng_seed(&state, 0);
+
+ for (int i = 0; i < nitems - 1; i++)
+ {
+ int j = shuffle_randrange(&state, i, nitems - 1);
+ ItemPointerData t = itemptr[j];
+
+ itemptr[j] = itemptr[i];
+ itemptr[i] = t;
+ }
+}
+
+static ItemPointer
+generate_tids(BlockNumber minblk, BlockNumber maxblk, int ntids_per_blk, uint64 *ntids_p,
+ bool random_block)
+{
+ ItemPointer tids;
+ uint64 maxitems;
+ uint64 ntids = 0;
+ pg_prng_state state;
+
+ maxitems = (maxblk - minblk + 1) * ntids_per_blk;
+ tids = MemoryContextAllocHuge(TopTransactionContext,
+ sizeof(ItemPointerData) * maxitems);
+
+ if (random_block)
+ pg_prng_seed(&state, 0x9E3779B185EBCA87);
+
+ for (BlockNumber blk = minblk; blk < maxblk; blk++)
+ {
+ if (random_block && !pg_prng_bool(&state))
+ continue;
+
+ for (OffsetNumber off = FirstOffsetNumber;
+ off <= ntids_per_blk; off++)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ ItemPointerSetBlockNumber(&(tids[ntids]), blk);
+ ItemPointerSetOffsetNumber(&(tids[ntids]), off);
+
+ ntids++;
+ }
+ }
+
+ *ntids_p = ntids;
+ return tids;
+}
+
+#ifdef MEASURE_BINARY_SEARCH
+static int
+vac_cmp_itemptr(const void *left, const void *right)
+{
+ BlockNumber lblk,
+ rblk;
+ OffsetNumber loff,
+ roff;
+
+ lblk = ItemPointerGetBlockNumber((ItemPointer) left);
+ rblk = ItemPointerGetBlockNumber((ItemPointer) right);
+
+ if (lblk < rblk)
+ return -1;
+ if (lblk > rblk)
+ return 1;
+
+ loff = ItemPointerGetOffsetNumber((ItemPointer) left);
+ roff = ItemPointerGetOffsetNumber((ItemPointer) right);
+
+ if (loff < roff)
+ return -1;
+ if (loff > roff)
+ return 1;
+
+ return 0;
+}
+#endif
+
+static Datum
+bench_search(FunctionCallInfo fcinfo, bool shuffle)
+{
+ BlockNumber minblk = PG_GETARG_INT32(0);
+ BlockNumber maxblk = PG_GETARG_INT32(1);
+ bool random_block = PG_GETARG_BOOL(2);
+ rt_radix_tree *rt = NULL;
+ uint64 ntids;
+ uint64 key;
+ uint64 last_key = PG_UINT64_MAX;
+ uint64 val = 0;
+ ItemPointer tids;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_load_ms,
+ rt_search_ms;
+ Datum values[7];
+ bool nulls[7];
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ tids = generate_tids(minblk, maxblk, TIDS_PER_BLOCK_FOR_LOAD, &ntids, random_block);
+
+ /* measure the load time of the radix tree */
+ rt = rt_create(CurrentMemoryContext);
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ uint32 off;
+
+ CHECK_FOR_INTERRUPTS();
+
+ key = tid_to_key_off(tid, &off);
+
+ if (last_key != PG_UINT64_MAX && last_key != key)
+ {
+ rt_set(rt, last_key, &val);
+ val = 0;
+ }
+
+ last_key = key;
+ val |= (uint64) 1 << off;
+ }
+ if (last_key != PG_UINT64_MAX)
+ rt_set(rt, last_key, &val);
+
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_load_ms = secs * 1000 + usecs / 1000;
+
+ rt_stats(rt);
+
+ if (shuffle)
+ shuffle_itemptrs(tids, ntids);
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* meaure the serach time of the radix tree */
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ uint32 off;
+ volatile bool ret; /* prevent calling rt_search from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ key = tid_to_key_off(tid, &off);
+
+ ret = rt_search(rt, key, &val);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_search_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int64GetDatum(rt_num_entries(rt));
+ values[1] = Int64GetDatum(rt_memory_usage(rt));
+ values[2] = Int64GetDatum(sizeof(ItemPointerData) * ntids);
+ values[3] = Int64GetDatum(rt_load_ms);
+ nulls[4] = true; /* ar_load_ms */
+ values[5] = Int64GetDatum(rt_search_ms);
+ nulls[6] = true; /* ar_search_ms */
+
+#ifdef MEASURE_BINARY_SEARCH
+ {
+ ItemPointer itemptrs = NULL;
+
+ int64 ar_load_ms,
+ ar_search_ms;
+
+ /* measure the load time of the array */
+ itemptrs = MemoryContextAllocHuge(CurrentMemoryContext,
+ sizeof(ItemPointerData) * ntids);
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointerSetBlockNumber(&(itemptrs[i]),
+ ItemPointerGetBlockNumber(&(tids[i])));
+ ItemPointerSetOffsetNumber(&(itemptrs[i]),
+ ItemPointerGetOffsetNumber(&(tids[i])));
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ ar_load_ms = secs * 1000 + usecs / 1000;
+
+ /* next, measure the serach time of the array */
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ volatile bool ret; /* prevent calling bsearch from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ ret = bsearch((void *) tid,
+ (void *) itemptrs,
+ ntids,
+ sizeof(ItemPointerData),
+ vac_cmp_itemptr);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ ar_search_ms = secs * 1000 + usecs / 1000;
+
+ /* set the result */
+ nulls[4] = false;
+ values[4] = Int64GetDatum(ar_load_ms);
+ nulls[6] = false;
+ values[6] = Int64GetDatum(ar_search_ms);
+ }
+#endif
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_seq_search(PG_FUNCTION_ARGS)
+{
+ return bench_search(fcinfo, false);
+}
+
+Datum
+bench_shuffle_search(PG_FUNCTION_ARGS)
+{
+ return bench_search(fcinfo, true);
+}
+
+Datum
+bench_load_random_int(PG_FUNCTION_ARGS)
+{
+ uint64 cnt = (uint64) PG_GETARG_INT64(0);
+ rt_radix_tree *rt;
+ pg_prng_state state;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_time_ms;
+ Datum values[2];
+ bool nulls[2];
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ pg_prng_seed(&state, 0);
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 key = pg_prng_uint64(&state);
+
+ rt_set(rt, key, &key);
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_time_ms = secs * 1000 + usecs / 1000;
+
+ rt_stats(rt);
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int64GetDatum(rt_memory_usage(rt));
+ values[1] = Int64GetDatum(load_time_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+/* copy of splitmix64() */
+static uint64
+hash64(uint64 x)
+{
+ x ^= x >> 30;
+ x *= UINT64CONST(0xbf58476d1ce4e5b9);
+ x ^= x >> 27;
+ x *= UINT64CONST(0x94d049bb133111eb);
+ x ^= x >> 31;
+ return x;
+}
+
+/* attempts to have a relatively even population of node kinds */
+Datum
+bench_search_random_nodes(PG_FUNCTION_ARGS)
+{
+ uint64 cnt = (uint64) PG_GETARG_INT64(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 search_time_ms;
+ Datum values[2] = {0};
+ bool nulls[2] = {0};
+ /* from trial and error */
+ uint64 filter = (((uint64) 0x7F<<32) | (0x07<<24) | (0xFF<<16) | 0xFF);
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (!PG_ARGISNULL(1))
+ {
+ char *filter_str = text_to_cstring(PG_GETARG_TEXT_P(1));
+
+ if (sscanf(filter_str, "0x%lX", &filter) == 0)
+ elog(ERROR, "invalid filter string %s", filter_str);
+ }
+ elog(NOTICE, "bench with filter 0x%lX", filter);
+
+ rt = rt_create(CurrentMemoryContext);
+
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ const uint64 hash = hash64(i);
+ const uint64 key = hash & filter;
+
+ rt_set(rt, key, &key);
+ }
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 hash = hash64(i);
+ uint64 key = hash & filter;
+ uint64 val;
+ volatile bool ret; /* prevent calling rt_search from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ ret = rt_search(rt, key, &val);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ search_time_ms = secs * 1000 + usecs / 1000;
+
+ rt_stats(rt);
+
+ values[0] = Int64GetDatum(rt_memory_usage(rt));
+ values[1] = Int64GetDatum(search_time_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_fixed_height_search(PG_FUNCTION_ARGS)
+{
+ int fanout = PG_GETARG_INT32(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_load_ms,
+ rt_search_ms;
+ Datum values[5];
+ bool nulls[5];
+
+ /* test boundary between vector and iteration */
+ const int n_keys = 5 * 16 * 16 * 16 * 16;
+ uint64 r,
+ h,
+ i,
+ j,
+ k;
+ uint64 key_id;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+
+ key_id = 0;
+
+ /*
+ * lower nodes have limited fanout, the top is only limited by
+ * bits-per-byte
+ */
+ for (r = 1;; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ for (i = 1; i <= fanout; i++)
+ {
+ for (j = 1; j <= fanout; j++)
+ {
+ for (k = 1; k <= fanout; k++)
+ {
+ uint64 key;
+
+ key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+ CHECK_FOR_INTERRUPTS();
+
+ key_id++;
+ if (key_id > n_keys)
+ goto finish_set;
+
+ rt_set(rt, key, &key_id);
+ }
+ }
+ }
+ }
+ }
+finish_set:
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_load_ms = secs * 1000 + usecs / 1000;
+
+ rt_stats(rt);
+
+ /* meaure the search time of the radix tree */
+ start_time = GetCurrentTimestamp();
+
+ key_id = 0;
+ for (r = 1;; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ for (i = 1; i <= fanout; i++)
+ {
+ for (j = 1; j <= fanout; j++)
+ {
+ for (k = 1; k <= fanout; k++)
+ {
+ uint64 key,
+ val;
+
+ key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+ CHECK_FOR_INTERRUPTS();
+
+ key_id++;
+ if (key_id > n_keys)
+ goto finish_search;
+
+ rt_search(rt, key, &val);
+ }
+ }
+ }
+ }
+ }
+finish_search:
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_search_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int32GetDatum(fanout);
+ values[1] = Int64GetDatum(rt_num_entries(rt));
+ values[2] = Int64GetDatum(rt_memory_usage(rt));
+ values[3] = Int64GetDatum(rt_load_ms);
+ values[4] = Int64GetDatum(rt_search_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_node128_load(PG_FUNCTION_ARGS)
+{
+ int fanout = PG_GETARG_INT32(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_sparseload_ms;
+ Datum values[5];
+ bool nulls[5];
+
+ uint64 r,
+ h;
+ uint64 key_id;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ rt = rt_create(CurrentMemoryContext);
+
+ key_id = 0;
+
+ for (r = 1; r <= fanout; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (h);
+
+ key_id++;
+ rt_set(rt, key, &key_id);
+ }
+ }
+
+ rt_stats(rt);
+
+ /* measure sparse deletion and re-loading */
+ start_time = GetCurrentTimestamp();
+
+ for (int t = 0; t<10000; t++)
+ {
+ /* delete one key in each leaf */
+ for (r = 1; r <= fanout; r++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (fanout);
+
+ rt_delete(rt, key);
+ }
+
+ /* add them all back */
+ key_id = 0;
+ for (r = 1; r <= fanout; r++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (fanout);
+
+ key_id++;
+ rt_set(rt, key, &key_id);
+ }
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_sparseload_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int32GetDatum(fanout);
+ values[1] = Int64GetDatum(rt_num_entries(rt));
+ values[2] = Int64GetDatum(rt_memory_usage(rt));
+ values[3] = Int64GetDatum(rt_sparseload_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
diff --git a/contrib/bench_radix_tree/bench_radix_tree.control b/contrib/bench_radix_tree/bench_radix_tree.control
new file mode 100644
index 0000000000..1d988e6c9a
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.control
@@ -0,0 +1,6 @@
+# bench_radix_tree extension
+comment = 'benchmark suits for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/bench_radix_tree'
+relocatable = true
+trusted = true
diff --git a/contrib/bench_radix_tree/expected/bench.out b/contrib/bench_radix_tree/expected/bench.out
new file mode 100644
index 0000000000..60c303892e
--- /dev/null
+++ b/contrib/bench_radix_tree/expected/bench.out
@@ -0,0 +1,13 @@
+create extension bench_radix_tree;
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/bench_radix_tree/meson.build b/contrib/bench_radix_tree/meson.build
new file mode 100644
index 0000000000..332c1ae7df
--- /dev/null
+++ b/contrib/bench_radix_tree/meson.build
@@ -0,0 +1,33 @@
+bench_radix_tree_sources = files(
+ 'bench_radix_tree.c',
+)
+
+if host_system == 'windows'
+ bench_radix_tree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'bench_radix_tree',
+ '--FILEDESC', 'bench_radix_tree - performance test code for radix tree',])
+endif
+
+bench_radix_tree = shared_module('bench_radix_tree',
+ bench_radix_tree_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += bench_radix_tree
+
+install_data(
+ 'bench_radix_tree.control',
+ 'bench_radix_tree--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'bench_radix_tree',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'bench_radix_tree',
+ ],
+ },
+}
diff --git a/contrib/bench_radix_tree/sql/bench.sql b/contrib/bench_radix_tree/sql/bench.sql
new file mode 100644
index 0000000000..a46018c9d4
--- /dev/null
+++ b/contrib/bench_radix_tree/sql/bench.sql
@@ -0,0 +1,16 @@
+create extension bench_radix_tree;
+
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/meson.build b/contrib/meson.build
index bd4a57c43c..52253de793 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,6 +12,7 @@ subdir('amcheck')
subdir('auth_delay')
subdir('auto_explain')
subdir('basic_archive')
+#subdir('bench_radix_tree')
subdir('bloom')
subdir('basebackup_to_shell')
subdir('bool_plperl')
--
2.39.1
[text/x-patch] v26-0003-Add-radixtree-template.patch (116.9K, ../../CAFBsxsE7FLxfc2bfPf-DwdJ6RmDcpUcNRyUnHeuR26LVi6jzGA@mail.gmail.com/8-v26-0003-Add-radixtree-template.patch)
download | inline diff:
From 4c4cbb9b13da160b8883e6c7f861516f3eedac6a Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 14 Sep 2022 12:38:51 +0000
Subject: [PATCH v26 3/9] Add radixtree template
WIP: commit message based on template comments
---
src/backend/utils/mmgr/dsa.c | 12 +
src/include/lib/radixtree.h | 2516 +++++++++++++++++
src/include/lib/radixtree_delete_impl.h | 122 +
src/include/lib/radixtree_insert_impl.h | 332 +++
src/include/lib/radixtree_iter_impl.h | 153 +
src/include/lib/radixtree_search_impl.h | 138 +
src/include/utils/dsa.h | 1 +
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_radixtree/.gitignore | 4 +
src/test/modules/test_radixtree/Makefile | 23 +
src/test/modules/test_radixtree/README | 7 +
.../expected/test_radixtree.out | 36 +
src/test/modules/test_radixtree/meson.build | 35 +
.../test_radixtree/sql/test_radixtree.sql | 7 +
.../test_radixtree/test_radixtree--1.0.sql | 8 +
.../modules/test_radixtree/test_radixtree.c | 674 +++++
.../test_radixtree/test_radixtree.control | 4 +
src/tools/pginclude/cpluspluscheck | 6 +
src/tools/pginclude/headerscheck | 6 +
20 files changed, 4086 insertions(+)
create mode 100644 src/include/lib/radixtree.h
create mode 100644 src/include/lib/radixtree_delete_impl.h
create mode 100644 src/include/lib/radixtree_insert_impl.h
create mode 100644 src/include/lib/radixtree_iter_impl.h
create mode 100644 src/include/lib/radixtree_search_impl.h
create mode 100644 src/test/modules/test_radixtree/.gitignore
create mode 100644 src/test/modules/test_radixtree/Makefile
create mode 100644 src/test/modules/test_radixtree/README
create mode 100644 src/test/modules/test_radixtree/expected/test_radixtree.out
create mode 100644 src/test/modules/test_radixtree/meson.build
create mode 100644 src/test/modules/test_radixtree/sql/test_radixtree.sql
create mode 100644 src/test/modules/test_radixtree/test_radixtree--1.0.sql
create mode 100644 src/test/modules/test_radixtree/test_radixtree.c
create mode 100644 src/test/modules/test_radixtree/test_radixtree.control
diff --git a/src/backend/utils/mmgr/dsa.c b/src/backend/utils/mmgr/dsa.c
index f5a62061a3..80555aefff 100644
--- a/src/backend/utils/mmgr/dsa.c
+++ b/src/backend/utils/mmgr/dsa.c
@@ -1024,6 +1024,18 @@ dsa_set_size_limit(dsa_area *area, size_t limit)
LWLockRelease(DSA_AREA_LOCK(area));
}
+size_t
+dsa_get_total_size(dsa_area *area)
+{
+ size_t size;
+
+ LWLockAcquire(DSA_AREA_LOCK(area), LW_SHARED);
+ size = area->control->total_segment_size;
+ LWLockRelease(DSA_AREA_LOCK(area));
+
+ return size;
+}
+
/*
* Aggressively free all spare memory in the hope of returning DSM segments to
* the operating system.
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
new file mode 100644
index 0000000000..d6919aef08
--- /dev/null
+++ b/src/include/lib/radixtree.h
@@ -0,0 +1,2516 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree.h
+ * Template for adaptive radix tree.
+ *
+ * This module employs the idea from the paper "The Adaptive Radix Tree: ARTful
+ * Indexing for Main-Memory Databases" by Viktor Leis, Alfons Kemper, and Thomas
+ * Neumann, 2013. The radix tree uses adaptive node sizes, a small number of node
+ * types, each with a different numbers of elements. Depending on the number of
+ * children, the appropriate node type is used.
+ *
+ * WIP: notes about traditional radix tree trading off span vs height...
+ *
+ * There are two kinds of nodes, inner nodes and leaves. Inner nodes
+ * map partial keys to child pointers.
+ *
+ * The ART paper mentions three ways to implement leaves:
+ *
+ * "- Single-value leaves: The values are stored using an addi-
+ * tional leaf node type which stores one value.
+ * - Multi-value leaves: The values are stored in one of four
+ * different leaf node types, which mirror the structure of
+ * inner nodes, but contain values instead of pointers.
+ * - Combined pointer/value slots: If values fit into point-
+ * ers, no separate node types are necessary. Instead, each
+ * pointer storage location in an inner node can either
+ * store a pointer or a value."
+ *
+ * We chose "multi-value leaves" to avoid the additional pointer traversal
+ * required by "single-value leaves"
+ *
+ * For simplicity, the key is assumed to be 64-bit unsigned integer. The
+ * tree doesn't need to contain paths where the highest bytes of all keys
+ * are zero. That way, the tree's height adapts to the distribution of keys.
+ *
+ * TODO: In the future it might be worthwhile to offer configurability of
+ * leaf implementation for different use cases. Single-values leaves would
+ * give more flexibility in key type, including variable-length keys.
+ *
+ * There are some optimizations not yet implemented, particularly path
+ * compression and lazy path expansion.
+ *
+ * To handle concurrency, we use a single reader-writer lock for the radix
+ * tree. The radix tree is exclusively locked during write operations such
+ * as RT_SET() and RT_DELETE(), and shared locked during read operations
+ * such as RT_SEARCH(). An iteration also holds the shared lock on the radix
+ * tree until it is completed.
+ *
+ * TODO: The current locking mechanism is not optimized for high concurrency
+ * with mixed read-write workloads. In the future it might be worthwhile
+ * to replace it with the Optimistic Lock Coupling or ROWEX mentioned in
+ * the paper "The ART of Practical Synchronization" by the same authors as
+ * the ART paper, 2016.
+ *
+ * WIP: the radix tree nodes don't shrink.
+ *
+ * To generate a radix tree and associated functions for a use case several
+ * macros have to be #define'ed before this file is included. Including
+ * the file #undef's all those, so a new radix tree can be generated
+ * afterwards.
+ * The relevant parameters are:
+ * - RT_PREFIX - prefix for all symbol names generated. A prefix of 'foo'
+ * will result in radix tree type 'foo_radix_tree' and functions like
+ * 'foo_create'/'foo_free' and so forth.
+ * - RT_DECLARE - if defined function prototypes and type declarations are
+ * generated
+ * - RT_DEFINE - if defined function definitions are generated
+ * - RT_SCOPE - in which scope (e.g. extern, static inline) do function
+ * declarations reside
+ * - RT_VALUE_TYPE - the type of the value.
+ *
+ * Optional parameters:
+ * - RT_SHMEM - if defined, the radix tree is created in the DSA area
+ * so that multiple processes can access it simultaneously.
+ * - RT_DEBUG - if defined add stats tracking and debugging functions
+ *
+ * Interface
+ * ---------
+ *
+ * RT_CREATE - Create a new, empty radix tree
+ * RT_FREE - Free the radix tree
+ * RT_SEARCH - Search a key-value pair
+ * RT_SET - Set a key-value pair
+ * RT_BEGIN_ITERATE - Begin iterating through all key-value pairs
+ * RT_ITERATE_NEXT - Return next key-value pair, if any
+ * RT_END_ITER - End iteration
+ * RT_MEMORY_USAGE - Get the memory usage
+ *
+ * Interface for Shared Memory
+ * ---------
+ *
+ * RT_ATTACH - Attach to the radix tree
+ * RT_DETACH - Detach from the radix tree
+ * RT_GET_HANDLE - Return the handle of the radix tree
+ *
+ * Optional Interface
+ * ---------
+ *
+ * RT_DELETE - Delete a key-value pair. Declared/define if RT_USE_DELETE is defined
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/include/lib/radixtree.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/stringinfo.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "port/pg_bitutils.h"
+#include "port/simd.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/* helpers */
+#define RT_MAKE_PREFIX(a) CppConcat(a,_)
+#define RT_MAKE_NAME(name) RT_MAKE_NAME_(RT_MAKE_PREFIX(RT_PREFIX),name)
+#define RT_MAKE_NAME_(a,b) CppConcat(a,b)
+
+/* function declarations */
+#define RT_CREATE RT_MAKE_NAME(create)
+#define RT_FREE RT_MAKE_NAME(free)
+#define RT_SEARCH RT_MAKE_NAME(search)
+#ifdef RT_SHMEM
+#define RT_ATTACH RT_MAKE_NAME(attach)
+#define RT_DETACH RT_MAKE_NAME(detach)
+#define RT_GET_HANDLE RT_MAKE_NAME(get_handle)
+#endif
+#define RT_SET RT_MAKE_NAME(set)
+#define RT_BEGIN_ITERATE RT_MAKE_NAME(begin_iterate)
+#define RT_ITERATE_NEXT RT_MAKE_NAME(iterate_next)
+#define RT_END_ITERATE RT_MAKE_NAME(end_iterate)
+#ifdef RT_USE_DELETE
+#define RT_DELETE RT_MAKE_NAME(delete)
+#endif
+#define RT_MEMORY_USAGE RT_MAKE_NAME(memory_usage)
+#ifdef RT_DEBUG
+#define RT_DUMP RT_MAKE_NAME(dump)
+#define RT_DUMP_NODE RT_MAKE_NAME(dump_node)
+#define RT_DUMP_SEARCH RT_MAKE_NAME(dump_search)
+#define RT_STATS RT_MAKE_NAME(stats)
+#endif
+
+/* internal helper functions (no externally visible prototypes) */
+#define RT_NEW_ROOT RT_MAKE_NAME(new_root)
+#define RT_ALLOC_NODE RT_MAKE_NAME(alloc_node)
+#define RT_INIT_NODE RT_MAKE_NAME(init_node)
+#define RT_FREE_NODE RT_MAKE_NAME(free_node)
+#define RT_FREE_RECURSE RT_MAKE_NAME(free_recurse)
+#define RT_EXTEND RT_MAKE_NAME(extend)
+#define RT_SET_EXTEND RT_MAKE_NAME(set_extend)
+#define RT_SWITCH_NODE_KIND RT_MAKE_NAME(grow_node_kind)
+#define RT_COPY_NODE RT_MAKE_NAME(copy_node)
+#define RT_REPLACE_NODE RT_MAKE_NAME(replace_node)
+#define RT_PTR_GET_LOCAL RT_MAKE_NAME(ptr_get_local)
+#define RT_PTR_ALLOC_IS_VALID RT_MAKE_NAME(ptr_stored_is_valid)
+#define RT_NODE_3_SEARCH_EQ RT_MAKE_NAME(node_3_search_eq)
+#define RT_NODE_32_SEARCH_EQ RT_MAKE_NAME(node_32_search_eq)
+#define RT_NODE_3_GET_INSERTPOS RT_MAKE_NAME(node_3_get_insertpos)
+#define RT_NODE_32_GET_INSERTPOS RT_MAKE_NAME(node_32_get_insertpos)
+#define RT_CHUNK_CHILDREN_ARRAY_SHIFT RT_MAKE_NAME(chunk_children_array_shift)
+#define RT_CHUNK_VALUES_ARRAY_SHIFT RT_MAKE_NAME(chunk_values_array_shift)
+#define RT_CHUNK_CHILDREN_ARRAY_DELETE RT_MAKE_NAME(chunk_children_array_delete)
+#define RT_CHUNK_VALUES_ARRAY_DELETE RT_MAKE_NAME(chunk_values_array_delete)
+#define RT_CHUNK_CHILDREN_ARRAY_COPY RT_MAKE_NAME(chunk_children_array_copy)
+#define RT_CHUNK_VALUES_ARRAY_COPY RT_MAKE_NAME(chunk_values_array_copy)
+#define RT_NODE_125_IS_CHUNK_USED RT_MAKE_NAME(node_125_is_chunk_used)
+#define RT_NODE_INNER_125_GET_CHILD RT_MAKE_NAME(node_inner_125_get_child)
+#define RT_NODE_LEAF_125_GET_VALUE RT_MAKE_NAME(node_leaf_125_get_value)
+#define RT_NODE_INNER_256_IS_CHUNK_USED RT_MAKE_NAME(node_inner_256_is_chunk_used)
+#define RT_NODE_LEAF_256_IS_CHUNK_USED RT_MAKE_NAME(node_leaf_256_is_chunk_used)
+#define RT_NODE_INNER_256_GET_CHILD RT_MAKE_NAME(node_inner_256_get_child)
+#define RT_NODE_LEAF_256_GET_VALUE RT_MAKE_NAME(node_leaf_256_get_value)
+#define RT_NODE_INNER_256_SET RT_MAKE_NAME(node_inner_256_set)
+#define RT_NODE_LEAF_256_SET RT_MAKE_NAME(node_leaf_256_set)
+#define RT_NODE_INNER_256_DELETE RT_MAKE_NAME(node_inner_256_delete)
+#define RT_NODE_LEAF_256_DELETE RT_MAKE_NAME(node_leaf_256_delete)
+#define RT_KEY_GET_SHIFT RT_MAKE_NAME(key_get_shift)
+#define RT_SHIFT_GET_MAX_VAL RT_MAKE_NAME(shift_get_max_val)
+#define RT_NODE_SEARCH_INNER RT_MAKE_NAME(node_search_inner)
+#define RT_NODE_SEARCH_LEAF RT_MAKE_NAME(node_search_leaf)
+#define RT_NODE_UPDATE_INNER RT_MAKE_NAME(node_update_inner)
+#define RT_NODE_DELETE_INNER RT_MAKE_NAME(node_delete_inner)
+#define RT_NODE_DELETE_LEAF RT_MAKE_NAME(node_delete_leaf)
+#define RT_NODE_INSERT_INNER RT_MAKE_NAME(node_insert_inner)
+#define RT_NODE_INSERT_LEAF RT_MAKE_NAME(node_insert_leaf)
+#define RT_NODE_INNER_ITERATE_NEXT RT_MAKE_NAME(node_inner_iterate_next)
+#define RT_NODE_LEAF_ITERATE_NEXT RT_MAKE_NAME(node_leaf_iterate_next)
+#define RT_UPDATE_ITER_STACK RT_MAKE_NAME(update_iter_stack)
+#define RT_ITER_UPDATE_KEY RT_MAKE_NAME(iter_update_key)
+#define RT_VERIFY_NODE RT_MAKE_NAME(verify_node)
+
+/* type declarations */
+#define RT_RADIX_TREE RT_MAKE_NAME(radix_tree)
+#define RT_RADIX_TREE_CONTROL RT_MAKE_NAME(radix_tree_control)
+#define RT_ITER RT_MAKE_NAME(iter)
+#ifdef RT_SHMEM
+#define RT_HANDLE RT_MAKE_NAME(handle)
+#endif
+#define RT_NODE RT_MAKE_NAME(node)
+#define RT_NODE_ITER RT_MAKE_NAME(node_iter)
+#define RT_NODE_BASE_3 RT_MAKE_NAME(node_base_3)
+#define RT_NODE_BASE_32 RT_MAKE_NAME(node_base_32)
+#define RT_NODE_BASE_125 RT_MAKE_NAME(node_base_125)
+#define RT_NODE_BASE_256 RT_MAKE_NAME(node_base_256)
+#define RT_NODE_INNER_3 RT_MAKE_NAME(node_inner_3)
+#define RT_NODE_INNER_32 RT_MAKE_NAME(node_inner_32)
+#define RT_NODE_INNER_125 RT_MAKE_NAME(node_inner_125)
+#define RT_NODE_INNER_256 RT_MAKE_NAME(node_inner_256)
+#define RT_NODE_LEAF_3 RT_MAKE_NAME(node_leaf_3)
+#define RT_NODE_LEAF_32 RT_MAKE_NAME(node_leaf_32)
+#define RT_NODE_LEAF_125 RT_MAKE_NAME(node_leaf_125)
+#define RT_NODE_LEAF_256 RT_MAKE_NAME(node_leaf_256)
+#define RT_SIZE_CLASS RT_MAKE_NAME(size_class)
+#define RT_SIZE_CLASS_ELEM RT_MAKE_NAME(size_class_elem)
+#define RT_SIZE_CLASS_INFO RT_MAKE_NAME(size_class_info)
+#define RT_CLASS_3 RT_MAKE_NAME(class_3)
+#define RT_CLASS_32_MIN RT_MAKE_NAME(class_32_min)
+#define RT_CLASS_32_MAX RT_MAKE_NAME(class_32_max)
+#define RT_CLASS_125 RT_MAKE_NAME(class_125)
+#define RT_CLASS_256 RT_MAKE_NAME(class_256)
+
+/* generate forward declarations necessary to use the radix tree */
+#ifdef RT_DECLARE
+
+typedef struct RT_RADIX_TREE RT_RADIX_TREE;
+typedef struct RT_ITER RT_ITER;
+
+#ifdef RT_SHMEM
+typedef dsa_pointer RT_HANDLE;
+#endif
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id);
+RT_SCOPE RT_RADIX_TREE * RT_ATTACH(dsa_area *dsa, dsa_pointer dp);
+RT_SCOPE void RT_DETACH(RT_RADIX_TREE *tree);
+RT_SCOPE RT_HANDLE RT_GET_HANDLE(RT_RADIX_TREE *tree);
+#else
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx);
+#endif
+RT_SCOPE void RT_FREE(RT_RADIX_TREE *tree);
+
+RT_SCOPE bool RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p);
+RT_SCOPE bool RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p);
+#ifdef RT_USE_DELETE
+RT_SCOPE bool RT_DELETE(RT_RADIX_TREE *tree, uint64 key);
+#endif
+
+RT_SCOPE RT_ITER * RT_BEGIN_ITERATE(RT_RADIX_TREE *tree);
+RT_SCOPE bool RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p);
+RT_SCOPE void RT_END_ITERATE(RT_ITER *iter);
+
+RT_SCOPE uint64 RT_MEMORY_USAGE(RT_RADIX_TREE *tree);
+
+#ifdef RT_DEBUG
+RT_SCOPE void RT_DUMP(RT_RADIX_TREE *tree);
+RT_SCOPE void RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key);
+RT_SCOPE void RT_STATS(RT_RADIX_TREE *tree);
+#endif
+
+#endif /* RT_DECLARE */
+
+
+/* generate implementation of the radix tree */
+#ifdef RT_DEFINE
+
+/* The number of bits encoded in one tree level */
+#define RT_NODE_SPAN BITS_PER_BYTE
+
+/* The number of maximum slots in the node */
+#define RT_NODE_MAX_SLOTS (1 << RT_NODE_SPAN)
+
+/* Mask for extracting a chunk from the key */
+#define RT_CHUNK_MASK ((1 << RT_NODE_SPAN) - 1)
+
+/* Maximum shift the radix tree uses */
+#define RT_MAX_SHIFT RT_KEY_GET_SHIFT(UINT64_MAX)
+
+/* Tree level the radix tree uses */
+#define RT_MAX_LEVEL ((sizeof(uint64) * BITS_PER_BYTE) / RT_NODE_SPAN)
+
+/*
+ * Number of bits necessary for isset array in the slot-index node.
+ * Since bitmapword can be 64 bits, the only values that make sense
+ * here are 64 and 128.
+ */
+#define RT_SLOT_IDX_LIMIT (RT_NODE_MAX_SLOTS / 2)
+
+/* Invalid index used in node-125 */
+#define RT_INVALID_SLOT_IDX 0xFF
+
+/* Get a chunk from the key */
+#define RT_GET_KEY_CHUNK(key, shift) ((uint8) (((key) >> (shift)) & RT_CHUNK_MASK))
+
+/* For accessing bitmaps */
+#define RT_BM_IDX(x) ((x) / BITS_PER_BITMAPWORD)
+#define RT_BM_BIT(x) ((x) % BITS_PER_BITMAPWORD)
+
+/*
+ * Node kinds
+ *
+ * The different node kinds are what make the tree "adaptive".
+ *
+ * Each node kind is associated with a different datatype and different
+ * search/set/delete/iterate algorithms adapted for its size. The largest
+ * kind, node256 is basically the same as a traditional radix tree,
+ * and would be most wasteful of memory when sparsely populated. The
+ * smaller nodes expend some additional CPU time to enable a smaller
+ * memory footprint.
+ *
+ * XXX There are 4 node kinds, and this should never be increased,
+ * for several reasons:
+ * 1. With 5 or more kinds, gcc tends to use a jump table for switch
+ * statements.
+ * 2. The 4 kinds can be represented with 2 bits, so we have the option
+ * in the future to tag the node pointer with the kind, even on
+ * platforms with 32-bit pointers. This might speed up node traversal
+ * in trees with highly random node kinds.
+ * 3. We can have multiple size classes per node kind.
+ */
+#define RT_NODE_KIND_3 0x00
+#define RT_NODE_KIND_32 0x01
+#define RT_NODE_KIND_125 0x02
+#define RT_NODE_KIND_256 0x03
+#define RT_NODE_KIND_COUNT 4
+
+/*
+ * Calculate the slab blocksize so that we can allocate at least 32 chunks
+ * from the block.
+ */
+#define RT_SLAB_BLOCK_SIZE(size) \
+ Max((SLAB_DEFAULT_BLOCK_SIZE / (size)) * (size), (size) * 32)
+
+/* Common type for all nodes types */
+typedef struct RT_NODE
+{
+ /*
+ * Number of children. We use uint16 to be able to indicate 256 children
+ * at the fanout of 8.
+ */
+ uint16 count;
+
+ /*
+ * Max capacity for the current size class. Storing this in the
+ * node enables multiple size classes per node kind.
+ * Technically, kinds with a single size class don't need this, so we could
+ * keep this in the individual base types, but the code is simpler this way.
+ * Note: node256 is unique in that it cannot possibly have more than a
+ * single size class, so for that kind we store zero, and uint8 is
+ * sufficient for other kinds.
+ */
+ uint8 fanout;
+
+ /*
+ * Shift indicates which part of the key space is represented by this
+ * node. That is, the key is shifted by 'shift' and the lowest
+ * RT_NODE_SPAN bits are then represented in chunk.
+ */
+ uint8 shift;
+
+ /* Node kind, one per search/set algorithm */
+ uint8 kind;
+} RT_NODE;
+
+
+#define RT_PTR_LOCAL RT_NODE *
+
+#ifdef RT_SHMEM
+#define RT_PTR_ALLOC dsa_pointer
+#else
+#define RT_PTR_ALLOC RT_PTR_LOCAL
+#endif
+
+
+#ifdef RT_SHMEM
+#define RT_INVALID_PTR_ALLOC InvalidDsaPointer
+#else
+#define RT_INVALID_PTR_ALLOC NULL
+#endif
+
+#ifdef RT_SHMEM
+#define RT_LOCK_EXCLUSIVE(tree) LWLockAcquire(&tree->ctl->lock, LW_EXCLUSIVE)
+#define RT_LOCK_SHARED(tree) LWLockAcquire(&tree->ctl->lock, LW_SHARED)
+#define RT_UNLOCK(tree) LWLockRelease(&tree->ctl->lock);
+#else
+#define RT_LOCK_EXCLUSIVE(tree) ((void) 0)
+#define RT_LOCK_SHARED(tree) ((void) 0)
+#define RT_UNLOCK(tree) ((void) 0)
+#endif
+
+/*
+ * Inner nodes and leaf nodes have analogous structure. To distinguish
+ * them at runtime, we take advantage of the fact that the key chunk
+ * is accessed by shifting: Inner tree nodes (shift > 0), store the
+ * pointer to its child node in the slot. In leaf nodes (shift == 0),
+ * the slot contains the value corresponding to the key.
+ */
+#define RT_NODE_IS_LEAF(n) (((RT_PTR_LOCAL) (n))->shift == 0)
+
+#define RT_NODE_MUST_GROW(node) \
+ ((node)->base.n.count == (node)->base.n.fanout)
+
+/*
+ * Base type of each node kinds for leaf and inner nodes.
+ * The base types must be a be able to accommodate the largest size
+ * class for variable-sized node kinds.
+ */
+typedef struct RT_NODE_BASE_3
+{
+ RT_NODE n;
+
+ /* 3 children, for key chunks */
+ uint8 chunks[3];
+} RT_NODE_BASE_3;
+
+typedef struct RT_NODE_BASE_32
+{
+ RT_NODE n;
+
+ /* 32 children, for key chunks */
+ uint8 chunks[32];
+} RT_NODE_BASE_32;
+
+/*
+ * node-125 uses slot_idx array, an array of RT_NODE_MAX_SLOTS length
+ * to store indexes into a second array that contains the values (or
+ * child pointers).
+ */
+typedef struct RT_NODE_BASE_125
+{
+ RT_NODE n;
+
+ /* The index of slots for each fanout */
+ uint8 slot_idxs[RT_NODE_MAX_SLOTS];
+
+ /* bitmap to track which slots are in use */
+ bitmapword isset[RT_BM_IDX(RT_SLOT_IDX_LIMIT)];
+} RT_NODE_BASE_125;
+
+typedef struct RT_NODE_BASE_256
+{
+ RT_NODE n;
+} RT_NODE_BASE_256;
+
+/*
+ * Inner and leaf nodes.
+ *
+ * These are separate because the value type might be different than
+ * something fitting into a pointer-width type.
+ */
+typedef struct RT_NODE_INNER_3
+{
+ RT_NODE_BASE_3 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_3;
+
+typedef struct RT_NODE_LEAF_3
+{
+ RT_NODE_BASE_3 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_3;
+
+typedef struct RT_NODE_INNER_32
+{
+ RT_NODE_BASE_32 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_32;
+
+typedef struct RT_NODE_LEAF_32
+{
+ RT_NODE_BASE_32 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_32;
+
+typedef struct RT_NODE_INNER_125
+{
+ RT_NODE_BASE_125 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_125;
+
+typedef struct RT_NODE_LEAF_125
+{
+ RT_NODE_BASE_125 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_125;
+
+/*
+ * node-256 is the largest node type. This node has an array
+ * for directly storing values (or child pointers in inner nodes).
+ * Unlike other node kinds, it's array size is by definition
+ * fixed.
+ */
+typedef struct RT_NODE_INNER_256
+{
+ RT_NODE_BASE_256 base;
+
+ /* Slots for 256 children */
+ RT_PTR_ALLOC children[RT_NODE_MAX_SLOTS];
+} RT_NODE_INNER_256;
+
+typedef struct RT_NODE_LEAF_256
+{
+ RT_NODE_BASE_256 base;
+
+ /*
+ * Unlike with inner256, zero is a valid value here, so we use a
+ * bitmap to track which slots are in use.
+ */
+ bitmapword isset[RT_BM_IDX(RT_NODE_MAX_SLOTS)];
+
+ /* Slots for 256 values */
+ RT_VALUE_TYPE values[RT_NODE_MAX_SLOTS];
+} RT_NODE_LEAF_256;
+
+/*
+ * Node size classes
+ *
+ * Nodes of different kinds necessarily belong to different size classes.
+ * The main innovation in our implementation compared to the ART paper
+ * is decoupling the notion of size class from kind.
+ *
+ * The size classes within a given node kind have the same underlying
+ * type, but a variable number of children/values. This is possible
+ * because the base type contains small fixed data structures that
+ * work the same way regardless of how full the node is. We store the
+ * node's allocated capacity in the "fanout" member of RT_NODE, to allow
+ * runtime introspection.
+ *
+ * Growing from one node kind to another requires special code for each
+ * case, but growing from one size class to another within the same kind
+ * is basically just allocate + memcpy.
+ *
+ * The size classes have been chosen so that inner nodes on platforms
+ * with 64-bit pointers (and leaf nodes when using a 64-bit key) are
+ * equal to or slightly smaller than some DSA size class.
+ */
+typedef enum RT_SIZE_CLASS
+{
+ RT_CLASS_3 = 0,
+ RT_CLASS_32_MIN,
+ RT_CLASS_32_MAX,
+ RT_CLASS_125,
+ RT_CLASS_256
+} RT_SIZE_CLASS;
+
+/* Information for each size class */
+typedef struct RT_SIZE_CLASS_ELEM
+{
+ const char *name;
+ int fanout;
+
+ /* slab chunk size */
+ Size inner_size;
+ Size leaf_size;
+} RT_SIZE_CLASS_ELEM;
+
+static const RT_SIZE_CLASS_ELEM RT_SIZE_CLASS_INFO[] = {
+ [RT_CLASS_3] = {
+ .name = "radix tree node 3",
+ .fanout = 3,
+ .inner_size = sizeof(RT_NODE_INNER_3) + 3 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_3) + 3 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_32_MIN] = {
+ .name = "radix tree node 15",
+ .fanout = 15,
+ .inner_size = sizeof(RT_NODE_INNER_32) + 15 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_32) + 15 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_32_MAX] = {
+ .name = "radix tree node 32",
+ .fanout = 32,
+ .inner_size = sizeof(RT_NODE_INNER_32) + 32 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_32) + 32 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_125] = {
+ .name = "radix tree node 125",
+ .fanout = 125,
+ .inner_size = sizeof(RT_NODE_INNER_125) + 125 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_125) + 125 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_256] = {
+ .name = "radix tree node 256",
+ .fanout = 256,
+ .inner_size = sizeof(RT_NODE_INNER_256),
+ .leaf_size = sizeof(RT_NODE_LEAF_256),
+ },
+};
+
+#define RT_SIZE_CLASS_COUNT lengthof(RT_SIZE_CLASS_INFO)
+
+#ifdef RT_SHMEM
+/* A magic value used to identify our radix tree */
+#define RT_RADIX_TREE_MAGIC 0x54A48167
+#endif
+
+/* Contains the actual tree and ancillary info */
+// WIP: this name is a bit strange
+typedef struct RT_RADIX_TREE_CONTROL
+{
+#ifdef RT_SHMEM
+ RT_HANDLE handle;
+ uint32 magic;
+ LWLock lock;
+#endif
+
+ RT_PTR_ALLOC root;
+ uint64 max_val;
+ uint64 num_keys;
+
+ /* statistics */
+#ifdef RT_DEBUG
+ int32 cnt[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE_CONTROL;
+
+/* Entry point for allocating and accessing the tree */
+typedef struct RT_RADIX_TREE
+{
+ MemoryContext context;
+
+ /* pointing to either local memory or DSA */
+ RT_RADIX_TREE_CONTROL *ctl;
+
+#ifdef RT_SHMEM
+ dsa_area *dsa;
+#else
+ MemoryContextData *inner_slabs[RT_SIZE_CLASS_COUNT];
+ MemoryContextData *leaf_slabs[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE;
+
+/*
+ * Iteration support.
+ *
+ * Iterating the radix tree returns each pair of key and value in the ascending
+ * order of the key. To support this, the we iterate nodes of each level.
+ *
+ * RT_NODE_ITER struct is used to track the iteration within a node.
+ *
+ * RT_ITER is the struct for iteration of the radix tree, and uses RT_NODE_ITER
+ * in order to track the iteration of each level. During iteration, we also
+ * construct the key whenever updating the node iteration information, e.g., when
+ * advancing the current index within the node or when moving to the next node
+ * at the same level.
+ *
+ * XXX: Currently we allow only one process to do iteration. Therefore, rt_node_iter
+ * has the local pointers to nodes, rather than RT_PTR_ALLOC.
+ * We need either a safeguard to disallow other processes to begin the iteration
+ * while one process is doing or to allow multiple processes to do the iteration.
+ */
+typedef struct RT_NODE_ITER
+{
+ RT_PTR_LOCAL node; /* current node being iterated */
+ int current_idx; /* current position. -1 for initial value */
+} RT_NODE_ITER;
+
+typedef struct RT_ITER
+{
+ RT_RADIX_TREE *tree;
+
+ /* Track the iteration on nodes of each level */
+ RT_NODE_ITER stack[RT_MAX_LEVEL];
+ int stack_len;
+
+ /* The key is constructed during iteration */
+ uint64 key;
+} RT_ITER;
+
+
+static bool RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_PTR_ALLOC child);
+static bool RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_VALUE_TYPE *value_p);
+
+/* verification (available only with assertion) */
+static void RT_VERIFY_NODE(RT_PTR_LOCAL node);
+
+/* Get the local address of an allocated node */
+static inline RT_PTR_LOCAL
+RT_PTR_GET_LOCAL(RT_RADIX_TREE *tree, RT_PTR_ALLOC node)
+{
+#ifdef RT_SHMEM
+ return dsa_get_address(tree->dsa, (dsa_pointer) node);
+#else
+ return node;
+#endif
+}
+
+static inline bool
+RT_PTR_ALLOC_IS_VALID(RT_PTR_ALLOC ptr)
+{
+#ifdef RT_SHMEM
+ return DsaPointerIsValid(ptr);
+#else
+ return PointerIsValid(ptr);
+#endif
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_3_SEARCH_EQ(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+ int idx = -1;
+
+ for (int i = 0; i < node->n.count; i++)
+ {
+ if (node->chunks[i] == chunk)
+ {
+ idx = i;
+ break;
+ }
+ }
+
+ return idx;
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_3_GET_INSERTPOS(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+ int idx;
+
+ for (idx = 0; idx < node->n.count; idx++)
+ {
+ if (node->chunks[idx] >= chunk)
+ break;
+ }
+
+ return idx;
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_32_SEARCH_EQ(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+ int count = node->n.count;
+#ifndef USE_NO_SIMD
+ Vector8 spread_chunk;
+ Vector8 haystack1;
+ Vector8 haystack2;
+ Vector8 cmp1;
+ Vector8 cmp2;
+ uint32 bitfield;
+ int index_simd = -1;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+ int index = -1;
+
+ for (int i = 0; i < count; i++)
+ {
+ if (node->chunks[i] == chunk)
+ {
+ index = i;
+ break;
+ }
+ }
+#endif
+
+#ifndef USE_NO_SIMD
+ /* replicate the search key */
+ spread_chunk = vector8_broadcast(chunk);
+
+ /* compare to all 32 keys stored in the node */
+ vector8_load(&haystack1, &node->chunks[0]);
+ vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+ cmp1 = vector8_eq(spread_chunk, haystack1);
+ cmp2 = vector8_eq(spread_chunk, haystack2);
+
+ /* convert comparison to a bitfield */
+ bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+
+ /* mask off invalid entries */
+ bitfield &= ((UINT64CONST(1) << count) - 1);
+
+ /* convert bitfield to index by counting trailing zeros */
+ if (bitfield)
+ index_simd = pg_rightmost_one_pos32(bitfield);
+
+ Assert(index_simd == index);
+ return index_simd;
+#else
+ return index;
+#endif
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_32_GET_INSERTPOS(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+ int count = node->n.count;
+#ifndef USE_NO_SIMD
+ Vector8 spread_chunk;
+ Vector8 haystack1;
+ Vector8 haystack2;
+ Vector8 cmp1;
+ Vector8 cmp2;
+ Vector8 min1;
+ Vector8 min2;
+ uint32 bitfield;
+ int index_simd;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+ int index;
+
+ for (index = 0; index < count; index++)
+ {
+ /*
+ * This is coded with '>=' to match what we can do with SIMD,
+ * with an assert to keep us honest.
+ */
+ if (node->chunks[index] >= chunk)
+ {
+ Assert(node->chunks[index] != chunk);
+ break;
+ }
+ }
+#endif
+
+#ifndef USE_NO_SIMD
+ /*
+ * This is a bit more complicated than RT_NODE_32_SEARCH_EQ(), because
+ * no unsigned uint8 comparison instruction exists, at least for SSE2. So
+ * we need to play some trickery using vector8_min() to effectively get
+ * >=. There'll never be any equal elements in current uses, but that's
+ * what we get here...
+ */
+ spread_chunk = vector8_broadcast(chunk);
+ vector8_load(&haystack1, &node->chunks[0]);
+ vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+ min1 = vector8_min(spread_chunk, haystack1);
+ min2 = vector8_min(spread_chunk, haystack2);
+ cmp1 = vector8_eq(spread_chunk, min1);
+ cmp2 = vector8_eq(spread_chunk, min2);
+ bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+ bitfield &= ((UINT64CONST(1) << count) - 1);
+
+ if (bitfield)
+ index_simd = pg_rightmost_one_pos32(bitfield);
+ else
+ index_simd = count;
+
+ Assert(index_simd == index);
+ return index_simd;
+#else
+ return index;
+#endif
+}
+
+
+/*
+ * Functions to manipulate both chunks array and children/values array.
+ * These are used for node-3 and node-32.
+ */
+
+/* Shift the elements right at 'idx' by one */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_SHIFT(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+ memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+ memmove(&(children[idx + 1]), &(children[idx]), sizeof(RT_PTR_ALLOC) * (count - idx));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_SHIFT(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+ memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+ memmove(&(values[idx + 1]), &(values[idx]), sizeof(RT_VALUE_TYPE) * (count - idx));
+}
+
+/* Delete the element at 'idx' */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_DELETE(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+ memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+ memmove(&(children[idx]), &(children[idx + 1]), sizeof(RT_PTR_ALLOC) * (count - idx - 1));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_DELETE(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+ memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+ memmove(&(values[idx]), &(values[idx + 1]), sizeof(RT_VALUE_TYPE) * (count - idx - 1));
+}
+
+/* Copy both chunks and children/values arrays */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_COPY(uint8 *src_chunks, RT_PTR_ALLOC *src_children,
+ uint8 *dst_chunks, RT_PTR_ALLOC *dst_children)
+{
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const Size chunk_size = sizeof(uint8) * fanout;
+ const Size children_size = sizeof(RT_PTR_ALLOC) * fanout;
+
+ memcpy(dst_chunks, src_chunks, chunk_size);
+ memcpy(dst_children, src_children, children_size);
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_COPY(uint8 *src_chunks, RT_VALUE_TYPE *src_values,
+ uint8 *dst_chunks, RT_VALUE_TYPE *dst_values)
+{
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const Size chunk_size = sizeof(uint8) * fanout;
+ const Size values_size = sizeof(RT_VALUE_TYPE) * fanout;
+
+ memcpy(dst_chunks, src_chunks, chunk_size);
+ memcpy(dst_values, src_values, values_size);
+}
+
+/* Functions to manipulate inner and leaf node-125 */
+
+/* Does the given chunk in the node has the value? */
+static inline bool
+RT_NODE_125_IS_CHUNK_USED(RT_NODE_BASE_125 *node, uint8 chunk)
+{
+ return node->slot_idxs[chunk] != RT_INVALID_SLOT_IDX;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_125_GET_CHILD(RT_NODE_INNER_125 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ return node->children[node->base.slot_idxs[chunk]];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_125_GET_VALUE(RT_NODE_LEAF_125 *node, uint8 chunk)
+{
+ Assert(RT_NODE_IS_LEAF(node));
+ Assert(((RT_NODE_BASE_125 *) node)->slot_idxs[chunk] != RT_INVALID_SLOT_IDX);
+ return node->values[node->base.slot_idxs[chunk]];
+}
+
+/* Functions to manipulate inner and leaf node-256 */
+
+/* Return true if the slot corresponding to the given chunk is in use */
+static inline bool
+RT_NODE_INNER_256_IS_CHUNK_USED(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ return node->children[chunk] != RT_INVALID_PTR_ALLOC;
+}
+
+static inline bool
+RT_NODE_LEAF_256_IS_CHUNK_USED(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ return (node->isset[idx] & ((bitmapword) 1 << bitnum)) != 0;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_256_GET_CHILD(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ Assert(RT_NODE_INNER_256_IS_CHUNK_USED(node, chunk));
+ return node->children[chunk];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_256_GET_VALUE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ Assert(RT_NODE_IS_LEAF(node));
+ Assert(RT_NODE_LEAF_256_IS_CHUNK_USED(node, chunk));
+ return node->values[chunk];
+}
+
+/* Set the child in the node-256 */
+static inline void
+RT_NODE_INNER_256_SET(RT_NODE_INNER_256 *node, uint8 chunk, RT_PTR_ALLOC child)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ node->children[chunk] = child;
+}
+
+/* Set the value in the node-256 */
+static inline void
+RT_NODE_LEAF_256_SET(RT_NODE_LEAF_256 *node, uint8 chunk, RT_VALUE_TYPE value)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ node->isset[idx] |= ((bitmapword) 1 << bitnum);
+ node->values[chunk] = value;
+}
+
+/* Set the slot at the given chunk position */
+static inline void
+RT_NODE_INNER_256_DELETE(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ node->children[chunk] = RT_INVALID_PTR_ALLOC;
+}
+
+static inline void
+RT_NODE_LEAF_256_DELETE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ node->isset[idx] &= ~((bitmapword) 1 << bitnum);
+}
+
+/*
+ * Return the largest shift that will allowing storing the given key.
+ */
+static inline int
+RT_KEY_GET_SHIFT(uint64 key)
+{
+ if (key == 0)
+ return 0;
+ else
+ return (pg_leftmost_one_pos64(key) / RT_NODE_SPAN) * RT_NODE_SPAN;
+}
+
+/*
+ * Return the max value that can be stored in the tree with the given shift.
+ */
+static uint64
+RT_SHIFT_GET_MAX_VAL(int shift)
+{
+ if (shift == RT_MAX_SHIFT)
+ return UINT64_MAX;
+
+ return (UINT64CONST(1) << (shift + RT_NODE_SPAN)) - 1;
+}
+
+/*
+ * Allocate a new node with the given node kind.
+ */
+static RT_PTR_ALLOC
+RT_ALLOC_NODE(RT_RADIX_TREE *tree, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+ RT_PTR_ALLOC allocnode;
+ size_t allocsize;
+
+ if (is_leaf)
+ allocsize = RT_SIZE_CLASS_INFO[size_class].leaf_size;
+ else
+ allocsize = RT_SIZE_CLASS_INFO[size_class].inner_size;
+
+#ifdef RT_SHMEM
+ allocnode = dsa_allocate(tree->dsa, allocsize);
+#else
+ if (is_leaf)
+ allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->leaf_slabs[size_class],
+ allocsize);
+ else
+ allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->inner_slabs[size_class],
+ allocsize);
+#endif
+
+#ifdef RT_DEBUG
+ /* update the statistics */
+ tree->ctl->cnt[size_class]++;
+#endif
+
+ return allocnode;
+}
+
+/* Initialize the node contents */
+static inline void
+RT_INIT_NODE(RT_PTR_LOCAL node, uint8 kind, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+ if (is_leaf)
+ MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].leaf_size);
+ else
+ MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].inner_size);
+
+ node->kind = kind;
+
+ if (kind == RT_NODE_KIND_256)
+ /* See comment for the RT_NODE type */
+ Assert(node->fanout == 0);
+ else
+ node->fanout = RT_SIZE_CLASS_INFO[size_class].fanout;
+
+ /* Initialize slot_idxs to invalid values */
+ if (kind == RT_NODE_KIND_125)
+ {
+ RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+
+ memset(n125->slot_idxs, RT_INVALID_SLOT_IDX, sizeof(n125->slot_idxs));
+ }
+}
+
+/*
+ * Create a new node as the root. Subordinate nodes will be created during
+ * the insertion.
+ */
+static void
+RT_NEW_ROOT(RT_RADIX_TREE *tree, uint64 key)
+{
+ int shift = RT_KEY_GET_SHIFT(key);
+ bool is_leaf = shift == 0;
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(newnode, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ newnode->shift = shift;
+ tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(shift);
+ tree->ctl->root = allocnode;
+}
+
+static inline void
+RT_COPY_NODE(RT_PTR_LOCAL newnode, RT_PTR_LOCAL oldnode)
+{
+ newnode->shift = oldnode->shift;
+ newnode->count = oldnode->count;
+}
+
+/*
+ * Given a new allocated node and an old node, initalize the new
+ * node with the necessary fields and return its local pointer.
+ */
+static inline RT_PTR_LOCAL
+RT_SWITCH_NODE_KIND(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, RT_PTR_LOCAL node,
+ uint8 new_kind, uint8 new_class, bool is_leaf)
+{
+ RT_PTR_LOCAL newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(newnode, new_kind, new_class, is_leaf);
+ RT_COPY_NODE(newnode, node);
+
+ return newnode;
+}
+
+/* Free the given node */
+static void
+RT_FREE_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode)
+{
+ /* If we're deleting the root node, make the tree empty */
+ if (tree->ctl->root == allocnode)
+ {
+ tree->ctl->root = RT_INVALID_PTR_ALLOC;
+ tree->ctl->max_val = 0;
+ }
+
+#ifdef RT_DEBUG
+ {
+ int i;
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+ /* update the statistics */
+ for (i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ if (node->fanout == RT_SIZE_CLASS_INFO[i].fanout)
+ break;
+ }
+
+ /* fanout of node256 is intentionally 0 */
+ if (i == RT_SIZE_CLASS_COUNT)
+ i = RT_CLASS_256;
+
+ tree->ctl->cnt[i]--;
+ Assert(tree->ctl->cnt[i] >= 0);
+ }
+#endif
+
+#ifdef RT_SHMEM
+ dsa_free(tree->dsa, allocnode);
+#else
+ pfree(allocnode);
+#endif
+}
+
+/* Update the parent's pointer when growing a node */
+static inline void
+RT_NODE_UPDATE_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC new_child)
+{
+#define RT_ACTION_UPDATE
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+#undef RT_ACTION_UPDATE
+}
+
+/*
+ * Replace old_child with new_child, and free the old one.
+ */
+static void
+RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
+ RT_PTR_ALLOC stored_old_child, RT_PTR_LOCAL old_child,
+ RT_PTR_ALLOC new_child, uint64 key)
+{
+#ifdef USE_ASSERT_CHECKING
+ RT_PTR_LOCAL new = RT_PTR_GET_LOCAL(tree, new_child);
+
+ Assert(old_child->shift == new->shift);
+ Assert(old_child->count == new->count);
+#endif
+
+ if (parent == old_child)
+ {
+ /* Replace the root node with the new larger node */
+ tree->ctl->root = new_child;
+ }
+ else
+ RT_NODE_UPDATE_INNER(parent, key, new_child);
+
+ RT_FREE_NODE(tree, stored_old_child);
+}
+
+/*
+ * The radix tree doesn't have sufficient height. Extend the radix tree so
+ * it can store the key.
+ */
+static void
+RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
+{
+ int target_shift;
+ RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+ int shift = root->shift + RT_NODE_SPAN;
+
+ target_shift = RT_KEY_GET_SHIFT(key);
+
+ /* Grow tree from 'shift' to 'target_shift' */
+ while (shift <= target_shift)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL node;
+ RT_NODE_INNER_3 *n3;
+
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, true);
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(node, RT_NODE_KIND_3, RT_CLASS_3, true);
+ node->shift = shift;
+ node->count = 1;
+
+ n3 = (RT_NODE_INNER_3 *) node;
+ n3->base.chunks[0] = 0;
+ n3->children[0] = tree->ctl->root;
+
+ /* Update the root */
+ tree->ctl->root = allocnode;
+
+ shift += RT_NODE_SPAN;
+ }
+
+ tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(target_shift);
+}
+
+/*
+ * The radix tree doesn't have inner and leaf nodes for given key-value pair.
+ * Insert inner and leaf nodes from 'node' to bottom.
+ */
+static inline void
+RT_SET_EXTEND(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
+ RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node)
+{
+ int shift = node->shift;
+
+ Assert(RT_PTR_GET_LOCAL(tree, stored_node) == node);
+
+ while (shift >= RT_NODE_SPAN)
+ {
+ RT_PTR_ALLOC allocchild;
+ RT_PTR_LOCAL newchild;
+ int newshift = shift - RT_NODE_SPAN;
+ bool is_leaf = newshift == 0;
+
+ allocchild = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ newchild = RT_PTR_GET_LOCAL(tree, allocchild);
+ RT_INIT_NODE(newchild, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ newchild->shift = newshift;
+ RT_NODE_INSERT_INNER(tree, parent, stored_node, node, key, allocchild);
+
+ parent = node;
+ node = newchild;
+ stored_node = allocchild;
+ shift -= RT_NODE_SPAN;
+ }
+
+ RT_NODE_INSERT_LEAF(tree, parent, stored_node, node, key, value_p);
+ tree->ctl->num_keys++;
+}
+
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the child
+ * pointer is set to child_p.
+ */
+static inline bool
+RT_NODE_SEARCH_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC *child_p)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the pointer
+ * to the value is set to value_p.
+ */
+static inline bool
+RT_NODE_SEARCH_LEAF(RT_PTR_LOCAL node, uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_INNER(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_LEAF(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+#endif
+
+/*
+ * Insert "child" into "node".
+ *
+ * "parent" is the parent of "node", so the grandparent of the child.
+ * If the node we're inserting into needs to grow, we update the parent's
+ * child pointer with the pointer to the new larger node.
+ */
+static bool
+RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_PTR_ALLOC child)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/* Like RT_NODE_INSERT_INNER, but for leaf nodes */
+static bool
+RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Create the radix tree in the given memory context and return it.
+ */
+RT_SCOPE RT_RADIX_TREE *
+#ifdef RT_SHMEM
+RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id)
+#else
+RT_CREATE(MemoryContext ctx)
+#endif
+{
+ RT_RADIX_TREE *tree;
+ MemoryContext old_ctx;
+#ifdef RT_SHMEM
+ dsa_pointer dp;
+#endif
+
+ old_ctx = MemoryContextSwitchTo(ctx);
+
+ tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+ tree->context = ctx;
+
+#ifdef RT_SHMEM
+ tree->dsa = dsa;
+ dp = dsa_allocate0(dsa, sizeof(RT_RADIX_TREE_CONTROL));
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, dp);
+ tree->ctl->handle = dp;
+ tree->ctl->magic = RT_RADIX_TREE_MAGIC;
+ LWLockInitialize(&tree->ctl->lock, tranche_id);
+#else
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) palloc0(sizeof(RT_RADIX_TREE_CONTROL));
+
+ /* Create a slab context for each size class */
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ RT_SIZE_CLASS_ELEM size_class = RT_SIZE_CLASS_INFO[i];
+ size_t inner_blocksize = RT_SLAB_BLOCK_SIZE(size_class.inner_size);
+ size_t leaf_blocksize = RT_SLAB_BLOCK_SIZE(size_class.leaf_size);
+
+ tree->inner_slabs[i] = SlabContextCreate(ctx,
+ size_class.name,
+ inner_blocksize,
+ size_class.inner_size);
+ tree->leaf_slabs[i] = SlabContextCreate(ctx,
+ size_class.name,
+ leaf_blocksize,
+ size_class.leaf_size);
+ }
+#endif
+
+ tree->ctl->root = RT_INVALID_PTR_ALLOC;
+
+ MemoryContextSwitchTo(old_ctx);
+
+ return tree;
+}
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE *
+RT_ATTACH(dsa_area *dsa, RT_HANDLE handle)
+{
+ RT_RADIX_TREE *tree;
+ dsa_pointer control;
+
+ tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+
+ /* Find the control object in shard memory */
+ control = handle;
+
+ tree->dsa = dsa;
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, control);
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+ return tree;
+}
+
+RT_SCOPE void
+RT_DETACH(RT_RADIX_TREE *tree)
+{
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ pfree(tree);
+}
+
+RT_SCOPE RT_HANDLE
+RT_GET_HANDLE(RT_RADIX_TREE *tree)
+{
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ return tree->ctl->handle;
+}
+
+/*
+ * Recursively free all nodes allocated to the DSA area.
+ */
+static inline void
+RT_FREE_RECURSE(RT_RADIX_TREE *tree, RT_PTR_ALLOC ptr)
+{
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, ptr);
+
+ check_stack_depth();
+ CHECK_FOR_INTERRUPTS();
+
+ /* The leaf node doesn't have child pointers */
+ if (RT_NODE_IS_LEAF(node))
+ {
+ dsa_free(tree->dsa, ptr);
+ return;
+ }
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+ for (int i = 0; i < n3->base.n.count; i++)
+ RT_FREE_RECURSE(tree, n3->children[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+ for (int i = 0; i < n32->base.n.count; i++)
+ RT_FREE_RECURSE(tree, n32->children[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) node;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+ continue;
+
+ RT_FREE_RECURSE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+ }
+
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ RT_FREE_RECURSE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+ }
+
+ break;
+ }
+ }
+
+ /* Free the inner node */
+ dsa_free(tree->dsa, ptr);
+}
+#endif
+
+/*
+ * Free the given radix tree.
+ */
+RT_SCOPE void
+RT_FREE(RT_RADIX_TREE *tree)
+{
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+ /* Free all memory used for radix tree nodes */
+ if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ RT_FREE_RECURSE(tree, tree->ctl->root);
+
+ /*
+ * Vandalize the control block to help catch programming error where
+ * other backends access the memory formerly occupied by this radix tree.
+ */
+ tree->ctl->magic = 0;
+ dsa_free(tree->dsa, tree->ctl->handle);
+#else
+ pfree(tree->ctl);
+
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ MemoryContextDelete(tree->inner_slabs[i]);
+ MemoryContextDelete(tree->leaf_slabs[i]);
+ }
+#endif
+
+ pfree(tree);
+}
+
+/*
+ * Set key to value. If the entry already exists, we update its value to 'value'
+ * and return true. Returns false if entry doesn't yet exist.
+ */
+RT_SCOPE bool
+RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+ int shift;
+ bool updated;
+ RT_PTR_LOCAL parent;
+ RT_PTR_ALLOC stored_child;
+ RT_PTR_LOCAL child;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ RT_LOCK_EXCLUSIVE(tree);
+
+ /* Empty tree, create the root */
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ RT_NEW_ROOT(tree, key);
+
+ /* Extend the tree if necessary */
+ if (key > tree->ctl->max_val)
+ RT_EXTEND(tree, key);
+
+ stored_child = tree->ctl->root;
+ parent = RT_PTR_GET_LOCAL(tree, stored_child);
+ shift = parent->shift;
+
+ /* Descend the tree until we reach a leaf node */
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC new_child = RT_INVALID_PTR_ALLOC;;
+
+ child = RT_PTR_GET_LOCAL(tree, stored_child);
+
+ if (RT_NODE_IS_LEAF(child))
+ break;
+
+ if (!RT_NODE_SEARCH_INNER(child, key, &new_child))
+ {
+ RT_SET_EXTEND(tree, key, value_p, parent, stored_child, child);
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ parent = child;
+ stored_child = new_child;
+ shift -= RT_NODE_SPAN;
+ }
+
+ updated = RT_NODE_INSERT_LEAF(tree, parent, stored_child, child, key, value_p);
+
+ /* Update the statistics */
+ if (!updated)
+ tree->ctl->num_keys++;
+
+ RT_UNLOCK(tree);
+ return updated;
+}
+
+/*
+ * Search the given key in the radix tree. Return true if there is the key,
+ * otherwise return false. On success, we set the value to *value_p so it must
+ * not be NULL.
+ */
+RT_SCOPE bool
+RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+ RT_PTR_LOCAL node;
+ int shift;
+ bool found;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+ Assert(value_p != NULL);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ node = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+ shift = node->shift;
+
+ /* Descend the tree until a leaf node */
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+ if (RT_NODE_IS_LEAF(node))
+ break;
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ node = RT_PTR_GET_LOCAL(tree, child);
+ shift -= RT_NODE_SPAN;
+ }
+
+ found = RT_NODE_SEARCH_LEAF(node, key, value_p);
+
+ RT_UNLOCK(tree);
+ return found;
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Delete the given key from the radix tree. Return true if the key is found (and
+ * deleted), otherwise do nothing and return false.
+ */
+RT_SCOPE bool
+RT_DELETE(RT_RADIX_TREE *tree, uint64 key)
+{
+ RT_PTR_LOCAL node;
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_ALLOC stack[RT_MAX_LEVEL] = {0};
+ int shift;
+ int level;
+ bool deleted;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ RT_LOCK_EXCLUSIVE(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ /*
+ * Descend the tree to search the key while building a stack of nodes we
+ * visited.
+ */
+ allocnode = tree->ctl->root;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift = node->shift;
+ level = -1;
+ while (shift > 0)
+ {
+ RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+ /* Push the current node to the stack */
+ stack[++level] = allocnode;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ allocnode = child;
+ shift -= RT_NODE_SPAN;
+ }
+
+ /* Delete the key from the leaf node if exists */
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ deleted = RT_NODE_DELETE_LEAF(node, key);
+
+ if (!deleted)
+ {
+ /* no key is found in the leaf node */
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ /* Found the key to delete. Update the statistics */
+ tree->ctl->num_keys--;
+
+ /*
+ * Return if the leaf node still has keys and we don't need to delete the
+ * node.
+ */
+ if (node->count > 0)
+ {
+ RT_UNLOCK(tree);
+ return true;
+ }
+
+ /* Free the empty leaf node */
+ RT_FREE_NODE(tree, allocnode);
+
+ /* Delete the key in inner nodes recursively */
+ while (level >= 0)
+ {
+ allocnode = stack[level--];
+
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ deleted = RT_NODE_DELETE_INNER(node, key);
+ Assert(deleted);
+
+ /* If the node didn't become empty, we stop deleting the key */
+ if (node->count > 0)
+ break;
+
+ /* The node became empty */
+ RT_FREE_NODE(tree, allocnode);
+ }
+
+ RT_UNLOCK(tree);
+ return true;
+}
+#endif
+
+static inline void
+RT_ITER_UPDATE_KEY(RT_ITER *iter, uint8 chunk, uint8 shift)
+{
+ iter->key &= ~(((uint64) RT_CHUNK_MASK) << shift);
+ iter->key |= (((uint64) chunk) << shift);
+}
+
+/*
+ * Advance the slot in the inner node. Return the child if exists, otherwise
+ * null.
+ */
+static inline RT_PTR_LOCAL
+RT_NODE_INNER_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Advance the slot in the leaf node. On success, return true and the value
+ * is set to value_p, otherwise return false.
+ */
+static inline bool
+RT_NODE_LEAF_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter,
+ RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Update each node_iter for inner nodes in the iterator node stack.
+ */
+static void
+RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
+{
+ int level = from;
+ RT_PTR_LOCAL node = from_node;
+
+ for (;;)
+ {
+ RT_NODE_ITER *node_iter = &(iter->stack[level--]);
+
+ node_iter->node = node;
+ node_iter->current_idx = -1;
+
+ /* We don't advance the leaf node iterator here */
+ if (RT_NODE_IS_LEAF(node))
+ return;
+
+ /* Advance to the next slot in the inner node */
+ node = RT_NODE_INNER_ITERATE_NEXT(iter, node_iter);
+
+ /* We must find the first children in the node */
+ Assert(node);
+ }
+}
+
+/*
+ * Create and return the iterator for the given radix tree.
+ *
+ * The radix tree is locked in shared mode during the iteration, so
+ * RT_END_ITERATE needs to be called when finished to release the lock.
+ */
+RT_SCOPE RT_ITER *
+RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
+{
+ MemoryContext old_ctx;
+ RT_ITER *iter;
+ RT_PTR_LOCAL root;
+ int top_level;
+
+ old_ctx = MemoryContextSwitchTo(tree->context);
+
+ iter = (RT_ITER *) palloc0(sizeof(RT_ITER));
+ iter->tree = tree;
+
+ RT_LOCK_SHARED(tree);
+
+ /* empty tree */
+ if (!iter->tree->ctl->root)
+ return iter;
+
+ root = RT_PTR_GET_LOCAL(tree, iter->tree->ctl->root);
+ top_level = root->shift / RT_NODE_SPAN;
+ iter->stack_len = top_level;
+
+ /*
+ * Descend to the left most leaf node from the root. The key is being
+ * constructed while descending to the leaf.
+ */
+ RT_UPDATE_ITER_STACK(iter, root, top_level);
+
+ MemoryContextSwitchTo(old_ctx);
+
+ return iter;
+}
+
+/*
+ * Return true with setting key_p and value_p if there is next key. Otherwise
+ * return false.
+ */
+RT_SCOPE bool
+RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p)
+{
+ /* Empty tree */
+ if (!iter->tree->ctl->root)
+ return false;
+
+ for (;;)
+ {
+ RT_PTR_LOCAL child = NULL;
+ RT_VALUE_TYPE value;
+ int level;
+ bool found;
+
+ /* Advance the leaf node iterator to get next key-value pair */
+ found = RT_NODE_LEAF_ITERATE_NEXT(iter, &(iter->stack[0]), &value);
+
+ if (found)
+ {
+ *key_p = iter->key;
+ *value_p = value;
+ return true;
+ }
+
+ /*
+ * We've visited all values in the leaf node, so advance inner node
+ * iterators from the level=1 until we find the next child node.
+ */
+ for (level = 1; level <= iter->stack_len; level++)
+ {
+ child = RT_NODE_INNER_ITERATE_NEXT(iter, &(iter->stack[level]));
+
+ if (child)
+ break;
+ }
+
+ /* the iteration finished */
+ if (!child)
+ return false;
+
+ /*
+ * Set the node to the node iterator and update the iterator stack
+ * from this node.
+ */
+ RT_UPDATE_ITER_STACK(iter, child, level - 1);
+
+ /* Node iterators are updated, so try again from the leaf */
+ }
+
+ return false;
+}
+
+/*
+ * Terminate the iteration and release the lock.
+ *
+ * This function needs to be called after finishing or when exiting an
+ * iteration.
+ */
+RT_SCOPE void
+RT_END_ITERATE(RT_ITER *iter)
+{
+#ifdef RT_SHMEM
+ Assert(LWLockHeldByMe(&iter->tree->ctl->lock));
+#endif
+
+ RT_UNLOCK(iter->tree);
+ pfree(iter);
+}
+
+/*
+ * Return the statistics of the amount of memory used by the radix tree.
+ */
+RT_SCOPE uint64
+RT_MEMORY_USAGE(RT_RADIX_TREE *tree)
+{
+ Size total = 0;
+
+ RT_LOCK_SHARED(tree);
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ total = dsa_get_total_size(tree->dsa);
+#else
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ total += MemoryContextMemAllocated(tree->inner_slabs[i], true);
+ total += MemoryContextMemAllocated(tree->leaf_slabs[i], true);
+ }
+#endif
+
+ RT_UNLOCK(tree);
+ return total;
+}
+
+/*
+ * Verify the radix tree node.
+ */
+static void
+RT_VERIFY_NODE(RT_PTR_LOCAL node)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(node->count >= 0);
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE_BASE_3 *n3 = (RT_NODE_BASE_3 *) node;
+
+ for (int i = 1; i < n3->n.count; i++)
+ Assert(n3->chunks[i - 1] < n3->chunks[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE_BASE_32 *n32 = (RT_NODE_BASE_32 *) node;
+
+ for (int i = 1; i < n32->n.count; i++)
+ Assert(n32->chunks[i - 1] < n32->chunks[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+ int cnt = 0;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ uint8 slot = n125->slot_idxs[i];
+ int idx = RT_BM_IDX(slot);
+ int bitnum = RT_BM_BIT(slot);
+
+ if (!RT_NODE_125_IS_CHUNK_USED(n125, i))
+ continue;
+
+ /* Check if the corresponding slot is used */
+ Assert(slot < node->fanout);
+ Assert((n125->isset[idx] & ((bitmapword) 1 << bitnum)) != 0);
+
+ cnt++;
+ }
+
+ Assert(n125->n.count == cnt);
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+ int cnt = 0;
+
+ for (int i = 0; i < RT_BM_IDX(RT_NODE_MAX_SLOTS); i++)
+ cnt += bmw_popcount(n256->isset[i]);
+
+ /* Check if the number of used chunk matches */
+ Assert(n256->base.n.count == cnt);
+
+ break;
+ }
+ }
+ }
+#endif
+}
+
+/***************** DEBUG FUNCTIONS *****************/
+#ifdef RT_DEBUG
+
+#define RT_UINT64_FORMAT_HEX "%" INT64_MODIFIER "X"
+
+RT_SCOPE void
+RT_STATS(RT_RADIX_TREE *tree)
+{
+ RT_LOCK_SHARED(tree);
+
+ fprintf(stderr, "max_val = " UINT64_FORMAT "\n", tree->ctl->max_val);
+ fprintf(stderr, "num_keys = " UINT64_FORMAT "\n", tree->ctl->num_keys);
+
+#ifdef RT_SHMEM
+ fprintf(stderr, "handle = " UINT64_FORMAT "\n", tree->ctl->handle);
+#endif
+
+ if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+
+ fprintf(stderr, "height = %d, n3 = %u, n15 = %u, n32 = %u, n125 = %u, n256 = %u\n",
+ root->shift / RT_NODE_SPAN,
+ tree->ctl->cnt[RT_CLASS_3],
+ tree->ctl->cnt[RT_CLASS_32_MIN],
+ tree->ctl->cnt[RT_CLASS_32_MAX],
+ tree->ctl->cnt[RT_CLASS_125],
+ tree->ctl->cnt[RT_CLASS_256]);
+ }
+
+ RT_UNLOCK(tree);
+}
+
+static void
+RT_DUMP_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, int level,
+ bool recurse, StringInfo buf)
+{
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+ StringInfoData spaces;
+
+ initStringInfo(&spaces);
+ appendStringInfoSpaces(&spaces, (level * 4) + 1);
+
+ appendStringInfo(buf, "%s%s[%s] kind %d, fanout %d, count %u, shift %u:\n",
+ spaces.data,
+ level == 0 ? "" : "-> ",
+ RT_NODE_IS_LEAF(node) ? "LEAF" : "INNR",
+ (node->kind == RT_NODE_KIND_3) ? 3 :
+ (node->kind == RT_NODE_KIND_32) ? 32 :
+ (node->kind == RT_NODE_KIND_125) ? 125 : 256,
+ node->fanout == 0 ? 256 : node->fanout,
+ node->count, node->shift);
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ for (int i = 0; i < node->count; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_3 *n3 = (RT_NODE_LEAF_3 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+ spaces.data, i, n3->base.chunks[i]);
+ }
+ else
+ {
+ RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X",
+ spaces.data, i, n3->base.chunks[i]);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, n3->children[i], level + 1,
+ recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ for (int i = 0; i < node->count; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_32 *n32 = (RT_NODE_LEAF_32 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+ spaces.data, i, n32->base.chunks[i]);
+ }
+ else
+ {
+ RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X",
+ spaces.data, i, n32->base.chunks[i]);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, n32->children[i], level + 1,
+ recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_BASE_125 *b125 = (RT_NODE_BASE_125 *) node;
+ char *sep = "";
+
+ appendStringInfo(buf, "%sslot_idxs: ", spaces.data);
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+ continue;
+
+ appendStringInfo(buf, "%s[%d]=%d ",
+ sep, i, b125->slot_idxs[i]);
+ sep = ",";
+ }
+
+ appendStringInfo(buf, "\n%sisset-bitmap: ", spaces.data);
+ for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+ appendStringInfo(buf, "%X ", ((uint8 *) b125->isset)[i]);
+ appendStringInfo(buf, "\n");
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+ continue;
+
+ if (RT_NODE_IS_LEAF(node))
+ appendStringInfo(buf, "%schunk 0x%X\n",
+ spaces.data, i);
+ else
+ {
+ RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) b125;
+
+ appendStringInfo(buf, "%schunk 0x%X",
+ spaces.data, i);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i),
+ level + 1, recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+ appendStringInfo(buf, "%sisset-bitmap: ", spaces.data);
+ for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+ appendStringInfo(buf, "%X ", ((uint8 *) n256->isset)[i]);
+ appendStringInfo(buf, "\n");
+ }
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ appendStringInfo(buf, "%schunk 0x%X\n",
+ spaces.data, i);
+ }
+ else
+ {
+ RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ appendStringInfo(buf, "%schunk 0x%X",
+ spaces.data, i);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i),
+ level + 1, recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ }
+}
+
+RT_SCOPE void
+RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key)
+{
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL node;
+ StringInfoData buf;
+ int shift;
+ int level = 0;
+
+ RT_STATS(tree);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "empty tree\n");
+ return;
+ }
+
+ if (key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "key " UINT64_FORMAT "(0x" RT_UINT64_FORMAT_HEX ") is larger than max val\n",
+ key, key);
+ return;
+ }
+
+ initStringInfo(&buf);
+ allocnode = tree->ctl->root;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift = node->shift;
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC child;
+
+ RT_DUMP_NODE(tree, allocnode, level, false, &buf);
+
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_VALUE_TYPE dummy;
+
+ /* We reached at a leaf node, find the corresponding slot */
+ RT_NODE_SEARCH_LEAF(node, key, &dummy);
+
+ break;
+ }
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ break;
+
+ allocnode = child;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift -= RT_NODE_SPAN;
+ level++;
+ }
+ RT_UNLOCK(tree);
+
+ fprintf(stderr, "%s", buf.data);
+}
+
+RT_SCOPE void
+RT_DUMP(RT_RADIX_TREE *tree)
+{
+ StringInfoData buf;
+
+ RT_STATS(tree);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "empty tree\n");
+ return;
+ }
+
+ initStringInfo(&buf);
+
+ RT_DUMP_NODE(tree, tree->ctl->root, 0, true, &buf);
+ RT_UNLOCK(tree);
+
+ fprintf(stderr, "%s",buf.data);
+}
+#endif
+
+#endif /* RT_DEFINE */
+
+
+/* undefine external parameters, so next radix tree can be defined */
+#undef RT_PREFIX
+#undef RT_SCOPE
+#undef RT_DECLARE
+#undef RT_DEFINE
+#undef RT_VALUE_TYPE
+
+/* locally declared macros */
+#undef RT_MAKE_PREFIX
+#undef RT_MAKE_NAME
+#undef RT_MAKE_NAME_
+#undef RT_NODE_SPAN
+#undef RT_NODE_MAX_SLOTS
+#undef RT_CHUNK_MASK
+#undef RT_MAX_SHIFT
+#undef RT_MAX_LEVEL
+#undef RT_GET_KEY_CHUNK
+#undef RT_BM_IDX
+#undef RT_BM_BIT
+#undef RT_LOCK_EXCLUSIVE
+#undef RT_LOCK_SHARED
+#undef RT_UNLOCK
+#undef RT_NODE_IS_LEAF
+#undef RT_NODE_MUST_GROW
+#undef RT_NODE_KIND_COUNT
+#undef RT_SIZE_CLASS_COUNT
+#undef RT_SLOT_IDX_LIMIT
+#undef RT_INVALID_SLOT_IDX
+#undef RT_SLAB_BLOCK_SIZE
+#undef RT_RADIX_TREE_MAGIC
+#undef RT_UINT64_FORMAT_HEX
+
+/* type declarations */
+#undef RT_RADIX_TREE
+#undef RT_RADIX_TREE_CONTROL
+#undef RT_PTR_LOCAL
+#undef RT_PTR_ALLOC
+#undef RT_INVALID_PTR_ALLOC
+#undef RT_HANDLE
+#undef RT_ITER
+#undef RT_NODE
+#undef RT_NODE_ITER
+#undef RT_NODE_KIND_3
+#undef RT_NODE_KIND_32
+#undef RT_NODE_KIND_125
+#undef RT_NODE_KIND_256
+#undef RT_NODE_BASE_3
+#undef RT_NODE_BASE_32
+#undef RT_NODE_BASE_125
+#undef RT_NODE_BASE_256
+#undef RT_NODE_INNER_3
+#undef RT_NODE_INNER_32
+#undef RT_NODE_INNER_125
+#undef RT_NODE_INNER_256
+#undef RT_NODE_LEAF_3
+#undef RT_NODE_LEAF_32
+#undef RT_NODE_LEAF_125
+#undef RT_NODE_LEAF_256
+#undef RT_SIZE_CLASS
+#undef RT_SIZE_CLASS_ELEM
+#undef RT_SIZE_CLASS_INFO
+#undef RT_CLASS_3
+#undef RT_CLASS_32_MIN
+#undef RT_CLASS_32_MAX
+#undef RT_CLASS_125
+#undef RT_CLASS_256
+
+/* function declarations */
+#undef RT_CREATE
+#undef RT_FREE
+#undef RT_ATTACH
+#undef RT_DETACH
+#undef RT_GET_HANDLE
+#undef RT_SEARCH
+#undef RT_SET
+#undef RT_BEGIN_ITERATE
+#undef RT_ITERATE_NEXT
+#undef RT_END_ITERATE
+#undef RT_USE_DELETE
+#undef RT_DELETE
+#undef RT_MEMORY_USAGE
+#undef RT_DUMP
+#undef RT_DUMP_NODE
+#undef RT_DUMP_SEARCH
+#undef RT_STATS
+
+/* internal helper functions */
+#undef RT_NEW_ROOT
+#undef RT_ALLOC_NODE
+#undef RT_INIT_NODE
+#undef RT_FREE_NODE
+#undef RT_FREE_RECURSE
+#undef RT_EXTEND
+#undef RT_SET_EXTEND
+#undef RT_SWITCH_NODE_KIND
+#undef RT_COPY_NODE
+#undef RT_REPLACE_NODE
+#undef RT_PTR_GET_LOCAL
+#undef RT_PTR_ALLOC_IS_VALID
+#undef RT_NODE_3_SEARCH_EQ
+#undef RT_NODE_32_SEARCH_EQ
+#undef RT_NODE_3_GET_INSERTPOS
+#undef RT_NODE_32_GET_INSERTPOS
+#undef RT_CHUNK_CHILDREN_ARRAY_SHIFT
+#undef RT_CHUNK_VALUES_ARRAY_SHIFT
+#undef RT_CHUNK_CHILDREN_ARRAY_DELETE
+#undef RT_CHUNK_VALUES_ARRAY_DELETE
+#undef RT_CHUNK_CHILDREN_ARRAY_COPY
+#undef RT_CHUNK_VALUES_ARRAY_COPY
+#undef RT_NODE_125_IS_CHUNK_USED
+#undef RT_NODE_INNER_125_GET_CHILD
+#undef RT_NODE_LEAF_125_GET_VALUE
+#undef RT_NODE_INNER_256_IS_CHUNK_USED
+#undef RT_NODE_LEAF_256_IS_CHUNK_USED
+#undef RT_NODE_INNER_256_GET_CHILD
+#undef RT_NODE_LEAF_256_GET_VALUE
+#undef RT_NODE_INNER_256_SET
+#undef RT_NODE_LEAF_256_SET
+#undef RT_NODE_INNER_256_DELETE
+#undef RT_NODE_LEAF_256_DELETE
+#undef RT_KEY_GET_SHIFT
+#undef RT_SHIFT_GET_MAX_VAL
+#undef RT_NODE_SEARCH_INNER
+#undef RT_NODE_SEARCH_LEAF
+#undef RT_NODE_UPDATE_INNER
+#undef RT_NODE_DELETE_INNER
+#undef RT_NODE_DELETE_LEAF
+#undef RT_NODE_INSERT_INNER
+#undef RT_NODE_INSERT_LEAF
+#undef RT_NODE_INNER_ITERATE_NEXT
+#undef RT_NODE_LEAF_ITERATE_NEXT
+#undef RT_UPDATE_ITER_STACK
+#undef RT_ITER_UPDATE_KEY
+#undef RT_VERIFY_NODE
+
+#undef RT_DEBUG
diff --git a/src/include/lib/radixtree_delete_impl.h b/src/include/lib/radixtree_delete_impl.h
new file mode 100644
index 0000000000..5f6dda1f12
--- /dev/null
+++ b/src/include/lib/radixtree_delete_impl.h
@@ -0,0 +1,122 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_delete_impl.h
+ * Common implementation for deletion in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ * TODO: Shrink nodes when deletion would allow them to fit in a smaller
+ * size class.
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_delete_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_DELETE(n3->base.chunks, n3->values,
+ n3->base.n.count, idx);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_DELETE(n3->base.chunks, n3->children,
+ n3->base.n.count, idx);
+#endif
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_DELETE(n32->base.chunks, n32->values,
+ n32->base.n.count, idx);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_DELETE(n32->base.chunks, n32->children,
+ n32->base.n.count, idx);
+#endif
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+ int idx;
+ int bitnum;
+
+ if (slotpos == RT_INVALID_SLOT_IDX)
+ return false;
+
+ idx = RT_BM_IDX(slotpos);
+ bitnum = RT_BM_BIT(slotpos);
+ n125->base.isset[idx] &= ~((bitmapword) 1 << bitnum);
+ n125->base.slot_idxs[chunk] = RT_INVALID_SLOT_IDX;
+
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_DELETE(n256, chunk);
+#else
+ RT_NODE_INNER_256_DELETE(n256, chunk);
+#endif
+ break;
+ }
+ }
+
+ /* update statistics */
+ node->count--;
+
+ return true;
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
new file mode 100644
index 0000000000..c18e26b537
--- /dev/null
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -0,0 +1,332 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_insert_impl.h
+ * Common implementation for insertion in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_insert_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+ bool chunk_exists = false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ const bool is_leaf = true;
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+ const bool is_leaf = false;
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx;
+
+ idx = RT_NODE_3_SEARCH_EQ(&n3->base, chunk);
+ if (idx != -1)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+#ifdef RT_NODE_LEVEL_LEAF
+ n3->values[idx] = *value_p;
+#else
+ n3->children[idx] = child;
+#endif
+ break;
+ }
+
+ if (unlikely(RT_NODE_MUST_GROW(n3)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE32_TYPE *new32;
+ const uint8 new_kind = RT_NODE_KIND_32;
+ const RT_SIZE_CLASS new_class = RT_CLASS_32_MIN;
+
+ /* grow node from 3 to 32 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_COPY(n3->base.chunks, n3->values,
+ new32->base.chunks, new32->values);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_COPY(n3->base.chunks, n3->children,
+ new32->base.chunks, new32->children);
+#endif
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
+ int count = n3->base.n.count;
+
+ /* shift chunks and children */
+ if (insertpos < count)
+ {
+ Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_SHIFT(n3->base.chunks, n3->values,
+ count, insertpos);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_SHIFT(n3->base.chunks, n3->children,
+ count, insertpos);
+#endif
+ }
+
+ n3->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+ n3->values[insertpos] = *value_p;
+#else
+ n3->children[insertpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_32:
+ {
+ const RT_SIZE_CLASS_ELEM class32_max = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX];
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx;
+
+ idx = RT_NODE_32_SEARCH_EQ(&n32->base, chunk);
+ if (idx != -1)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+#ifdef RT_NODE_LEVEL_LEAF
+ n32->values[idx] = *value_p;
+#else
+ n32->children[idx] = child;
+#endif
+ break;
+ }
+
+ if (unlikely(RT_NODE_MUST_GROW(n32)) &&
+ n32->base.n.fanout < class32_max.fanout)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ const RT_SIZE_CLASS_ELEM class32_min = RT_SIZE_CLASS_INFO[RT_CLASS_32_MIN];
+ const RT_SIZE_CLASS new_class = RT_CLASS_32_MAX;
+
+ Assert(n32->base.n.fanout == class32_min.fanout);
+
+ /* grow to the next size class of this kind */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ n32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ memcpy(newnode, node, class32_min.leaf_size);
+#else
+ memcpy(newnode, node, class32_min.inner_size);
+#endif
+ newnode->fanout = class32_max.fanout;
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+
+ if (unlikely(RT_NODE_MUST_GROW(n32)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE125_TYPE *new125;
+ const uint8 new_kind = RT_NODE_KIND_125;
+ const RT_SIZE_CLASS new_class = RT_CLASS_125;
+
+ Assert(n32->base.n.fanout == class32_max.fanout);
+
+ /* grow node from 32 to 125 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new125 = (RT_NODE125_TYPE *) newnode;
+
+ for (int i = 0; i < class32_max.fanout; i++)
+ {
+ new125->base.slot_idxs[n32->base.chunks[i]] = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ new125->values[i] = n32->values[i];
+#else
+ new125->children[i] = n32->children[i];
+#endif
+ }
+
+ /*
+ * Since we just copied a dense array, we can set the bits
+ * using a single store, provided the length of that array
+ * is at most the number of bits in a bitmapword.
+ */
+ Assert(class32_max.fanout <= sizeof(bitmapword) * BITS_PER_BYTE);
+ new125->base.isset[0] = (bitmapword) (((uint64) 1 << class32_max.fanout) - 1);
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
+ int count = n32->base.n.count;
+
+ if (insertpos < count)
+ {
+ Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_SHIFT(n32->base.chunks, n32->values,
+ count, insertpos);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_SHIFT(n32->base.chunks, n32->children,
+ count, insertpos);
+#endif
+ }
+
+ n32->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+ n32->values[insertpos] = *value_p;
+#else
+ n32->children[insertpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+ int cnt = 0;
+
+ if (slotpos != RT_INVALID_SLOT_IDX)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+#ifdef RT_NODE_LEVEL_LEAF
+ n125->values[slotpos] = *value_p;
+#else
+ n125->children[slotpos] = child;
+#endif
+ break;
+ }
+
+ if (unlikely(RT_NODE_MUST_GROW(n125)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE256_TYPE *new256;
+ const uint8 new_kind = RT_NODE_KIND_256;
+ const RT_SIZE_CLASS new_class = RT_CLASS_256;
+
+ /* grow node from 125 to 256 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new256 = (RT_NODE256_TYPE *) newnode;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS && cnt < n125->base.n.count; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+ continue;
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_SET(new256, i, RT_NODE_LEAF_125_GET_VALUE(n125, i));
+#else
+ RT_NODE_INNER_256_SET(new256, i, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+ cnt++;
+ }
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int idx;
+ bitmapword inverse;
+
+ /* get the first word with at least one bit not set */
+ for (idx = 0; idx < RT_BM_IDX(RT_SLOT_IDX_LIMIT); idx++)
+ {
+ if (n125->base.isset[idx] < ~((bitmapword) 0))
+ break;
+ }
+
+ /* To get the first unset bit in X, get the first set bit in ~X */
+ inverse = ~(n125->base.isset[idx]);
+ slotpos = idx * BITS_PER_BITMAPWORD;
+ slotpos += bmw_rightmost_one_pos(inverse);
+ Assert(slotpos < node->fanout);
+
+ /* mark the slot used */
+ n125->base.isset[idx] |= bmw_rightmost_one(inverse);
+ n125->base.slot_idxs[chunk] = slotpos;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ n125->values[slotpos] = *value_p;
+#else
+ n125->children[slotpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ chunk_exists = RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk);
+#else
+ chunk_exists = RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk);
+#endif
+ Assert(chunk_exists || node->count < RT_NODE_MAX_SLOTS);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_SET(n256, chunk, *value_p);
+#else
+ RT_NODE_INNER_256_SET(n256, chunk, child);
+#endif
+ break;
+ }
+ }
+
+ /* Update statistics */
+ if (!chunk_exists)
+ node->count++;
+
+ /*
+ * Done. Finally, verify the chunk and value is inserted or replaced
+ * properly in the node.
+ */
+ RT_VERIFY_NODE(node);
+
+ return chunk_exists;
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_iter_impl.h b/src/include/lib/radixtree_iter_impl.h
new file mode 100644
index 0000000000..98c78eb237
--- /dev/null
+++ b/src/include/lib/radixtree_iter_impl.h
@@ -0,0 +1,153 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_iter_impl.h
+ * Common implementation for iteration in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_iter_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ bool found = false;
+ uint8 key_chunk;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_VALUE_TYPE value;
+
+ Assert(RT_NODE_IS_LEAF(node_iter->node));
+#else
+ RT_PTR_LOCAL child = NULL;
+
+ Assert(!RT_NODE_IS_LEAF(node_iter->node));
+#endif
+
+#ifdef RT_SHMEM
+ Assert(iter->tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ switch (node_iter->node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node_iter->node;
+
+ node_iter->current_idx++;
+ if (node_iter->current_idx >= n3->base.n.count)
+ break;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = n3->values[node_iter->current_idx];
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, n3->children[node_iter->current_idx]);
+#endif
+ key_chunk = n3->base.chunks[node_iter->current_idx];
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node_iter->node;
+
+ node_iter->current_idx++;
+ if (node_iter->current_idx >= n32->base.n.count)
+ break;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ value = n32->values[node_iter->current_idx];
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, n32->children[node_iter->current_idx]);
+#endif
+ key_chunk = n32->base.chunks[node_iter->current_idx];
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node_iter->node;
+ int i;
+
+ for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (RT_NODE_125_IS_CHUNK_USED((RT_NODE_BASE_125 *) n125, i))
+ break;
+ }
+
+ if (i >= RT_NODE_MAX_SLOTS)
+ break;
+
+ node_iter->current_idx = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = RT_NODE_LEAF_125_GET_VALUE(n125, i);
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+ key_chunk = i;
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node_iter->node;
+ int i;
+
+ for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ {
+#ifdef RT_NODE_LEVEL_LEAF
+ if (RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+#else
+ if (RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+#endif
+ break;
+ }
+
+ if (i >= RT_NODE_MAX_SLOTS)
+ break;
+
+ node_iter->current_idx = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = RT_NODE_LEAF_256_GET_VALUE(n256, i);
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+#endif
+ key_chunk = i;
+ found = true;
+ break;
+ }
+ }
+
+ if (found)
+ {
+ RT_ITER_UPDATE_KEY(iter, key_chunk, node_iter->node->shift);
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = value;
+#endif
+ }
+
+#ifdef RT_NODE_LEVEL_LEAF
+ return found;
+#else
+ return child;
+#endif
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_search_impl.h b/src/include/lib/radixtree_search_impl.h
new file mode 100644
index 0000000000..a8925c75d0
--- /dev/null
+++ b/src/include/lib/radixtree_search_impl.h
@@ -0,0 +1,138 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_search_impl.h
+ * Common implementation for search in leaf and inner nodes, plus
+ * update for inner nodes only.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_search_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(value_p != NULL);
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+#ifndef RT_ACTION_UPDATE
+ Assert(child_p != NULL);
+#endif
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+#ifdef RT_ACTION_UPDATE
+ Assert(idx >= 0);
+ n3->children[idx] = new_child;
+#else
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n3->values[idx];
+#else
+ *child_p = n3->children[idx];
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+#ifdef RT_ACTION_UPDATE
+ Assert(idx >= 0);
+ n32->children[idx] = new_child;
+#else
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n32->values[idx];
+#else
+ *child_p = n32->children[idx];
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+
+#ifdef RT_ACTION_UPDATE
+ Assert(slotpos != RT_INVALID_SLOT_IDX);
+ n125->children[slotpos] = new_child;
+#else
+ if (slotpos == RT_INVALID_SLOT_IDX)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_125_GET_VALUE(n125, chunk);
+#else
+ *child_p = RT_NODE_INNER_125_GET_CHILD(n125, chunk);
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_ACTION_UPDATE
+ RT_NODE_INNER_256_SET(n256, chunk, new_child);
+#else
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_256_GET_VALUE(n256, chunk);
+#else
+ *child_p = RT_NODE_INNER_256_GET_CHILD(n256, chunk);
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ }
+
+#ifdef RT_ACTION_UPDATE
+ return;
+#else
+ return true;
+#endif /* RT_ACTION_UPDATE */
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/utils/dsa.h b/src/include/utils/dsa.h
index 3ce4ee300a..2af215484f 100644
--- a/src/include/utils/dsa.h
+++ b/src/include/utils/dsa.h
@@ -121,6 +121,7 @@ extern dsa_handle dsa_get_handle(dsa_area *area);
extern dsa_pointer dsa_allocate_extended(dsa_area *area, size_t size, int flags);
extern void dsa_free(dsa_area *area, dsa_pointer dp);
extern void *dsa_get_address(dsa_area *area, dsa_pointer dp);
+extern size_t dsa_get_total_size(dsa_area *area);
extern void dsa_trim(dsa_area *area);
extern void dsa_dump(dsa_area *area);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index c629cbe383..9659eb85d7 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -28,6 +28,7 @@ SUBDIRS = \
test_pg_db_role_setting \
test_pg_dump \
test_predtest \
+ test_radixtree \
test_rbtree \
test_regex \
test_rls_hooks \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 1baa6b558d..232cbdac80 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -24,6 +24,7 @@ subdir('test_parser')
subdir('test_pg_db_role_setting')
subdir('test_pg_dump')
subdir('test_predtest')
+subdir('test_radixtree')
subdir('test_rbtree')
subdir('test_regex')
subdir('test_rls_hooks')
diff --git a/src/test/modules/test_radixtree/.gitignore b/src/test/modules/test_radixtree/.gitignore
new file mode 100644
index 0000000000..5dcb3ff972
--- /dev/null
+++ b/src/test/modules/test_radixtree/.gitignore
@@ -0,0 +1,4 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
diff --git a/src/test/modules/test_radixtree/Makefile b/src/test/modules/test_radixtree/Makefile
new file mode 100644
index 0000000000..da06b93da3
--- /dev/null
+++ b/src/test/modules/test_radixtree/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_radixtree/Makefile
+
+MODULE_big = test_radixtree
+OBJS = \
+ $(WIN32RES) \
+ test_radixtree.o
+PGFILEDESC = "test_radixtree - test code for src/backend/lib/radixtree.c"
+
+EXTENSION = test_radixtree
+DATA = test_radixtree--1.0.sql
+
+REGRESS = test_radixtree
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_radixtree
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_radixtree/README b/src/test/modules/test_radixtree/README
new file mode 100644
index 0000000000..a8b271869a
--- /dev/null
+++ b/src/test/modules/test_radixtree/README
@@ -0,0 +1,7 @@
+test_integerset contains unit tests for testing the integer set implementation
+in src/backend/lib/integerset.c.
+
+The tests verify the correctness of the implementation, but they can also be
+used as a micro-benchmark. If you set the 'intset_test_stats' flag in
+test_integerset.c, the tests will print extra information about execution time
+and memory usage.
diff --git a/src/test/modules/test_radixtree/expected/test_radixtree.out b/src/test/modules/test_radixtree/expected/test_radixtree.out
new file mode 100644
index 0000000000..ce645cb8b5
--- /dev/null
+++ b/src/test/modules/test_radixtree/expected/test_radixtree.out
@@ -0,0 +1,36 @@
+CREATE EXTENSION test_radixtree;
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
+NOTICE: testing basic operations with leaf node 4
+NOTICE: testing basic operations with inner node 4
+NOTICE: testing basic operations with leaf node 32
+NOTICE: testing basic operations with inner node 32
+NOTICE: testing basic operations with leaf node 125
+NOTICE: testing basic operations with inner node 125
+NOTICE: testing basic operations with leaf node 256
+NOTICE: testing basic operations with inner node 256
+NOTICE: testing radix tree node types with shift "0"
+NOTICE: testing radix tree node types with shift "8"
+NOTICE: testing radix tree node types with shift "16"
+NOTICE: testing radix tree node types with shift "24"
+NOTICE: testing radix tree node types with shift "32"
+NOTICE: testing radix tree node types with shift "40"
+NOTICE: testing radix tree node types with shift "48"
+NOTICE: testing radix tree node types with shift "56"
+NOTICE: testing radix tree with pattern "all ones"
+NOTICE: testing radix tree with pattern "alternating bits"
+NOTICE: testing radix tree with pattern "clusters of ten"
+NOTICE: testing radix tree with pattern "clusters of hundred"
+NOTICE: testing radix tree with pattern "one-every-64k"
+NOTICE: testing radix tree with pattern "sparse"
+NOTICE: testing radix tree with pattern "single values, distance > 2^32"
+NOTICE: testing radix tree with pattern "clusters, distance > 2^32"
+NOTICE: testing radix tree with pattern "clusters, distance > 2^60"
+ test_radixtree
+----------------
+
+(1 row)
+
diff --git a/src/test/modules/test_radixtree/meson.build b/src/test/modules/test_radixtree/meson.build
new file mode 100644
index 0000000000..6add06bbdb
--- /dev/null
+++ b/src/test/modules/test_radixtree/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_radixtree_sources = files(
+ 'test_radixtree.c',
+)
+
+if host_system == 'windows'
+ test_radixtree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_radixtree',
+ '--FILEDESC', 'test_radixtree - test code for src/include//lib/radixtree.h',])
+endif
+
+test_radixtree = shared_module('test_radixtree',
+ test_radixtree_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += test_radixtree
+
+install_data(
+ 'test_radixtree.control',
+ 'test_radixtree--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'test_radixtree',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'test_radixtree',
+ ],
+ },
+}
diff --git a/src/test/modules/test_radixtree/sql/test_radixtree.sql b/src/test/modules/test_radixtree/sql/test_radixtree.sql
new file mode 100644
index 0000000000..41ece5e9f5
--- /dev/null
+++ b/src/test/modules/test_radixtree/sql/test_radixtree.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_radixtree;
+
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
diff --git a/src/test/modules/test_radixtree/test_radixtree--1.0.sql b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
new file mode 100644
index 0000000000..074a5a7ea7
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_radixtree/test_radixtree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_radixtree" to load this file. \quit
+
+CREATE FUNCTION test_radixtree()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
new file mode 100644
index 0000000000..f944945db9
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -0,0 +1,674 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_radixtree.c
+ * Test radixtree set data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_radixtree/test_radixtree.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "storage/lwlock.h"
+#include "utils/memutils.h"
+#include "utils/timestamp.h"
+
+#define UINT64_HEX_FORMAT "%" INT64_MODIFIER "X"
+
+/*
+ * The tests pass with uint32, but build with warnings because the string
+ * format expects uint64.
+ */
+typedef uint64 TestValueType;
+
+/*
+ * If you enable this, the "pattern" tests will print information about
+ * how long populating, probing, and iterating the test set takes, and
+ * how much memory the test set consumed. That can be used as
+ * micro-benchmark of various operations and input patterns (you might
+ * want to increase the number of values used in each of the test, if
+ * you do that, to reduce noise).
+ *
+ * The information is printed to the server's stderr, mostly because
+ * that's where MemoryContextStats() output goes.
+ */
+static const bool rt_test_stats = false;
+
+static int rt_node_kind_fanouts[] = {
+ 0,
+ 4, /* RT_NODE_KIND_4 */
+ 32, /* RT_NODE_KIND_32 */
+ 125, /* RT_NODE_KIND_125 */
+ 256 /* RT_NODE_KIND_256 */
+};
+/*
+ * A struct to define a pattern of integers, for use with the test_pattern()
+ * function.
+ */
+typedef struct
+{
+ char *test_name; /* short name of the test, for humans */
+ char *pattern_str; /* a bit pattern */
+ uint64 spacing; /* pattern repeats at this interval */
+ uint64 num_values; /* number of integers to set in total */
+} test_spec;
+
+/* Test patterns borrowed from test_integerset.c */
+static const test_spec test_specs[] = {
+ {
+ "all ones", "1111111111",
+ 10, 1000000
+ },
+ {
+ "alternating bits", "0101010101",
+ 10, 1000000
+ },
+ {
+ "clusters of ten", "1111111111",
+ 10000, 1000000
+ },
+ {
+ "clusters of hundred",
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111",
+ 10000, 1000000
+ },
+ {
+ "one-every-64k", "1",
+ 65536, 1000000
+ },
+ {
+ "sparse", "100000000000000000000000000000001",
+ 10000000, 1000000
+ },
+ {
+ "single values, distance > 2^32", "1",
+ UINT64CONST(10000000000), 100000
+ },
+ {
+ "clusters, distance > 2^32", "10101010",
+ UINT64CONST(10000000000), 1000000
+ },
+ {
+ "clusters, distance > 2^60", "10101010",
+ UINT64CONST(2000000000000000000),
+ 23 /* can't be much higher than this, or we
+ * overflow uint64 */
+ }
+};
+
+/* define the radix tree implementation to test */
+#define RT_PREFIX rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE TestValueType
+// WIP: compiles with warnings because rt_attach is defined but not used
+// #define RT_SHMEM
+#include "lib/radixtree.h"
+
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+ return tree->ctl->num_keys;
+}
+
+PG_MODULE_MAGIC;
+
+PG_FUNCTION_INFO_V1(test_radixtree);
+
+static void
+test_empty(void)
+{
+ rt_radix_tree *radixtree;
+ rt_iter *iter;
+ TestValueType dummy;
+ uint64 key;
+ TestValueType val;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+ dsa = dsa_create(tranche_id);
+
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ if (rt_search(radixtree, 0, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_search(radixtree, 1, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_search(radixtree, PG_UINT64_MAX, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_delete(radixtree, 0))
+ elog(ERROR, "rt_delete on empty tree returned true");
+
+ if (rt_num_entries(radixtree) != 0)
+ elog(ERROR, "rt_num_entries on empty tree return non-zero");
+
+ iter = rt_begin_iterate(radixtree);
+
+ if (rt_iterate_next(iter, &key, &val))
+ elog(ERROR, "rt_itereate_next on empty tree returned true");
+
+ rt_end_iterate(iter);
+
+ rt_free(radixtree);
+
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+static void
+test_basic(int children, bool test_inner)
+{
+ rt_radix_tree *radixtree;
+ uint64 *keys;
+ int shift = test_inner ? 8 : 0;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing basic operations with %s node %d",
+ test_inner ? "inner" : "leaf", children);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ /* prepare keys in order like 1, 32, 2, 31, 2, ... */
+ keys = palloc(sizeof(uint64) * children);
+ for (int i = 0; i < children; i++)
+ {
+ if (i % 2 == 0)
+ keys[i] = (uint64) ((i / 2) + 1) << shift;
+ else
+ keys[i] = (uint64) (children - (i / 2)) << shift;
+ }
+
+ /* insert keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (rt_set(radixtree, keys[i], (TestValueType*) &keys[i]))
+ elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+ }
+
+ /* look up keys */
+ for (int i = 0; i < children; i++)
+ {
+ TestValueType value;
+
+ if (!rt_search(radixtree, keys[i], &value))
+ elog(ERROR, "could not find key 0x" UINT64_HEX_FORMAT, keys[i]);
+ if (value != (TestValueType) keys[i])
+ elog(ERROR, "rt_search returned 0x" UINT64_HEX_FORMAT ", expected " UINT64_HEX_FORMAT,
+ value, (TestValueType) keys[i]);
+ }
+
+ /* update keys */
+ for (int i = 0; i < children; i++)
+ {
+ TestValueType update = keys[i] + 1;
+ if (!rt_set(radixtree, keys[i], (TestValueType*) &update))
+ elog(ERROR, "could not update key 0x" UINT64_HEX_FORMAT, keys[i]);
+ }
+
+ /* repeat deleting and inserting keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (!rt_delete(radixtree, keys[i]))
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, keys[i]);
+ if (rt_set(radixtree, keys[i], (TestValueType*) &keys[i]))
+ elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+ }
+
+ pfree(keys);
+ rt_free(radixtree);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Check if keys from start to end with the shift exist in the tree.
+ */
+static void
+check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end,
+ int incr)
+{
+ for (int i = start; i < end; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ TestValueType val;
+
+ if (!rt_search(radixtree, key, &val))
+ elog(ERROR, "key 0x" UINT64_HEX_FORMAT " is not found on node-%d",
+ key, end);
+ if (val != (TestValueType) key)
+ elog(ERROR, "rt_search with key 0x" UINT64_HEX_FORMAT " returns 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+ key, val, key);
+ }
+}
+
+static void
+test_node_types_insert(rt_radix_tree *radixtree, uint8 shift, bool insert_asc)
+{
+ uint64 num_entries;
+ int ninserted = 0;
+ int start = insert_asc ? 0 : 256;
+ int incr = insert_asc ? 1 : -1;
+ int end = insert_asc ? 256 : 0;
+ int node_kind_idx = 1;
+
+ for (int i = start; i != end; i += incr)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_set(radixtree, key, (TestValueType*) &key);
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " is found", key);
+
+ /*
+ * After filling all slots in each node type, check if the values
+ * are stored properly.
+ */
+ if (ninserted == rt_node_kind_fanouts[node_kind_idx] - 1)
+ {
+ int check_start = insert_asc
+ ? rt_node_kind_fanouts[node_kind_idx - 1]
+ : rt_node_kind_fanouts[node_kind_idx];
+ int check_end = insert_asc
+ ? rt_node_kind_fanouts[node_kind_idx]
+ : rt_node_kind_fanouts[node_kind_idx - 1];
+
+ check_search_on_node(radixtree, shift, check_start, check_end, incr);
+ node_kind_idx++;
+ }
+
+ ninserted++;
+ }
+
+ num_entries = rt_num_entries(radixtree);
+
+ if (num_entries != 256)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+static void
+test_node_types_delete(rt_radix_tree *radixtree, uint8 shift)
+{
+ uint64 num_entries;
+
+ for (int i = 0; i < 256; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_delete(radixtree, key);
+
+ if (!found)
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, key);
+ }
+
+ num_entries = rt_num_entries(radixtree);
+
+ /* The tree must be empty */
+ if (num_entries != 0)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+/*
+ * Test for inserting and deleting key-value pairs to each node type at the given shift
+ * level.
+ */
+static void
+test_node_types(uint8 shift)
+{
+ rt_radix_tree *radixtree;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing radix tree node types with shift \"%d\"", shift);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ /*
+ * Insert and search entries for every node type at the 'shift' level,
+ * then delete all entries to make it empty, and insert and search entries
+ * again.
+ */
+ test_node_types_insert(radixtree, shift, true);
+ test_node_types_delete(radixtree, shift);
+ test_node_types_insert(radixtree, shift, false);
+
+ rt_free(radixtree);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Test with a repeating pattern, defined by the 'spec'.
+ */
+static void
+test_pattern(const test_spec * spec)
+{
+ rt_radix_tree *radixtree;
+ rt_iter *iter;
+ MemoryContext radixtree_ctx;
+ TimestampTz starttime;
+ TimestampTz endtime;
+ uint64 n;
+ uint64 last_int;
+ uint64 ndeleted;
+ uint64 nbefore;
+ uint64 nafter;
+ int patternlen;
+ uint64 *pattern_values;
+ uint64 pattern_num_values;
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing radix tree with pattern \"%s\"", spec->test_name);
+ if (rt_test_stats)
+ fprintf(stderr, "-----\ntesting radix tree with pattern \"%s\"\n", spec->test_name);
+
+ /* Pre-process the pattern, creating an array of integers from it. */
+ patternlen = strlen(spec->pattern_str);
+ pattern_values = palloc(patternlen * sizeof(uint64));
+ pattern_num_values = 0;
+ for (int i = 0; i < patternlen; i++)
+ {
+ if (spec->pattern_str[i] == '1')
+ pattern_values[pattern_num_values++] = i;
+ }
+
+ /*
+ * Allocate the radix tree.
+ *
+ * Allocate it in a separate memory context, so that we can print its
+ * memory usage easily.
+ */
+ radixtree_ctx = AllocSetContextCreate(CurrentMemoryContext,
+ "radixtree test",
+ ALLOCSET_SMALL_SIZES);
+ MemoryContextSetIdentifier(radixtree_ctx, spec->test_name);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(radixtree_ctx, dsa, tranche_id);
+#else
+ radixtree = rt_create(radixtree_ctx);
+#endif
+
+
+ /*
+ * Add values to the set.
+ */
+ starttime = GetCurrentTimestamp();
+
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ uint64 x = 0;
+
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ bool found;
+
+ x = last_int + pattern_values[i];
+
+ found = rt_set(radixtree, x, (TestValueType*) &x);
+
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " found", x);
+
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+
+ endtime = GetCurrentTimestamp();
+
+ if (rt_test_stats)
+ fprintf(stderr, "added " UINT64_FORMAT " values in %d ms\n",
+ spec->num_values, (int) (endtime - starttime) / 1000);
+
+ /*
+ * Print stats on the amount of memory used.
+ *
+ * We print the usage reported by rt_memory_usage(), as well as the stats
+ * from the memory context. They should be in the same ballpark, but it's
+ * hard to automate testing that, so if you're making changes to the
+ * implementation, just observe that manually.
+ */
+ if (rt_test_stats)
+ {
+ uint64 mem_usage;
+
+ /*
+ * Also print memory usage as reported by rt_memory_usage(). It
+ * should be in the same ballpark as the usage reported by
+ * MemoryContextStats().
+ */
+ mem_usage = rt_memory_usage(radixtree);
+ fprintf(stderr, "rt_memory_usage() reported " UINT64_FORMAT " (%0.2f bytes / integer)\n",
+ mem_usage, (double) mem_usage / spec->num_values);
+
+ MemoryContextStats(radixtree_ctx);
+ }
+
+ /* Check that rt_num_entries works */
+ n = rt_num_entries(radixtree);
+ if (n != spec->num_values)
+ elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT, n, spec->num_values);
+
+ /*
+ * Test random-access probes with rt_search()
+ */
+ starttime = GetCurrentTimestamp();
+
+ for (n = 0; n < 100000; n++)
+ {
+ bool found;
+ bool expected;
+ uint64 x;
+ TestValueType v;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit values that are actually in the set.
+ */
+ x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+ /* Do we expect this value to be present in the set? */
+ if (x >= last_int)
+ expected = false;
+ else
+ {
+ uint64 idx = x % spec->spacing;
+
+ if (idx >= patternlen)
+ expected = false;
+ else if (spec->pattern_str[idx] == '1')
+ expected = true;
+ else
+ expected = false;
+ }
+
+ /* Is it present according to rt_search() ? */
+ found = rt_search(radixtree, x, &v);
+
+ if (found != expected)
+ elog(ERROR, "mismatch at 0x" UINT64_HEX_FORMAT ": %d vs %d", x, found, expected);
+ if (found && (v != (TestValueType) x))
+ elog(ERROR, "found 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+ v, x);
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "probed " UINT64_FORMAT " values in %d ms\n",
+ n, (int) (endtime - starttime) / 1000);
+
+ /*
+ * Test iterator
+ */
+ starttime = GetCurrentTimestamp();
+
+ iter = rt_begin_iterate(radixtree);
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ uint64 expected = last_int + pattern_values[i];
+ uint64 x;
+ TestValueType val;
+
+ if (!rt_iterate_next(iter, &x, &val))
+ break;
+
+ if (x != expected)
+ elog(ERROR,
+ "iterate returned wrong key; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d",
+ x, expected, i);
+ if (val != (TestValueType) expected)
+ elog(ERROR,
+ "iterate returned wrong value; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d", x, expected, i);
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "iterated " UINT64_FORMAT " values in %d ms\n",
+ n, (int) (endtime - starttime) / 1000);
+
+ rt_end_iterate(iter);
+
+ if (n < spec->num_values)
+ elog(ERROR, "iterator stopped short after " UINT64_FORMAT " entries, expected " UINT64_FORMAT, n, spec->num_values);
+ if (n > spec->num_values)
+ elog(ERROR, "iterator returned " UINT64_FORMAT " entries, " UINT64_FORMAT " was expected", n, spec->num_values);
+
+ /*
+ * Test random-access probes with rt_delete()
+ */
+ starttime = GetCurrentTimestamp();
+
+ nbefore = rt_num_entries(radixtree);
+ ndeleted = 0;
+ for (n = 0; n < 1; n++)
+ {
+ bool found;
+ uint64 x;
+ TestValueType v;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit values that are actually in the set.
+ */
+ x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+ /* Is it present according to rt_search() ? */
+ found = rt_search(radixtree, x, &v);
+
+ if (!found)
+ continue;
+
+ /* If the key is found, delete it and check again */
+ if (!rt_delete(radixtree, x))
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, x);
+ if (rt_search(radixtree, x, &v))
+ elog(ERROR, "found deleted key 0x" UINT64_HEX_FORMAT, x);
+ if (rt_delete(radixtree, x))
+ elog(ERROR, "deleted already-deleted key 0x" UINT64_HEX_FORMAT, x);
+
+ ndeleted++;
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "deleted " UINT64_FORMAT " values in %d ms\n",
+ ndeleted, (int) (endtime - starttime) / 1000);
+
+ nafter = rt_num_entries(radixtree);
+
+ /* Check that rt_num_entries works */
+ if ((nbefore - ndeleted) != nafter)
+ elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT "after " UINT64_FORMAT " deletion",
+ nafter, (nbefore - ndeleted), ndeleted);
+
+ rt_free(radixtree);
+ MemoryContextDelete(radixtree_ctx);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+Datum
+test_radixtree(PG_FUNCTION_ARGS)
+{
+ test_empty();
+
+ for (int i = 1; i < lengthof(rt_node_kind_fanouts); i++)
+ {
+ test_basic(rt_node_kind_fanouts[i], false);
+ test_basic(rt_node_kind_fanouts[i], true);
+ }
+
+ for (int shift = 0; shift <= (64 - 8); shift += 8)
+ test_node_types(shift);
+
+ /* Test different test patterns, with lots of entries */
+ for (int i = 0; i < lengthof(test_specs); i++)
+ test_pattern(&test_specs[i]);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_radixtree/test_radixtree.control b/src/test/modules/test_radixtree/test_radixtree.control
new file mode 100644
index 0000000000..e53f2a3e0c
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.control
@@ -0,0 +1,4 @@
+comment = 'Test code for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/test_radixtree'
+relocatable = true
diff --git a/src/tools/pginclude/cpluspluscheck b/src/tools/pginclude/cpluspluscheck
index e52fe9f509..09fa6e7432 100755
--- a/src/tools/pginclude/cpluspluscheck
+++ b/src/tools/pginclude/cpluspluscheck
@@ -101,6 +101,12 @@ do
test "$f" = src/include/nodes/nodetags.h && continue
test "$f" = src/backend/nodes/nodetags.h && continue
+ # radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+ test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+ test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+ test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+ test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
# These files are not meant to be included standalone, because
# they contain lists that might have multiple use-cases.
test "$f" = src/include/access/rmgrlist.h && continue
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index abbba7aa63..d4d2f1da03 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -96,6 +96,12 @@ do
test "$f" = src/include/nodes/nodetags.h && continue
test "$f" = src/backend/nodes/nodetags.h && continue
+ # radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+ test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+ test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+ test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+ test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
# These files are not meant to be included standalone, because
# they contain lists that might have multiple use-cases.
test "$f" = src/include/access/rmgrlist.h && continue
--
2.39.1
[text/x-patch] v26-0006-Adjust-some-inlining-declarations.patch (2.1K, ../../CAFBsxsE7FLxfc2bfPf-DwdJ6RmDcpUcNRyUnHeuR26LVi6jzGA@mail.gmail.com/9-v26-0006-Adjust-some-inlining-declarations.patch)
download | inline diff:
From 0726bb6b4e0250a72ce399d945d250724b4a29ab Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Mon, 6 Feb 2023 21:04:14 +0700
Subject: [PATCH v26 6/9] Adjust some inlining declarations
---
src/include/lib/radixtree.h | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index d6919aef08..4bd0aaa810 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -1124,7 +1124,7 @@ RT_INIT_NODE(RT_PTR_LOCAL node, uint8 kind, RT_SIZE_CLASS size_class, bool is_le
* Create a new node as the root. Subordinate nodes will be created during
* the insertion.
*/
-static void
+static pg_noinline void
RT_NEW_ROOT(RT_RADIX_TREE *tree, uint64 key)
{
int shift = RT_KEY_GET_SHIFT(key);
@@ -1215,7 +1215,7 @@ RT_NODE_UPDATE_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC new_child)
/*
* Replace old_child with new_child, and free the old one.
*/
-static void
+static inline void
RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
RT_PTR_ALLOC stored_old_child, RT_PTR_LOCAL old_child,
RT_PTR_ALLOC new_child, uint64 key)
@@ -1242,7 +1242,7 @@ RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
* The radix tree doesn't have sufficient height. Extend the radix tree so
* it can store the key.
*/
-static void
+static pg_noinline void
RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
{
int target_shift;
@@ -1281,7 +1281,7 @@ RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
* The radix tree doesn't have inner and leaf nodes for given key-value pair.
* Insert inner and leaf nodes from 'node' to bottom.
*/
-static inline void
+static pg_noinline void
RT_SET_EXTEND(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node)
{
@@ -1486,7 +1486,7 @@ RT_GET_HANDLE(RT_RADIX_TREE *tree)
/*
* Recursively free all nodes allocated to the DSA area.
*/
-static inline void
+static void
RT_FREE_RECURSE(RT_RADIX_TREE *tree, RT_PTR_ALLOC ptr)
{
RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, ptr);
--
2.39.1
[text/x-patch] v26-0007-Skip-unnecessary-searches-in-RT_NODE_INSERT_INNE.patch (4.7K, ../../CAFBsxsE7FLxfc2bfPf-DwdJ6RmDcpUcNRyUnHeuR26LVi6jzGA@mail.gmail.com/10-v26-0007-Skip-unnecessary-searches-in-RT_NODE_INSERT_INNE.patch)
download | inline diff:
From 6831fe27a2c9c5765113b7903403c426f09f55f6 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Mon, 6 Feb 2023 22:04:50 +0700
Subject: [PATCH v26 7/9] Skip unnecessary searches in RT_NODE_INSERT_INNER
For inner nodes, we know the key chunk doesn't exist already,
otherwise we would have found it while descending the tree.
To reinforce this fact, declare this function to return void.
---
src/include/lib/radixtree.h | 4 +--
src/include/lib/radixtree_insert_impl.h | 48 ++++++++++++-------------
2 files changed, 24 insertions(+), 28 deletions(-)
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index 4bd0aaa810..1cdb995e54 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -685,7 +685,7 @@ typedef struct RT_ITER
} RT_ITER;
-static bool RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+static void RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
uint64 key, RT_PTR_ALLOC child);
static bool RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
uint64 key, RT_VALUE_TYPE *value_p);
@@ -1375,7 +1375,7 @@ RT_NODE_DELETE_LEAF(RT_PTR_LOCAL node, uint64 key)
* If the node we're inserting into needs to grow, we update the parent's
* child pointer with the pointer to the new larger node.
*/
-static bool
+static void
RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
uint64 key, RT_PTR_ALLOC child)
{
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
index c18e26b537..d56e58dcac 100644
--- a/src/include/lib/radixtree_insert_impl.h
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -28,10 +28,10 @@
#endif
uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
- bool chunk_exists = false;
#ifdef RT_NODE_LEVEL_LEAF
const bool is_leaf = true;
+ bool chunk_exists = false;
Assert(RT_NODE_IS_LEAF(node));
#else
const bool is_leaf = false;
@@ -43,21 +43,18 @@
case RT_NODE_KIND_3:
{
RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
- int idx;
- idx = RT_NODE_3_SEARCH_EQ(&n3->base, chunk);
+#ifdef RT_NODE_LEVEL_LEAF
+ int idx = RT_NODE_3_SEARCH_EQ(&n3->base, chunk);
+
if (idx != -1)
{
/* found the existing chunk */
chunk_exists = true;
-#ifdef RT_NODE_LEVEL_LEAF
n3->values[idx] = *value_p;
-#else
- n3->children[idx] = child;
-#endif
break;
}
-
+#endif
if (unlikely(RT_NODE_MUST_GROW(n3)))
{
RT_PTR_ALLOC allocnode;
@@ -113,21 +110,18 @@
{
const RT_SIZE_CLASS_ELEM class32_max = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX];
RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
- int idx;
- idx = RT_NODE_32_SEARCH_EQ(&n32->base, chunk);
+#ifdef RT_NODE_LEVEL_LEAF
+ int idx = RT_NODE_32_SEARCH_EQ(&n32->base, chunk);
+
if (idx != -1)
{
/* found the existing chunk */
chunk_exists = true;
-#ifdef RT_NODE_LEVEL_LEAF
n32->values[idx] = *value_p;
-#else
- n32->children[idx] = child;
-#endif
break;
}
-
+#endif
if (unlikely(RT_NODE_MUST_GROW(n32)) &&
n32->base.n.fanout < class32_max.fanout)
{
@@ -220,21 +214,19 @@
case RT_NODE_KIND_125:
{
RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
- int slotpos = n125->base.slot_idxs[chunk];
+ int slotpos;
int cnt = 0;
+#ifdef RT_NODE_LEVEL_LEAF
+ slotpos = n125->base.slot_idxs[chunk];
if (slotpos != RT_INVALID_SLOT_IDX)
{
/* found the existing chunk */
chunk_exists = true;
-#ifdef RT_NODE_LEVEL_LEAF
n125->values[slotpos] = *value_p;
-#else
- n125->children[slotpos] = child;
-#endif
break;
}
-
+#endif
if (unlikely(RT_NODE_MUST_GROW(n125)))
{
RT_PTR_ALLOC allocnode;
@@ -300,14 +292,10 @@
#ifdef RT_NODE_LEVEL_LEAF
chunk_exists = RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk);
-#else
- chunk_exists = RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk);
-#endif
Assert(chunk_exists || node->count < RT_NODE_MAX_SLOTS);
-
-#ifdef RT_NODE_LEVEL_LEAF
RT_NODE_LEAF_256_SET(n256, chunk, *value_p);
#else
+ Assert(node->count < RT_NODE_MAX_SLOTS);
RT_NODE_INNER_256_SET(n256, chunk, child);
#endif
break;
@@ -315,8 +303,12 @@
}
/* Update statistics */
+#ifdef RT_NODE_LEVEL_LEAF
if (!chunk_exists)
node->count++;
+#else
+ node->count++;
+#endif
/*
* Done. Finally, verify the chunk and value is inserted or replaced
@@ -324,7 +316,11 @@
*/
RT_VERIFY_NODE(node);
+#ifdef RT_NODE_LEVEL_LEAF
return chunk_exists;
+#else
+ return;
+#endif
#undef RT_NODE3_TYPE
#undef RT_NODE32_TYPE
--
2.39.1
[text/x-patch] v26-0008-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch (35.2K, ../../CAFBsxsE7FLxfc2bfPf-DwdJ6RmDcpUcNRyUnHeuR26LVi6jzGA@mail.gmail.com/11-v26-0008-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch)
download | inline diff:
From f17e983832736a1daa64e67a10f9a64189b68210 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 23 Dec 2022 23:50:39 +0900
Subject: [PATCH v26 8/9] Add TIDStore, to store sets of TIDs (ItemPointerData)
efficiently.
The TIDStore is designed to store large sets of TIDs efficiently, and
is backed by the radix tree. A TID is encoded into 64-bit key and
value and inserted to the radix tree.
The TIDStore is not used for anything yet, aside from the test code,
but the follow up patch integrates the TIDStore with lazy vacuum,
reducing lazy vacuum memory usage and lifting the 1GB limit on its
size, by storing the list of dead TIDs more efficiently.
This includes a unit test module, in src/test/modules/test_tidstore.
---
doc/src/sgml/monitoring.sgml | 4 +
src/backend/access/common/Makefile | 1 +
src/backend/access/common/meson.build | 1 +
src/backend/access/common/tidstore.c | 688 ++++++++++++++++++
src/backend/storage/lmgr/lwlock.c | 2 +
src/include/access/tidstore.h | 49 ++
src/include/storage/lwlock.h | 1 +
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_tidstore/Makefile | 23 +
.../test_tidstore/expected/test_tidstore.out | 13 +
src/test/modules/test_tidstore/meson.build | 35 +
.../test_tidstore/sql/test_tidstore.sql | 7 +
.../test_tidstore/test_tidstore--1.0.sql | 8 +
.../modules/test_tidstore/test_tidstore.c | 195 +++++
.../test_tidstore/test_tidstore.control | 4 +
16 files changed, 1033 insertions(+)
create mode 100644 src/backend/access/common/tidstore.c
create mode 100644 src/include/access/tidstore.h
create mode 100644 src/test/modules/test_tidstore/Makefile
create mode 100644 src/test/modules/test_tidstore/expected/test_tidstore.out
create mode 100644 src/test/modules/test_tidstore/meson.build
create mode 100644 src/test/modules/test_tidstore/sql/test_tidstore.sql
create mode 100644 src/test/modules/test_tidstore/test_tidstore--1.0.sql
create mode 100644 src/test/modules/test_tidstore/test_tidstore.c
create mode 100644 src/test/modules/test_tidstore/test_tidstore.control
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1756f1a4b6..d936aa3da3 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2192,6 +2192,10 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<entry>Waiting to access a shared TID bitmap during a parallel bitmap
index scan.</entry>
</row>
+ <row>
+ <entry><literal>SharedTidStore</literal></entry>
+ <entry>Waiting to access a shared TID store.</entry>
+ </row>
<row>
<entry><literal>SharedTupleStore</literal></entry>
<entry>Waiting to access a shared tuple store during parallel
diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile
index b9aff0ccfd..67b8cc6108 100644
--- a/src/backend/access/common/Makefile
+++ b/src/backend/access/common/Makefile
@@ -27,6 +27,7 @@ OBJS = \
syncscan.o \
toast_compression.o \
toast_internals.o \
+ tidstore.o \
tupconvert.o \
tupdesc.o
diff --git a/src/backend/access/common/meson.build b/src/backend/access/common/meson.build
index f5ac17b498..fce19c09ce 100644
--- a/src/backend/access/common/meson.build
+++ b/src/backend/access/common/meson.build
@@ -15,6 +15,7 @@ backend_sources += files(
'syncscan.c',
'toast_compression.c',
'toast_internals.c',
+ 'tidstore.c',
'tupconvert.c',
'tupdesc.c',
)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
new file mode 100644
index 0000000000..4c72673ce9
--- /dev/null
+++ b/src/backend/access/common/tidstore.c
@@ -0,0 +1,688 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.c
+ * Tid (ItemPointerData) storage implementation.
+ *
+ * This module provides a in-memory data structure to store Tids (ItemPointer).
+ * Internally, a tid is encoded as a pair of 64-bit key and 64-bit value, and
+ * stored in the radix tree.
+ *
+ * A TidStore can be shared among parallel worker processes by passing DSA area
+ * to tidstore_create(). Other backends can attach to the shared TidStore by
+ * tidstore_attach().
+ *
+ * Regarding the concurrency, it basically relies on the concurrency support in
+ * the radix tree, but we acquires the lock on a TidStore in some cases, for
+ * example, when to reset the store and when to access the number tids in the
+ * store (num_tids).
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/access/common/tidstore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/tidstore.h"
+#include "miscadmin.h"
+#include "port/pg_bitutils.h"
+#include "storage/lwlock.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/*
+ * For encoding purposes, tids are represented as a pair of 64-bit key and
+ * 64-bit value. First, we construct 64-bit unsigned integer by combining
+ * the block number and the offset number. The number of bits used for the
+ * offset number is specified by max_offsets in tidstore_create(). We are
+ * frugal with the bits, because smaller keys could help keeping the radix
+ * tree shallow.
+ *
+ * For example, a tid of heap with 8kB blocks uses the lowest 9 bits for
+ * the offset number and uses the next 32 bits for the block number. That
+ * is, only 41 bits are used:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ *
+ * X = bits used for offset number
+ * Y = bits used for block number
+ * u = unused bit
+ * (high on the left, low on the right)
+ *
+ * 9 bits are enough for the offset number, because MaxHeapTuplesPerPage < 2^9
+ * on 8kB blocks.
+ *
+ * The 64-bit value is the bitmap representation of the lowest 6 bits
+ * (TIDSTORE_VALUE_NBITS) of the integer, and the rest 35 bits are used
+ * as the key:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ * |----| value
+ * |---------------------------------------------| key
+ *
+ * The maximum height of the radix tree is 5 in this case.
+ *
+ * If the number of bits for offset number fits in a 64-bit value, we don't
+ * encode tids but directly use the block number and the offset number as key
+ * and value, respectively.
+ */
+#define TIDSTORE_VALUE_NBITS 6 /* log(64, 2) */
+
+/* A magic value used to identify our TidStores. */
+#define TIDSTORE_MAGIC 0x826f6a10
+
+#define RT_PREFIX local_rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE uint64
+#include "lib/radixtree.h"
+
+#define RT_PREFIX shared_rt
+#define RT_SHMEM
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE uint64
+#include "lib/radixtree.h"
+
+/* The control object for a TidStore */
+typedef struct TidStoreControl
+{
+ /* the number of tids in the store */
+ int64 num_tids;
+
+ /* These values are never changed after creation */
+ size_t max_bytes; /* the maximum bytes a TidStore can use */
+ int max_offset; /* the maximum offset number */
+ int offset_nbits; /* the number of bits required for max_offset */
+ bool encode_tids; /* do we use tid encoding? */
+ int offset_key_nbits; /* the number of bits of a offset number
+ * used for the key */
+
+ /* The below fields are used only in shared case */
+
+ uint32 magic;
+ LWLock lock;
+
+ /* handles for TidStore and radix tree */
+ tidstore_handle handle;
+ shared_rt_handle tree_handle;
+} TidStoreControl;
+
+/* Per-backend state for a TidStore */
+struct TidStore
+{
+ /*
+ * Control object. This is allocated in DSA area 'area' in the shared
+ * case, otherwise in backend-local memory.
+ */
+ TidStoreControl *control;
+
+ /* Storage for Tids. Use either one depending on TidStoreIsShared() */
+ union
+ {
+ local_rt_radix_tree *local;
+ shared_rt_radix_tree *shared;
+ } tree;
+
+ /* DSA area for TidStore if used */
+ dsa_area *area;
+};
+#define TidStoreIsShared(ts) ((ts)->area != NULL)
+
+/* Iterator for TidStore */
+typedef struct TidStoreIter
+{
+ TidStore *ts;
+
+ /* iterator of radix tree. Use either one depending on TidStoreIsShared() */
+ union
+ {
+ shared_rt_iter *shared;
+ local_rt_iter *local;
+ } tree_iter;
+
+ /* we returned all tids? */
+ bool finished;
+
+ /* save for the next iteration */
+ uint64 next_key;
+ uint64 next_val;
+
+ /* output for the caller */
+ TidStoreIterResult result;
+} TidStoreIter;
+
+static void tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val);
+static inline BlockNumber key_get_blkno(TidStore *ts, uint64 key);
+static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off);
+
+/*
+ * Create a TidStore. The returned object is allocated in backend-local memory.
+ * The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
+ */
+TidStore *
+tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
+{
+ TidStore *ts;
+
+ ts = palloc0(sizeof(TidStore));
+
+ /*
+ * Create the radix tree for the main storage.
+ *
+ * Memory consumption depends on the number of stored tids, but also on the
+ * distribution of them, how the radix tree stores, and the memory management
+ * that backed the radix tree. The maximum bytes that a TidStore can
+ * use is specified by the max_bytes in tidstore_create(). We want the total
+ * amount of memory consumption by a TidStore not to exceed the max_bytes.
+ *
+ * In local TidStore cases, the radix tree uses slab allocators for each kind
+ * of node class. The most memory consuming case while adding Tids associated
+ * with one page (i.e. during tidstore_add_tids()) is that we allocate a new
+ * slab block for a new radix tree node, which is approximately 70kB. Therefore,
+ * we deduct 70kB from the max_bytes.
+ *
+ * In shared cases, DSA allocates the memory segments big enough to follow
+ * a geometric series that approximately doubles the total DSA size (see
+ * make_new_segment() in dsa.c). We simulated the how DSA increases segment
+ * size and the simulation revealed, the 75% threshold for the maximum bytes
+ * perfectly works in case where the max_bytes is a power-of-2, and the 60%
+ * threshold works for other cases.
+ */
+ if (area != NULL)
+ {
+ dsa_pointer dp;
+ float ratio = ((max_bytes & (max_bytes - 1)) == 0) ? 0.75 : 0.6;
+
+ ts->tree.shared = shared_rt_create(CurrentMemoryContext, area,
+ LWTRANCHE_SHARED_TIDSTORE);
+
+ dp = dsa_allocate0(area, sizeof(TidStoreControl));
+ ts->control = (TidStoreControl *) dsa_get_address(area, dp);
+ ts->control->max_bytes = (uint64) (max_bytes * ratio);
+ ts->area = area;
+
+ ts->control->magic = TIDSTORE_MAGIC;
+ LWLockInitialize(&ts->control->lock, LWTRANCHE_SHARED_TIDSTORE);
+ ts->control->handle = dp;
+ ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+ }
+ else
+ {
+ ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+ ts->control = (TidStoreControl *) palloc0(sizeof(TidStoreControl));
+ ts->control->max_bytes = max_bytes - (70 * 1024);
+ }
+
+ ts->control->max_offset = max_offset;
+ ts->control->offset_nbits = pg_ceil_log2_32(max_offset);
+
+ /*
+ * We use tid encoding if the number of bits for the offset number doesn't
+ * fix in a value, uint64.
+ */
+ if (ts->control->offset_nbits > TIDSTORE_VALUE_NBITS)
+ {
+ ts->control->encode_tids = true;
+ ts->control->offset_key_nbits =
+ ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
+ }
+ else
+ {
+ ts->control->encode_tids = false;
+ ts->control->offset_key_nbits = 0;
+ }
+
+ return ts;
+}
+
+/*
+ * Attach to the shared TidStore using a handle. The returned object is
+ * allocated in backend-local memory using the CurrentMemoryContext.
+ */
+TidStore *
+tidstore_attach(dsa_area *area, tidstore_handle handle)
+{
+ TidStore *ts;
+ dsa_pointer control;
+
+ Assert(area != NULL);
+ Assert(DsaPointerIsValid(handle));
+
+ /* create per-backend state */
+ ts = palloc0(sizeof(TidStore));
+
+ /* Find the control object in shared memory */
+ control = handle;
+
+ /* Set up the TidStore */
+ ts->control = (TidStoreControl *) dsa_get_address(area, control);
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ ts->tree.shared = shared_rt_attach(area, ts->control->tree_handle);
+ ts->area = area;
+
+ return ts;
+}
+
+/*
+ * Detach from a TidStore. This detaches from radix tree and frees the
+ * backend-local resources. The radix tree will continue to exist until
+ * it is either explicitly destroyed, or the area that backs it is returned
+ * to the operating system.
+ */
+void
+tidstore_detach(TidStore *ts)
+{
+ Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+ shared_rt_detach(ts->tree.shared);
+ pfree(ts);
+}
+
+/*
+ * Destroy a TidStore, returning all memory.
+ *
+ * TODO: The caller must be certain that no other backend will attempt to
+ * access the TidStore before calling this function. Other backend must
+ * explicitly call tidstore_detach to free up backend-local memory associated
+ * with the TidStore. The backend that calls tidstore_destroy must not call
+ * tidstore_detach.
+ */
+void
+tidstore_destroy(TidStore *ts)
+{
+ if (TidStoreIsShared(ts))
+ {
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ /*
+ * Vandalize the control block to help catch programming error where
+ * other backends access the memory formerly occupied by this radix
+ * tree.
+ */
+ ts->control->magic = 0;
+ dsa_free(ts->area, ts->control->handle);
+ shared_rt_free(ts->tree.shared);
+ }
+ else
+ {
+ pfree(ts->control);
+ local_rt_free(ts->tree.local);
+ }
+
+ pfree(ts);
+}
+
+/*
+ * Forget all collected Tids. It's similar to tidstore_destroy but we don't free
+ * entire TidStore but recreate only the radix tree storage.
+ */
+void
+tidstore_reset(TidStore *ts)
+{
+ if (TidStoreIsShared(ts))
+ {
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+ /*
+ * Free the radix tree and return allocated DSA segments to
+ * the operating system.
+ */
+ shared_rt_free(ts->tree.shared);
+ dsa_trim(ts->area);
+
+ /* Recreate the radix tree */
+ ts->tree.shared = shared_rt_create(CurrentMemoryContext, ts->area,
+ LWTRANCHE_SHARED_TIDSTORE);
+
+ /* update the radix tree handle as we recreated it */
+ ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
+
+ LWLockRelease(&ts->control->lock);
+ }
+ else
+ {
+ local_rt_free(ts->tree.local);
+ ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
+ }
+}
+
+/* Add Tids on a block to TidStore */
+void
+tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets)
+{
+ ItemPointerData tid;
+ uint64 key_base;
+ uint64 *values;
+ int nkeys;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ if (ts->control->encode_tids)
+ {
+ key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
+ nkeys = UINT64CONST(1) << ts->control->offset_key_nbits;
+ }
+ else
+ {
+ key_base = (uint64) blkno;
+ nkeys = 1;
+ }
+ values = palloc0(sizeof(uint64) * nkeys);
+
+ ItemPointerSetBlockNumber(&tid, blkno);
+ for (int i = 0; i < num_offsets; i++)
+ {
+ uint64 key;
+ uint32 off;
+ int idx;
+
+ ItemPointerSetOffsetNumber(&tid, offsets[i]);
+
+ /* encode the tid to key and val */
+ key = tid_to_key_off(ts, &tid, &off);
+
+ idx = key - key_base;
+ Assert(idx >= 0 && idx < nkeys);
+
+ values[idx] |= UINT64CONST(1) << off;
+ }
+
+ if (TidStoreIsShared(ts))
+ LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+ /* insert the calculated key-values to the tree */
+ for (int i = 0; i < nkeys; i++)
+ {
+ if (values[i])
+ {
+ uint64 key = key_base + i;
+
+ if (TidStoreIsShared(ts))
+ shared_rt_set(ts->tree.shared, key, &values[i]);
+ else
+ local_rt_set(ts->tree.local, key, &values[i]);
+ }
+ }
+
+ /* update statistics */
+ ts->control->num_tids += num_offsets;
+
+ if (TidStoreIsShared(ts))
+ LWLockRelease(&ts->control->lock);
+
+ pfree(values);
+}
+
+/* Return true if the given tid is present in the TidStore */
+bool
+tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
+{
+ uint64 key;
+ uint64 val = 0;
+ uint32 off;
+ bool found;
+
+ key = tid_to_key_off(ts, tid, &off);
+
+ if (TidStoreIsShared(ts))
+ found = shared_rt_search(ts->tree.shared, key, &val);
+ else
+ found = local_rt_search(ts->tree.local, key, &val);
+
+ if (!found)
+ return false;
+
+ return (val & (UINT64CONST(1) << off)) != 0;
+}
+
+/*
+ * Prepare to iterate through a TidStore. Since the radix tree is locked during the
+ * iteration, so tidstore_end_iterate() needs to called when finished.
+ *
+ * Concurrent updates during the iteration will be blocked when inserting a
+ * key-value to the radix tree.
+ */
+TidStoreIter *
+tidstore_begin_iterate(TidStore *ts)
+{
+ TidStoreIter *iter;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ iter = palloc0(sizeof(TidStoreIter));
+ iter->ts = ts;
+
+ iter->result.blkno = InvalidBlockNumber;
+ iter->result.offsets = palloc(sizeof(OffsetNumber) * ts->control->max_offset);
+
+ if (TidStoreIsShared(ts))
+ iter->tree_iter.shared = shared_rt_begin_iterate(ts->tree.shared);
+ else
+ iter->tree_iter.local = local_rt_begin_iterate(ts->tree.local);
+
+ /* If the TidStore is empty, there is no business */
+ if (tidstore_num_tids(ts) == 0)
+ iter->finished = true;
+
+ return iter;
+}
+
+static inline bool
+tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
+{
+ if (TidStoreIsShared(iter->ts))
+ return shared_rt_iterate_next(iter->tree_iter.shared, key, val);
+
+ return local_rt_iterate_next(iter->tree_iter.local, key, val);
+}
+
+/*
+ * Scan the TidStore and return a pointer to TidStoreIterResult that has tids
+ * in one block. We return the block numbers in ascending order and the offset
+ * numbers in each result is also sorted in ascending order.
+ */
+TidStoreIterResult *
+tidstore_iterate_next(TidStoreIter *iter)
+{
+ uint64 key;
+ uint64 val;
+ TidStoreIterResult *result = &(iter->result);
+
+ if (iter->finished)
+ return NULL;
+
+ if (BlockNumberIsValid(result->blkno))
+ {
+ /* Process the previously collected key-value */
+ result->num_offsets = 0;
+ tidstore_iter_extract_tids(iter, iter->next_key, iter->next_val);
+ }
+
+ while (tidstore_iter_kv(iter, &key, &val))
+ {
+ BlockNumber blkno;
+
+ blkno = key_get_blkno(iter->ts, key);
+
+ if (BlockNumberIsValid(result->blkno) && result->blkno != blkno)
+ {
+ /*
+ * We got a key-value pair for a different block. So return the
+ * collected tids, and remember the key-value for the next iteration.
+ */
+ iter->next_key = key;
+ iter->next_val = val;
+ return result;
+ }
+
+ /* Collect tids extracted from the key-value pair */
+ tidstore_iter_extract_tids(iter, key, val);
+ }
+
+ iter->finished = true;
+ return result;
+}
+
+/*
+ * Finish an iteration over TidStore. This needs to be called after finishing
+ * or when existing an iteration.
+ */
+void
+tidstore_end_iterate(TidStoreIter *iter)
+{
+ if (TidStoreIsShared(iter->ts))
+ shared_rt_end_iterate(iter->tree_iter.shared);
+ else
+ local_rt_end_iterate(iter->tree_iter.local);
+
+ pfree(iter->result.offsets);
+ pfree(iter);
+}
+
+/* Return the number of tids we collected so far */
+int64
+tidstore_num_tids(TidStore *ts)
+{
+ uint64 num_tids;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ if (!TidStoreIsShared(ts))
+ return ts->control->num_tids;
+
+ LWLockAcquire(&ts->control->lock, LW_SHARED);
+ num_tids = ts->control->num_tids;
+ LWLockRelease(&ts->control->lock);
+
+ return num_tids;
+}
+
+/* Return true if the current memory usage of TidStore exceeds the limit */
+bool
+tidstore_is_full(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ return (tidstore_memory_usage(ts) > ts->control->max_bytes);
+}
+
+/* Return the maximum memory TidStore can use */
+size_t
+tidstore_max_memory(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ return ts->control->max_bytes;
+}
+
+/* Return the memory usage of TidStore */
+size_t
+tidstore_memory_usage(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ /*
+ * In the shared case, TidStoreControl and radix_tree are backed by the
+ * same DSA area and rt_memory_usage() returns the value including both.
+ * So we don't need to add the size of TidStoreControl separately.
+ */
+ if (TidStoreIsShared(ts))
+ return sizeof(TidStore) + shared_rt_memory_usage(ts->tree.shared);
+
+ return sizeof(TidStore) + sizeof(TidStore) + local_rt_memory_usage(ts->tree.local);
+}
+
+/*
+ * Get a handle that can be used by other processes to attach to this TidStore
+ */
+tidstore_handle
+tidstore_get_handle(TidStore *ts)
+{
+ Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+ return ts->control->handle;
+}
+
+/* Extract tids from the given key-value pair */
+static void
+tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
+{
+ TidStoreIterResult *result = (&iter->result);
+
+ for (int i = 0; i < sizeof(uint64) * BITS_PER_BYTE; i++)
+ {
+ uint64 tid_i;
+ OffsetNumber off;
+
+ if (i > iter->ts->control->max_offset)
+ {
+ Assert(!iter->ts->control->encode_tids);
+ break;
+ }
+
+ if ((val & (UINT64CONST(1) << i)) == 0)
+ continue;
+
+ tid_i = key << TIDSTORE_VALUE_NBITS;
+ tid_i |= i;
+
+ off = tid_i & ((UINT64CONST(1) << iter->ts->control->offset_nbits) - 1);
+
+ Assert(result->num_offsets < iter->ts->control->max_offset);
+ result->offsets[result->num_offsets++] = off;
+ }
+
+ result->blkno = key_get_blkno(iter->ts, key);
+}
+
+/* Get block number from the given key */
+static inline BlockNumber
+key_get_blkno(TidStore *ts, uint64 key)
+{
+ if (ts->control->encode_tids)
+ return (BlockNumber) (key >> ts->control->offset_key_nbits);
+
+ return (BlockNumber) key;
+}
+
+/* Encode a tid to key and offset */
+static inline uint64
+tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off)
+{
+ uint64 key;
+ uint64 tid_i;
+
+ if (!ts->control->encode_tids)
+ {
+ *off = ItemPointerGetOffsetNumber(tid);
+
+ /* Use the block number as the key */
+ return (int64) ItemPointerGetBlockNumber(tid);
+ }
+
+ tid_i = ItemPointerGetOffsetNumber(tid);
+ tid_i |= (uint64) ItemPointerGetBlockNumber(tid) << ts->control->offset_nbits;
+
+ *off = tid_i & ((UINT64CONST(1) << TIDSTORE_VALUE_NBITS) - 1);
+ key = tid_i >> TIDSTORE_VALUE_NBITS;
+ Assert(*off < (sizeof(uint64) * BITS_PER_BYTE));
+
+ return key;
+}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index d2ec396045..55b3a04097 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -176,6 +176,8 @@ static const char *const BuiltinTrancheNames[] = {
"SharedTupleStore",
/* LWTRANCHE_SHARED_TIDBITMAP: */
"SharedTidBitmap",
+ /* LWTRANCHE_SHARED_TIDSTORE: */
+ "SharedTidStore",
/* LWTRANCHE_PARALLEL_APPEND: */
"ParallelAppend",
/* LWTRANCHE_PER_XACT_PREDICATE_LIST: */
diff --git a/src/include/access/tidstore.h b/src/include/access/tidstore.h
new file mode 100644
index 0000000000..a35a52124a
--- /dev/null
+++ b/src/include/access/tidstore.h
@@ -0,0 +1,49 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.h
+ * Tid storage.
+ *
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/tidstore.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef TIDSTORE_H
+#define TIDSTORE_H
+
+#include "storage/itemptr.h"
+#include "utils/dsa.h"
+
+typedef dsa_pointer tidstore_handle;
+
+typedef struct TidStore TidStore;
+typedef struct TidStoreIter TidStoreIter;
+
+typedef struct TidStoreIterResult
+{
+ BlockNumber blkno;
+ OffsetNumber *offsets;
+ int num_offsets;
+} TidStoreIterResult;
+
+extern TidStore *tidstore_create(size_t max_bytes, int max_offset, dsa_area *dsa);
+extern TidStore *tidstore_attach(dsa_area *dsa, dsa_pointer handle);
+extern void tidstore_detach(TidStore *ts);
+extern void tidstore_destroy(TidStore *ts);
+extern void tidstore_reset(TidStore *ts);
+extern void tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets);
+extern bool tidstore_lookup_tid(TidStore *ts, ItemPointer tid);
+extern TidStoreIter * tidstore_begin_iterate(TidStore *ts);
+extern TidStoreIterResult *tidstore_iterate_next(TidStoreIter *iter);
+extern void tidstore_end_iterate(TidStoreIter *iter);
+extern int64 tidstore_num_tids(TidStore *ts);
+extern bool tidstore_is_full(TidStore *ts);
+extern size_t tidstore_max_memory(TidStore *ts);
+extern size_t tidstore_memory_usage(TidStore *ts);
+extern tidstore_handle tidstore_get_handle(TidStore *ts);
+
+#endif /* TIDSTORE_H */
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index d2c7afb8f4..07002fdfbe 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -199,6 +199,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_PER_SESSION_RECORD_TYPMOD,
LWTRANCHE_SHARED_TUPLESTORE,
LWTRANCHE_SHARED_TIDBITMAP,
+ LWTRANCHE_SHARED_TIDSTORE,
LWTRANCHE_PARALLEL_APPEND,
LWTRANCHE_PER_XACT_PREDICATE_LIST,
LWTRANCHE_PGSTATS_DSA,
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 9659eb85d7..bddc16ada7 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
test_rls_hooks \
test_shm_mq \
test_slru \
+ test_tidstore \
unsafe_tests \
worker_spi
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 232cbdac80..c0d5645ad8 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -30,5 +30,6 @@ subdir('test_regex')
subdir('test_rls_hooks')
subdir('test_shm_mq')
subdir('test_slru')
+subdir('test_tidstore')
subdir('unsafe_tests')
subdir('worker_spi')
diff --git a/src/test/modules/test_tidstore/Makefile b/src/test/modules/test_tidstore/Makefile
new file mode 100644
index 0000000000..dab107d70c
--- /dev/null
+++ b/src/test/modules/test_tidstore/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_tidstore/Makefile
+
+MODULE_big = test_tidstore
+OBJS = \
+ $(WIN32RES) \
+ test_tidstore.o
+PGFILEDESC = "test_tidstore - test code for src/backend/access/common/tidstore.c"
+
+EXTENSION = test_tidstore
+DATA = test_tidstore--1.0.sql
+
+REGRESS = test_tidstore
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_tidstore
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_tidstore/expected/test_tidstore.out b/src/test/modules/test_tidstore/expected/test_tidstore.out
new file mode 100644
index 0000000000..7ff2f9af87
--- /dev/null
+++ b/src/test/modules/test_tidstore/expected/test_tidstore.out
@@ -0,0 +1,13 @@
+CREATE EXTENSION test_tidstore;
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
+NOTICE: testing empty tidstore
+NOTICE: testing basic operations
+ test_tidstore
+---------------
+
+(1 row)
+
diff --git a/src/test/modules/test_tidstore/meson.build b/src/test/modules/test_tidstore/meson.build
new file mode 100644
index 0000000000..31f2da7b61
--- /dev/null
+++ b/src/test/modules/test_tidstore/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_tidstore_sources = files(
+ 'test_tidstore.c',
+)
+
+if host_system == 'windows'
+ test_tidstore_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_tidstore',
+ '--FILEDESC', 'test_tidstore - test code for src/backend/access/common/tidstore.c',])
+endif
+
+test_tidstore = shared_module('test_tidstore',
+ test_tidstore_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += test_tidstore
+
+install_data(
+ 'test_tidstore.control',
+ 'test_tidstore--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'test_tidstore',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'test_tidstore',
+ ],
+ },
+}
diff --git a/src/test/modules/test_tidstore/sql/test_tidstore.sql b/src/test/modules/test_tidstore/sql/test_tidstore.sql
new file mode 100644
index 0000000000..03aea31815
--- /dev/null
+++ b/src/test/modules/test_tidstore/sql/test_tidstore.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_tidstore;
+
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
diff --git a/src/test/modules/test_tidstore/test_tidstore--1.0.sql b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
new file mode 100644
index 0000000000..47e9149900
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_tidstore/test_tidstore--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_tidstore" to load this file. \quit
+
+CREATE FUNCTION test_tidstore()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_tidstore/test_tidstore.c b/src/test/modules/test_tidstore/test_tidstore.c
new file mode 100644
index 0000000000..9b849ae8e8
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.c
@@ -0,0 +1,195 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_tidstore.c
+ * Test TidStore data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_tidstore/test_tidstore.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/tidstore.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "utils/memutils.h"
+
+PG_MODULE_MAGIC;
+
+#define TEST_TIDSTORE_MAX_BYTES (2 * 1024 * 1024L) /* 2MB */
+
+PG_FUNCTION_INFO_V1(test_tidstore);
+
+static void
+check_tid(TidStore *ts, BlockNumber blkno, OffsetNumber off, bool expect)
+{
+ ItemPointerData tid;
+ bool found;
+
+ ItemPointerSet(&tid, blkno, off);
+
+ found = tidstore_lookup_tid(ts, &tid);
+
+ if (found != expect)
+ elog(ERROR, "lookup TID (%u, %u) returned %d, expected %d",
+ blkno, off, found, expect);
+}
+
+static void
+test_basic(int max_offset)
+{
+#define TEST_TIDSTORE_NUM_BLOCKS 5
+#define TEST_TIDSTORE_NUM_OFFSETS 5
+
+ TidStore *ts;
+ TidStoreIter *iter;
+ TidStoreIterResult *iter_result;
+ BlockNumber blks[TEST_TIDSTORE_NUM_BLOCKS] = {
+ 0, MaxBlockNumber, MaxBlockNumber - 1, 1, MaxBlockNumber / 2,
+ };
+ BlockNumber blks_sorted[TEST_TIDSTORE_NUM_BLOCKS] = {
+ 0, 1, MaxBlockNumber / 2, MaxBlockNumber - 1, MaxBlockNumber
+ };
+ OffsetNumber offs[TEST_TIDSTORE_NUM_OFFSETS];
+ int blk_idx;
+
+ /* prepare the offset array */
+ offs[0] = FirstOffsetNumber;
+ offs[1] = FirstOffsetNumber + 1;
+ offs[2] = max_offset / 2;
+ offs[3] = max_offset - 1;
+ offs[4] = max_offset;
+
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
+
+ /* add tids */
+ for (int i = 0; i < TEST_TIDSTORE_NUM_BLOCKS; i++)
+ tidstore_add_tids(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
+
+ /* lookup test */
+ for (OffsetNumber off = FirstOffsetNumber ; off < max_offset; off++)
+ {
+ bool expect = false;
+ for (int i = 0; i < TEST_TIDSTORE_NUM_OFFSETS; i++)
+ {
+ if (offs[i] == off)
+ {
+ expect = true;
+ break;
+ }
+ }
+
+ check_tid(ts, 0, off, expect);
+ check_tid(ts, 2, off, false);
+ check_tid(ts, MaxBlockNumber - 2, off, false);
+ check_tid(ts, MaxBlockNumber, off, expect);
+ }
+
+ /* test the number of tids */
+ if (tidstore_num_tids(ts) != (TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS))
+ elog(ERROR, "tidstore_num_tids returned " UINT64_FORMAT ", expected %d",
+ tidstore_num_tids(ts),
+ TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS);
+
+ /* iteration test */
+ iter = tidstore_begin_iterate(ts);
+ blk_idx = 0;
+ while ((iter_result = tidstore_iterate_next(iter)) != NULL)
+ {
+ /* check the returned block number */
+ if (blks_sorted[blk_idx] != iter_result->blkno)
+ elog(ERROR, "tidstore_iterate_next returned block number %u, expected %u",
+ iter_result->blkno, blks_sorted[blk_idx]);
+
+ /* check the returned offset numbers */
+ if (TEST_TIDSTORE_NUM_OFFSETS != iter_result->num_offsets)
+ elog(ERROR, "tidstore_iterate_next returned %u offsets, expected %u",
+ iter_result->num_offsets, TEST_TIDSTORE_NUM_OFFSETS);
+
+ for (int i = 0; i < iter_result->num_offsets; i++)
+ {
+ if (offs[i] != iter_result->offsets[i])
+ elog(ERROR, "tidstore_iterate_next returned offset number %u on block %u, expected %u",
+ iter_result->offsets[i], iter_result->blkno, offs[i]);
+ }
+
+ blk_idx++;
+ }
+
+ if (blk_idx != TEST_TIDSTORE_NUM_BLOCKS)
+ elog(ERROR, "tidstore_iterate_next returned %d blocks, expected %d",
+ blk_idx, TEST_TIDSTORE_NUM_BLOCKS);
+
+ /* remove all tids */
+ tidstore_reset(ts);
+
+ /* test the number of tids */
+ if (tidstore_num_tids(ts) != 0)
+ elog(ERROR, "tidstore_num_tids on empty store returned non-zero");
+
+ /* lookup test for empty store */
+ for (OffsetNumber off = FirstOffsetNumber ; off < MaxHeapTuplesPerPage;
+ off++)
+ {
+ check_tid(ts, 0, off, false);
+ check_tid(ts, 2, off, false);
+ check_tid(ts, MaxBlockNumber - 2, off, false);
+ check_tid(ts, MaxBlockNumber, off, false);
+ }
+
+ tidstore_destroy(ts);
+}
+
+static void
+test_empty(void)
+{
+ TidStore *ts;
+ TidStoreIter *iter;
+ ItemPointerData tid;
+
+ elog(NOTICE, "testing empty tidstore");
+
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
+
+ ItemPointerSet(&tid, 0, FirstOffsetNumber);
+ if (tidstore_lookup_tid(ts, &tid))
+ elog(ERROR, "tidstore_lookup_tid for (0,1) on empty store returned true");
+
+ ItemPointerSet(&tid, MaxBlockNumber, MaxOffsetNumber);
+ if (tidstore_lookup_tid(ts, &tid))
+ elog(ERROR, "tidstore_lookup_tid for (%u,%u) on empty store returned true",
+ MaxBlockNumber, MaxOffsetNumber);
+
+ if (tidstore_num_tids(ts) != 0)
+ elog(ERROR, "tidstore_num_entries on empty store returned non-zero");
+
+ if (tidstore_is_full(ts))
+ elog(ERROR, "tidstore_is_full on empty store returned true");
+
+ iter = tidstore_begin_iterate(ts);
+
+ if (tidstore_iterate_next(iter) != NULL)
+ elog(ERROR, "tidstore_iterate_next on empty store returned TIDs");
+
+ tidstore_end_iterate(iter);
+
+ tidstore_destroy(ts);
+}
+
+Datum
+test_tidstore(PG_FUNCTION_ARGS)
+{
+ test_empty();
+
+ elog(NOTICE, "testing basic operations");
+ test_basic(MaxHeapTuplesPerPage);
+ test_basic(10);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_tidstore/test_tidstore.control b/src/test/modules/test_tidstore/test_tidstore.control
new file mode 100644
index 0000000000..9b6bd4638f
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.control
@@ -0,0 +1,4 @@
+comment = 'Test code for tidstore'
+default_version = '1.0'
+module_pathname = '$libdir/test_tidstore'
+relocatable = true
--
2.39.1
[text/x-patch] v26-0009-Use-TIDStore-for-storing-dead-tuple-TID-during-l.patch (48.1K, ../../CAFBsxsE7FLxfc2bfPf-DwdJ6RmDcpUcNRyUnHeuR26LVi6jzGA@mail.gmail.com/12-v26-0009-Use-TIDStore-for-storing-dead-tuple-TID-during-l.patch)
download | inline diff:
From ed8115b5f5c1b0745e35a0d6d72064ad9df4cf42 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Tue, 7 Feb 2023 17:19:29 +0700
Subject: [PATCH v26 9/9] Use TIDStore for storing dead tuple TID during lazy
vacuum
Previously, we used an array of ItemPointerData to store dead tuple
TIDs, which was not space efficient and slow to lookup. Also, we had
the 1GB limit on its size.
Now we use TIDStore to store dead tuple TIDs. Since the TIDStore,
backed by the radix tree, incrementaly allocates the memory, we get
rid of the 1GB limit.
Since we are no longer able to exactly estimate the maximum number of
TIDs can be stored the pg_stat_progress_vacuum shows the progress
information based on the amount of memory in bytes. The column names
are also changed to max_dead_tuple_bytes and num_dead_tuple_bytes.
In addition, since the TIDStore use the radix tree internally, the
minimum amount of memory required by TIDStore is 1MB, the inital DSA
segment size. Due to that, we increase the minimum value of
maintenance_work_mem (also autovacuum_work_mem) from 1MB to 2MB.
XXX: needs to bump catalog version
---
doc/src/sgml/monitoring.sgml | 8 +-
src/backend/access/heap/vacuumlazy.c | 278 ++++++++-------------
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/vacuum.c | 78 +-----
src/backend/commands/vacuumparallel.c | 73 +++---
src/backend/postmaster/autovacuum.c | 6 +-
src/backend/storage/lmgr/lwlock.c | 2 +
src/backend/utils/misc/guc_tables.c | 2 +-
src/include/commands/progress.h | 4 +-
src/include/commands/vacuum.h | 25 +-
src/include/storage/lwlock.h | 1 +
src/test/regress/expected/cluster.out | 2 +-
src/test/regress/expected/create_index.out | 2 +-
src/test/regress/expected/rules.out | 4 +-
src/test/regress/sql/cluster.sql | 2 +-
src/test/regress/sql/create_index.sql | 2 +-
16 files changed, 177 insertions(+), 314 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index d936aa3da3..0230c74e3d 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6870,10 +6870,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>max_dead_tuples</structfield> <type>bigint</type>
+ <structfield>max_dead_tuple_bytes</structfield> <type>bigint</type>
</para>
<para>
- Number of dead tuples that we can store before needing to perform
+ Amount of dead tuple data that we can store before needing to perform
an index vacuum cycle, based on
<xref linkend="guc-maintenance-work-mem"/>.
</para></entry>
@@ -6881,10 +6881,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>num_dead_tuples</structfield> <type>bigint</type>
+ <structfield>num_dead_tuple_bytes</structfield> <type>bigint</type>
</para>
<para>
- Number of dead tuples collected since the last index vacuum cycle.
+ Amount of dead tuple data collected since the last index vacuum cycle.
</para></entry>
</row>
</tbody>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 8f14cf85f3..b4e40423a8 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3,18 +3,18 @@
* vacuumlazy.c
* Concurrent ("lazy") vacuuming.
*
- * The major space usage for vacuuming is storage for the array of dead TIDs
+ * The major space usage for vacuuming is TidStore, a storage for dead TIDs
* that are to be removed from indexes. We want to ensure we can vacuum even
* the very largest relations with finite memory space usage. To do that, we
- * set upper bounds on the number of TIDs we can keep track of at once.
+ * set upper bounds on the maximum memory that can be used for keeping track
+ * of dead TIDs at once.
*
* We are willing to use at most maintenance_work_mem (or perhaps
* autovacuum_work_mem) memory space to keep track of dead TIDs. We initially
- * allocate an array of TIDs of that size, with an upper limit that depends on
- * table size (this limit ensures we don't allocate a huge area uselessly for
- * vacuuming small tables). If the array threatens to overflow, we must call
- * lazy_vacuum to vacuum indexes (and to vacuum the pages that we've pruned).
- * This frees up the memory space dedicated to storing dead TIDs.
+ * create a TidStore with the maximum bytes that can be used by the TidStore.
+ * If the TidStore is full, we must call lazy_vacuum to vacuum indexes (and to
+ * vacuum the pages that we've pruned). This frees up the memory space dedicated
+ * to storing dead TIDs.
*
* In practice VACUUM will often complete its initial pass over the target
* heap relation without ever running out of space to store TIDs. This means
@@ -40,6 +40,7 @@
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
#include "access/multixact.h"
+#include "access/tidstore.h"
#include "access/transam.h"
#include "access/visibilitymap.h"
#include "access/xact.h"
@@ -188,7 +189,7 @@ typedef struct LVRelState
* lazy_vacuum_heap_rel, which marks the same LP_DEAD line pointers as
* LP_UNUSED during second heap pass.
*/
- VacDeadItems *dead_items; /* TIDs whose index tuples we'll delete */
+ TidStore *dead_items; /* TIDs whose index tuples we'll delete */
BlockNumber rel_pages; /* total number of pages */
BlockNumber scanned_pages; /* # pages examined (not skipped via VM) */
BlockNumber removed_pages; /* # pages removed by relation truncation */
@@ -220,11 +221,14 @@ typedef struct LVRelState
typedef struct LVPagePruneState
{
bool hastup; /* Page prevents rel truncation? */
- bool has_lpdead_items; /* includes existing LP_DEAD items */
+
+ /* collected offsets of LP_DEAD items including existing ones */
+ OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
+ int num_offsets;
/*
* State describes the proper VM bit states to set for the page following
- * pruning and freezing. all_visible implies !has_lpdead_items, but don't
+ * pruning and freezing. all_visible implies num_offsets == 0, but don't
* trust all_frozen result unless all_visible is also set to true.
*/
bool all_visible; /* Every item visible to all? */
@@ -259,8 +263,9 @@ static bool lazy_scan_noprune(LVRelState *vacrel, Buffer buf,
static void lazy_vacuum(LVRelState *vacrel);
static bool lazy_vacuum_all_indexes(LVRelState *vacrel);
static void lazy_vacuum_heap_rel(LVRelState *vacrel);
-static int lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
- Buffer buffer, int index, Buffer vmbuffer);
+static void lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
+ OffsetNumber *offsets, int num_offsets,
+ Buffer buffer, Buffer vmbuffer);
static bool lazy_check_wraparound_failsafe(LVRelState *vacrel);
static void lazy_cleanup_all_indexes(LVRelState *vacrel);
static IndexBulkDeleteResult *lazy_vacuum_one_index(Relation indrel,
@@ -487,11 +492,11 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
}
/*
- * Allocate dead_items array memory using dead_items_alloc. This handles
- * parallel VACUUM initialization as part of allocating shared memory
- * space used for dead_items. (But do a failsafe precheck first, to
- * ensure that parallel VACUUM won't be attempted at all when relfrozenxid
- * is already dangerously old.)
+ * Allocate dead_items memory using dead_items_alloc. This handles parallel
+ * VACUUM initialization as part of allocating shared memory space used for
+ * dead_items. (But do a failsafe precheck first, to ensure that parallel
+ * VACUUM won't be attempted at all when relfrozenxid is already dangerously
+ * old.)
*/
lazy_check_wraparound_failsafe(vacrel);
dead_items_alloc(vacrel, params->nworkers);
@@ -797,7 +802,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* have collected the TIDs whose index tuples need to be removed.
*
* Finally, invokes lazy_vacuum_heap_rel to vacuum heap pages, which
- * largely consists of marking LP_DEAD items (from collected TID array)
+ * largely consists of marking LP_DEAD items (from vacrel->dead_items)
* as LP_UNUSED. This has to happen in a second, final pass over the
* heap, to preserve a basic invariant that all index AMs rely on: no
* extant index tuple can ever be allowed to contain a TID that points to
@@ -825,21 +830,21 @@ lazy_scan_heap(LVRelState *vacrel)
blkno,
next_unskippable_block,
next_fsm_block_to_vacuum = 0;
- VacDeadItems *dead_items = vacrel->dead_items;
+ TidStore *dead_items = vacrel->dead_items;
Buffer vmbuffer = InvalidBuffer;
bool next_unskippable_allvis,
skipping_current_range;
const int initprog_index[] = {
PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_TOTAL_HEAP_BLKS,
- PROGRESS_VACUUM_MAX_DEAD_TUPLES
+ PROGRESS_VACUUM_MAX_DEAD_TUPLE_BYTES
};
int64 initprog_val[3];
/* Report that we're scanning the heap, advertising total # of blocks */
initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
initprog_val[1] = rel_pages;
- initprog_val[2] = dead_items->max_items;
+ initprog_val[2] = tidstore_max_memory(vacrel->dead_items);
pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
/* Set up an initial range of skippable blocks using the visibility map */
@@ -906,8 +911,7 @@ lazy_scan_heap(LVRelState *vacrel)
* dead_items TIDs, pause and do a cycle of vacuuming before we tackle
* this page.
*/
- Assert(dead_items->max_items >= MaxHeapTuplesPerPage);
- if (dead_items->max_items - dead_items->num_items < MaxHeapTuplesPerPage)
+ if (tidstore_is_full(vacrel->dead_items))
{
/*
* Before beginning index vacuuming, we release any pin we may
@@ -969,7 +973,7 @@ lazy_scan_heap(LVRelState *vacrel)
continue;
}
- /* Collect LP_DEAD items in dead_items array, count tuples */
+ /* Collect LP_DEAD items in dead_items, count tuples */
if (lazy_scan_noprune(vacrel, buf, blkno, page, &hastup,
&recordfreespace))
{
@@ -1011,14 +1015,14 @@ lazy_scan_heap(LVRelState *vacrel)
* Prune, freeze, and count tuples.
*
* Accumulates details of remaining LP_DEAD line pointers on page in
- * dead_items array. This includes LP_DEAD line pointers that we
- * pruned ourselves, as well as existing LP_DEAD line pointers that
- * were pruned some time earlier. Also considers freezing XIDs in the
- * tuple headers of remaining items with storage.
+ * dead_items. This includes LP_DEAD line pointers that we pruned
+ * ourselves, as well as existing LP_DEAD line pointers that were pruned
+ * some time earlier. Also considers freezing XIDs in the tuple headers
+ * of remaining items with storage.
*/
lazy_scan_prune(vacrel, buf, blkno, page, &prunestate);
- Assert(!prunestate.all_visible || !prunestate.has_lpdead_items);
+ Assert(!prunestate.all_visible || (prunestate.num_offsets == 0));
/* Remember the location of the last page with nonremovable tuples */
if (prunestate.hastup)
@@ -1034,14 +1038,12 @@ lazy_scan_heap(LVRelState *vacrel)
* performed here can be thought of as the one-pass equivalent of
* a call to lazy_vacuum().
*/
- if (prunestate.has_lpdead_items)
+ if (prunestate.num_offsets > 0)
{
Size freespace;
- lazy_vacuum_heap_page(vacrel, blkno, buf, 0, vmbuffer);
-
- /* Forget the LP_DEAD items that we just vacuumed */
- dead_items->num_items = 0;
+ lazy_vacuum_heap_page(vacrel, blkno, prunestate.deadoffsets,
+ prunestate.num_offsets, buf, vmbuffer);
/*
* Periodically perform FSM vacuuming to make newly-freed
@@ -1078,7 +1080,16 @@ lazy_scan_heap(LVRelState *vacrel)
* with prunestate-driven visibility map and FSM steps (just like
* the two-pass strategy).
*/
- Assert(dead_items->num_items == 0);
+ Assert(tidstore_num_tids(dead_items) == 0);
+ }
+ else if (prunestate.num_offsets > 0)
+ {
+ /* Save details of the LP_DEAD items from the page in dead_items */
+ tidstore_add_tids(dead_items, blkno, prunestate.deadoffsets,
+ prunestate.num_offsets);
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
+ tidstore_memory_usage(dead_items));
}
/*
@@ -1145,7 +1156,7 @@ lazy_scan_heap(LVRelState *vacrel)
* There should never be LP_DEAD items on a page with PD_ALL_VISIBLE
* set, however.
*/
- else if (prunestate.has_lpdead_items && PageIsAllVisible(page))
+ else if ((prunestate.num_offsets > 0) && PageIsAllVisible(page))
{
elog(WARNING, "page containing LP_DEAD items is marked as all-visible in relation \"%s\" page %u",
vacrel->relname, blkno);
@@ -1193,7 +1204,7 @@ lazy_scan_heap(LVRelState *vacrel)
* Final steps for block: drop cleanup lock, record free space in the
* FSM
*/
- if (prunestate.has_lpdead_items && vacrel->do_index_vacuuming)
+ if ((prunestate.num_offsets > 0) && vacrel->do_index_vacuuming)
{
/*
* Wait until lazy_vacuum_heap_rel() to save free space. This
@@ -1249,7 +1260,7 @@ lazy_scan_heap(LVRelState *vacrel)
* Do index vacuuming (call each index's ambulkdelete routine), then do
* related heap vacuuming
*/
- if (dead_items->num_items > 0)
+ if (tidstore_num_tids(dead_items) > 0)
lazy_vacuum(vacrel);
/*
@@ -1524,9 +1535,9 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* The approach we take now is to restart pruning when the race condition is
* detected. This allows heap_page_prune() to prune the tuples inserted by
* the now-aborted transaction. This is a little crude, but it guarantees
- * that any items that make it into the dead_items array are simple LP_DEAD
- * line pointers, and that every remaining item with tuple storage is
- * considered as a candidate for freezing.
+ * that any items that make it into the dead_items are simple LP_DEAD line
+ * pointers, and that every remaining item with tuple storage is considered
+ * as a candidate for freezing.
*/
static void
lazy_scan_prune(LVRelState *vacrel,
@@ -1543,13 +1554,11 @@ lazy_scan_prune(LVRelState *vacrel,
HTSV_Result res;
int tuples_deleted,
tuples_frozen,
- lpdead_items,
live_tuples,
recently_dead_tuples;
int nnewlpdead;
HeapPageFreeze pagefrz;
int64 fpi_before = pgWalUsage.wal_fpi;
- OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
@@ -1571,7 +1580,6 @@ retry:
pagefrz.NoFreezePageRelminMxid = vacrel->NewRelminMxid;
tuples_deleted = 0;
tuples_frozen = 0;
- lpdead_items = 0;
live_tuples = 0;
recently_dead_tuples = 0;
@@ -1580,9 +1588,9 @@ retry:
*
* We count tuples removed by the pruning step as tuples_deleted. Its
* final value can be thought of as the number of tuples that have been
- * deleted from the table. It should not be confused with lpdead_items;
- * lpdead_items's final value can be thought of as the number of tuples
- * that were deleted from indexes.
+ * deleted from the table. It should not be confused with
+ * prunestate->deadoffsets; prunestate->deadoffsets's final value can
+ * be thought of as the number of tuples that were deleted from indexes.
*/
tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest,
InvalidTransactionId, 0, &nnewlpdead,
@@ -1593,7 +1601,7 @@ retry:
* requiring freezing among remaining tuples with storage
*/
prunestate->hastup = false;
- prunestate->has_lpdead_items = false;
+ prunestate->num_offsets = 0;
prunestate->all_visible = true;
prunestate->all_frozen = true;
prunestate->visibility_cutoff_xid = InvalidTransactionId;
@@ -1638,7 +1646,7 @@ retry:
* (This is another case where it's useful to anticipate that any
* LP_DEAD items will become LP_UNUSED during the ongoing VACUUM.)
*/
- deadoffsets[lpdead_items++] = offnum;
+ prunestate->deadoffsets[prunestate->num_offsets++] = offnum;
continue;
}
@@ -1875,7 +1883,7 @@ retry:
*/
#ifdef USE_ASSERT_CHECKING
/* Note that all_frozen value does not matter when !all_visible */
- if (prunestate->all_visible && lpdead_items == 0)
+ if (prunestate->all_visible && prunestate->num_offsets == 0)
{
TransactionId cutoff;
bool all_frozen;
@@ -1888,28 +1896,9 @@ retry:
}
#endif
- /*
- * Now save details of the LP_DEAD items from the page in vacrel
- */
- if (lpdead_items > 0)
+ if (prunestate->num_offsets > 0)
{
- VacDeadItems *dead_items = vacrel->dead_items;
- ItemPointerData tmp;
-
vacrel->lpdead_item_pages++;
- prunestate->has_lpdead_items = true;
-
- ItemPointerSetBlockNumber(&tmp, blkno);
-
- for (int i = 0; i < lpdead_items; i++)
- {
- ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
- dead_items->items[dead_items->num_items++] = tmp;
- }
-
- Assert(dead_items->num_items <= dead_items->max_items);
- pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
- dead_items->num_items);
/*
* It was convenient to ignore LP_DEAD items in all_visible earlier on
@@ -1928,7 +1917,7 @@ retry:
/* Finally, add page-local counts to whole-VACUUM counts */
vacrel->tuples_deleted += tuples_deleted;
vacrel->tuples_frozen += tuples_frozen;
- vacrel->lpdead_items += lpdead_items;
+ vacrel->lpdead_items += prunestate->num_offsets;
vacrel->live_tuples += live_tuples;
vacrel->recently_dead_tuples += recently_dead_tuples;
}
@@ -1940,7 +1929,7 @@ retry:
* lazy_scan_prune, which requires a full cleanup lock. While pruning isn't
* performed here, it's quite possible that an earlier opportunistic pruning
* operation left LP_DEAD items behind. We'll at least collect any such items
- * in the dead_items array for removal from indexes.
+ * in the dead_items for removal from indexes.
*
* For aggressive VACUUM callers, we may return false to indicate that a full
* cleanup lock is required for processing by lazy_scan_prune. This is only
@@ -2099,7 +2088,7 @@ lazy_scan_noprune(LVRelState *vacrel,
vacrel->NewRelfrozenXid = NoFreezePageRelfrozenXid;
vacrel->NewRelminMxid = NoFreezePageRelminMxid;
- /* Save any LP_DEAD items found on the page in dead_items array */
+ /* Save any LP_DEAD items found on the page in dead_items */
if (vacrel->nindexes == 0)
{
/* Using one-pass strategy (since table has no indexes) */
@@ -2129,8 +2118,7 @@ lazy_scan_noprune(LVRelState *vacrel,
}
else
{
- VacDeadItems *dead_items = vacrel->dead_items;
- ItemPointerData tmp;
+ TidStore *dead_items = vacrel->dead_items;
/*
* Page has LP_DEAD items, and so any references/TIDs that remain in
@@ -2139,17 +2127,10 @@ lazy_scan_noprune(LVRelState *vacrel,
*/
vacrel->lpdead_item_pages++;
- ItemPointerSetBlockNumber(&tmp, blkno);
+ tidstore_add_tids(dead_items, blkno, deadoffsets, lpdead_items);
- for (int i = 0; i < lpdead_items; i++)
- {
- ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
- dead_items->items[dead_items->num_items++] = tmp;
- }
-
- Assert(dead_items->num_items <= dead_items->max_items);
- pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
- dead_items->num_items);
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
+ tidstore_memory_usage(dead_items));
vacrel->lpdead_items += lpdead_items;
@@ -2198,7 +2179,7 @@ lazy_vacuum(LVRelState *vacrel)
if (!vacrel->do_index_vacuuming)
{
Assert(!vacrel->do_index_cleanup);
- vacrel->dead_items->num_items = 0;
+ tidstore_reset(vacrel->dead_items);
return;
}
@@ -2227,7 +2208,7 @@ lazy_vacuum(LVRelState *vacrel)
BlockNumber threshold;
Assert(vacrel->num_index_scans == 0);
- Assert(vacrel->lpdead_items == vacrel->dead_items->num_items);
+ Assert(vacrel->lpdead_items == tidstore_num_tids(vacrel->dead_items));
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2254,8 +2235,8 @@ lazy_vacuum(LVRelState *vacrel)
* cases then this may need to be reconsidered.
*/
threshold = (double) vacrel->rel_pages * BYPASS_THRESHOLD_PAGES;
- bypass = (vacrel->lpdead_item_pages < threshold &&
- vacrel->lpdead_items < MAXDEADITEMS(32L * 1024L * 1024L));
+ bypass = (vacrel->lpdead_item_pages < threshold) &&
+ tidstore_memory_usage(vacrel->dead_items) < (32L * 1024L * 1024L);
}
if (bypass)
@@ -2300,7 +2281,7 @@ lazy_vacuum(LVRelState *vacrel)
* Forget the LP_DEAD items that we just vacuumed (or just decided to not
* vacuum)
*/
- vacrel->dead_items->num_items = 0;
+ tidstore_reset(vacrel->dead_items);
}
/*
@@ -2373,7 +2354,7 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
* place).
*/
Assert(vacrel->num_index_scans > 0 ||
- vacrel->dead_items->num_items == vacrel->lpdead_items);
+ tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items);
Assert(allindexes || vacrel->failsafe_active);
/*
@@ -2392,9 +2373,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
/*
* lazy_vacuum_heap_rel() -- second pass over the heap for two pass strategy
*
- * This routine marks LP_DEAD items in vacrel->dead_items array as LP_UNUSED.
- * Pages that never had lazy_scan_prune record LP_DEAD items are not visited
- * at all.
+ * This routine marks LP_DEAD items in vacrel->dead_items as LP_UNUSED. Pages
+ * that never had lazy_scan_prune record LP_DEAD items are not visited at all.
*
* We may also be able to truncate the line pointer array of the heap pages we
* visit. If there is a contiguous group of LP_UNUSED items at the end of the
@@ -2410,10 +2390,11 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
static void
lazy_vacuum_heap_rel(LVRelState *vacrel)
{
- int index = 0;
BlockNumber vacuumed_pages = 0;
Buffer vmbuffer = InvalidBuffer;
LVSavedErrInfo saved_err_info;
+ TidStoreIter *iter;
+ TidStoreIterResult *result;
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2428,7 +2409,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
VACUUM_ERRCB_PHASE_VACUUM_HEAP,
InvalidBlockNumber, InvalidOffsetNumber);
- while (index < vacrel->dead_items->num_items)
+ iter = tidstore_begin_iterate(vacrel->dead_items);
+ while ((result = tidstore_iterate_next(iter)) != NULL)
{
BlockNumber blkno;
Buffer buf;
@@ -2437,7 +2419,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
vacuum_delay_point();
- blkno = ItemPointerGetBlockNumber(&vacrel->dead_items->items[index]);
+ blkno = result->blkno;
vacrel->blkno = blkno;
/*
@@ -2451,7 +2433,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
buf = ReadBufferExtended(vacrel->rel, MAIN_FORKNUM, blkno, RBM_NORMAL,
vacrel->bstrategy);
LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
- index = lazy_vacuum_heap_page(vacrel, blkno, buf, index, vmbuffer);
+ lazy_vacuum_heap_page(vacrel, blkno, result->offsets, result->num_offsets,
+ buf, vmbuffer);
/* Now that we've vacuumed the page, record its available space */
page = BufferGetPage(buf);
@@ -2461,6 +2444,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
vacuumed_pages++;
}
+ tidstore_end_iterate(iter);
vacrel->blkno = InvalidBlockNumber;
if (BufferIsValid(vmbuffer))
@@ -2470,36 +2454,31 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
* We set all LP_DEAD items from the first heap pass to LP_UNUSED during
* the second heap pass. No more, no less.
*/
- Assert(index > 0);
Assert(vacrel->num_index_scans > 1 ||
- (index == vacrel->lpdead_items &&
+ (tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items &&
vacuumed_pages == vacrel->lpdead_item_pages));
ereport(DEBUG2,
- (errmsg("table \"%s\": removed %lld dead item identifiers in %u pages",
- vacrel->relname, (long long) index, vacuumed_pages)));
+ (errmsg("table \"%s\": removed " UINT64_FORMAT "dead item identifiers in %u pages",
+ vacrel->relname, tidstore_num_tids(vacrel->dead_items),
+ vacuumed_pages)));
/* Revert to the previous phase information for error traceback */
restore_vacuum_error_info(vacrel, &saved_err_info);
}
/*
- * lazy_vacuum_heap_page() -- free page's LP_DEAD items listed in the
- * vacrel->dead_items array.
+ * lazy_vacuum_heap_page() -- free page's LP_DEAD items.
*
* Caller must have an exclusive buffer lock on the buffer (though a full
* cleanup lock is also acceptable). vmbuffer must be valid and already have
* a pin on blkno's visibility map page.
- *
- * index is an offset into the vacrel->dead_items array for the first listed
- * LP_DEAD item on the page. The return value is the first index immediately
- * after all LP_DEAD items for the same page in the array.
*/
-static int
-lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
- int index, Buffer vmbuffer)
+static void
+lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
+ OffsetNumber *deadoffsets, int num_offsets, Buffer buffer,
+ Buffer vmbuffer)
{
- VacDeadItems *dead_items = vacrel->dead_items;
Page page = BufferGetPage(buffer);
OffsetNumber unused[MaxHeapTuplesPerPage];
int nunused = 0;
@@ -2518,16 +2497,11 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
START_CRIT_SECTION();
- for (; index < dead_items->num_items; index++)
+ for (int i = 0; i < num_offsets; i++)
{
- BlockNumber tblk;
- OffsetNumber toff;
ItemId itemid;
+ OffsetNumber toff = deadoffsets[i];
- tblk = ItemPointerGetBlockNumber(&dead_items->items[index]);
- if (tblk != blkno)
- break; /* past end of tuples for this block */
- toff = ItemPointerGetOffsetNumber(&dead_items->items[index]);
itemid = PageGetItemId(page, toff);
Assert(ItemIdIsDead(itemid) && !ItemIdHasStorage(itemid));
@@ -2597,7 +2571,6 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
/* Revert to the previous phase information for error traceback */
restore_vacuum_error_info(vacrel, &saved_err_info);
- return index;
}
/*
@@ -2687,8 +2660,8 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
* lazy_vacuum_one_index() -- vacuum index relation.
*
* Delete all the index tuples containing a TID collected in
- * vacrel->dead_items array. Also update running statistics.
- * Exact details depend on index AM's ambulkdelete routine.
+ * vacrel->dead_items. Also update running statistics. Exact
+ * details depend on index AM's ambulkdelete routine.
*
* reltuples is the number of heap tuples to be passed to the
* bulkdelete callback. It's always assumed to be estimated.
@@ -3094,48 +3067,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
}
/*
- * Returns the number of dead TIDs that VACUUM should allocate space to
- * store, given a heap rel of size vacrel->rel_pages, and given current
- * maintenance_work_mem setting (or current autovacuum_work_mem setting,
- * when applicable).
- *
- * See the comments at the head of this file for rationale.
- */
-static int
-dead_items_max_items(LVRelState *vacrel)
-{
- int64 max_items;
- int vac_work_mem = IsAutoVacuumWorkerProcess() &&
- autovacuum_work_mem != -1 ?
- autovacuum_work_mem : maintenance_work_mem;
-
- if (vacrel->nindexes > 0)
- {
- BlockNumber rel_pages = vacrel->rel_pages;
-
- max_items = MAXDEADITEMS(vac_work_mem * 1024L);
- max_items = Min(max_items, INT_MAX);
- max_items = Min(max_items, MAXDEADITEMS(MaxAllocSize));
-
- /* curious coding here to ensure the multiplication can't overflow */
- if ((BlockNumber) (max_items / MaxHeapTuplesPerPage) > rel_pages)
- max_items = rel_pages * MaxHeapTuplesPerPage;
-
- /* stay sane if small maintenance_work_mem */
- max_items = Max(max_items, MaxHeapTuplesPerPage);
- }
- else
- {
- /* One-pass case only stores a single heap page's TIDs at a time */
- max_items = MaxHeapTuplesPerPage;
- }
-
- return (int) max_items;
-}
-
-/*
- * Allocate dead_items (either using palloc, or in dynamic shared memory).
- * Sets dead_items in vacrel for caller.
+ * Allocate a (local or shared) TidStore for storing dead TIDs. Sets dead_items
+ * in vacrel for caller.
*
* Also handles parallel initialization as part of allocating dead_items in
* DSM when required.
@@ -3143,11 +3076,9 @@ dead_items_max_items(LVRelState *vacrel)
static void
dead_items_alloc(LVRelState *vacrel, int nworkers)
{
- VacDeadItems *dead_items;
- int max_items;
-
- max_items = dead_items_max_items(vacrel);
- Assert(max_items >= MaxHeapTuplesPerPage);
+ int vac_work_mem = IsAutoVacuumWorkerProcess() &&
+ autovacuum_work_mem != -1 ?
+ autovacuum_work_mem * 1024L : maintenance_work_mem * 1024L;
/*
* Initialize state for a parallel vacuum. As of now, only one worker can
@@ -3174,7 +3105,7 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
else
vacrel->pvs = parallel_vacuum_init(vacrel->rel, vacrel->indrels,
vacrel->nindexes, nworkers,
- max_items,
+ vac_work_mem, MaxHeapTuplesPerPage,
vacrel->verbose ? INFO : DEBUG2,
vacrel->bstrategy);
@@ -3187,11 +3118,8 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
}
/* Serial VACUUM case */
- dead_items = (VacDeadItems *) palloc(vac_max_items_to_alloc_size(max_items));
- dead_items->max_items = max_items;
- dead_items->num_items = 0;
-
- vacrel->dead_items = dead_items;
+ vacrel->dead_items = tidstore_create(vac_work_mem, MaxHeapTuplesPerPage,
+ NULL);
}
/*
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 8608e3fa5b..a526e607fe 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1165,7 +1165,7 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuple_bytes, S.param7 AS dead_tuple_bytes
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index aa79d9de4d..d8e680ca20 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -97,7 +97,6 @@ static bool vacuum_rel(Oid relid, RangeVar *relation, VacuumParams *params,
static double compute_parallel_delay(void);
static VacOptValue get_vacoptval_from_boolean(DefElem *def);
static bool vac_tid_reaped(ItemPointer itemptr, void *state);
-static int vac_cmp_itemptr(const void *left, const void *right);
/*
* Primary entry point for manual VACUUM and ANALYZE commands
@@ -2303,16 +2302,16 @@ get_vacoptval_from_boolean(DefElem *def)
*/
IndexBulkDeleteResult *
vac_bulkdel_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat,
- VacDeadItems *dead_items)
+ TidStore *dead_items)
{
/* Do bulk deletion */
istat = index_bulk_delete(ivinfo, istat, vac_tid_reaped,
(void *) dead_items);
ereport(ivinfo->message_level,
- (errmsg("scanned index \"%s\" to remove %d row versions",
+ (errmsg("scanned index \"%s\" to remove " UINT64_FORMAT " row versions",
RelationGetRelationName(ivinfo->index),
- dead_items->num_items)));
+ tidstore_num_tids(dead_items))));
return istat;
}
@@ -2343,82 +2342,15 @@ vac_cleanup_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat)
return istat;
}
-/*
- * Returns the total required space for VACUUM's dead_items array given a
- * max_items value.
- */
-Size
-vac_max_items_to_alloc_size(int max_items)
-{
- Assert(max_items <= MAXDEADITEMS(MaxAllocSize));
-
- return offsetof(VacDeadItems, items) + sizeof(ItemPointerData) * max_items;
-}
-
/*
* vac_tid_reaped() -- is a particular tid deletable?
*
* This has the right signature to be an IndexBulkDeleteCallback.
- *
- * Assumes dead_items array is sorted (in ascending TID order).
*/
static bool
vac_tid_reaped(ItemPointer itemptr, void *state)
{
- VacDeadItems *dead_items = (VacDeadItems *) state;
- int64 litem,
- ritem,
- item;
- ItemPointer res;
-
- litem = itemptr_encode(&dead_items->items[0]);
- ritem = itemptr_encode(&dead_items->items[dead_items->num_items - 1]);
- item = itemptr_encode(itemptr);
-
- /*
- * Doing a simple bound check before bsearch() is useful to avoid the
- * extra cost of bsearch(), especially if dead items on the heap are
- * concentrated in a certain range. Since this function is called for
- * every index tuple, it pays to be really fast.
- */
- if (item < litem || item > ritem)
- return false;
-
- res = (ItemPointer) bsearch(itemptr,
- dead_items->items,
- dead_items->num_items,
- sizeof(ItemPointerData),
- vac_cmp_itemptr);
-
- return (res != NULL);
-}
-
-/*
- * Comparator routines for use with qsort() and bsearch().
- */
-static int
-vac_cmp_itemptr(const void *left, const void *right)
-{
- BlockNumber lblk,
- rblk;
- OffsetNumber loff,
- roff;
-
- lblk = ItemPointerGetBlockNumber((ItemPointer) left);
- rblk = ItemPointerGetBlockNumber((ItemPointer) right);
-
- if (lblk < rblk)
- return -1;
- if (lblk > rblk)
- return 1;
-
- loff = ItemPointerGetOffsetNumber((ItemPointer) left);
- roff = ItemPointerGetOffsetNumber((ItemPointer) right);
-
- if (loff < roff)
- return -1;
- if (loff > roff)
- return 1;
+ TidStore *dead_items = (TidStore *) state;
- return 0;
+ return tidstore_lookup_tid(dead_items, itemptr);
}
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index bcd40c80a1..d653683693 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -9,12 +9,11 @@
* In a parallel vacuum, we perform both index bulk deletion and index cleanup
* with parallel worker processes. Individual indexes are processed by one
* vacuum process. ParalleVacuumState contains shared information as well as
- * the memory space for storing dead items allocated in the DSM segment. We
- * launch parallel worker processes at the start of parallel index
- * bulk-deletion and index cleanup and once all indexes are processed, the
- * parallel worker processes exit. Each time we process indexes in parallel,
- * the parallel context is re-initialized so that the same DSM can be used for
- * multiple passes of index bulk-deletion and index cleanup.
+ * the shared TidStore. We launch parallel worker processes at the start of
+ * parallel index bulk-deletion and index cleanup and once all indexes are
+ * processed, the parallel worker processes exit. Each time we process indexes
+ * in parallel, the parallel context is re-initialized so that the same DSM can
+ * be used for multiple passes of index bulk-deletion and index cleanup.
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -103,6 +102,9 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /* Handle of the shared TidStore */
+ tidstore_handle dead_items_handle;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -166,7 +168,8 @@ struct ParallelVacuumState
PVIndStats *indstats;
/* Shared dead items space among parallel vacuum workers */
- VacDeadItems *dead_items;
+ TidStore *dead_items;
+ dsa_area *dead_items_area;
/* Points to buffer usage area in DSM */
BufferUsage *buffer_usage;
@@ -222,20 +225,23 @@ static void parallel_vacuum_error_callback(void *arg);
*/
ParallelVacuumState *
parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
- int nrequested_workers, int max_items,
- int elevel, BufferAccessStrategy bstrategy)
+ int nrequested_workers, int vac_work_mem,
+ int max_offset, int elevel,
+ BufferAccessStrategy bstrategy)
{
ParallelVacuumState *pvs;
ParallelContext *pcxt;
PVShared *shared;
- VacDeadItems *dead_items;
+ TidStore *dead_items;
PVIndStats *indstats;
BufferUsage *buffer_usage;
WalUsage *wal_usage;
+ void *area_space;
+ dsa_area *dead_items_dsa;
bool *will_parallel_vacuum;
Size est_indstats_len;
Size est_shared_len;
- Size est_dead_items_len;
+ Size dsa_minsize = dsa_minimum_size();
int nindexes_mwm = 0;
int parallel_workers = 0;
int querylen;
@@ -283,9 +289,8 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_estimate_chunk(&pcxt->estimator, est_shared_len);
shm_toc_estimate_keys(&pcxt->estimator, 1);
- /* Estimate size for dead_items -- PARALLEL_VACUUM_KEY_DEAD_ITEMS */
- est_dead_items_len = vac_max_items_to_alloc_size(max_items);
- shm_toc_estimate_chunk(&pcxt->estimator, est_dead_items_len);
+ /* Estimate size for dead tuple DSA -- PARALLEL_VACUUM_KEY_DSA */
+ shm_toc_estimate_chunk(&pcxt->estimator, dsa_minsize);
shm_toc_estimate_keys(&pcxt->estimator, 1);
/*
@@ -351,6 +356,16 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_INDEX_STATS, indstats);
pvs->indstats = indstats;
+ /* Prepare DSA space for dead items */
+ area_space = shm_toc_allocate(pcxt->toc, dsa_minsize);
+ shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, area_space);
+ dead_items_dsa = dsa_create_in_place(area_space, dsa_minsize,
+ LWTRANCHE_PARALLEL_VACUUM_DSA,
+ pcxt->seg);
+ dead_items = tidstore_create(vac_work_mem, max_offset, dead_items_dsa);
+ pvs->dead_items = dead_items;
+ pvs->dead_items_area = dead_items_dsa;
+
/* Prepare shared information */
shared = (PVShared *) shm_toc_allocate(pcxt->toc, est_shared_len);
MemSet(shared, 0, est_shared_len);
@@ -360,6 +375,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
+ shared->dead_items_handle = tidstore_get_handle(dead_items);
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
@@ -368,15 +384,6 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_SHARED, shared);
pvs->shared = shared;
- /* Prepare the dead_items space */
- dead_items = (VacDeadItems *) shm_toc_allocate(pcxt->toc,
- est_dead_items_len);
- dead_items->max_items = max_items;
- dead_items->num_items = 0;
- MemSet(dead_items->items, 0, sizeof(ItemPointerData) * max_items);
- shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, dead_items);
- pvs->dead_items = dead_items;
-
/*
* Allocate space for each worker's BufferUsage and WalUsage; no need to
* initialize
@@ -434,6 +441,9 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
istats[i] = NULL;
}
+ tidstore_destroy(pvs->dead_items);
+ dsa_detach(pvs->dead_items_area);
+
DestroyParallelContext(pvs->pcxt);
ExitParallelMode();
@@ -442,7 +452,7 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
}
/* Returns the dead items space */
-VacDeadItems *
+TidStore *
parallel_vacuum_get_dead_items(ParallelVacuumState *pvs)
{
return pvs->dead_items;
@@ -940,7 +950,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
Relation *indrels;
PVIndStats *indstats;
PVShared *shared;
- VacDeadItems *dead_items;
+ TidStore *dead_items;
+ void *area_space;
+ dsa_area *dead_items_area;
BufferUsage *buffer_usage;
WalUsage *wal_usage;
int nindexes;
@@ -984,10 +996,10 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
PARALLEL_VACUUM_KEY_INDEX_STATS,
false);
- /* Set dead_items space */
- dead_items = (VacDeadItems *) shm_toc_lookup(toc,
- PARALLEL_VACUUM_KEY_DEAD_ITEMS,
- false);
+ /* Set dead items */
+ area_space = shm_toc_lookup(toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, false);
+ dead_items_area = dsa_attach_in_place(area_space, seg);
+ dead_items = tidstore_attach(dead_items_area, shared->dead_items_handle);
/* Set cost-based vacuum delay */
VacuumCostActive = (VacuumCostDelay > 0);
@@ -1033,6 +1045,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
InstrEndParallelQuery(&buffer_usage[ParallelWorkerNumber],
&wal_usage[ParallelWorkerNumber]);
+ tidstore_detach(pvs.dead_items);
+ dsa_detach(dead_items_area);
+
/* Pop the error context stack */
error_context_stack = errcallback.previous;
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index ff6149a179..a371f6fbba 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -3397,12 +3397,12 @@ check_autovacuum_work_mem(int *newval, void **extra, GucSource source)
return true;
/*
- * We clamp manually-set values to at least 1MB. Since
+ * We clamp manually-set values to at least 2MB. Since
* maintenance_work_mem is always set to at least this value, do the same
* here.
*/
- if (*newval < 1024)
- *newval = 1024;
+ if (*newval < 2048)
+ *newval = 2048;
return true;
}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 55b3a04097..c223a7dc94 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -192,6 +192,8 @@ static const char *const BuiltinTrancheNames[] = {
"LogicalRepLauncherDSA",
/* LWTRANCHE_LAUNCHER_HASH: */
"LogicalRepLauncherHash",
+ /* LWTRANCHE_PARALLEL_VACUUM_DSA: */
+ "ParallelVacuumDSA",
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index b46e3b8c55..27a88b9369 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2312,7 +2312,7 @@ struct config_int ConfigureNamesInt[] =
GUC_UNIT_KB
},
&maintenance_work_mem,
- 65536, 1024, MAX_KILOBYTES,
+ 65536, 2048, MAX_KILOBYTES,
NULL, NULL, NULL
},
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index e5add41352..b209d3cf84 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -23,8 +23,8 @@
#define PROGRESS_VACUUM_HEAP_BLKS_SCANNED 2
#define PROGRESS_VACUUM_HEAP_BLKS_VACUUMED 3
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
-#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
-#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_MAX_DEAD_TUPLE_BYTES 5
+#define PROGRESS_VACUUM_DEAD_TUPLE_BYTES 6
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 689dbb7702..a3ebb169ef 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -17,6 +17,7 @@
#include "access/htup.h"
#include "access/genam.h"
#include "access/parallel.h"
+#include "access/tidstore.h"
#include "catalog/pg_class.h"
#include "catalog/pg_statistic.h"
#include "catalog/pg_type.h"
@@ -276,21 +277,6 @@ struct VacuumCutoffs
MultiXactId MultiXactCutoff;
};
-/*
- * VacDeadItems stores TIDs whose index tuples are deleted by index vacuuming.
- */
-typedef struct VacDeadItems
-{
- int max_items; /* # slots allocated in array */
- int num_items; /* current # of entries */
-
- /* Sorted array of TIDs to delete from indexes */
- ItemPointerData items[FLEXIBLE_ARRAY_MEMBER];
-} VacDeadItems;
-
-#define MAXDEADITEMS(avail_mem) \
- (((avail_mem) - offsetof(VacDeadItems, items)) / sizeof(ItemPointerData))
-
/* GUC parameters */
extern PGDLLIMPORT int default_statistics_target; /* PGDLLIMPORT for PostGIS */
extern PGDLLIMPORT int vacuum_freeze_min_age;
@@ -339,18 +325,17 @@ extern Relation vacuum_open_relation(Oid relid, RangeVar *relation,
LOCKMODE lmode);
extern IndexBulkDeleteResult *vac_bulkdel_one_index(IndexVacuumInfo *ivinfo,
IndexBulkDeleteResult *istat,
- VacDeadItems *dead_items);
+ TidStore *dead_items);
extern IndexBulkDeleteResult *vac_cleanup_one_index(IndexVacuumInfo *ivinfo,
IndexBulkDeleteResult *istat);
-extern Size vac_max_items_to_alloc_size(int max_items);
/* in commands/vacuumparallel.c */
extern ParallelVacuumState *parallel_vacuum_init(Relation rel, Relation *indrels,
int nindexes, int nrequested_workers,
- int max_items, int elevel,
- BufferAccessStrategy bstrategy);
+ int vac_work_mem, int max_offset,
+ int elevel, BufferAccessStrategy bstrategy);
extern void parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats);
-extern VacDeadItems *parallel_vacuum_get_dead_items(ParallelVacuumState *pvs);
+extern TidStore *parallel_vacuum_get_dead_items(ParallelVacuumState *pvs);
extern void parallel_vacuum_bulkdel_all_indexes(ParallelVacuumState *pvs,
long num_table_tuples,
int num_index_scans);
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index 07002fdfbe..537b34b30c 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -207,6 +207,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_PGSTATS_DATA,
LWTRANCHE_LAUNCHER_DSA,
LWTRANCHE_LAUNCHER_HASH,
+ LWTRANCHE_PARALLEL_VACUUM_DSA,
LWTRANCHE_FIRST_USER_DEFINED
} BuiltinTrancheIds;
diff --git a/src/test/regress/expected/cluster.out b/src/test/regress/expected/cluster.out
index 2eec483eaa..e04f50726f 100644
--- a/src/test/regress/expected/cluster.out
+++ b/src/test/regress/expected/cluster.out
@@ -526,7 +526,7 @@ create index cluster_sort on clstr_4 (hundred, thousand, tenthous);
-- ensure we don't use the index in CLUSTER nor the checking SELECTs
set enable_indexscan = off;
-- Use external sort:
-set maintenance_work_mem = '1MB';
+set maintenance_work_mem = '2MB';
cluster clstr_4 using cluster_sort;
select * from
(select hundred, lag(hundred) over () as lhundred,
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 6cd57e3eaa..d1889b9d10 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -1214,7 +1214,7 @@ DROP TABLE unlogged_hash_table;
-- CREATE INDEX hash_ovfl_index ON hash_ovfl_heap USING hash (x int4_ops);
-- Test hash index build tuplesorting. Force hash tuplesort using low
-- maintenance_work_mem setting and fillfactor:
-SET maintenance_work_mem = '1MB';
+SET maintenance_work_mem = '2MB';
CREATE INDEX hash_tuplesort_idx ON tenk1 USING hash (stringu1 name_ops) WITH (fillfactor = 10);
EXPLAIN (COSTS OFF)
SELECT count(*) FROM tenk1 WHERE stringu1 = 'TVAAAA';
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index e7a2f5856a..f6ae02eb14 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2020,8 +2020,8 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param3 AS heap_blks_scanned,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
- s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param6 AS max_dead_tuple_bytes,
+ s.param7 AS dead_tuple_bytes
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_recovery_prefetch| SELECT stats_reset,
diff --git a/src/test/regress/sql/cluster.sql b/src/test/regress/sql/cluster.sql
index a4cfaae807..a4cb5b98a5 100644
--- a/src/test/regress/sql/cluster.sql
+++ b/src/test/regress/sql/cluster.sql
@@ -258,7 +258,7 @@ create index cluster_sort on clstr_4 (hundred, thousand, tenthous);
set enable_indexscan = off;
-- Use external sort:
-set maintenance_work_mem = '1MB';
+set maintenance_work_mem = '2MB';
cluster clstr_4 using cluster_sort;
select * from
(select hundred, lag(hundred) over () as lhundred,
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index a3738833b2..edb5e4b4f3 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -367,7 +367,7 @@ DROP TABLE unlogged_hash_table;
-- Test hash index build tuplesorting. Force hash tuplesort using low
-- maintenance_work_mem setting and fillfactor:
-SET maintenance_work_mem = '1MB';
+SET maintenance_work_mem = '2MB';
CREATE INDEX hash_tuplesort_idx ON tenk1 USING hash (stringu1 name_ops) WITH (fillfactor = 10);
EXPLAIN (COSTS OFF)
SELECT count(*) FROM tenk1 WHERE stringu1 = 'TVAAAA';
--
2.39.1
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-02-09 07:08 ` Masahiko Sawada <[email protected]>
2023-02-09 12:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-10 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
1 sibling, 3 replies; 78+ messages in thread
From: Masahiko Sawada @ 2023-02-09 07:08 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
Hi,
On Tue, Feb 7, 2023 at 6:25 PM John Naylor <[email protected]> wrote:
>
>
> On Tue, Jan 31, 2023 at 9:43 PM Masahiko Sawada <[email protected]> wrote:
>
> > I've attached v24 patches. The locking support patch is separated
> > (0005 patch). Also I kept the updates for TidStore and the vacuum
> > integration from v23 separate.
>
> Okay, that's a lot more simple, and closer to what I imagined. For v25, I squashed v24's additions and added a couple of my own. I've kept the CF status at "needs review" because no specific action is required at the moment.
>
> I did start to review the TID store some more, but that's on hold because something else came up: On a lark I decided to re-run some benchmarks to see if anything got lost in converting to a template, and that led me down a rabbit hole -- some good and bad news on that below.
>
> 0001:
>
> I removed the uint64 case, as discussed. There is now a brief commit message, but needs to be fleshed out a bit. I took another look at the Arm optimization that Nathan found some month ago, for forming the highbit mask, but that doesn't play nicely with how node32 uses it, so I decided against it. I added a comment to describe the reasoning in case someone else gets a similar idea.
>
> I briefly looked into "separate-commit TODO: move non-SIMD fallbacks to their own header to clean up the #ifdef maze.", but decided it wasn't such a clear win to justify starting the work now. It's still in the back of my mind, but I removed the reminder from the commit message.
The changes make sense to me.
>
> 0003:
>
> The template now requires the value to be passed as a pointer. That was a pretty trivial change, but affected multiple other patches, so not sent separately. Also adds a forgotten RT_ prefix to the bitmap macros and adds a top comment to the *_impl.h headers. There are some comment fixes. The changes were either trivial or discussed earlier, so also not sent separately.
Great.
>
> 0004/5: I wanted to measure the load time as well as search time in bench_search_random_nodes(). That's kept separate to make it easier to test other patch versions.
>
> The bad news is that the speed of loading TIDs in bench_seq/shuffle_search() has regressed noticeably. I can't reproduce this in any other bench function and was the reason for writing 0005 to begin with. More confusingly, my efforts to fix this improved *other* functions, but the former didn't budge at all. First the patches:
>
> 0006 adds and removes some "inline" declarations (where it made sense), and added some for "pg_noinline" based on Andres' advice some months ago.
Agreed.
>
> 0007 removes some dead code. RT_NODE_INSERT_INNER is only called during RT_SET_EXTEND, so it can't possibly find an existing key. This kind of change is much easier with the inner/node cases handled together in a template, as far as being sure of how those cases are different. I thought about trying the search in assert builds and verifying it doesn't exist, but thought yet another #ifdef would be too messy.
Agreed.
>
> v25-addendum-try-no-maintain-order.txt -- It makes optional keeping the key chunks in order for the linear-search nodes. I believe the TID store no longer cares about the ordering, but this is a text file for now because I don't want to clutter the CI with a behavior change. Also, the second ART paper (on concurrency) mentioned that some locking schemes don't allow these arrays to be shifted. So it might make sense to give up entirely on guaranteeing ordered iteration, or at least make it optional as in the patch.
I think it's still important for lazy vacuum that an iteration over a
TID store returns TIDs in ascending order, because otherwise a heap
vacuum does random writes. That being said, we can have
RT_ITERATE_NEXT() return key-value pairs in an order regardless of how
the key chunks are stored in a node.
> ========================================
> psql -c "select rt_load_ms, rt_search_ms from bench_seq_search(0, 1 * 1000 * 1000)"
> (min load time of three)
>
> v15:
> rt_load_ms | rt_search_ms
> ------------+--------------
> 113 | 455
>
> v25-0005:
> rt_load_ms | rt_search_ms
> ------------+--------------
> 135 | 456
>
> v25-0006 (inlining or not):
> rt_load_ms | rt_search_ms
> ------------+--------------
> 136 | 455
>
> v25-0007 (remove dead code):
> rt_load_ms | rt_search_ms
> ------------+--------------
> 135 | 455
>
> v25-addendum...txt (no ordering):
> rt_load_ms | rt_search_ms
> ------------+--------------
> 134 | 455
>
> Note: The regression seems to have started in v17, which is the first with a full template.
>
> Nothing so far has helped here, and previous experience has shown that trying to profile 100ms will not be useful. Instead of putting more effort into diving deeper, it seems a better use of time to write a benchmark that calls the tid store itself. That's more realistic, since this function was intended to test load and search of tids, but the tid store doesn't quite operate so simply anymore. What do you think, Masahiko?
Yeah, that's more realistic. TidStore now encodes TIDs slightly
differently from the benchmark test.
I've attached the patch that adds a simple benchmark test using
TidStore. With this test, I got similar trends of results to yours
with gcc, but I've not analyzed them in depth yet.
query: select * from bench_tidstore_load(0, 10 * 1000 * 1000)
v15:
load_ms
---------
816
v25-0007 (remove dead code):
load_ms
---------
839
v25-addendum...txt (no ordering):
load_ms
---------
820
BTW it would be better to remove the RT_DEBUG macro from bench_radix_tree.c.
>
> I'm inclined to keep 0006, because it might give a slight boost, and 0007 because it's never a bad idea to remove dead code.
Yeah, these two changes make sense to me too.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
From e056133360436e115a434a8a21685a99602a5b5d Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 8 Feb 2023 15:53:14 +0900
Subject: [PATCH] Add bench_tidstore_load()
---
.../bench_radix_tree--1.0.sql | 10 ++++
contrib/bench_radix_tree/bench_radix_tree.c | 46 +++++++++++++++++++
2 files changed, 56 insertions(+)
diff --git a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
index 95eedbbe10..fbf51c1086 100644
--- a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
+++ b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
@@ -75,3 +75,13 @@ OUT rt_sparseload_ms int8
returns record
as 'MODULE_PATHNAME'
LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_tidstore_load(
+minblk int4,
+maxblk int4,
+OUT mem_allocated int8,
+OUT load_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
index 7d1e2eee57..3c2caa3b90 100644
--- a/contrib/bench_radix_tree/bench_radix_tree.c
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -9,6 +9,7 @@
*/
#include "postgres.h"
+#include "access/tidstore.h"
#include "common/pg_prng.h"
#include "fmgr.h"
#include "funcapi.h"
@@ -54,6 +55,7 @@ PG_FUNCTION_INFO_V1(bench_load_random_int);
PG_FUNCTION_INFO_V1(bench_fixed_height_search);
PG_FUNCTION_INFO_V1(bench_search_random_nodes);
PG_FUNCTION_INFO_V1(bench_node128_load);
+PG_FUNCTION_INFO_V1(bench_tidstore_load);
static uint64
tid_to_key_off(ItemPointer tid, uint32 *off)
@@ -168,6 +170,50 @@ vac_cmp_itemptr(const void *left, const void *right)
}
#endif
+Datum
+bench_tidstore_load(PG_FUNCTION_ARGS)
+{
+ BlockNumber minblk = PG_GETARG_INT32(0);
+ BlockNumber maxblk = PG_GETARG_INT32(1);
+ TidStore *ts;
+ OffsetNumber *offs;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_ms;
+ TupleDesc tupdesc;
+ Datum values[2];
+ bool nulls[2] = {false};
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ offs = palloc(sizeof(OffsetNumber) * TIDS_PER_BLOCK_FOR_LOAD);
+ for (int i = 0; i < TIDS_PER_BLOCK_FOR_LOAD; i++)
+ offs[i] = i + 1; /* FirstOffsetNumber is 1 */
+
+ ts = tidstore_create(1 * 1024L * 1024L * 1024L, MaxHeapTuplesPerPage, NULL);
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* load tids */
+ start_time = GetCurrentTimestamp();
+ for (BlockNumber blkno = minblk; blkno < maxblk; blkno++)
+ tidstore_add_tids(ts, blkno, offs, TIDS_PER_BLOCK_FOR_LOAD);
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_ms = secs * 1000 + usecs / 1000;
+
+ values[0] = Int64GetDatum(tidstore_memory_usage(ts));
+ values[1] = Int64GetDatum(load_ms);
+
+ tidstore_destroy(ts);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
static Datum
bench_search(FunctionCallInfo fcinfo, bool shuffle)
{
--
2.31.1
Attachments:
[text/plain] 0001-Add-bench_tidstore_load.patch.txt (3.1K, ../../CAD21AoDePt8x14Vrg+xLFd2akRAKHS_8j2uVmyoRVMYOze-VSw@mail.gmail.com/2-0001-Add-bench_tidstore_load.patch.txt)
download | inline diff:
From e056133360436e115a434a8a21685a99602a5b5d Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 8 Feb 2023 15:53:14 +0900
Subject: [PATCH] Add bench_tidstore_load()
---
.../bench_radix_tree--1.0.sql | 10 ++++
contrib/bench_radix_tree/bench_radix_tree.c | 46 +++++++++++++++++++
2 files changed, 56 insertions(+)
diff --git a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
index 95eedbbe10..fbf51c1086 100644
--- a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
+++ b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
@@ -75,3 +75,13 @@ OUT rt_sparseload_ms int8
returns record
as 'MODULE_PATHNAME'
LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_tidstore_load(
+minblk int4,
+maxblk int4,
+OUT mem_allocated int8,
+OUT load_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
index 7d1e2eee57..3c2caa3b90 100644
--- a/contrib/bench_radix_tree/bench_radix_tree.c
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -9,6 +9,7 @@
*/
#include "postgres.h"
+#include "access/tidstore.h"
#include "common/pg_prng.h"
#include "fmgr.h"
#include "funcapi.h"
@@ -54,6 +55,7 @@ PG_FUNCTION_INFO_V1(bench_load_random_int);
PG_FUNCTION_INFO_V1(bench_fixed_height_search);
PG_FUNCTION_INFO_V1(bench_search_random_nodes);
PG_FUNCTION_INFO_V1(bench_node128_load);
+PG_FUNCTION_INFO_V1(bench_tidstore_load);
static uint64
tid_to_key_off(ItemPointer tid, uint32 *off)
@@ -168,6 +170,50 @@ vac_cmp_itemptr(const void *left, const void *right)
}
#endif
+Datum
+bench_tidstore_load(PG_FUNCTION_ARGS)
+{
+ BlockNumber minblk = PG_GETARG_INT32(0);
+ BlockNumber maxblk = PG_GETARG_INT32(1);
+ TidStore *ts;
+ OffsetNumber *offs;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_ms;
+ TupleDesc tupdesc;
+ Datum values[2];
+ bool nulls[2] = {false};
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ offs = palloc(sizeof(OffsetNumber) * TIDS_PER_BLOCK_FOR_LOAD);
+ for (int i = 0; i < TIDS_PER_BLOCK_FOR_LOAD; i++)
+ offs[i] = i + 1; /* FirstOffsetNumber is 1 */
+
+ ts = tidstore_create(1 * 1024L * 1024L * 1024L, MaxHeapTuplesPerPage, NULL);
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* load tids */
+ start_time = GetCurrentTimestamp();
+ for (BlockNumber blkno = minblk; blkno < maxblk; blkno++)
+ tidstore_add_tids(ts, blkno, offs, TIDS_PER_BLOCK_FOR_LOAD);
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_ms = secs * 1000 + usecs / 1000;
+
+ values[0] = Int64GetDatum(tidstore_memory_usage(ts));
+ values[1] = Int64GetDatum(load_ms);
+
+ tidstore_destroy(ts);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
static Datum
bench_search(FunctionCallInfo fcinfo, bool shuffle)
{
--
2.31.1
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-02-09 12:56 ` John Naylor <[email protected]>
2 siblings, 0 replies; 78+ messages in thread
From: John Naylor @ 2023-02-09 12:56 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Thu, Feb 9, 2023 at 2:08 PM Masahiko Sawada <[email protected]>
wrote:
> I think it's still important for lazy vacuum that an iteration over a
> TID store returns TIDs in ascending order, because otherwise a heap
> vacuum does random writes. That being said, we can have
> RT_ITERATE_NEXT() return key-value pairs in an order regardless of how
> the key chunks are stored in a node.
Okay, we can keep that possibility in mind if we need to go there.
> > Note: The regression seems to have started in v17, which is the first
with a full template.
> > 0007 removes some dead code. RT_NODE_INSERT_INNER is only called during
RT_SET_EXTEND, so it can't possibly find an existing key. This kind of
change is much easier with the inner/node cases handled together in a
template, as far as being sure of how those cases are different. I thought
about trying the search in assert builds and verifying it doesn't exist,
but thought yet another #ifdef would be too messy.
It just occurred to me that these facts might be related. v17 was the first
use of the full template, and I decided then I liked one of your earlier
patches where replace_node() calls node_update_inner() better than calling
node_insert_inner() with a NULL parent, which was a bit hard to understand.
That now-dead code was actually used in the latter case for updating the
(original) parent. It's possible that trying to use separate paths
contributed to the regression. I'll try the other way and report back.
> I've attached the patch that adds a simple benchmark test using
> TidStore. With this test, I got similar trends of results to yours
> with gcc, but I've not analyzed them in depth yet.
Thanks for that! I'll take a look.
> BTW it would be better to remove the RT_DEBUG macro from
bench_radix_tree.c.
Absolutely.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-02-10 06:51 ` John Naylor <[email protected]>
2023-02-10 07:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2 siblings, 1 reply; 78+ messages in thread
From: John Naylor @ 2023-02-10 06:51 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Thu, Feb 9, 2023 at 2:08 PM Masahiko Sawada <[email protected]>
wrote:
>
> query: select * from bench_tidstore_load(0, 10 * 1000 * 1000)
>
> v15:
> load_ms
> ---------
> 816
How did you build the tid store and test on v15? I first tried to
apply v15-0009-PoC-lazy-vacuum-integration.patch, which conflicts with
vacuum now, so reset all that, but still getting build errors because the
tid store types and functions have changed.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-10 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-02-10 07:15 ` Masahiko Sawada <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Masahiko Sawada @ 2023-02-10 07:15 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Fri, Feb 10, 2023 at 3:51 PM John Naylor
<[email protected]> wrote:
>
>
> On Thu, Feb 9, 2023 at 2:08 PM Masahiko Sawada <[email protected]> wrote:
> >
> > query: select * from bench_tidstore_load(0, 10 * 1000 * 1000)
> >
> > v15:
> > load_ms
> > ---------
> > 816
>
> How did you build the tid store and test on v15? I first tried to apply v15-0009-PoC-lazy-vacuum-integration.patch, which conflicts with vacuum now, so reset all that, but still getting build errors because the tid store types and functions have changed.
I applied v26-0008-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch
on top of v15 radix tree and changed the TidStore so that it uses v15
(non-templated) radixtree. That way, we can test TidStore using v15
radix tree. I've attached the patch that I applied on top of
v26-0008-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] change_tidstore_for_v15.patch (10.7K, ../../CAD21AoC_XQFZTTfE63Yi7AwcSucEHJZX7+Mrg8SicrJbrDouZA@mail.gmail.com/2-change_tidstore_for_v15.patch)
download | inline diff:
commit f2d6acbce26d7e05e64666ae00fca030a657de76
Author: Masahiko Sawada <[email protected]>
Date: Wed Feb 8 15:52:47 2023 +0900
Add TidStore from v26 patch.
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 4c72673ce9..5048400a9f 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -29,6 +29,7 @@
#include "access/tidstore.h"
#include "miscadmin.h"
#include "port/pg_bitutils.h"
+#include "lib/radixtree.h"
#include "storage/lwlock.h"
#include "utils/dsa.h"
#include "utils/memutils.h"
@@ -74,21 +75,6 @@
/* A magic value used to identify our TidStores. */
#define TIDSTORE_MAGIC 0x826f6a10
-#define RT_PREFIX local_rt
-#define RT_SCOPE static
-#define RT_DECLARE
-#define RT_DEFINE
-#define RT_VALUE_TYPE uint64
-#include "lib/radixtree.h"
-
-#define RT_PREFIX shared_rt
-#define RT_SHMEM
-#define RT_SCOPE static
-#define RT_DECLARE
-#define RT_DEFINE
-#define RT_VALUE_TYPE uint64
-#include "lib/radixtree.h"
-
/* The control object for a TidStore */
typedef struct TidStoreControl
{
@@ -110,7 +96,6 @@ typedef struct TidStoreControl
/* handles for TidStore and radix tree */
tidstore_handle handle;
- shared_rt_handle tree_handle;
} TidStoreControl;
/* Per-backend state for a TidStore */
@@ -125,14 +110,9 @@ struct TidStore
/* Storage for Tids. Use either one depending on TidStoreIsShared() */
union
{
- local_rt_radix_tree *local;
- shared_rt_radix_tree *shared;
+ radix_tree *local;
} tree;
-
- /* DSA area for TidStore if used */
- dsa_area *area;
};
-#define TidStoreIsShared(ts) ((ts)->area != NULL)
/* Iterator for TidStore */
typedef struct TidStoreIter
@@ -142,8 +122,8 @@ typedef struct TidStoreIter
/* iterator of radix tree. Use either one depending on TidStoreIsShared() */
union
{
- shared_rt_iter *shared;
- local_rt_iter *local;
+ rt_iter *shared;
+ rt_iter *local;
} tree_iter;
/* we returned all tids? */
@@ -194,31 +174,10 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
* perfectly works in case where the max_bytes is a power-of-2, and the 60%
* threshold works for other cases.
*/
- if (area != NULL)
- {
- dsa_pointer dp;
- float ratio = ((max_bytes & (max_bytes - 1)) == 0) ? 0.75 : 0.6;
-
- ts->tree.shared = shared_rt_create(CurrentMemoryContext, area,
- LWTRANCHE_SHARED_TIDSTORE);
-
- dp = dsa_allocate0(area, sizeof(TidStoreControl));
- ts->control = (TidStoreControl *) dsa_get_address(area, dp);
- ts->control->max_bytes = (uint64) (max_bytes * ratio);
- ts->area = area;
+ ts->tree.local = rt_create(CurrentMemoryContext);
- ts->control->magic = TIDSTORE_MAGIC;
- LWLockInitialize(&ts->control->lock, LWTRANCHE_SHARED_TIDSTORE);
- ts->control->handle = dp;
- ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
- }
- else
- {
- ts->tree.local = local_rt_create(CurrentMemoryContext);
-
- ts->control = (TidStoreControl *) palloc0(sizeof(TidStoreControl));
- ts->control->max_bytes = max_bytes - (70 * 1024);
- }
+ ts->control = (TidStoreControl *) palloc0(sizeof(TidStoreControl));
+ ts->control->max_bytes = max_bytes - (70 * 1024);
ts->control->max_offset = max_offset;
ts->control->offset_nbits = pg_ceil_log2_32(max_offset);
@@ -242,50 +201,6 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
return ts;
}
-/*
- * Attach to the shared TidStore using a handle. The returned object is
- * allocated in backend-local memory using the CurrentMemoryContext.
- */
-TidStore *
-tidstore_attach(dsa_area *area, tidstore_handle handle)
-{
- TidStore *ts;
- dsa_pointer control;
-
- Assert(area != NULL);
- Assert(DsaPointerIsValid(handle));
-
- /* create per-backend state */
- ts = palloc0(sizeof(TidStore));
-
- /* Find the control object in shared memory */
- control = handle;
-
- /* Set up the TidStore */
- ts->control = (TidStoreControl *) dsa_get_address(area, control);
- Assert(ts->control->magic == TIDSTORE_MAGIC);
-
- ts->tree.shared = shared_rt_attach(area, ts->control->tree_handle);
- ts->area = area;
-
- return ts;
-}
-
-/*
- * Detach from a TidStore. This detaches from radix tree and frees the
- * backend-local resources. The radix tree will continue to exist until
- * it is either explicitly destroyed, or the area that backs it is returned
- * to the operating system.
- */
-void
-tidstore_detach(TidStore *ts)
-{
- Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
-
- shared_rt_detach(ts->tree.shared);
- pfree(ts);
-}
-
/*
* Destroy a TidStore, returning all memory.
*
@@ -298,25 +213,8 @@ tidstore_detach(TidStore *ts)
void
tidstore_destroy(TidStore *ts)
{
- if (TidStoreIsShared(ts))
- {
- Assert(ts->control->magic == TIDSTORE_MAGIC);
-
- /*
- * Vandalize the control block to help catch programming error where
- * other backends access the memory formerly occupied by this radix
- * tree.
- */
- ts->control->magic = 0;
- dsa_free(ts->area, ts->control->handle);
- shared_rt_free(ts->tree.shared);
- }
- else
- {
- pfree(ts->control);
- local_rt_free(ts->tree.local);
- }
-
+ pfree(ts->control);
+ rt_free(ts->tree.local);
pfree(ts);
}
@@ -327,39 +225,11 @@ tidstore_destroy(TidStore *ts)
void
tidstore_reset(TidStore *ts)
{
- if (TidStoreIsShared(ts))
- {
- Assert(ts->control->magic == TIDSTORE_MAGIC);
-
- LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
-
- /*
- * Free the radix tree and return allocated DSA segments to
- * the operating system.
- */
- shared_rt_free(ts->tree.shared);
- dsa_trim(ts->area);
+ rt_free(ts->tree.local);
+ ts->tree.local = rt_create(CurrentMemoryContext);
- /* Recreate the radix tree */
- ts->tree.shared = shared_rt_create(CurrentMemoryContext, ts->area,
- LWTRANCHE_SHARED_TIDSTORE);
-
- /* update the radix tree handle as we recreated it */
- ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
-
- /* Reset the statistics */
- ts->control->num_tids = 0;
-
- LWLockRelease(&ts->control->lock);
- }
- else
- {
- local_rt_free(ts->tree.local);
- ts->tree.local = local_rt_create(CurrentMemoryContext);
-
- /* Reset the statistics */
- ts->control->num_tids = 0;
- }
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
}
/* Add Tids on a block to TidStore */
@@ -372,8 +242,6 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
uint64 *values;
int nkeys;
- Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
-
if (ts->control->encode_tids)
{
key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
@@ -404,9 +272,6 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
values[idx] |= UINT64CONST(1) << off;
}
- if (TidStoreIsShared(ts))
- LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
-
/* insert the calculated key-values to the tree */
for (int i = 0; i < nkeys; i++)
{
@@ -414,19 +279,13 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
{
uint64 key = key_base + i;
- if (TidStoreIsShared(ts))
- shared_rt_set(ts->tree.shared, key, &values[i]);
- else
- local_rt_set(ts->tree.local, key, &values[i]);
+ rt_set(ts->tree.local, key, values[i]);
}
}
/* update statistics */
ts->control->num_tids += num_offsets;
- if (TidStoreIsShared(ts))
- LWLockRelease(&ts->control->lock);
-
pfree(values);
}
@@ -441,10 +300,7 @@ tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
key = tid_to_key_off(ts, tid, &off);
- if (TidStoreIsShared(ts))
- found = shared_rt_search(ts->tree.shared, key, &val);
- else
- found = local_rt_search(ts->tree.local, key, &val);
+ found = rt_search(ts->tree.local, key, &val);
if (!found)
return false;
@@ -464,18 +320,13 @@ tidstore_begin_iterate(TidStore *ts)
{
TidStoreIter *iter;
- Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
-
iter = palloc0(sizeof(TidStoreIter));
iter->ts = ts;
iter->result.blkno = InvalidBlockNumber;
iter->result.offsets = palloc(sizeof(OffsetNumber) * ts->control->max_offset);
- if (TidStoreIsShared(ts))
- iter->tree_iter.shared = shared_rt_begin_iterate(ts->tree.shared);
- else
- iter->tree_iter.local = local_rt_begin_iterate(ts->tree.local);
+ iter->tree_iter.local = rt_begin_iterate(ts->tree.local);
/* If the TidStore is empty, there is no business */
if (tidstore_num_tids(ts) == 0)
@@ -487,10 +338,7 @@ tidstore_begin_iterate(TidStore *ts)
static inline bool
tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
{
- if (TidStoreIsShared(iter->ts))
- return shared_rt_iterate_next(iter->tree_iter.shared, key, val);
-
- return local_rt_iterate_next(iter->tree_iter.local, key, val);
+ return rt_iterate_next(iter->tree_iter.local, key, val);
}
/*
@@ -547,10 +395,7 @@ tidstore_iterate_next(TidStoreIter *iter)
void
tidstore_end_iterate(TidStoreIter *iter)
{
- if (TidStoreIsShared(iter->ts))
- shared_rt_end_iterate(iter->tree_iter.shared);
- else
- local_rt_end_iterate(iter->tree_iter.local);
+ rt_end_iterate(iter->tree_iter.local);
pfree(iter->result.offsets);
pfree(iter);
@@ -560,26 +405,13 @@ tidstore_end_iterate(TidStoreIter *iter)
int64
tidstore_num_tids(TidStore *ts)
{
- uint64 num_tids;
-
- Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
-
- if (!TidStoreIsShared(ts))
- return ts->control->num_tids;
-
- LWLockAcquire(&ts->control->lock, LW_SHARED);
- num_tids = ts->control->num_tids;
- LWLockRelease(&ts->control->lock);
-
- return num_tids;
+ return ts->control->num_tids;
}
/* Return true if the current memory usage of TidStore exceeds the limit */
bool
tidstore_is_full(TidStore *ts)
{
- Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
-
return (tidstore_memory_usage(ts) > ts->control->max_bytes);
}
@@ -587,8 +419,6 @@ tidstore_is_full(TidStore *ts)
size_t
tidstore_max_memory(TidStore *ts)
{
- Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
-
return ts->control->max_bytes;
}
@@ -596,17 +426,7 @@ tidstore_max_memory(TidStore *ts)
size_t
tidstore_memory_usage(TidStore *ts)
{
- Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
-
- /*
- * In the shared case, TidStoreControl and radix_tree are backed by the
- * same DSA area and rt_memory_usage() returns the value including both.
- * So we don't need to add the size of TidStoreControl separately.
- */
- if (TidStoreIsShared(ts))
- return sizeof(TidStore) + shared_rt_memory_usage(ts->tree.shared);
-
- return sizeof(TidStore) + sizeof(TidStore) + local_rt_memory_usage(ts->tree.local);
+ return sizeof(TidStore) + sizeof(TidStore) + rt_memory_usage(ts->tree.local);
}
/*
@@ -615,7 +435,6 @@ tidstore_memory_usage(TidStore *ts)
tidstore_handle
tidstore_get_handle(TidStore *ts)
{
- Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
return ts->control->handle;
}
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-02-11 05:33 ` John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2 siblings, 1 reply; 78+ messages in thread
From: John Naylor @ 2023-02-11 05:33 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
I didn't get any closer to radix-tree regression, but I did find some
inefficiencies in tidstore_add_tids() that are worth talking about first,
addressed in a rough fashion in the attached .txt addendums that I can
clean up and incorporate later.
To start, I can reproduce the regression with this test as well:
select * from bench_tidstore_load(0, 10 * 1000 * 1000);
v15 + v26 store + adjustments:
mem_allocated | load_ms
---------------+---------
98202152 | 1676
v26 0001-0008
mem_allocated | load_ms
---------------+---------
98202032 | 1826
...and reverting to the alternate way to update the parent didn't help:
v26 0001-6, 0008, insert_inner w/ null parent
mem_allocated | load_ms
---------------+---------
98202032 | 1825
...and I'm kind of glad that wasn't the problem, because going back to that
would be a pain for the shmem case.
Running perf doesn't show anything much different in the proportions (note
that rt_set must have been inlined when declared locally in v26):
v15 + v26 store + adjustments:
65.88% postgres postgres [.] tidstore_add_tids
10.74% postgres postgres [.] rt_set
9.20% postgres postgres [.] palloc0
6.49% postgres postgres [.] rt_node_insert_leaf
v26 0001-0008
78.50% postgres postgres [.] tidstore_add_tids
8.88% postgres postgres [.] palloc0
6.24% postgres postgres [.] local_rt_node_insert_leaf
v2699-0001: The first thing I noticed is that palloc0 is taking way more
time than it should, and it's because the compiler doesn't know the
values[] array is small. One reason we need to zero the array is to make
the algorithm agnostic about what order the offsets come in, as I requested
in a previous review. Thinking some more, I was way too paranoid about
that. As long as access methods scan the line pointer array in the usual
way, maybe we can just assert that the keys we create are in order, and
zero any unused array entries as we find them. (I admit I can't actually
think of a reason we would ever encounter offsets out of order.) Also, we
can keep track of the last key we need to consider for insertion into the
radix tree, and ignore the rest. That might shave a few cycles during the
exclusive lock when the max offset of an LP_DEAD item < 64 on a given page,
which I think would be common in the wild. I also got rid of the special
case for non-encoding, since shifting by zero should work the same way.
These together led to a nice speedup on the v26 branch:
mem_allocated | load_ms
---------------+---------
98202032 | 1386
v2699-0002: The next thing I noticed is forming a full ItemIdPointer to
pass to tid_to_key_off(). That's bad for tidstore_add_tids() because
ItemPointerSetBlockNumber() must do this in order to allow the struct to be
SHORTALIGN'd:
static inline void
BlockIdSet(BlockIdData *blockId, BlockNumber blockNumber)
{
blockId->bi_hi = blockNumber >> 16;
blockId->bi_lo = blockNumber & 0xffff;
}
Then, tid_to_key_off() calls ItemPointerGetBlockNumber(), which must
reverse the above process:
static inline BlockNumber
BlockIdGetBlockNumber(const BlockIdData *blockId)
{
return (((BlockNumber) blockId->bi_hi) << 16) | ((BlockNumber)
blockId->bi_lo);
}
There is no reason to do any of this if we're not reading/writing directly
to/from an on-disk tid etc. To avoid this, I created a new function
encode_key_off() [name could be better], which deals with the raw block
number that we already have. Then turn tid_to_key_off() into a wrapper
around that, since we still need the full conversion for
tidstore_lookup_tid().
v2699-0003: Get rid of all the remaining special cases for encoding/or not.
I am unaware of the need to optimize that case or treat it in any way
differently. I haven't tested this on an installation with non-default
blocksize and didn't measure this separately, but 0002+0003 gives:
mem_allocated | load_ms
---------------+---------
98202032 | 1259
If these are acceptable, I can incorporate them into a later patchset. In
any case, speeding up tidstore_add_tids() will make any regressions in the
backing radix tree more obvious. I will take a look at that next week.
--
John Naylor
EDB: http://www.enterprisedb.com
From 6bdd33fa4f55757b54d16ce00dc60a21b929606e Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 11 Feb 2023 10:45:21 +0700
Subject: [PATCH v2699 2/3] Do less work when encoding key/value
---
src/backend/access/common/tidstore.c | 25 +++++++++++++++----------
1 file changed, 15 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 5d24680737..3d384cf645 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -159,6 +159,7 @@ typedef struct TidStoreIter
static void tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val);
static inline BlockNumber key_get_blkno(TidStore *ts, uint64 key);
+static inline uint64 encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint32 *off);
static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off);
/*
@@ -367,7 +368,6 @@ void
tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
int num_offsets)
{
- ItemPointerData tid;
uint64 *values;
uint64 key;
uint64 prev_key;
@@ -381,16 +381,12 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
values = palloc(sizeof(uint64) * nkeys);
key = prev_key = key_base;
- ItemPointerSetBlockNumber(&tid, blkno);
-
for (int i = 0; i < num_offsets; i++)
{
uint32 off;
- ItemPointerSetOffsetNumber(&tid, offsets[i]);
-
/* encode the tid to key and val */
- key = tid_to_key_off(ts, &tid, &off);
+ key = encode_key_off(ts, blkno, offsets[i], &off);
/* make sure we scanned the line pointer array in order */
Assert(key >= prev_key);
@@ -681,20 +677,29 @@ key_get_blkno(TidStore *ts, uint64 key)
/* Encode a tid to key and offset */
static inline uint64
tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off)
+{
+ uint32 offset = ItemPointerGetOffsetNumber(tid);
+ BlockNumber block = ItemPointerGetBlockNumber(tid);
+
+ return encode_key_off(ts, block, offset, off);
+}
+
+/* encode a block and offset to a key and partial offset */
+static inline uint64
+encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint32 *off)
{
uint64 key;
uint64 tid_i;
if (!ts->control->encode_tids)
{
- *off = ItemPointerGetOffsetNumber(tid);
+ *off = offset;
/* Use the block number as the key */
- return (int64) ItemPointerGetBlockNumber(tid);
+ return (int64) block;
}
- tid_i = ItemPointerGetOffsetNumber(tid);
- tid_i |= (uint64) ItemPointerGetBlockNumber(tid) << ts->control->offset_nbits;
+ tid_i = offset | ((uint64) block << ts->control->offset_nbits);
*off = tid_i & ((UINT64CONST(1) << TIDSTORE_VALUE_NBITS) - 1);
key = tid_i >> TIDSTORE_VALUE_NBITS;
--
2.39.1
From c0bc497f50318c8e31ccdf0c2a9186ffc736abeb Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Fri, 10 Feb 2023 19:56:01 +0700
Subject: [PATCH v2699 1/3] Miscellaneous optimizations for tidstore_add_tids()
- remove palloc0; it's expensive for lengths not known at compile-time
- optimize for case with only one key per heap block
- make some intializations const and branch-free
- when writing to the radix tree, stop at the last non-zero bitmap
---
src/backend/access/common/tidstore.c | 56 ++++++++++++++++++----------
1 file changed, 36 insertions(+), 20 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 4c72673ce9..5d24680737 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -368,51 +368,67 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
int num_offsets)
{
ItemPointerData tid;
- uint64 key_base;
uint64 *values;
- int nkeys;
+ uint64 key;
+ uint64 prev_key;
+ uint64 off_bitmap = 0;
+ int idx;
+ const uint64 key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
+ const int nkeys = UINT64CONST(1) << ts->control->offset_key_nbits;
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
- if (ts->control->encode_tids)
- {
- key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
- nkeys = UINT64CONST(1) << ts->control->offset_key_nbits;
- }
- else
- {
- key_base = (uint64) blkno;
- nkeys = 1;
- }
- values = palloc0(sizeof(uint64) * nkeys);
+ values = palloc(sizeof(uint64) * nkeys);
+ key = prev_key = key_base;
ItemPointerSetBlockNumber(&tid, blkno);
+
for (int i = 0; i < num_offsets; i++)
{
- uint64 key;
uint32 off;
- int idx;
ItemPointerSetOffsetNumber(&tid, offsets[i]);
/* encode the tid to key and val */
key = tid_to_key_off(ts, &tid, &off);
- idx = key - key_base;
- Assert(idx >= 0 && idx < nkeys);
+ /* make sure we scanned the line pointer array in order */
+ Assert(key >= prev_key);
- values[idx] |= UINT64CONST(1) << off;
+ if (key > prev_key)
+ {
+ idx = prev_key - key_base;
+ Assert(idx >= 0 && idx < nkeys);
+
+ /* write out offset bitmap for this key */
+ values[idx] = off_bitmap;
+
+ /* zero out any gaps up to the current key */
+ for (int empty_idx = idx + 1; empty_idx < key - key_base; empty_idx++)
+ values[empty_idx] = 0;
+
+ /* reset for current key -- the current offset will be handled below */
+ off_bitmap = 0;
+ prev_key = key;
+ }
+
+ off_bitmap |= UINT64CONST(1) << off;
}
+ /* save the final index for later */
+ idx = key - key_base;
+ /* write out last offset bitmap */
+ values[idx] = off_bitmap;
+
if (TidStoreIsShared(ts))
LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
/* insert the calculated key-values to the tree */
- for (int i = 0; i < nkeys; i++)
+ for (int i = 0; i <= idx; i++)
{
if (values[i])
{
- uint64 key = key_base + i;
+ key = key_base + i;
if (TidStoreIsShared(ts))
shared_rt_set(ts->tree.shared, key, &values[i]);
--
2.39.1
From 82c1f639aaa64cc943af3b53294a63d5d8f7a9b9 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 11 Feb 2023 11:51:32 +0700
Subject: [PATCH v2699 3/3] Force all callers to encode, no matter how small
the expected offset
---
src/backend/access/common/tidstore.c | 36 +++++-----------------------
1 file changed, 6 insertions(+), 30 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 3d384cf645..ff8e66936e 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -99,7 +99,6 @@ typedef struct TidStoreControl
size_t max_bytes; /* the maximum bytes a TidStore can use */
int max_offset; /* the maximum offset number */
int offset_nbits; /* the number of bits required for max_offset */
- bool encode_tids; /* do we use tid encoding? */
int offset_key_nbits; /* the number of bits of a offset number
* used for the key */
@@ -224,21 +223,15 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
ts->control->max_offset = max_offset;
ts->control->offset_nbits = pg_ceil_log2_32(max_offset);
+ if (ts->control->offset_nbits < TIDSTORE_VALUE_NBITS)
+ ts->control->offset_nbits = TIDSTORE_VALUE_NBITS;
+
/*
* We use tid encoding if the number of bits for the offset number doesn't
* fix in a value, uint64.
*/
- if (ts->control->offset_nbits > TIDSTORE_VALUE_NBITS)
- {
- ts->control->encode_tids = true;
- ts->control->offset_key_nbits =
- ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
- }
- else
- {
- ts->control->encode_tids = false;
- ts->control->offset_key_nbits = 0;
- }
+ ts->control->offset_key_nbits =
+ ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
return ts;
}
@@ -643,12 +636,6 @@ tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
uint64 tid_i;
OffsetNumber off;
- if (i > iter->ts->control->max_offset)
- {
- Assert(!iter->ts->control->encode_tids);
- break;
- }
-
if ((val & (UINT64CONST(1) << i)) == 0)
continue;
@@ -668,10 +655,7 @@ tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
static inline BlockNumber
key_get_blkno(TidStore *ts, uint64 key)
{
- if (ts->control->encode_tids)
- return (BlockNumber) (key >> ts->control->offset_key_nbits);
-
- return (BlockNumber) key;
+ return (BlockNumber) (key >> ts->control->offset_key_nbits);
}
/* Encode a tid to key and offset */
@@ -691,14 +675,6 @@ encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint32 *off)
uint64 key;
uint64 tid_i;
- if (!ts->control->encode_tids)
- {
- *off = offset;
-
- /* Use the block number as the key */
- return (int64) block;
- }
-
tid_i = offset | ((uint64) block << ts->control->offset_nbits);
*off = tid_i & ((UINT64CONST(1) << TIDSTORE_VALUE_NBITS) - 1);
--
2.39.1
Attachments:
[text/plain] v2699-0002-Do-less-work-when-encoding-key-value.patch.txt (2.7K, ../../CAFBsxsFps1MReb3HQYEGc=M=1NXRjSu3TSaEVEXkVrCR3g-hxg@mail.gmail.com/3-v2699-0002-Do-less-work-when-encoding-key-value.patch.txt)
download | inline diff:
From 6bdd33fa4f55757b54d16ce00dc60a21b929606e Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 11 Feb 2023 10:45:21 +0700
Subject: [PATCH v2699 2/3] Do less work when encoding key/value
---
src/backend/access/common/tidstore.c | 25 +++++++++++++++----------
1 file changed, 15 insertions(+), 10 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 5d24680737..3d384cf645 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -159,6 +159,7 @@ typedef struct TidStoreIter
static void tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val);
static inline BlockNumber key_get_blkno(TidStore *ts, uint64 key);
+static inline uint64 encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint32 *off);
static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off);
/*
@@ -367,7 +368,6 @@ void
tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
int num_offsets)
{
- ItemPointerData tid;
uint64 *values;
uint64 key;
uint64 prev_key;
@@ -381,16 +381,12 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
values = palloc(sizeof(uint64) * nkeys);
key = prev_key = key_base;
- ItemPointerSetBlockNumber(&tid, blkno);
-
for (int i = 0; i < num_offsets; i++)
{
uint32 off;
- ItemPointerSetOffsetNumber(&tid, offsets[i]);
-
/* encode the tid to key and val */
- key = tid_to_key_off(ts, &tid, &off);
+ key = encode_key_off(ts, blkno, offsets[i], &off);
/* make sure we scanned the line pointer array in order */
Assert(key >= prev_key);
@@ -681,20 +677,29 @@ key_get_blkno(TidStore *ts, uint64 key)
/* Encode a tid to key and offset */
static inline uint64
tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off)
+{
+ uint32 offset = ItemPointerGetOffsetNumber(tid);
+ BlockNumber block = ItemPointerGetBlockNumber(tid);
+
+ return encode_key_off(ts, block, offset, off);
+}
+
+/* encode a block and offset to a key and partial offset */
+static inline uint64
+encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint32 *off)
{
uint64 key;
uint64 tid_i;
if (!ts->control->encode_tids)
{
- *off = ItemPointerGetOffsetNumber(tid);
+ *off = offset;
/* Use the block number as the key */
- return (int64) ItemPointerGetBlockNumber(tid);
+ return (int64) block;
}
- tid_i = ItemPointerGetOffsetNumber(tid);
- tid_i |= (uint64) ItemPointerGetBlockNumber(tid) << ts->control->offset_nbits;
+ tid_i = offset | ((uint64) block << ts->control->offset_nbits);
*off = tid_i & ((UINT64CONST(1) << TIDSTORE_VALUE_NBITS) - 1);
key = tid_i >> TIDSTORE_VALUE_NBITS;
--
2.39.1
[text/plain] v2699-0001-Miscellaneous-optimizations-for-tidstore_add_t.patch.txt (3.0K, ../../CAFBsxsFps1MReb3HQYEGc=M=1NXRjSu3TSaEVEXkVrCR3g-hxg@mail.gmail.com/4-v2699-0001-Miscellaneous-optimizations-for-tidstore_add_t.patch.txt)
download | inline diff:
From c0bc497f50318c8e31ccdf0c2a9186ffc736abeb Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Fri, 10 Feb 2023 19:56:01 +0700
Subject: [PATCH v2699 1/3] Miscellaneous optimizations for tidstore_add_tids()
- remove palloc0; it's expensive for lengths not known at compile-time
- optimize for case with only one key per heap block
- make some intializations const and branch-free
- when writing to the radix tree, stop at the last non-zero bitmap
---
src/backend/access/common/tidstore.c | 56 ++++++++++++++++++----------
1 file changed, 36 insertions(+), 20 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 4c72673ce9..5d24680737 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -368,51 +368,67 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
int num_offsets)
{
ItemPointerData tid;
- uint64 key_base;
uint64 *values;
- int nkeys;
+ uint64 key;
+ uint64 prev_key;
+ uint64 off_bitmap = 0;
+ int idx;
+ const uint64 key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
+ const int nkeys = UINT64CONST(1) << ts->control->offset_key_nbits;
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
- if (ts->control->encode_tids)
- {
- key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
- nkeys = UINT64CONST(1) << ts->control->offset_key_nbits;
- }
- else
- {
- key_base = (uint64) blkno;
- nkeys = 1;
- }
- values = palloc0(sizeof(uint64) * nkeys);
+ values = palloc(sizeof(uint64) * nkeys);
+ key = prev_key = key_base;
ItemPointerSetBlockNumber(&tid, blkno);
+
for (int i = 0; i < num_offsets; i++)
{
- uint64 key;
uint32 off;
- int idx;
ItemPointerSetOffsetNumber(&tid, offsets[i]);
/* encode the tid to key and val */
key = tid_to_key_off(ts, &tid, &off);
- idx = key - key_base;
- Assert(idx >= 0 && idx < nkeys);
+ /* make sure we scanned the line pointer array in order */
+ Assert(key >= prev_key);
- values[idx] |= UINT64CONST(1) << off;
+ if (key > prev_key)
+ {
+ idx = prev_key - key_base;
+ Assert(idx >= 0 && idx < nkeys);
+
+ /* write out offset bitmap for this key */
+ values[idx] = off_bitmap;
+
+ /* zero out any gaps up to the current key */
+ for (int empty_idx = idx + 1; empty_idx < key - key_base; empty_idx++)
+ values[empty_idx] = 0;
+
+ /* reset for current key -- the current offset will be handled below */
+ off_bitmap = 0;
+ prev_key = key;
+ }
+
+ off_bitmap |= UINT64CONST(1) << off;
}
+ /* save the final index for later */
+ idx = key - key_base;
+ /* write out last offset bitmap */
+ values[idx] = off_bitmap;
+
if (TidStoreIsShared(ts))
LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
/* insert the calculated key-values to the tree */
- for (int i = 0; i < nkeys; i++)
+ for (int i = 0; i <= idx; i++)
{
if (values[i])
{
- uint64 key = key_base + i;
+ key = key_base + i;
if (TidStoreIsShared(ts))
shared_rt_set(ts->tree.shared, key, &values[i]);
--
2.39.1
[text/plain] v2699-0003-Force-all-callers-to-encode-no-matter-how-smal.patch.txt (2.8K, ../../CAFBsxsFps1MReb3HQYEGc=M=1NXRjSu3TSaEVEXkVrCR3g-hxg@mail.gmail.com/5-v2699-0003-Force-all-callers-to-encode-no-matter-how-smal.patch.txt)
download | inline diff:
From 82c1f639aaa64cc943af3b53294a63d5d8f7a9b9 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sat, 11 Feb 2023 11:51:32 +0700
Subject: [PATCH v2699 3/3] Force all callers to encode, no matter how small
the expected offset
---
src/backend/access/common/tidstore.c | 36 +++++-----------------------
1 file changed, 6 insertions(+), 30 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 3d384cf645..ff8e66936e 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -99,7 +99,6 @@ typedef struct TidStoreControl
size_t max_bytes; /* the maximum bytes a TidStore can use */
int max_offset; /* the maximum offset number */
int offset_nbits; /* the number of bits required for max_offset */
- bool encode_tids; /* do we use tid encoding? */
int offset_key_nbits; /* the number of bits of a offset number
* used for the key */
@@ -224,21 +223,15 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
ts->control->max_offset = max_offset;
ts->control->offset_nbits = pg_ceil_log2_32(max_offset);
+ if (ts->control->offset_nbits < TIDSTORE_VALUE_NBITS)
+ ts->control->offset_nbits = TIDSTORE_VALUE_NBITS;
+
/*
* We use tid encoding if the number of bits for the offset number doesn't
* fix in a value, uint64.
*/
- if (ts->control->offset_nbits > TIDSTORE_VALUE_NBITS)
- {
- ts->control->encode_tids = true;
- ts->control->offset_key_nbits =
- ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
- }
- else
- {
- ts->control->encode_tids = false;
- ts->control->offset_key_nbits = 0;
- }
+ ts->control->offset_key_nbits =
+ ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
return ts;
}
@@ -643,12 +636,6 @@ tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
uint64 tid_i;
OffsetNumber off;
- if (i > iter->ts->control->max_offset)
- {
- Assert(!iter->ts->control->encode_tids);
- break;
- }
-
if ((val & (UINT64CONST(1) << i)) == 0)
continue;
@@ -668,10 +655,7 @@ tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
static inline BlockNumber
key_get_blkno(TidStore *ts, uint64 key)
{
- if (ts->control->encode_tids)
- return (BlockNumber) (key >> ts->control->offset_key_nbits);
-
- return (BlockNumber) key;
+ return (BlockNumber) (key >> ts->control->offset_key_nbits);
}
/* Encode a tid to key and offset */
@@ -691,14 +675,6 @@ encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint32 *off)
uint64 key;
uint64 tid_i;
- if (!ts->control->encode_tids)
- {
- *off = offset;
-
- /* Use the block number as the key */
- return (int64) block;
- }
-
tid_i = offset | ((uint64) block << ts->control->offset_nbits);
*off = tid_i & ((UINT64CONST(1) << TIDSTORE_VALUE_NBITS) - 1);
--
2.39.1
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-02-13 07:50 ` Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-02-13 07:50 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Sat, Feb 11, 2023 at 2:33 PM John Naylor
<[email protected]> wrote:
>
> I didn't get any closer to radix-tree regression,
Me neither. It seems that in v26, inserting chunks into node-32 is
slow but needs more analysis. I'll share if I found something
interesting.
> but I did find some inefficiencies in tidstore_add_tids() that are worth talking about first, addressed in a rough fashion in the attached .txt addendums that I can clean up and incorporate later.
>
> To start, I can reproduce the regression with this test as well:
>
> select * from bench_tidstore_load(0, 10 * 1000 * 1000);
>
> v15 + v26 store + adjustments:
> mem_allocated | load_ms
> ---------------+---------
> 98202152 | 1676
>
> v26 0001-0008
> mem_allocated | load_ms
> ---------------+---------
> 98202032 | 1826
>
> ...and reverting to the alternate way to update the parent didn't help:
>
> v26 0001-6, 0008, insert_inner w/ null parent
>
> mem_allocated | load_ms
> ---------------+---------
> 98202032 | 1825
>
> ...and I'm kind of glad that wasn't the problem, because going back to that would be a pain for the shmem case.
>
> Running perf doesn't show anything much different in the proportions (note that rt_set must have been inlined when declared locally in v26):
>
> v15 + v26 store + adjustments:
> 65.88% postgres postgres [.] tidstore_add_tids
> 10.74% postgres postgres [.] rt_set
> 9.20% postgres postgres [.] palloc0
> 6.49% postgres postgres [.] rt_node_insert_leaf
>
> v26 0001-0008
> 78.50% postgres postgres [.] tidstore_add_tids
> 8.88% postgres postgres [.] palloc0
> 6.24% postgres postgres [.] local_rt_node_insert_leaf
>
> v2699-0001: The first thing I noticed is that palloc0 is taking way more time than it should, and it's because the compiler doesn't know the values[] array is small. One reason we need to zero the array is to make the algorithm agnostic about what order the offsets come in, as I requested in a previous review. Thinking some more, I was way too paranoid about that. As long as access methods scan the line pointer array in the usual way, maybe we can just assert that the keys we create are in order, and zero any unused array entries as we find them. (I admit I can't actually think of a reason we would ever encounter offsets out of order.)
I can think that something like traversing a HOT chain could visit
offsets out of order. But fortunately we prune such collected TIDs
before heap vacuum in heap case.
> Also, we can keep track of the last key we need to consider for insertion into the radix tree, and ignore the rest. That might shave a few cycles during the exclusive lock when the max offset of an LP_DEAD item < 64 on a given page, which I think would be common in the wild. I also got rid of the special case for non-encoding, since shifting by zero should work the same way. These together led to a nice speedup on the v26 branch:
>
> mem_allocated | load_ms
> ---------------+---------
> 98202032 | 1386
>
> v2699-0002: The next thing I noticed is forming a full ItemIdPointer to pass to tid_to_key_off(). That's bad for tidstore_add_tids() because ItemPointerSetBlockNumber() must do this in order to allow the struct to be SHORTALIGN'd:
>
> static inline void
> BlockIdSet(BlockIdData *blockId, BlockNumber blockNumber)
> {
> blockId->bi_hi = blockNumber >> 16;
> blockId->bi_lo = blockNumber & 0xffff;
> }
>
> Then, tid_to_key_off() calls ItemPointerGetBlockNumber(), which must reverse the above process:
>
> static inline BlockNumber
> BlockIdGetBlockNumber(const BlockIdData *blockId)
> {
> return (((BlockNumber) blockId->bi_hi) << 16) | ((BlockNumber) blockId->bi_lo);
> }
>
> There is no reason to do any of this if we're not reading/writing directly to/from an on-disk tid etc. To avoid this, I created a new function encode_key_off() [name could be better], which deals with the raw block number that we already have. Then turn tid_to_key_off() into a wrapper around that, since we still need the full conversion for tidstore_lookup_tid().
>
> v2699-0003: Get rid of all the remaining special cases for encoding/or not. I am unaware of the need to optimize that case or treat it in any way differently. I haven't tested this on an installation with non-default blocksize and didn't measure this separately, but 0002+0003 gives:
>
> mem_allocated | load_ms
> ---------------+---------
> 98202032 | 1259
>
> If these are acceptable, I can incorporate them into a later patchset.
These are nice improvements! I agree with all changes.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-02-14 11:24 ` John Naylor <[email protected]>
2023-02-14 12:36 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
0 siblings, 2 replies; 78+ messages in thread
From: John Naylor @ 2023-02-14 11:24 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Mon, Feb 13, 2023 at 2:51 PM Masahiko Sawada <[email protected]>
wrote:
>
> On Sat, Feb 11, 2023 at 2:33 PM John Naylor
> <[email protected]> wrote:
> >
> > I didn't get any closer to radix-tree regression,
>
> Me neither. It seems that in v26, inserting chunks into node-32 is
> slow but needs more analysis. I'll share if I found something
> interesting.
If that were the case, then the other benchmarks I ran would likely have
slowed down as well, but they are the same or faster. There is one
microbenchmark I didn't run before: "select * from
bench_fixed_height_search(15)" (15 to reduce noise from growing size class,
and despite the name it measures load time as well). Trying this now shows
no difference: a few runs range 19 to 21ms in each version. That also
reinforces that update_inner is fine and that the move to value pointer API
didn't regress.
Changing TIDS_PER_BLOCK_FOR_LOAD to 1 to stress the tree more gives (min of
5, perf run separate from measurements):
v15 + v26 store:
mem_allocated | load_ms
---------------+---------
98202152 | 553
19.71% postgres postgres [.] tidstore_add_tids
+ 31.47% postgres postgres [.] rt_set
= 51.18%
20.62% postgres postgres [.] rt_node_insert_leaf
6.05% postgres postgres [.] AllocSetAlloc
4.74% postgres postgres [.] AllocSetFree
4.62% postgres postgres [.] palloc
2.23% postgres postgres [.] SlabAlloc
v26:
mem_allocated | load_ms
---------------+---------
98202032 | 617
57.45% postgres postgres [.] tidstore_add_tids
20.67% postgres postgres [.] local_rt_node_insert_leaf
5.99% postgres postgres [.] AllocSetAlloc
3.55% postgres postgres [.] palloc
3.05% postgres postgres [.] AllocSetFree
2.05% postgres postgres [.] SlabAlloc
So it seems the store itself got faster when we removed shared memory paths
from the v26 store to test it against v15.
I thought to favor the local memory case in the tidstore by controlling
inlining -- it's smaller and will be called much more often, so I tried the
following (done in 0007)
#define RT_PREFIX shared_rt
#define RT_SHMEM
-#define RT_SCOPE static
+#define RT_SCOPE static pg_noinline
That brings it down to
mem_allocated | load_ms
---------------+---------
98202032 | 590
That's better, but not still not within noise level. Perhaps some slowdown
is unavoidable, but it would be nice to understand why.
> I can think that something like traversing a HOT chain could visit
> offsets out of order. But fortunately we prune such collected TIDs
> before heap vacuum in heap case.
Further, currently we *already* assume we populate the tid array in order
(for binary search), so we can just continue assuming that (with an assert
added since it's more public in this form). I'm not sure why such basic
common sense evaded me a few versions ago...
> > If these are acceptable, I can incorporate them into a later patchset.
>
> These are nice improvements! I agree with all changes.
Great, I've squashed these into the tidstore patch (0004). Also added 0005,
which is just a simplification.
I squashed the earlier dead code removal into the radix tree patch.
v27-0008 measures tid store iteration performance and adds a stub function
to prevent spurious warnings, so the benchmarking module can always be
built.
Getting the list of offsets from the old array for a given block is always
trivial, but tidstore_iter_extract_tids() is doing a huge amount of
unnecessary work when TIDS_PER_BLOCK_FOR_LOAD is 1, enough to exceed the
load time:
mem_allocated | load_ms | iter_ms
---------------+---------+---------
98202032 | 589 | 915
Fortunately, it's an easy fix, done in 0009.
mem_allocated | load_ms | iter_ms
---------------+---------+---------
98202032 | 589 | 153
I'll soon resume more cosmetic review of the tid store, but this is enough
to post.
--
John Naylor
EDB: http://www.enterprisedb.com
Attachments:
[text/x-patch] v27-0001-Introduce-helper-SIMD-functions-for-small-byte-a.patch (2.9K, ../../CAFBsxsESfmZpWwk3XxWkpdg_DjawH2vizKW91fyawQ+wVFkOiQ@mail.gmail.com/3-v27-0001-Introduce-helper-SIMD-functions-for-small-byte-a.patch)
download | inline diff:
From d577ef9d9755e7ca4d3722c1a044381a81d66244 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 24 Oct 2022 14:07:09 +0900
Subject: [PATCH v27 1/9] Introduce helper SIMD functions for small byte arrays
vector8_min - helper for emulating ">=" semantics
vector8_highbit_mask - used to turn the result of a vector
comparison into a bitmask
Masahiko Sawada
Reviewed by Nathan Bossart, additional adjustments by me
Discussion: https://www.postgresql.org/message-id/CAD21AoDap240WDDdUDE0JMpCmuMMnGajrKrkCRxM7zn9Xk3JRA%40mail.gmail.com
---
src/include/port/simd.h | 47 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)
diff --git a/src/include/port/simd.h b/src/include/port/simd.h
index c836360d4b..350e2caaea 100644
--- a/src/include/port/simd.h
+++ b/src/include/port/simd.h
@@ -79,6 +79,7 @@ static inline bool vector8_has_le(const Vector8 v, const uint8 c);
static inline bool vector8_is_highbit_set(const Vector8 v);
#ifndef USE_NO_SIMD
static inline bool vector32_is_highbit_set(const Vector32 v);
+static inline uint32 vector8_highbit_mask(const Vector8 v);
#endif
/* arithmetic operations */
@@ -96,6 +97,7 @@ static inline Vector8 vector8_ssub(const Vector8 v1, const Vector8 v2);
*/
#ifndef USE_NO_SIMD
static inline Vector8 vector8_eq(const Vector8 v1, const Vector8 v2);
+static inline Vector8 vector8_min(const Vector8 v1, const Vector8 v2);
static inline Vector32 vector32_eq(const Vector32 v1, const Vector32 v2);
#endif
@@ -299,6 +301,36 @@ vector32_is_highbit_set(const Vector32 v)
}
#endif /* ! USE_NO_SIMD */
+/*
+ * Return a bitmask formed from the high-bit of each element.
+ */
+#ifndef USE_NO_SIMD
+static inline uint32
+vector8_highbit_mask(const Vector8 v)
+{
+#ifdef USE_SSE2
+ return (uint32) _mm_movemask_epi8(v);
+#elif defined(USE_NEON)
+ /*
+ * Note: There is a faster way to do this, but it returns a uint64 and
+ * and if the caller wanted to extract the bit position using CTZ,
+ * it would have to divide that result by 4.
+ */
+ static const uint8 mask[16] = {
+ 1 << 0, 1 << 1, 1 << 2, 1 << 3,
+ 1 << 4, 1 << 5, 1 << 6, 1 << 7,
+ 1 << 0, 1 << 1, 1 << 2, 1 << 3,
+ 1 << 4, 1 << 5, 1 << 6, 1 << 7,
+ };
+
+ uint8x16_t masked = vandq_u8(vld1q_u8(mask), (uint8x16_t) vshrq_n_s8(v, 7));
+ uint8x16_t maskedhi = vextq_u8(masked, masked, 8);
+
+ return (uint32) vaddvq_u16((uint16x8_t) vzip1q_u8(masked, maskedhi));
+#endif
+}
+#endif /* ! USE_NO_SIMD */
+
/*
* Return the bitwise OR of the inputs
*/
@@ -372,4 +404,19 @@ vector32_eq(const Vector32 v1, const Vector32 v2)
}
#endif /* ! USE_NO_SIMD */
+/*
+ * Given two vectors, return a vector with the minimum element of each.
+ */
+#ifndef USE_NO_SIMD
+static inline Vector8
+vector8_min(const Vector8 v1, const Vector8 v2)
+{
+#ifdef USE_SSE2
+ return _mm_min_epu8(v1, v2);
+#elif defined(USE_NEON)
+ return vminq_u8(v1, v2);
+#endif
+}
+#endif /* ! USE_NO_SIMD */
+
#endif /* SIMD_H */
--
2.39.1
[text/x-patch] v27-0002-Move-some-bitmap-logic-out-of-bitmapset.c.patch (6.1K, ../../CAFBsxsESfmZpWwk3XxWkpdg_DjawH2vizKW91fyawQ+wVFkOiQ@mail.gmail.com/4-v27-0002-Move-some-bitmap-logic-out-of-bitmapset.c.patch)
download | inline diff:
From 149a49f51f7a16b7c1eb762e704f1ec476ecb65a Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 6 Dec 2022 13:39:41 +0700
Subject: [PATCH v27 2/9] Move some bitmap logic out of bitmapset.c
Add pg_rightmost_one32/64 functions. This functionality was previously
private to bitmapset.c as the RIGHTMOST_ONE macro. It has practical
use in other contexts, so move to pg_bitutils.h.
Also move the logic for selecting appropriate pg_bitutils functions
based on word size to bitmapset.h for wider visibility and add
appropriate selection for bmw_rightmost_one().
Since the previous macro relied on casting to signedbitmapword,
and the new functions do not, remove that typedef.
Design input and review by Tom Lane
Discussion: https://www.postgresql.org/message-id/CAFBsxsFW2JjTo58jtDB%2B3sZhxMx3t-3evew8%3DAcr%2BGGhC%2BkFaA%40mail.gmail.com
---
src/backend/nodes/bitmapset.c | 36 ++------------------------------
src/include/nodes/bitmapset.h | 16 ++++++++++++--
src/include/port/pg_bitutils.h | 31 +++++++++++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 -
4 files changed, 47 insertions(+), 37 deletions(-)
diff --git a/src/backend/nodes/bitmapset.c b/src/backend/nodes/bitmapset.c
index 98660524ad..fcd8e2ccbc 100644
--- a/src/backend/nodes/bitmapset.c
+++ b/src/backend/nodes/bitmapset.c
@@ -32,39 +32,7 @@
#define BITMAPSET_SIZE(nwords) \
(offsetof(Bitmapset, words) + (nwords) * sizeof(bitmapword))
-/*----------
- * This is a well-known cute trick for isolating the rightmost one-bit
- * in a word. It assumes two's complement arithmetic. Consider any
- * nonzero value, and focus attention on the rightmost one. The value is
- * then something like
- * xxxxxx10000
- * where x's are unspecified bits. The two's complement negative is formed
- * by inverting all the bits and adding one. Inversion gives
- * yyyyyy01111
- * where each y is the inverse of the corresponding x. Incrementing gives
- * yyyyyy10000
- * and then ANDing with the original value gives
- * 00000010000
- * This works for all cases except original value = zero, where of course
- * we get zero.
- *----------
- */
-#define RIGHTMOST_ONE(x) ((signedbitmapword) (x) & -((signedbitmapword) (x)))
-
-#define HAS_MULTIPLE_ONES(x) ((bitmapword) RIGHTMOST_ONE(x) != (x))
-
-/* Select appropriate bit-twiddling functions for bitmap word size */
-#if BITS_PER_BITMAPWORD == 32
-#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w)
-#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w)
-#define bmw_popcount(w) pg_popcount32(w)
-#elif BITS_PER_BITMAPWORD == 64
-#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w)
-#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w)
-#define bmw_popcount(w) pg_popcount64(w)
-#else
-#error "invalid BITS_PER_BITMAPWORD"
-#endif
+#define HAS_MULTIPLE_ONES(x) (bmw_rightmost_one(x) != (x))
/*
@@ -1013,7 +981,7 @@ bms_first_member(Bitmapset *a)
{
int result;
- w = RIGHTMOST_ONE(w);
+ w = bmw_rightmost_one(w);
a->words[wordnum] &= ~w;
result = wordnum * BITS_PER_BITMAPWORD;
diff --git a/src/include/nodes/bitmapset.h b/src/include/nodes/bitmapset.h
index 3d2225e1ae..5f9a511b4a 100644
--- a/src/include/nodes/bitmapset.h
+++ b/src/include/nodes/bitmapset.h
@@ -38,13 +38,11 @@ struct List;
#define BITS_PER_BITMAPWORD 64
typedef uint64 bitmapword; /* must be an unsigned type */
-typedef int64 signedbitmapword; /* must be the matching signed type */
#else
#define BITS_PER_BITMAPWORD 32
typedef uint32 bitmapword; /* must be an unsigned type */
-typedef int32 signedbitmapword; /* must be the matching signed type */
#endif
@@ -75,6 +73,20 @@ typedef enum
BMS_MULTIPLE /* >1 member */
} BMS_Membership;
+/* Select appropriate bit-twiddling functions for bitmap word size */
+#if BITS_PER_BITMAPWORD == 32
+#define bmw_rightmost_one(w) pg_rightmost_one32(w)
+#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w)
+#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w)
+#define bmw_popcount(w) pg_popcount32(w)
+#elif BITS_PER_BITMAPWORD == 64
+#define bmw_rightmost_one(w) pg_rightmost_one64(w)
+#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w)
+#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w)
+#define bmw_popcount(w) pg_popcount64(w)
+#else
+#error "invalid BITS_PER_BITMAPWORD"
+#endif
/*
* function prototypes in nodes/bitmapset.c
diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h
index a49a9c03d9..7235ad25ee 100644
--- a/src/include/port/pg_bitutils.h
+++ b/src/include/port/pg_bitutils.h
@@ -17,6 +17,37 @@ extern PGDLLIMPORT const uint8 pg_leftmost_one_pos[256];
extern PGDLLIMPORT const uint8 pg_rightmost_one_pos[256];
extern PGDLLIMPORT const uint8 pg_number_of_ones[256];
+/*----------
+ * This is a well-known cute trick for isolating the rightmost one-bit
+ * in a word. It assumes two's complement arithmetic. Consider any
+ * nonzero value, and focus attention on the rightmost one. The value is
+ * then something like
+ * xxxxxx10000
+ * where x's are unspecified bits. The two's complement negative is formed
+ * by inverting all the bits and adding one. Inversion gives
+ * yyyyyy01111
+ * where each y is the inverse of the corresponding x. Incrementing gives
+ * yyyyyy10000
+ * and then ANDing with the original value gives
+ * 00000010000
+ * This works for all cases except original value = zero, where of course
+ * we get zero.
+ *----------
+ */
+static inline uint32
+pg_rightmost_one32(uint32 word)
+{
+ int32 result = (int32) word & -((int32) word);
+ return (uint32) result;
+}
+
+static inline uint64
+pg_rightmost_one64(uint64 word)
+{
+ int64 result = (int64) word & -((int64) word);
+ return (uint64) result;
+}
+
/*
* pg_leftmost_one_pos32
* Returns the position of the most significant set bit in "word",
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 36d1dc0117..a0c60feade 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3669,7 +3669,6 @@ shmem_request_hook_type
shmem_startup_hook_type
sig_atomic_t
sigjmp_buf
-signedbitmapword
sigset_t
size_t
slist_head
--
2.39.1
[text/x-patch] v27-0005-Do-bitmap-conversion-in-one-place-rather-than-fo.patch (3.8K, ../../CAFBsxsESfmZpWwk3XxWkpdg_DjawH2vizKW91fyawQ+wVFkOiQ@mail.gmail.com/5-v27-0005-Do-bitmap-conversion-in-one-place-rather-than-fo.patch)
download | inline diff:
From dba9497b5b587da873fbb2de89570ec8b36d604b Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sun, 12 Feb 2023 15:17:40 +0700
Subject: [PATCH v27 5/9] Do bitmap conversion in one place rather than forcing
callers to do it
---
src/backend/access/common/tidstore.c | 31 +++++++++++++++-------------
1 file changed, 17 insertions(+), 14 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index ff8e66936e..ad8c0866e2 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -70,6 +70,7 @@
* and value, respectively.
*/
#define TIDSTORE_VALUE_NBITS 6 /* log(64, 2) */
+#define TIDSTORE_OFFSET_MASK ((1 << TIDSTORE_VALUE_NBITS) - 1)
/* A magic value used to identify our TidStores. */
#define TIDSTORE_MAGIC 0x826f6a10
@@ -158,8 +159,8 @@ typedef struct TidStoreIter
static void tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val);
static inline BlockNumber key_get_blkno(TidStore *ts, uint64 key);
-static inline uint64 encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint32 *off);
-static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off);
+static inline uint64 encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit);
+static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit);
/*
* Create a TidStore. The returned object is allocated in backend-local memory.
@@ -376,10 +377,10 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
for (int i = 0; i < num_offsets; i++)
{
- uint32 off;
+ uint64 off_bit;
/* encode the tid to key and val */
- key = encode_key_off(ts, blkno, offsets[i], &off);
+ key = encode_key_off(ts, blkno, offsets[i], &off_bit);
/* make sure we scanned the line pointer array in order */
Assert(key >= prev_key);
@@ -401,7 +402,7 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
prev_key = key;
}
- off_bitmap |= UINT64CONST(1) << off;
+ off_bitmap |= off_bit;
}
/* save the final index for later */
@@ -441,10 +442,10 @@ tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
{
uint64 key;
uint64 val = 0;
- uint32 off;
+ uint64 off_bit;
bool found;
- key = tid_to_key_off(ts, tid, &off);
+ key = tid_to_key_off(ts, tid, &off_bit);
if (TidStoreIsShared(ts))
found = shared_rt_search(ts->tree.shared, key, &val);
@@ -454,7 +455,7 @@ tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
if (!found)
return false;
- return (val & (UINT64CONST(1) << off)) != 0;
+ return (val & off_bit) != 0;
}
/*
@@ -660,26 +661,28 @@ key_get_blkno(TidStore *ts, uint64 key)
/* Encode a tid to key and offset */
static inline uint64
-tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off)
+tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit)
{
uint32 offset = ItemPointerGetOffsetNumber(tid);
BlockNumber block = ItemPointerGetBlockNumber(tid);
- return encode_key_off(ts, block, offset, off);
+ return encode_key_off(ts, block, offset, off_bit);
}
/* encode a block and offset to a key and partial offset */
static inline uint64
-encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint32 *off)
+encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit)
{
uint64 key;
uint64 tid_i;
+ uint32 off_lower;
- tid_i = offset | ((uint64) block << ts->control->offset_nbits);
+ off_lower = offset & TIDSTORE_OFFSET_MASK;
+ Assert(off_lower < (sizeof(uint64) * BITS_PER_BYTE));
- *off = tid_i & ((UINT64CONST(1) << TIDSTORE_VALUE_NBITS) - 1);
+ *off_bit = UINT64CONST(1) << off_lower;
+ tid_i = offset | ((uint64) block << ts->control->offset_nbits);
key = tid_i >> TIDSTORE_VALUE_NBITS;
- Assert(*off < (sizeof(uint64) * BITS_PER_BYTE));
return key;
}
--
2.39.1
[text/x-patch] v27-0003-Add-radixtree-template.patch (116.8K, ../../CAFBsxsESfmZpWwk3XxWkpdg_DjawH2vizKW91fyawQ+wVFkOiQ@mail.gmail.com/6-v27-0003-Add-radixtree-template.patch)
download | inline diff:
From bf9d659187537b250683af321b0167d69c7fb18a Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 14 Sep 2022 12:38:51 +0000
Subject: [PATCH v27 3/9] Add radixtree template
WIP: commit message based on template comments
---
src/backend/utils/mmgr/dsa.c | 12 +
src/include/lib/radixtree.h | 2516 +++++++++++++++++
src/include/lib/radixtree_delete_impl.h | 122 +
src/include/lib/radixtree_insert_impl.h | 328 +++
src/include/lib/radixtree_iter_impl.h | 153 +
src/include/lib/radixtree_search_impl.h | 138 +
src/include/utils/dsa.h | 1 +
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_radixtree/.gitignore | 4 +
src/test/modules/test_radixtree/Makefile | 23 +
src/test/modules/test_radixtree/README | 7 +
.../expected/test_radixtree.out | 36 +
src/test/modules/test_radixtree/meson.build | 35 +
.../test_radixtree/sql/test_radixtree.sql | 7 +
.../test_radixtree/test_radixtree--1.0.sql | 8 +
.../modules/test_radixtree/test_radixtree.c | 674 +++++
.../test_radixtree/test_radixtree.control | 4 +
src/tools/pginclude/cpluspluscheck | 6 +
src/tools/pginclude/headerscheck | 6 +
20 files changed, 4082 insertions(+)
create mode 100644 src/include/lib/radixtree.h
create mode 100644 src/include/lib/radixtree_delete_impl.h
create mode 100644 src/include/lib/radixtree_insert_impl.h
create mode 100644 src/include/lib/radixtree_iter_impl.h
create mode 100644 src/include/lib/radixtree_search_impl.h
create mode 100644 src/test/modules/test_radixtree/.gitignore
create mode 100644 src/test/modules/test_radixtree/Makefile
create mode 100644 src/test/modules/test_radixtree/README
create mode 100644 src/test/modules/test_radixtree/expected/test_radixtree.out
create mode 100644 src/test/modules/test_radixtree/meson.build
create mode 100644 src/test/modules/test_radixtree/sql/test_radixtree.sql
create mode 100644 src/test/modules/test_radixtree/test_radixtree--1.0.sql
create mode 100644 src/test/modules/test_radixtree/test_radixtree.c
create mode 100644 src/test/modules/test_radixtree/test_radixtree.control
diff --git a/src/backend/utils/mmgr/dsa.c b/src/backend/utils/mmgr/dsa.c
index f5a62061a3..80555aefff 100644
--- a/src/backend/utils/mmgr/dsa.c
+++ b/src/backend/utils/mmgr/dsa.c
@@ -1024,6 +1024,18 @@ dsa_set_size_limit(dsa_area *area, size_t limit)
LWLockRelease(DSA_AREA_LOCK(area));
}
+size_t
+dsa_get_total_size(dsa_area *area)
+{
+ size_t size;
+
+ LWLockAcquire(DSA_AREA_LOCK(area), LW_SHARED);
+ size = area->control->total_segment_size;
+ LWLockRelease(DSA_AREA_LOCK(area));
+
+ return size;
+}
+
/*
* Aggressively free all spare memory in the hope of returning DSM segments to
* the operating system.
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
new file mode 100644
index 0000000000..1cdb995e54
--- /dev/null
+++ b/src/include/lib/radixtree.h
@@ -0,0 +1,2516 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree.h
+ * Template for adaptive radix tree.
+ *
+ * This module employs the idea from the paper "The Adaptive Radix Tree: ARTful
+ * Indexing for Main-Memory Databases" by Viktor Leis, Alfons Kemper, and Thomas
+ * Neumann, 2013. The radix tree uses adaptive node sizes, a small number of node
+ * types, each with a different numbers of elements. Depending on the number of
+ * children, the appropriate node type is used.
+ *
+ * WIP: notes about traditional radix tree trading off span vs height...
+ *
+ * There are two kinds of nodes, inner nodes and leaves. Inner nodes
+ * map partial keys to child pointers.
+ *
+ * The ART paper mentions three ways to implement leaves:
+ *
+ * "- Single-value leaves: The values are stored using an addi-
+ * tional leaf node type which stores one value.
+ * - Multi-value leaves: The values are stored in one of four
+ * different leaf node types, which mirror the structure of
+ * inner nodes, but contain values instead of pointers.
+ * - Combined pointer/value slots: If values fit into point-
+ * ers, no separate node types are necessary. Instead, each
+ * pointer storage location in an inner node can either
+ * store a pointer or a value."
+ *
+ * We chose "multi-value leaves" to avoid the additional pointer traversal
+ * required by "single-value leaves"
+ *
+ * For simplicity, the key is assumed to be 64-bit unsigned integer. The
+ * tree doesn't need to contain paths where the highest bytes of all keys
+ * are zero. That way, the tree's height adapts to the distribution of keys.
+ *
+ * TODO: In the future it might be worthwhile to offer configurability of
+ * leaf implementation for different use cases. Single-values leaves would
+ * give more flexibility in key type, including variable-length keys.
+ *
+ * There are some optimizations not yet implemented, particularly path
+ * compression and lazy path expansion.
+ *
+ * To handle concurrency, we use a single reader-writer lock for the radix
+ * tree. The radix tree is exclusively locked during write operations such
+ * as RT_SET() and RT_DELETE(), and shared locked during read operations
+ * such as RT_SEARCH(). An iteration also holds the shared lock on the radix
+ * tree until it is completed.
+ *
+ * TODO: The current locking mechanism is not optimized for high concurrency
+ * with mixed read-write workloads. In the future it might be worthwhile
+ * to replace it with the Optimistic Lock Coupling or ROWEX mentioned in
+ * the paper "The ART of Practical Synchronization" by the same authors as
+ * the ART paper, 2016.
+ *
+ * WIP: the radix tree nodes don't shrink.
+ *
+ * To generate a radix tree and associated functions for a use case several
+ * macros have to be #define'ed before this file is included. Including
+ * the file #undef's all those, so a new radix tree can be generated
+ * afterwards.
+ * The relevant parameters are:
+ * - RT_PREFIX - prefix for all symbol names generated. A prefix of 'foo'
+ * will result in radix tree type 'foo_radix_tree' and functions like
+ * 'foo_create'/'foo_free' and so forth.
+ * - RT_DECLARE - if defined function prototypes and type declarations are
+ * generated
+ * - RT_DEFINE - if defined function definitions are generated
+ * - RT_SCOPE - in which scope (e.g. extern, static inline) do function
+ * declarations reside
+ * - RT_VALUE_TYPE - the type of the value.
+ *
+ * Optional parameters:
+ * - RT_SHMEM - if defined, the radix tree is created in the DSA area
+ * so that multiple processes can access it simultaneously.
+ * - RT_DEBUG - if defined add stats tracking and debugging functions
+ *
+ * Interface
+ * ---------
+ *
+ * RT_CREATE - Create a new, empty radix tree
+ * RT_FREE - Free the radix tree
+ * RT_SEARCH - Search a key-value pair
+ * RT_SET - Set a key-value pair
+ * RT_BEGIN_ITERATE - Begin iterating through all key-value pairs
+ * RT_ITERATE_NEXT - Return next key-value pair, if any
+ * RT_END_ITER - End iteration
+ * RT_MEMORY_USAGE - Get the memory usage
+ *
+ * Interface for Shared Memory
+ * ---------
+ *
+ * RT_ATTACH - Attach to the radix tree
+ * RT_DETACH - Detach from the radix tree
+ * RT_GET_HANDLE - Return the handle of the radix tree
+ *
+ * Optional Interface
+ * ---------
+ *
+ * RT_DELETE - Delete a key-value pair. Declared/define if RT_USE_DELETE is defined
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/include/lib/radixtree.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/stringinfo.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "port/pg_bitutils.h"
+#include "port/simd.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/* helpers */
+#define RT_MAKE_PREFIX(a) CppConcat(a,_)
+#define RT_MAKE_NAME(name) RT_MAKE_NAME_(RT_MAKE_PREFIX(RT_PREFIX),name)
+#define RT_MAKE_NAME_(a,b) CppConcat(a,b)
+
+/* function declarations */
+#define RT_CREATE RT_MAKE_NAME(create)
+#define RT_FREE RT_MAKE_NAME(free)
+#define RT_SEARCH RT_MAKE_NAME(search)
+#ifdef RT_SHMEM
+#define RT_ATTACH RT_MAKE_NAME(attach)
+#define RT_DETACH RT_MAKE_NAME(detach)
+#define RT_GET_HANDLE RT_MAKE_NAME(get_handle)
+#endif
+#define RT_SET RT_MAKE_NAME(set)
+#define RT_BEGIN_ITERATE RT_MAKE_NAME(begin_iterate)
+#define RT_ITERATE_NEXT RT_MAKE_NAME(iterate_next)
+#define RT_END_ITERATE RT_MAKE_NAME(end_iterate)
+#ifdef RT_USE_DELETE
+#define RT_DELETE RT_MAKE_NAME(delete)
+#endif
+#define RT_MEMORY_USAGE RT_MAKE_NAME(memory_usage)
+#ifdef RT_DEBUG
+#define RT_DUMP RT_MAKE_NAME(dump)
+#define RT_DUMP_NODE RT_MAKE_NAME(dump_node)
+#define RT_DUMP_SEARCH RT_MAKE_NAME(dump_search)
+#define RT_STATS RT_MAKE_NAME(stats)
+#endif
+
+/* internal helper functions (no externally visible prototypes) */
+#define RT_NEW_ROOT RT_MAKE_NAME(new_root)
+#define RT_ALLOC_NODE RT_MAKE_NAME(alloc_node)
+#define RT_INIT_NODE RT_MAKE_NAME(init_node)
+#define RT_FREE_NODE RT_MAKE_NAME(free_node)
+#define RT_FREE_RECURSE RT_MAKE_NAME(free_recurse)
+#define RT_EXTEND RT_MAKE_NAME(extend)
+#define RT_SET_EXTEND RT_MAKE_NAME(set_extend)
+#define RT_SWITCH_NODE_KIND RT_MAKE_NAME(grow_node_kind)
+#define RT_COPY_NODE RT_MAKE_NAME(copy_node)
+#define RT_REPLACE_NODE RT_MAKE_NAME(replace_node)
+#define RT_PTR_GET_LOCAL RT_MAKE_NAME(ptr_get_local)
+#define RT_PTR_ALLOC_IS_VALID RT_MAKE_NAME(ptr_stored_is_valid)
+#define RT_NODE_3_SEARCH_EQ RT_MAKE_NAME(node_3_search_eq)
+#define RT_NODE_32_SEARCH_EQ RT_MAKE_NAME(node_32_search_eq)
+#define RT_NODE_3_GET_INSERTPOS RT_MAKE_NAME(node_3_get_insertpos)
+#define RT_NODE_32_GET_INSERTPOS RT_MAKE_NAME(node_32_get_insertpos)
+#define RT_CHUNK_CHILDREN_ARRAY_SHIFT RT_MAKE_NAME(chunk_children_array_shift)
+#define RT_CHUNK_VALUES_ARRAY_SHIFT RT_MAKE_NAME(chunk_values_array_shift)
+#define RT_CHUNK_CHILDREN_ARRAY_DELETE RT_MAKE_NAME(chunk_children_array_delete)
+#define RT_CHUNK_VALUES_ARRAY_DELETE RT_MAKE_NAME(chunk_values_array_delete)
+#define RT_CHUNK_CHILDREN_ARRAY_COPY RT_MAKE_NAME(chunk_children_array_copy)
+#define RT_CHUNK_VALUES_ARRAY_COPY RT_MAKE_NAME(chunk_values_array_copy)
+#define RT_NODE_125_IS_CHUNK_USED RT_MAKE_NAME(node_125_is_chunk_used)
+#define RT_NODE_INNER_125_GET_CHILD RT_MAKE_NAME(node_inner_125_get_child)
+#define RT_NODE_LEAF_125_GET_VALUE RT_MAKE_NAME(node_leaf_125_get_value)
+#define RT_NODE_INNER_256_IS_CHUNK_USED RT_MAKE_NAME(node_inner_256_is_chunk_used)
+#define RT_NODE_LEAF_256_IS_CHUNK_USED RT_MAKE_NAME(node_leaf_256_is_chunk_used)
+#define RT_NODE_INNER_256_GET_CHILD RT_MAKE_NAME(node_inner_256_get_child)
+#define RT_NODE_LEAF_256_GET_VALUE RT_MAKE_NAME(node_leaf_256_get_value)
+#define RT_NODE_INNER_256_SET RT_MAKE_NAME(node_inner_256_set)
+#define RT_NODE_LEAF_256_SET RT_MAKE_NAME(node_leaf_256_set)
+#define RT_NODE_INNER_256_DELETE RT_MAKE_NAME(node_inner_256_delete)
+#define RT_NODE_LEAF_256_DELETE RT_MAKE_NAME(node_leaf_256_delete)
+#define RT_KEY_GET_SHIFT RT_MAKE_NAME(key_get_shift)
+#define RT_SHIFT_GET_MAX_VAL RT_MAKE_NAME(shift_get_max_val)
+#define RT_NODE_SEARCH_INNER RT_MAKE_NAME(node_search_inner)
+#define RT_NODE_SEARCH_LEAF RT_MAKE_NAME(node_search_leaf)
+#define RT_NODE_UPDATE_INNER RT_MAKE_NAME(node_update_inner)
+#define RT_NODE_DELETE_INNER RT_MAKE_NAME(node_delete_inner)
+#define RT_NODE_DELETE_LEAF RT_MAKE_NAME(node_delete_leaf)
+#define RT_NODE_INSERT_INNER RT_MAKE_NAME(node_insert_inner)
+#define RT_NODE_INSERT_LEAF RT_MAKE_NAME(node_insert_leaf)
+#define RT_NODE_INNER_ITERATE_NEXT RT_MAKE_NAME(node_inner_iterate_next)
+#define RT_NODE_LEAF_ITERATE_NEXT RT_MAKE_NAME(node_leaf_iterate_next)
+#define RT_UPDATE_ITER_STACK RT_MAKE_NAME(update_iter_stack)
+#define RT_ITER_UPDATE_KEY RT_MAKE_NAME(iter_update_key)
+#define RT_VERIFY_NODE RT_MAKE_NAME(verify_node)
+
+/* type declarations */
+#define RT_RADIX_TREE RT_MAKE_NAME(radix_tree)
+#define RT_RADIX_TREE_CONTROL RT_MAKE_NAME(radix_tree_control)
+#define RT_ITER RT_MAKE_NAME(iter)
+#ifdef RT_SHMEM
+#define RT_HANDLE RT_MAKE_NAME(handle)
+#endif
+#define RT_NODE RT_MAKE_NAME(node)
+#define RT_NODE_ITER RT_MAKE_NAME(node_iter)
+#define RT_NODE_BASE_3 RT_MAKE_NAME(node_base_3)
+#define RT_NODE_BASE_32 RT_MAKE_NAME(node_base_32)
+#define RT_NODE_BASE_125 RT_MAKE_NAME(node_base_125)
+#define RT_NODE_BASE_256 RT_MAKE_NAME(node_base_256)
+#define RT_NODE_INNER_3 RT_MAKE_NAME(node_inner_3)
+#define RT_NODE_INNER_32 RT_MAKE_NAME(node_inner_32)
+#define RT_NODE_INNER_125 RT_MAKE_NAME(node_inner_125)
+#define RT_NODE_INNER_256 RT_MAKE_NAME(node_inner_256)
+#define RT_NODE_LEAF_3 RT_MAKE_NAME(node_leaf_3)
+#define RT_NODE_LEAF_32 RT_MAKE_NAME(node_leaf_32)
+#define RT_NODE_LEAF_125 RT_MAKE_NAME(node_leaf_125)
+#define RT_NODE_LEAF_256 RT_MAKE_NAME(node_leaf_256)
+#define RT_SIZE_CLASS RT_MAKE_NAME(size_class)
+#define RT_SIZE_CLASS_ELEM RT_MAKE_NAME(size_class_elem)
+#define RT_SIZE_CLASS_INFO RT_MAKE_NAME(size_class_info)
+#define RT_CLASS_3 RT_MAKE_NAME(class_3)
+#define RT_CLASS_32_MIN RT_MAKE_NAME(class_32_min)
+#define RT_CLASS_32_MAX RT_MAKE_NAME(class_32_max)
+#define RT_CLASS_125 RT_MAKE_NAME(class_125)
+#define RT_CLASS_256 RT_MAKE_NAME(class_256)
+
+/* generate forward declarations necessary to use the radix tree */
+#ifdef RT_DECLARE
+
+typedef struct RT_RADIX_TREE RT_RADIX_TREE;
+typedef struct RT_ITER RT_ITER;
+
+#ifdef RT_SHMEM
+typedef dsa_pointer RT_HANDLE;
+#endif
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id);
+RT_SCOPE RT_RADIX_TREE * RT_ATTACH(dsa_area *dsa, dsa_pointer dp);
+RT_SCOPE void RT_DETACH(RT_RADIX_TREE *tree);
+RT_SCOPE RT_HANDLE RT_GET_HANDLE(RT_RADIX_TREE *tree);
+#else
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx);
+#endif
+RT_SCOPE void RT_FREE(RT_RADIX_TREE *tree);
+
+RT_SCOPE bool RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p);
+RT_SCOPE bool RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p);
+#ifdef RT_USE_DELETE
+RT_SCOPE bool RT_DELETE(RT_RADIX_TREE *tree, uint64 key);
+#endif
+
+RT_SCOPE RT_ITER * RT_BEGIN_ITERATE(RT_RADIX_TREE *tree);
+RT_SCOPE bool RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p);
+RT_SCOPE void RT_END_ITERATE(RT_ITER *iter);
+
+RT_SCOPE uint64 RT_MEMORY_USAGE(RT_RADIX_TREE *tree);
+
+#ifdef RT_DEBUG
+RT_SCOPE void RT_DUMP(RT_RADIX_TREE *tree);
+RT_SCOPE void RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key);
+RT_SCOPE void RT_STATS(RT_RADIX_TREE *tree);
+#endif
+
+#endif /* RT_DECLARE */
+
+
+/* generate implementation of the radix tree */
+#ifdef RT_DEFINE
+
+/* The number of bits encoded in one tree level */
+#define RT_NODE_SPAN BITS_PER_BYTE
+
+/* The number of maximum slots in the node */
+#define RT_NODE_MAX_SLOTS (1 << RT_NODE_SPAN)
+
+/* Mask for extracting a chunk from the key */
+#define RT_CHUNK_MASK ((1 << RT_NODE_SPAN) - 1)
+
+/* Maximum shift the radix tree uses */
+#define RT_MAX_SHIFT RT_KEY_GET_SHIFT(UINT64_MAX)
+
+/* Tree level the radix tree uses */
+#define RT_MAX_LEVEL ((sizeof(uint64) * BITS_PER_BYTE) / RT_NODE_SPAN)
+
+/*
+ * Number of bits necessary for isset array in the slot-index node.
+ * Since bitmapword can be 64 bits, the only values that make sense
+ * here are 64 and 128.
+ */
+#define RT_SLOT_IDX_LIMIT (RT_NODE_MAX_SLOTS / 2)
+
+/* Invalid index used in node-125 */
+#define RT_INVALID_SLOT_IDX 0xFF
+
+/* Get a chunk from the key */
+#define RT_GET_KEY_CHUNK(key, shift) ((uint8) (((key) >> (shift)) & RT_CHUNK_MASK))
+
+/* For accessing bitmaps */
+#define RT_BM_IDX(x) ((x) / BITS_PER_BITMAPWORD)
+#define RT_BM_BIT(x) ((x) % BITS_PER_BITMAPWORD)
+
+/*
+ * Node kinds
+ *
+ * The different node kinds are what make the tree "adaptive".
+ *
+ * Each node kind is associated with a different datatype and different
+ * search/set/delete/iterate algorithms adapted for its size. The largest
+ * kind, node256 is basically the same as a traditional radix tree,
+ * and would be most wasteful of memory when sparsely populated. The
+ * smaller nodes expend some additional CPU time to enable a smaller
+ * memory footprint.
+ *
+ * XXX There are 4 node kinds, and this should never be increased,
+ * for several reasons:
+ * 1. With 5 or more kinds, gcc tends to use a jump table for switch
+ * statements.
+ * 2. The 4 kinds can be represented with 2 bits, so we have the option
+ * in the future to tag the node pointer with the kind, even on
+ * platforms with 32-bit pointers. This might speed up node traversal
+ * in trees with highly random node kinds.
+ * 3. We can have multiple size classes per node kind.
+ */
+#define RT_NODE_KIND_3 0x00
+#define RT_NODE_KIND_32 0x01
+#define RT_NODE_KIND_125 0x02
+#define RT_NODE_KIND_256 0x03
+#define RT_NODE_KIND_COUNT 4
+
+/*
+ * Calculate the slab blocksize so that we can allocate at least 32 chunks
+ * from the block.
+ */
+#define RT_SLAB_BLOCK_SIZE(size) \
+ Max((SLAB_DEFAULT_BLOCK_SIZE / (size)) * (size), (size) * 32)
+
+/* Common type for all nodes types */
+typedef struct RT_NODE
+{
+ /*
+ * Number of children. We use uint16 to be able to indicate 256 children
+ * at the fanout of 8.
+ */
+ uint16 count;
+
+ /*
+ * Max capacity for the current size class. Storing this in the
+ * node enables multiple size classes per node kind.
+ * Technically, kinds with a single size class don't need this, so we could
+ * keep this in the individual base types, but the code is simpler this way.
+ * Note: node256 is unique in that it cannot possibly have more than a
+ * single size class, so for that kind we store zero, and uint8 is
+ * sufficient for other kinds.
+ */
+ uint8 fanout;
+
+ /*
+ * Shift indicates which part of the key space is represented by this
+ * node. That is, the key is shifted by 'shift' and the lowest
+ * RT_NODE_SPAN bits are then represented in chunk.
+ */
+ uint8 shift;
+
+ /* Node kind, one per search/set algorithm */
+ uint8 kind;
+} RT_NODE;
+
+
+#define RT_PTR_LOCAL RT_NODE *
+
+#ifdef RT_SHMEM
+#define RT_PTR_ALLOC dsa_pointer
+#else
+#define RT_PTR_ALLOC RT_PTR_LOCAL
+#endif
+
+
+#ifdef RT_SHMEM
+#define RT_INVALID_PTR_ALLOC InvalidDsaPointer
+#else
+#define RT_INVALID_PTR_ALLOC NULL
+#endif
+
+#ifdef RT_SHMEM
+#define RT_LOCK_EXCLUSIVE(tree) LWLockAcquire(&tree->ctl->lock, LW_EXCLUSIVE)
+#define RT_LOCK_SHARED(tree) LWLockAcquire(&tree->ctl->lock, LW_SHARED)
+#define RT_UNLOCK(tree) LWLockRelease(&tree->ctl->lock);
+#else
+#define RT_LOCK_EXCLUSIVE(tree) ((void) 0)
+#define RT_LOCK_SHARED(tree) ((void) 0)
+#define RT_UNLOCK(tree) ((void) 0)
+#endif
+
+/*
+ * Inner nodes and leaf nodes have analogous structure. To distinguish
+ * them at runtime, we take advantage of the fact that the key chunk
+ * is accessed by shifting: Inner tree nodes (shift > 0), store the
+ * pointer to its child node in the slot. In leaf nodes (shift == 0),
+ * the slot contains the value corresponding to the key.
+ */
+#define RT_NODE_IS_LEAF(n) (((RT_PTR_LOCAL) (n))->shift == 0)
+
+#define RT_NODE_MUST_GROW(node) \
+ ((node)->base.n.count == (node)->base.n.fanout)
+
+/*
+ * Base type of each node kinds for leaf and inner nodes.
+ * The base types must be a be able to accommodate the largest size
+ * class for variable-sized node kinds.
+ */
+typedef struct RT_NODE_BASE_3
+{
+ RT_NODE n;
+
+ /* 3 children, for key chunks */
+ uint8 chunks[3];
+} RT_NODE_BASE_3;
+
+typedef struct RT_NODE_BASE_32
+{
+ RT_NODE n;
+
+ /* 32 children, for key chunks */
+ uint8 chunks[32];
+} RT_NODE_BASE_32;
+
+/*
+ * node-125 uses slot_idx array, an array of RT_NODE_MAX_SLOTS length
+ * to store indexes into a second array that contains the values (or
+ * child pointers).
+ */
+typedef struct RT_NODE_BASE_125
+{
+ RT_NODE n;
+
+ /* The index of slots for each fanout */
+ uint8 slot_idxs[RT_NODE_MAX_SLOTS];
+
+ /* bitmap to track which slots are in use */
+ bitmapword isset[RT_BM_IDX(RT_SLOT_IDX_LIMIT)];
+} RT_NODE_BASE_125;
+
+typedef struct RT_NODE_BASE_256
+{
+ RT_NODE n;
+} RT_NODE_BASE_256;
+
+/*
+ * Inner and leaf nodes.
+ *
+ * These are separate because the value type might be different than
+ * something fitting into a pointer-width type.
+ */
+typedef struct RT_NODE_INNER_3
+{
+ RT_NODE_BASE_3 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_3;
+
+typedef struct RT_NODE_LEAF_3
+{
+ RT_NODE_BASE_3 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_3;
+
+typedef struct RT_NODE_INNER_32
+{
+ RT_NODE_BASE_32 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_32;
+
+typedef struct RT_NODE_LEAF_32
+{
+ RT_NODE_BASE_32 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_32;
+
+typedef struct RT_NODE_INNER_125
+{
+ RT_NODE_BASE_125 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_125;
+
+typedef struct RT_NODE_LEAF_125
+{
+ RT_NODE_BASE_125 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_125;
+
+/*
+ * node-256 is the largest node type. This node has an array
+ * for directly storing values (or child pointers in inner nodes).
+ * Unlike other node kinds, it's array size is by definition
+ * fixed.
+ */
+typedef struct RT_NODE_INNER_256
+{
+ RT_NODE_BASE_256 base;
+
+ /* Slots for 256 children */
+ RT_PTR_ALLOC children[RT_NODE_MAX_SLOTS];
+} RT_NODE_INNER_256;
+
+typedef struct RT_NODE_LEAF_256
+{
+ RT_NODE_BASE_256 base;
+
+ /*
+ * Unlike with inner256, zero is a valid value here, so we use a
+ * bitmap to track which slots are in use.
+ */
+ bitmapword isset[RT_BM_IDX(RT_NODE_MAX_SLOTS)];
+
+ /* Slots for 256 values */
+ RT_VALUE_TYPE values[RT_NODE_MAX_SLOTS];
+} RT_NODE_LEAF_256;
+
+/*
+ * Node size classes
+ *
+ * Nodes of different kinds necessarily belong to different size classes.
+ * The main innovation in our implementation compared to the ART paper
+ * is decoupling the notion of size class from kind.
+ *
+ * The size classes within a given node kind have the same underlying
+ * type, but a variable number of children/values. This is possible
+ * because the base type contains small fixed data structures that
+ * work the same way regardless of how full the node is. We store the
+ * node's allocated capacity in the "fanout" member of RT_NODE, to allow
+ * runtime introspection.
+ *
+ * Growing from one node kind to another requires special code for each
+ * case, but growing from one size class to another within the same kind
+ * is basically just allocate + memcpy.
+ *
+ * The size classes have been chosen so that inner nodes on platforms
+ * with 64-bit pointers (and leaf nodes when using a 64-bit key) are
+ * equal to or slightly smaller than some DSA size class.
+ */
+typedef enum RT_SIZE_CLASS
+{
+ RT_CLASS_3 = 0,
+ RT_CLASS_32_MIN,
+ RT_CLASS_32_MAX,
+ RT_CLASS_125,
+ RT_CLASS_256
+} RT_SIZE_CLASS;
+
+/* Information for each size class */
+typedef struct RT_SIZE_CLASS_ELEM
+{
+ const char *name;
+ int fanout;
+
+ /* slab chunk size */
+ Size inner_size;
+ Size leaf_size;
+} RT_SIZE_CLASS_ELEM;
+
+static const RT_SIZE_CLASS_ELEM RT_SIZE_CLASS_INFO[] = {
+ [RT_CLASS_3] = {
+ .name = "radix tree node 3",
+ .fanout = 3,
+ .inner_size = sizeof(RT_NODE_INNER_3) + 3 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_3) + 3 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_32_MIN] = {
+ .name = "radix tree node 15",
+ .fanout = 15,
+ .inner_size = sizeof(RT_NODE_INNER_32) + 15 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_32) + 15 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_32_MAX] = {
+ .name = "radix tree node 32",
+ .fanout = 32,
+ .inner_size = sizeof(RT_NODE_INNER_32) + 32 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_32) + 32 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_125] = {
+ .name = "radix tree node 125",
+ .fanout = 125,
+ .inner_size = sizeof(RT_NODE_INNER_125) + 125 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_125) + 125 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_256] = {
+ .name = "radix tree node 256",
+ .fanout = 256,
+ .inner_size = sizeof(RT_NODE_INNER_256),
+ .leaf_size = sizeof(RT_NODE_LEAF_256),
+ },
+};
+
+#define RT_SIZE_CLASS_COUNT lengthof(RT_SIZE_CLASS_INFO)
+
+#ifdef RT_SHMEM
+/* A magic value used to identify our radix tree */
+#define RT_RADIX_TREE_MAGIC 0x54A48167
+#endif
+
+/* Contains the actual tree and ancillary info */
+// WIP: this name is a bit strange
+typedef struct RT_RADIX_TREE_CONTROL
+{
+#ifdef RT_SHMEM
+ RT_HANDLE handle;
+ uint32 magic;
+ LWLock lock;
+#endif
+
+ RT_PTR_ALLOC root;
+ uint64 max_val;
+ uint64 num_keys;
+
+ /* statistics */
+#ifdef RT_DEBUG
+ int32 cnt[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE_CONTROL;
+
+/* Entry point for allocating and accessing the tree */
+typedef struct RT_RADIX_TREE
+{
+ MemoryContext context;
+
+ /* pointing to either local memory or DSA */
+ RT_RADIX_TREE_CONTROL *ctl;
+
+#ifdef RT_SHMEM
+ dsa_area *dsa;
+#else
+ MemoryContextData *inner_slabs[RT_SIZE_CLASS_COUNT];
+ MemoryContextData *leaf_slabs[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE;
+
+/*
+ * Iteration support.
+ *
+ * Iterating the radix tree returns each pair of key and value in the ascending
+ * order of the key. To support this, the we iterate nodes of each level.
+ *
+ * RT_NODE_ITER struct is used to track the iteration within a node.
+ *
+ * RT_ITER is the struct for iteration of the radix tree, and uses RT_NODE_ITER
+ * in order to track the iteration of each level. During iteration, we also
+ * construct the key whenever updating the node iteration information, e.g., when
+ * advancing the current index within the node or when moving to the next node
+ * at the same level.
+ *
+ * XXX: Currently we allow only one process to do iteration. Therefore, rt_node_iter
+ * has the local pointers to nodes, rather than RT_PTR_ALLOC.
+ * We need either a safeguard to disallow other processes to begin the iteration
+ * while one process is doing or to allow multiple processes to do the iteration.
+ */
+typedef struct RT_NODE_ITER
+{
+ RT_PTR_LOCAL node; /* current node being iterated */
+ int current_idx; /* current position. -1 for initial value */
+} RT_NODE_ITER;
+
+typedef struct RT_ITER
+{
+ RT_RADIX_TREE *tree;
+
+ /* Track the iteration on nodes of each level */
+ RT_NODE_ITER stack[RT_MAX_LEVEL];
+ int stack_len;
+
+ /* The key is constructed during iteration */
+ uint64 key;
+} RT_ITER;
+
+
+static void RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_PTR_ALLOC child);
+static bool RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_VALUE_TYPE *value_p);
+
+/* verification (available only with assertion) */
+static void RT_VERIFY_NODE(RT_PTR_LOCAL node);
+
+/* Get the local address of an allocated node */
+static inline RT_PTR_LOCAL
+RT_PTR_GET_LOCAL(RT_RADIX_TREE *tree, RT_PTR_ALLOC node)
+{
+#ifdef RT_SHMEM
+ return dsa_get_address(tree->dsa, (dsa_pointer) node);
+#else
+ return node;
+#endif
+}
+
+static inline bool
+RT_PTR_ALLOC_IS_VALID(RT_PTR_ALLOC ptr)
+{
+#ifdef RT_SHMEM
+ return DsaPointerIsValid(ptr);
+#else
+ return PointerIsValid(ptr);
+#endif
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_3_SEARCH_EQ(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+ int idx = -1;
+
+ for (int i = 0; i < node->n.count; i++)
+ {
+ if (node->chunks[i] == chunk)
+ {
+ idx = i;
+ break;
+ }
+ }
+
+ return idx;
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_3_GET_INSERTPOS(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+ int idx;
+
+ for (idx = 0; idx < node->n.count; idx++)
+ {
+ if (node->chunks[idx] >= chunk)
+ break;
+ }
+
+ return idx;
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_32_SEARCH_EQ(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+ int count = node->n.count;
+#ifndef USE_NO_SIMD
+ Vector8 spread_chunk;
+ Vector8 haystack1;
+ Vector8 haystack2;
+ Vector8 cmp1;
+ Vector8 cmp2;
+ uint32 bitfield;
+ int index_simd = -1;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+ int index = -1;
+
+ for (int i = 0; i < count; i++)
+ {
+ if (node->chunks[i] == chunk)
+ {
+ index = i;
+ break;
+ }
+ }
+#endif
+
+#ifndef USE_NO_SIMD
+ /* replicate the search key */
+ spread_chunk = vector8_broadcast(chunk);
+
+ /* compare to all 32 keys stored in the node */
+ vector8_load(&haystack1, &node->chunks[0]);
+ vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+ cmp1 = vector8_eq(spread_chunk, haystack1);
+ cmp2 = vector8_eq(spread_chunk, haystack2);
+
+ /* convert comparison to a bitfield */
+ bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+
+ /* mask off invalid entries */
+ bitfield &= ((UINT64CONST(1) << count) - 1);
+
+ /* convert bitfield to index by counting trailing zeros */
+ if (bitfield)
+ index_simd = pg_rightmost_one_pos32(bitfield);
+
+ Assert(index_simd == index);
+ return index_simd;
+#else
+ return index;
+#endif
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_32_GET_INSERTPOS(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+ int count = node->n.count;
+#ifndef USE_NO_SIMD
+ Vector8 spread_chunk;
+ Vector8 haystack1;
+ Vector8 haystack2;
+ Vector8 cmp1;
+ Vector8 cmp2;
+ Vector8 min1;
+ Vector8 min2;
+ uint32 bitfield;
+ int index_simd;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+ int index;
+
+ for (index = 0; index < count; index++)
+ {
+ /*
+ * This is coded with '>=' to match what we can do with SIMD,
+ * with an assert to keep us honest.
+ */
+ if (node->chunks[index] >= chunk)
+ {
+ Assert(node->chunks[index] != chunk);
+ break;
+ }
+ }
+#endif
+
+#ifndef USE_NO_SIMD
+ /*
+ * This is a bit more complicated than RT_NODE_32_SEARCH_EQ(), because
+ * no unsigned uint8 comparison instruction exists, at least for SSE2. So
+ * we need to play some trickery using vector8_min() to effectively get
+ * >=. There'll never be any equal elements in current uses, but that's
+ * what we get here...
+ */
+ spread_chunk = vector8_broadcast(chunk);
+ vector8_load(&haystack1, &node->chunks[0]);
+ vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+ min1 = vector8_min(spread_chunk, haystack1);
+ min2 = vector8_min(spread_chunk, haystack2);
+ cmp1 = vector8_eq(spread_chunk, min1);
+ cmp2 = vector8_eq(spread_chunk, min2);
+ bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+ bitfield &= ((UINT64CONST(1) << count) - 1);
+
+ if (bitfield)
+ index_simd = pg_rightmost_one_pos32(bitfield);
+ else
+ index_simd = count;
+
+ Assert(index_simd == index);
+ return index_simd;
+#else
+ return index;
+#endif
+}
+
+
+/*
+ * Functions to manipulate both chunks array and children/values array.
+ * These are used for node-3 and node-32.
+ */
+
+/* Shift the elements right at 'idx' by one */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_SHIFT(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+ memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+ memmove(&(children[idx + 1]), &(children[idx]), sizeof(RT_PTR_ALLOC) * (count - idx));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_SHIFT(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+ memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+ memmove(&(values[idx + 1]), &(values[idx]), sizeof(RT_VALUE_TYPE) * (count - idx));
+}
+
+/* Delete the element at 'idx' */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_DELETE(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+ memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+ memmove(&(children[idx]), &(children[idx + 1]), sizeof(RT_PTR_ALLOC) * (count - idx - 1));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_DELETE(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+ memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+ memmove(&(values[idx]), &(values[idx + 1]), sizeof(RT_VALUE_TYPE) * (count - idx - 1));
+}
+
+/* Copy both chunks and children/values arrays */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_COPY(uint8 *src_chunks, RT_PTR_ALLOC *src_children,
+ uint8 *dst_chunks, RT_PTR_ALLOC *dst_children)
+{
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const Size chunk_size = sizeof(uint8) * fanout;
+ const Size children_size = sizeof(RT_PTR_ALLOC) * fanout;
+
+ memcpy(dst_chunks, src_chunks, chunk_size);
+ memcpy(dst_children, src_children, children_size);
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_COPY(uint8 *src_chunks, RT_VALUE_TYPE *src_values,
+ uint8 *dst_chunks, RT_VALUE_TYPE *dst_values)
+{
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const Size chunk_size = sizeof(uint8) * fanout;
+ const Size values_size = sizeof(RT_VALUE_TYPE) * fanout;
+
+ memcpy(dst_chunks, src_chunks, chunk_size);
+ memcpy(dst_values, src_values, values_size);
+}
+
+/* Functions to manipulate inner and leaf node-125 */
+
+/* Does the given chunk in the node has the value? */
+static inline bool
+RT_NODE_125_IS_CHUNK_USED(RT_NODE_BASE_125 *node, uint8 chunk)
+{
+ return node->slot_idxs[chunk] != RT_INVALID_SLOT_IDX;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_125_GET_CHILD(RT_NODE_INNER_125 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ return node->children[node->base.slot_idxs[chunk]];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_125_GET_VALUE(RT_NODE_LEAF_125 *node, uint8 chunk)
+{
+ Assert(RT_NODE_IS_LEAF(node));
+ Assert(((RT_NODE_BASE_125 *) node)->slot_idxs[chunk] != RT_INVALID_SLOT_IDX);
+ return node->values[node->base.slot_idxs[chunk]];
+}
+
+/* Functions to manipulate inner and leaf node-256 */
+
+/* Return true if the slot corresponding to the given chunk is in use */
+static inline bool
+RT_NODE_INNER_256_IS_CHUNK_USED(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ return node->children[chunk] != RT_INVALID_PTR_ALLOC;
+}
+
+static inline bool
+RT_NODE_LEAF_256_IS_CHUNK_USED(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ return (node->isset[idx] & ((bitmapword) 1 << bitnum)) != 0;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_256_GET_CHILD(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ Assert(RT_NODE_INNER_256_IS_CHUNK_USED(node, chunk));
+ return node->children[chunk];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_256_GET_VALUE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ Assert(RT_NODE_IS_LEAF(node));
+ Assert(RT_NODE_LEAF_256_IS_CHUNK_USED(node, chunk));
+ return node->values[chunk];
+}
+
+/* Set the child in the node-256 */
+static inline void
+RT_NODE_INNER_256_SET(RT_NODE_INNER_256 *node, uint8 chunk, RT_PTR_ALLOC child)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ node->children[chunk] = child;
+}
+
+/* Set the value in the node-256 */
+static inline void
+RT_NODE_LEAF_256_SET(RT_NODE_LEAF_256 *node, uint8 chunk, RT_VALUE_TYPE value)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ node->isset[idx] |= ((bitmapword) 1 << bitnum);
+ node->values[chunk] = value;
+}
+
+/* Set the slot at the given chunk position */
+static inline void
+RT_NODE_INNER_256_DELETE(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ node->children[chunk] = RT_INVALID_PTR_ALLOC;
+}
+
+static inline void
+RT_NODE_LEAF_256_DELETE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ node->isset[idx] &= ~((bitmapword) 1 << bitnum);
+}
+
+/*
+ * Return the largest shift that will allowing storing the given key.
+ */
+static inline int
+RT_KEY_GET_SHIFT(uint64 key)
+{
+ if (key == 0)
+ return 0;
+ else
+ return (pg_leftmost_one_pos64(key) / RT_NODE_SPAN) * RT_NODE_SPAN;
+}
+
+/*
+ * Return the max value that can be stored in the tree with the given shift.
+ */
+static uint64
+RT_SHIFT_GET_MAX_VAL(int shift)
+{
+ if (shift == RT_MAX_SHIFT)
+ return UINT64_MAX;
+
+ return (UINT64CONST(1) << (shift + RT_NODE_SPAN)) - 1;
+}
+
+/*
+ * Allocate a new node with the given node kind.
+ */
+static RT_PTR_ALLOC
+RT_ALLOC_NODE(RT_RADIX_TREE *tree, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+ RT_PTR_ALLOC allocnode;
+ size_t allocsize;
+
+ if (is_leaf)
+ allocsize = RT_SIZE_CLASS_INFO[size_class].leaf_size;
+ else
+ allocsize = RT_SIZE_CLASS_INFO[size_class].inner_size;
+
+#ifdef RT_SHMEM
+ allocnode = dsa_allocate(tree->dsa, allocsize);
+#else
+ if (is_leaf)
+ allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->leaf_slabs[size_class],
+ allocsize);
+ else
+ allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->inner_slabs[size_class],
+ allocsize);
+#endif
+
+#ifdef RT_DEBUG
+ /* update the statistics */
+ tree->ctl->cnt[size_class]++;
+#endif
+
+ return allocnode;
+}
+
+/* Initialize the node contents */
+static inline void
+RT_INIT_NODE(RT_PTR_LOCAL node, uint8 kind, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+ if (is_leaf)
+ MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].leaf_size);
+ else
+ MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].inner_size);
+
+ node->kind = kind;
+
+ if (kind == RT_NODE_KIND_256)
+ /* See comment for the RT_NODE type */
+ Assert(node->fanout == 0);
+ else
+ node->fanout = RT_SIZE_CLASS_INFO[size_class].fanout;
+
+ /* Initialize slot_idxs to invalid values */
+ if (kind == RT_NODE_KIND_125)
+ {
+ RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+
+ memset(n125->slot_idxs, RT_INVALID_SLOT_IDX, sizeof(n125->slot_idxs));
+ }
+}
+
+/*
+ * Create a new node as the root. Subordinate nodes will be created during
+ * the insertion.
+ */
+static pg_noinline void
+RT_NEW_ROOT(RT_RADIX_TREE *tree, uint64 key)
+{
+ int shift = RT_KEY_GET_SHIFT(key);
+ bool is_leaf = shift == 0;
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(newnode, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ newnode->shift = shift;
+ tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(shift);
+ tree->ctl->root = allocnode;
+}
+
+static inline void
+RT_COPY_NODE(RT_PTR_LOCAL newnode, RT_PTR_LOCAL oldnode)
+{
+ newnode->shift = oldnode->shift;
+ newnode->count = oldnode->count;
+}
+
+/*
+ * Given a new allocated node and an old node, initalize the new
+ * node with the necessary fields and return its local pointer.
+ */
+static inline RT_PTR_LOCAL
+RT_SWITCH_NODE_KIND(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, RT_PTR_LOCAL node,
+ uint8 new_kind, uint8 new_class, bool is_leaf)
+{
+ RT_PTR_LOCAL newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(newnode, new_kind, new_class, is_leaf);
+ RT_COPY_NODE(newnode, node);
+
+ return newnode;
+}
+
+/* Free the given node */
+static void
+RT_FREE_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode)
+{
+ /* If we're deleting the root node, make the tree empty */
+ if (tree->ctl->root == allocnode)
+ {
+ tree->ctl->root = RT_INVALID_PTR_ALLOC;
+ tree->ctl->max_val = 0;
+ }
+
+#ifdef RT_DEBUG
+ {
+ int i;
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+ /* update the statistics */
+ for (i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ if (node->fanout == RT_SIZE_CLASS_INFO[i].fanout)
+ break;
+ }
+
+ /* fanout of node256 is intentionally 0 */
+ if (i == RT_SIZE_CLASS_COUNT)
+ i = RT_CLASS_256;
+
+ tree->ctl->cnt[i]--;
+ Assert(tree->ctl->cnt[i] >= 0);
+ }
+#endif
+
+#ifdef RT_SHMEM
+ dsa_free(tree->dsa, allocnode);
+#else
+ pfree(allocnode);
+#endif
+}
+
+/* Update the parent's pointer when growing a node */
+static inline void
+RT_NODE_UPDATE_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC new_child)
+{
+#define RT_ACTION_UPDATE
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+#undef RT_ACTION_UPDATE
+}
+
+/*
+ * Replace old_child with new_child, and free the old one.
+ */
+static inline void
+RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
+ RT_PTR_ALLOC stored_old_child, RT_PTR_LOCAL old_child,
+ RT_PTR_ALLOC new_child, uint64 key)
+{
+#ifdef USE_ASSERT_CHECKING
+ RT_PTR_LOCAL new = RT_PTR_GET_LOCAL(tree, new_child);
+
+ Assert(old_child->shift == new->shift);
+ Assert(old_child->count == new->count);
+#endif
+
+ if (parent == old_child)
+ {
+ /* Replace the root node with the new larger node */
+ tree->ctl->root = new_child;
+ }
+ else
+ RT_NODE_UPDATE_INNER(parent, key, new_child);
+
+ RT_FREE_NODE(tree, stored_old_child);
+}
+
+/*
+ * The radix tree doesn't have sufficient height. Extend the radix tree so
+ * it can store the key.
+ */
+static pg_noinline void
+RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
+{
+ int target_shift;
+ RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+ int shift = root->shift + RT_NODE_SPAN;
+
+ target_shift = RT_KEY_GET_SHIFT(key);
+
+ /* Grow tree from 'shift' to 'target_shift' */
+ while (shift <= target_shift)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL node;
+ RT_NODE_INNER_3 *n3;
+
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, true);
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(node, RT_NODE_KIND_3, RT_CLASS_3, true);
+ node->shift = shift;
+ node->count = 1;
+
+ n3 = (RT_NODE_INNER_3 *) node;
+ n3->base.chunks[0] = 0;
+ n3->children[0] = tree->ctl->root;
+
+ /* Update the root */
+ tree->ctl->root = allocnode;
+
+ shift += RT_NODE_SPAN;
+ }
+
+ tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(target_shift);
+}
+
+/*
+ * The radix tree doesn't have inner and leaf nodes for given key-value pair.
+ * Insert inner and leaf nodes from 'node' to bottom.
+ */
+static pg_noinline void
+RT_SET_EXTEND(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
+ RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node)
+{
+ int shift = node->shift;
+
+ Assert(RT_PTR_GET_LOCAL(tree, stored_node) == node);
+
+ while (shift >= RT_NODE_SPAN)
+ {
+ RT_PTR_ALLOC allocchild;
+ RT_PTR_LOCAL newchild;
+ int newshift = shift - RT_NODE_SPAN;
+ bool is_leaf = newshift == 0;
+
+ allocchild = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ newchild = RT_PTR_GET_LOCAL(tree, allocchild);
+ RT_INIT_NODE(newchild, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ newchild->shift = newshift;
+ RT_NODE_INSERT_INNER(tree, parent, stored_node, node, key, allocchild);
+
+ parent = node;
+ node = newchild;
+ stored_node = allocchild;
+ shift -= RT_NODE_SPAN;
+ }
+
+ RT_NODE_INSERT_LEAF(tree, parent, stored_node, node, key, value_p);
+ tree->ctl->num_keys++;
+}
+
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the child
+ * pointer is set to child_p.
+ */
+static inline bool
+RT_NODE_SEARCH_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC *child_p)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the pointer
+ * to the value is set to value_p.
+ */
+static inline bool
+RT_NODE_SEARCH_LEAF(RT_PTR_LOCAL node, uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_INNER(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_LEAF(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+#endif
+
+/*
+ * Insert "child" into "node".
+ *
+ * "parent" is the parent of "node", so the grandparent of the child.
+ * If the node we're inserting into needs to grow, we update the parent's
+ * child pointer with the pointer to the new larger node.
+ */
+static void
+RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_PTR_ALLOC child)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/* Like RT_NODE_INSERT_INNER, but for leaf nodes */
+static bool
+RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Create the radix tree in the given memory context and return it.
+ */
+RT_SCOPE RT_RADIX_TREE *
+#ifdef RT_SHMEM
+RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id)
+#else
+RT_CREATE(MemoryContext ctx)
+#endif
+{
+ RT_RADIX_TREE *tree;
+ MemoryContext old_ctx;
+#ifdef RT_SHMEM
+ dsa_pointer dp;
+#endif
+
+ old_ctx = MemoryContextSwitchTo(ctx);
+
+ tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+ tree->context = ctx;
+
+#ifdef RT_SHMEM
+ tree->dsa = dsa;
+ dp = dsa_allocate0(dsa, sizeof(RT_RADIX_TREE_CONTROL));
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, dp);
+ tree->ctl->handle = dp;
+ tree->ctl->magic = RT_RADIX_TREE_MAGIC;
+ LWLockInitialize(&tree->ctl->lock, tranche_id);
+#else
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) palloc0(sizeof(RT_RADIX_TREE_CONTROL));
+
+ /* Create a slab context for each size class */
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ RT_SIZE_CLASS_ELEM size_class = RT_SIZE_CLASS_INFO[i];
+ size_t inner_blocksize = RT_SLAB_BLOCK_SIZE(size_class.inner_size);
+ size_t leaf_blocksize = RT_SLAB_BLOCK_SIZE(size_class.leaf_size);
+
+ tree->inner_slabs[i] = SlabContextCreate(ctx,
+ size_class.name,
+ inner_blocksize,
+ size_class.inner_size);
+ tree->leaf_slabs[i] = SlabContextCreate(ctx,
+ size_class.name,
+ leaf_blocksize,
+ size_class.leaf_size);
+ }
+#endif
+
+ tree->ctl->root = RT_INVALID_PTR_ALLOC;
+
+ MemoryContextSwitchTo(old_ctx);
+
+ return tree;
+}
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE *
+RT_ATTACH(dsa_area *dsa, RT_HANDLE handle)
+{
+ RT_RADIX_TREE *tree;
+ dsa_pointer control;
+
+ tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+
+ /* Find the control object in shard memory */
+ control = handle;
+
+ tree->dsa = dsa;
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, control);
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+ return tree;
+}
+
+RT_SCOPE void
+RT_DETACH(RT_RADIX_TREE *tree)
+{
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ pfree(tree);
+}
+
+RT_SCOPE RT_HANDLE
+RT_GET_HANDLE(RT_RADIX_TREE *tree)
+{
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ return tree->ctl->handle;
+}
+
+/*
+ * Recursively free all nodes allocated to the DSA area.
+ */
+static void
+RT_FREE_RECURSE(RT_RADIX_TREE *tree, RT_PTR_ALLOC ptr)
+{
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, ptr);
+
+ check_stack_depth();
+ CHECK_FOR_INTERRUPTS();
+
+ /* The leaf node doesn't have child pointers */
+ if (RT_NODE_IS_LEAF(node))
+ {
+ dsa_free(tree->dsa, ptr);
+ return;
+ }
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+ for (int i = 0; i < n3->base.n.count; i++)
+ RT_FREE_RECURSE(tree, n3->children[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+ for (int i = 0; i < n32->base.n.count; i++)
+ RT_FREE_RECURSE(tree, n32->children[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) node;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+ continue;
+
+ RT_FREE_RECURSE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+ }
+
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ RT_FREE_RECURSE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+ }
+
+ break;
+ }
+ }
+
+ /* Free the inner node */
+ dsa_free(tree->dsa, ptr);
+}
+#endif
+
+/*
+ * Free the given radix tree.
+ */
+RT_SCOPE void
+RT_FREE(RT_RADIX_TREE *tree)
+{
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+ /* Free all memory used for radix tree nodes */
+ if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ RT_FREE_RECURSE(tree, tree->ctl->root);
+
+ /*
+ * Vandalize the control block to help catch programming error where
+ * other backends access the memory formerly occupied by this radix tree.
+ */
+ tree->ctl->magic = 0;
+ dsa_free(tree->dsa, tree->ctl->handle);
+#else
+ pfree(tree->ctl);
+
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ MemoryContextDelete(tree->inner_slabs[i]);
+ MemoryContextDelete(tree->leaf_slabs[i]);
+ }
+#endif
+
+ pfree(tree);
+}
+
+/*
+ * Set key to value. If the entry already exists, we update its value to 'value'
+ * and return true. Returns false if entry doesn't yet exist.
+ */
+RT_SCOPE bool
+RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+ int shift;
+ bool updated;
+ RT_PTR_LOCAL parent;
+ RT_PTR_ALLOC stored_child;
+ RT_PTR_LOCAL child;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ RT_LOCK_EXCLUSIVE(tree);
+
+ /* Empty tree, create the root */
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ RT_NEW_ROOT(tree, key);
+
+ /* Extend the tree if necessary */
+ if (key > tree->ctl->max_val)
+ RT_EXTEND(tree, key);
+
+ stored_child = tree->ctl->root;
+ parent = RT_PTR_GET_LOCAL(tree, stored_child);
+ shift = parent->shift;
+
+ /* Descend the tree until we reach a leaf node */
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC new_child = RT_INVALID_PTR_ALLOC;;
+
+ child = RT_PTR_GET_LOCAL(tree, stored_child);
+
+ if (RT_NODE_IS_LEAF(child))
+ break;
+
+ if (!RT_NODE_SEARCH_INNER(child, key, &new_child))
+ {
+ RT_SET_EXTEND(tree, key, value_p, parent, stored_child, child);
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ parent = child;
+ stored_child = new_child;
+ shift -= RT_NODE_SPAN;
+ }
+
+ updated = RT_NODE_INSERT_LEAF(tree, parent, stored_child, child, key, value_p);
+
+ /* Update the statistics */
+ if (!updated)
+ tree->ctl->num_keys++;
+
+ RT_UNLOCK(tree);
+ return updated;
+}
+
+/*
+ * Search the given key in the radix tree. Return true if there is the key,
+ * otherwise return false. On success, we set the value to *value_p so it must
+ * not be NULL.
+ */
+RT_SCOPE bool
+RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+ RT_PTR_LOCAL node;
+ int shift;
+ bool found;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+ Assert(value_p != NULL);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ node = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+ shift = node->shift;
+
+ /* Descend the tree until a leaf node */
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+ if (RT_NODE_IS_LEAF(node))
+ break;
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ node = RT_PTR_GET_LOCAL(tree, child);
+ shift -= RT_NODE_SPAN;
+ }
+
+ found = RT_NODE_SEARCH_LEAF(node, key, value_p);
+
+ RT_UNLOCK(tree);
+ return found;
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Delete the given key from the radix tree. Return true if the key is found (and
+ * deleted), otherwise do nothing and return false.
+ */
+RT_SCOPE bool
+RT_DELETE(RT_RADIX_TREE *tree, uint64 key)
+{
+ RT_PTR_LOCAL node;
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_ALLOC stack[RT_MAX_LEVEL] = {0};
+ int shift;
+ int level;
+ bool deleted;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ RT_LOCK_EXCLUSIVE(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ /*
+ * Descend the tree to search the key while building a stack of nodes we
+ * visited.
+ */
+ allocnode = tree->ctl->root;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift = node->shift;
+ level = -1;
+ while (shift > 0)
+ {
+ RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+ /* Push the current node to the stack */
+ stack[++level] = allocnode;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ allocnode = child;
+ shift -= RT_NODE_SPAN;
+ }
+
+ /* Delete the key from the leaf node if exists */
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ deleted = RT_NODE_DELETE_LEAF(node, key);
+
+ if (!deleted)
+ {
+ /* no key is found in the leaf node */
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ /* Found the key to delete. Update the statistics */
+ tree->ctl->num_keys--;
+
+ /*
+ * Return if the leaf node still has keys and we don't need to delete the
+ * node.
+ */
+ if (node->count > 0)
+ {
+ RT_UNLOCK(tree);
+ return true;
+ }
+
+ /* Free the empty leaf node */
+ RT_FREE_NODE(tree, allocnode);
+
+ /* Delete the key in inner nodes recursively */
+ while (level >= 0)
+ {
+ allocnode = stack[level--];
+
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ deleted = RT_NODE_DELETE_INNER(node, key);
+ Assert(deleted);
+
+ /* If the node didn't become empty, we stop deleting the key */
+ if (node->count > 0)
+ break;
+
+ /* The node became empty */
+ RT_FREE_NODE(tree, allocnode);
+ }
+
+ RT_UNLOCK(tree);
+ return true;
+}
+#endif
+
+static inline void
+RT_ITER_UPDATE_KEY(RT_ITER *iter, uint8 chunk, uint8 shift)
+{
+ iter->key &= ~(((uint64) RT_CHUNK_MASK) << shift);
+ iter->key |= (((uint64) chunk) << shift);
+}
+
+/*
+ * Advance the slot in the inner node. Return the child if exists, otherwise
+ * null.
+ */
+static inline RT_PTR_LOCAL
+RT_NODE_INNER_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Advance the slot in the leaf node. On success, return true and the value
+ * is set to value_p, otherwise return false.
+ */
+static inline bool
+RT_NODE_LEAF_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter,
+ RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Update each node_iter for inner nodes in the iterator node stack.
+ */
+static void
+RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
+{
+ int level = from;
+ RT_PTR_LOCAL node = from_node;
+
+ for (;;)
+ {
+ RT_NODE_ITER *node_iter = &(iter->stack[level--]);
+
+ node_iter->node = node;
+ node_iter->current_idx = -1;
+
+ /* We don't advance the leaf node iterator here */
+ if (RT_NODE_IS_LEAF(node))
+ return;
+
+ /* Advance to the next slot in the inner node */
+ node = RT_NODE_INNER_ITERATE_NEXT(iter, node_iter);
+
+ /* We must find the first children in the node */
+ Assert(node);
+ }
+}
+
+/*
+ * Create and return the iterator for the given radix tree.
+ *
+ * The radix tree is locked in shared mode during the iteration, so
+ * RT_END_ITERATE needs to be called when finished to release the lock.
+ */
+RT_SCOPE RT_ITER *
+RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
+{
+ MemoryContext old_ctx;
+ RT_ITER *iter;
+ RT_PTR_LOCAL root;
+ int top_level;
+
+ old_ctx = MemoryContextSwitchTo(tree->context);
+
+ iter = (RT_ITER *) palloc0(sizeof(RT_ITER));
+ iter->tree = tree;
+
+ RT_LOCK_SHARED(tree);
+
+ /* empty tree */
+ if (!iter->tree->ctl->root)
+ return iter;
+
+ root = RT_PTR_GET_LOCAL(tree, iter->tree->ctl->root);
+ top_level = root->shift / RT_NODE_SPAN;
+ iter->stack_len = top_level;
+
+ /*
+ * Descend to the left most leaf node from the root. The key is being
+ * constructed while descending to the leaf.
+ */
+ RT_UPDATE_ITER_STACK(iter, root, top_level);
+
+ MemoryContextSwitchTo(old_ctx);
+
+ return iter;
+}
+
+/*
+ * Return true with setting key_p and value_p if there is next key. Otherwise
+ * return false.
+ */
+RT_SCOPE bool
+RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p)
+{
+ /* Empty tree */
+ if (!iter->tree->ctl->root)
+ return false;
+
+ for (;;)
+ {
+ RT_PTR_LOCAL child = NULL;
+ RT_VALUE_TYPE value;
+ int level;
+ bool found;
+
+ /* Advance the leaf node iterator to get next key-value pair */
+ found = RT_NODE_LEAF_ITERATE_NEXT(iter, &(iter->stack[0]), &value);
+
+ if (found)
+ {
+ *key_p = iter->key;
+ *value_p = value;
+ return true;
+ }
+
+ /*
+ * We've visited all values in the leaf node, so advance inner node
+ * iterators from the level=1 until we find the next child node.
+ */
+ for (level = 1; level <= iter->stack_len; level++)
+ {
+ child = RT_NODE_INNER_ITERATE_NEXT(iter, &(iter->stack[level]));
+
+ if (child)
+ break;
+ }
+
+ /* the iteration finished */
+ if (!child)
+ return false;
+
+ /*
+ * Set the node to the node iterator and update the iterator stack
+ * from this node.
+ */
+ RT_UPDATE_ITER_STACK(iter, child, level - 1);
+
+ /* Node iterators are updated, so try again from the leaf */
+ }
+
+ return false;
+}
+
+/*
+ * Terminate the iteration and release the lock.
+ *
+ * This function needs to be called after finishing or when exiting an
+ * iteration.
+ */
+RT_SCOPE void
+RT_END_ITERATE(RT_ITER *iter)
+{
+#ifdef RT_SHMEM
+ Assert(LWLockHeldByMe(&iter->tree->ctl->lock));
+#endif
+
+ RT_UNLOCK(iter->tree);
+ pfree(iter);
+}
+
+/*
+ * Return the statistics of the amount of memory used by the radix tree.
+ */
+RT_SCOPE uint64
+RT_MEMORY_USAGE(RT_RADIX_TREE *tree)
+{
+ Size total = 0;
+
+ RT_LOCK_SHARED(tree);
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ total = dsa_get_total_size(tree->dsa);
+#else
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ total += MemoryContextMemAllocated(tree->inner_slabs[i], true);
+ total += MemoryContextMemAllocated(tree->leaf_slabs[i], true);
+ }
+#endif
+
+ RT_UNLOCK(tree);
+ return total;
+}
+
+/*
+ * Verify the radix tree node.
+ */
+static void
+RT_VERIFY_NODE(RT_PTR_LOCAL node)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(node->count >= 0);
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE_BASE_3 *n3 = (RT_NODE_BASE_3 *) node;
+
+ for (int i = 1; i < n3->n.count; i++)
+ Assert(n3->chunks[i - 1] < n3->chunks[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE_BASE_32 *n32 = (RT_NODE_BASE_32 *) node;
+
+ for (int i = 1; i < n32->n.count; i++)
+ Assert(n32->chunks[i - 1] < n32->chunks[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+ int cnt = 0;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ uint8 slot = n125->slot_idxs[i];
+ int idx = RT_BM_IDX(slot);
+ int bitnum = RT_BM_BIT(slot);
+
+ if (!RT_NODE_125_IS_CHUNK_USED(n125, i))
+ continue;
+
+ /* Check if the corresponding slot is used */
+ Assert(slot < node->fanout);
+ Assert((n125->isset[idx] & ((bitmapword) 1 << bitnum)) != 0);
+
+ cnt++;
+ }
+
+ Assert(n125->n.count == cnt);
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+ int cnt = 0;
+
+ for (int i = 0; i < RT_BM_IDX(RT_NODE_MAX_SLOTS); i++)
+ cnt += bmw_popcount(n256->isset[i]);
+
+ /* Check if the number of used chunk matches */
+ Assert(n256->base.n.count == cnt);
+
+ break;
+ }
+ }
+ }
+#endif
+}
+
+/***************** DEBUG FUNCTIONS *****************/
+#ifdef RT_DEBUG
+
+#define RT_UINT64_FORMAT_HEX "%" INT64_MODIFIER "X"
+
+RT_SCOPE void
+RT_STATS(RT_RADIX_TREE *tree)
+{
+ RT_LOCK_SHARED(tree);
+
+ fprintf(stderr, "max_val = " UINT64_FORMAT "\n", tree->ctl->max_val);
+ fprintf(stderr, "num_keys = " UINT64_FORMAT "\n", tree->ctl->num_keys);
+
+#ifdef RT_SHMEM
+ fprintf(stderr, "handle = " UINT64_FORMAT "\n", tree->ctl->handle);
+#endif
+
+ if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+
+ fprintf(stderr, "height = %d, n3 = %u, n15 = %u, n32 = %u, n125 = %u, n256 = %u\n",
+ root->shift / RT_NODE_SPAN,
+ tree->ctl->cnt[RT_CLASS_3],
+ tree->ctl->cnt[RT_CLASS_32_MIN],
+ tree->ctl->cnt[RT_CLASS_32_MAX],
+ tree->ctl->cnt[RT_CLASS_125],
+ tree->ctl->cnt[RT_CLASS_256]);
+ }
+
+ RT_UNLOCK(tree);
+}
+
+static void
+RT_DUMP_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, int level,
+ bool recurse, StringInfo buf)
+{
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+ StringInfoData spaces;
+
+ initStringInfo(&spaces);
+ appendStringInfoSpaces(&spaces, (level * 4) + 1);
+
+ appendStringInfo(buf, "%s%s[%s] kind %d, fanout %d, count %u, shift %u:\n",
+ spaces.data,
+ level == 0 ? "" : "-> ",
+ RT_NODE_IS_LEAF(node) ? "LEAF" : "INNR",
+ (node->kind == RT_NODE_KIND_3) ? 3 :
+ (node->kind == RT_NODE_KIND_32) ? 32 :
+ (node->kind == RT_NODE_KIND_125) ? 125 : 256,
+ node->fanout == 0 ? 256 : node->fanout,
+ node->count, node->shift);
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ for (int i = 0; i < node->count; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_3 *n3 = (RT_NODE_LEAF_3 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+ spaces.data, i, n3->base.chunks[i]);
+ }
+ else
+ {
+ RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X",
+ spaces.data, i, n3->base.chunks[i]);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, n3->children[i], level + 1,
+ recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ for (int i = 0; i < node->count; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_32 *n32 = (RT_NODE_LEAF_32 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+ spaces.data, i, n32->base.chunks[i]);
+ }
+ else
+ {
+ RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X",
+ spaces.data, i, n32->base.chunks[i]);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, n32->children[i], level + 1,
+ recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_BASE_125 *b125 = (RT_NODE_BASE_125 *) node;
+ char *sep = "";
+
+ appendStringInfo(buf, "%sslot_idxs: ", spaces.data);
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+ continue;
+
+ appendStringInfo(buf, "%s[%d]=%d ",
+ sep, i, b125->slot_idxs[i]);
+ sep = ",";
+ }
+
+ appendStringInfo(buf, "\n%sisset-bitmap: ", spaces.data);
+ for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+ appendStringInfo(buf, "%X ", ((uint8 *) b125->isset)[i]);
+ appendStringInfo(buf, "\n");
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+ continue;
+
+ if (RT_NODE_IS_LEAF(node))
+ appendStringInfo(buf, "%schunk 0x%X\n",
+ spaces.data, i);
+ else
+ {
+ RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) b125;
+
+ appendStringInfo(buf, "%schunk 0x%X",
+ spaces.data, i);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i),
+ level + 1, recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+ appendStringInfo(buf, "%sisset-bitmap: ", spaces.data);
+ for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+ appendStringInfo(buf, "%X ", ((uint8 *) n256->isset)[i]);
+ appendStringInfo(buf, "\n");
+ }
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ appendStringInfo(buf, "%schunk 0x%X\n",
+ spaces.data, i);
+ }
+ else
+ {
+ RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ appendStringInfo(buf, "%schunk 0x%X",
+ spaces.data, i);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i),
+ level + 1, recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ }
+}
+
+RT_SCOPE void
+RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key)
+{
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL node;
+ StringInfoData buf;
+ int shift;
+ int level = 0;
+
+ RT_STATS(tree);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "empty tree\n");
+ return;
+ }
+
+ if (key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "key " UINT64_FORMAT "(0x" RT_UINT64_FORMAT_HEX ") is larger than max val\n",
+ key, key);
+ return;
+ }
+
+ initStringInfo(&buf);
+ allocnode = tree->ctl->root;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift = node->shift;
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC child;
+
+ RT_DUMP_NODE(tree, allocnode, level, false, &buf);
+
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_VALUE_TYPE dummy;
+
+ /* We reached at a leaf node, find the corresponding slot */
+ RT_NODE_SEARCH_LEAF(node, key, &dummy);
+
+ break;
+ }
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ break;
+
+ allocnode = child;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift -= RT_NODE_SPAN;
+ level++;
+ }
+ RT_UNLOCK(tree);
+
+ fprintf(stderr, "%s", buf.data);
+}
+
+RT_SCOPE void
+RT_DUMP(RT_RADIX_TREE *tree)
+{
+ StringInfoData buf;
+
+ RT_STATS(tree);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "empty tree\n");
+ return;
+ }
+
+ initStringInfo(&buf);
+
+ RT_DUMP_NODE(tree, tree->ctl->root, 0, true, &buf);
+ RT_UNLOCK(tree);
+
+ fprintf(stderr, "%s",buf.data);
+}
+#endif
+
+#endif /* RT_DEFINE */
+
+
+/* undefine external parameters, so next radix tree can be defined */
+#undef RT_PREFIX
+#undef RT_SCOPE
+#undef RT_DECLARE
+#undef RT_DEFINE
+#undef RT_VALUE_TYPE
+
+/* locally declared macros */
+#undef RT_MAKE_PREFIX
+#undef RT_MAKE_NAME
+#undef RT_MAKE_NAME_
+#undef RT_NODE_SPAN
+#undef RT_NODE_MAX_SLOTS
+#undef RT_CHUNK_MASK
+#undef RT_MAX_SHIFT
+#undef RT_MAX_LEVEL
+#undef RT_GET_KEY_CHUNK
+#undef RT_BM_IDX
+#undef RT_BM_BIT
+#undef RT_LOCK_EXCLUSIVE
+#undef RT_LOCK_SHARED
+#undef RT_UNLOCK
+#undef RT_NODE_IS_LEAF
+#undef RT_NODE_MUST_GROW
+#undef RT_NODE_KIND_COUNT
+#undef RT_SIZE_CLASS_COUNT
+#undef RT_SLOT_IDX_LIMIT
+#undef RT_INVALID_SLOT_IDX
+#undef RT_SLAB_BLOCK_SIZE
+#undef RT_RADIX_TREE_MAGIC
+#undef RT_UINT64_FORMAT_HEX
+
+/* type declarations */
+#undef RT_RADIX_TREE
+#undef RT_RADIX_TREE_CONTROL
+#undef RT_PTR_LOCAL
+#undef RT_PTR_ALLOC
+#undef RT_INVALID_PTR_ALLOC
+#undef RT_HANDLE
+#undef RT_ITER
+#undef RT_NODE
+#undef RT_NODE_ITER
+#undef RT_NODE_KIND_3
+#undef RT_NODE_KIND_32
+#undef RT_NODE_KIND_125
+#undef RT_NODE_KIND_256
+#undef RT_NODE_BASE_3
+#undef RT_NODE_BASE_32
+#undef RT_NODE_BASE_125
+#undef RT_NODE_BASE_256
+#undef RT_NODE_INNER_3
+#undef RT_NODE_INNER_32
+#undef RT_NODE_INNER_125
+#undef RT_NODE_INNER_256
+#undef RT_NODE_LEAF_3
+#undef RT_NODE_LEAF_32
+#undef RT_NODE_LEAF_125
+#undef RT_NODE_LEAF_256
+#undef RT_SIZE_CLASS
+#undef RT_SIZE_CLASS_ELEM
+#undef RT_SIZE_CLASS_INFO
+#undef RT_CLASS_3
+#undef RT_CLASS_32_MIN
+#undef RT_CLASS_32_MAX
+#undef RT_CLASS_125
+#undef RT_CLASS_256
+
+/* function declarations */
+#undef RT_CREATE
+#undef RT_FREE
+#undef RT_ATTACH
+#undef RT_DETACH
+#undef RT_GET_HANDLE
+#undef RT_SEARCH
+#undef RT_SET
+#undef RT_BEGIN_ITERATE
+#undef RT_ITERATE_NEXT
+#undef RT_END_ITERATE
+#undef RT_USE_DELETE
+#undef RT_DELETE
+#undef RT_MEMORY_USAGE
+#undef RT_DUMP
+#undef RT_DUMP_NODE
+#undef RT_DUMP_SEARCH
+#undef RT_STATS
+
+/* internal helper functions */
+#undef RT_NEW_ROOT
+#undef RT_ALLOC_NODE
+#undef RT_INIT_NODE
+#undef RT_FREE_NODE
+#undef RT_FREE_RECURSE
+#undef RT_EXTEND
+#undef RT_SET_EXTEND
+#undef RT_SWITCH_NODE_KIND
+#undef RT_COPY_NODE
+#undef RT_REPLACE_NODE
+#undef RT_PTR_GET_LOCAL
+#undef RT_PTR_ALLOC_IS_VALID
+#undef RT_NODE_3_SEARCH_EQ
+#undef RT_NODE_32_SEARCH_EQ
+#undef RT_NODE_3_GET_INSERTPOS
+#undef RT_NODE_32_GET_INSERTPOS
+#undef RT_CHUNK_CHILDREN_ARRAY_SHIFT
+#undef RT_CHUNK_VALUES_ARRAY_SHIFT
+#undef RT_CHUNK_CHILDREN_ARRAY_DELETE
+#undef RT_CHUNK_VALUES_ARRAY_DELETE
+#undef RT_CHUNK_CHILDREN_ARRAY_COPY
+#undef RT_CHUNK_VALUES_ARRAY_COPY
+#undef RT_NODE_125_IS_CHUNK_USED
+#undef RT_NODE_INNER_125_GET_CHILD
+#undef RT_NODE_LEAF_125_GET_VALUE
+#undef RT_NODE_INNER_256_IS_CHUNK_USED
+#undef RT_NODE_LEAF_256_IS_CHUNK_USED
+#undef RT_NODE_INNER_256_GET_CHILD
+#undef RT_NODE_LEAF_256_GET_VALUE
+#undef RT_NODE_INNER_256_SET
+#undef RT_NODE_LEAF_256_SET
+#undef RT_NODE_INNER_256_DELETE
+#undef RT_NODE_LEAF_256_DELETE
+#undef RT_KEY_GET_SHIFT
+#undef RT_SHIFT_GET_MAX_VAL
+#undef RT_NODE_SEARCH_INNER
+#undef RT_NODE_SEARCH_LEAF
+#undef RT_NODE_UPDATE_INNER
+#undef RT_NODE_DELETE_INNER
+#undef RT_NODE_DELETE_LEAF
+#undef RT_NODE_INSERT_INNER
+#undef RT_NODE_INSERT_LEAF
+#undef RT_NODE_INNER_ITERATE_NEXT
+#undef RT_NODE_LEAF_ITERATE_NEXT
+#undef RT_UPDATE_ITER_STACK
+#undef RT_ITER_UPDATE_KEY
+#undef RT_VERIFY_NODE
+
+#undef RT_DEBUG
diff --git a/src/include/lib/radixtree_delete_impl.h b/src/include/lib/radixtree_delete_impl.h
new file mode 100644
index 0000000000..5f6dda1f12
--- /dev/null
+++ b/src/include/lib/radixtree_delete_impl.h
@@ -0,0 +1,122 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_delete_impl.h
+ * Common implementation for deletion in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ * TODO: Shrink nodes when deletion would allow them to fit in a smaller
+ * size class.
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_delete_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_DELETE(n3->base.chunks, n3->values,
+ n3->base.n.count, idx);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_DELETE(n3->base.chunks, n3->children,
+ n3->base.n.count, idx);
+#endif
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_DELETE(n32->base.chunks, n32->values,
+ n32->base.n.count, idx);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_DELETE(n32->base.chunks, n32->children,
+ n32->base.n.count, idx);
+#endif
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+ int idx;
+ int bitnum;
+
+ if (slotpos == RT_INVALID_SLOT_IDX)
+ return false;
+
+ idx = RT_BM_IDX(slotpos);
+ bitnum = RT_BM_BIT(slotpos);
+ n125->base.isset[idx] &= ~((bitmapword) 1 << bitnum);
+ n125->base.slot_idxs[chunk] = RT_INVALID_SLOT_IDX;
+
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_DELETE(n256, chunk);
+#else
+ RT_NODE_INNER_256_DELETE(n256, chunk);
+#endif
+ break;
+ }
+ }
+
+ /* update statistics */
+ node->count--;
+
+ return true;
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
new file mode 100644
index 0000000000..d56e58dcac
--- /dev/null
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -0,0 +1,328 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_insert_impl.h
+ * Common implementation for insertion in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_insert_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ const bool is_leaf = true;
+ bool chunk_exists = false;
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+ const bool is_leaf = false;
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ int idx = RT_NODE_3_SEARCH_EQ(&n3->base, chunk);
+
+ if (idx != -1)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n3->values[idx] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n3)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE32_TYPE *new32;
+ const uint8 new_kind = RT_NODE_KIND_32;
+ const RT_SIZE_CLASS new_class = RT_CLASS_32_MIN;
+
+ /* grow node from 3 to 32 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_COPY(n3->base.chunks, n3->values,
+ new32->base.chunks, new32->values);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_COPY(n3->base.chunks, n3->children,
+ new32->base.chunks, new32->children);
+#endif
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
+ int count = n3->base.n.count;
+
+ /* shift chunks and children */
+ if (insertpos < count)
+ {
+ Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_SHIFT(n3->base.chunks, n3->values,
+ count, insertpos);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_SHIFT(n3->base.chunks, n3->children,
+ count, insertpos);
+#endif
+ }
+
+ n3->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+ n3->values[insertpos] = *value_p;
+#else
+ n3->children[insertpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_32:
+ {
+ const RT_SIZE_CLASS_ELEM class32_max = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX];
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ int idx = RT_NODE_32_SEARCH_EQ(&n32->base, chunk);
+
+ if (idx != -1)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n32->values[idx] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n32)) &&
+ n32->base.n.fanout < class32_max.fanout)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ const RT_SIZE_CLASS_ELEM class32_min = RT_SIZE_CLASS_INFO[RT_CLASS_32_MIN];
+ const RT_SIZE_CLASS new_class = RT_CLASS_32_MAX;
+
+ Assert(n32->base.n.fanout == class32_min.fanout);
+
+ /* grow to the next size class of this kind */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ n32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ memcpy(newnode, node, class32_min.leaf_size);
+#else
+ memcpy(newnode, node, class32_min.inner_size);
+#endif
+ newnode->fanout = class32_max.fanout;
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+
+ if (unlikely(RT_NODE_MUST_GROW(n32)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE125_TYPE *new125;
+ const uint8 new_kind = RT_NODE_KIND_125;
+ const RT_SIZE_CLASS new_class = RT_CLASS_125;
+
+ Assert(n32->base.n.fanout == class32_max.fanout);
+
+ /* grow node from 32 to 125 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new125 = (RT_NODE125_TYPE *) newnode;
+
+ for (int i = 0; i < class32_max.fanout; i++)
+ {
+ new125->base.slot_idxs[n32->base.chunks[i]] = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ new125->values[i] = n32->values[i];
+#else
+ new125->children[i] = n32->children[i];
+#endif
+ }
+
+ /*
+ * Since we just copied a dense array, we can set the bits
+ * using a single store, provided the length of that array
+ * is at most the number of bits in a bitmapword.
+ */
+ Assert(class32_max.fanout <= sizeof(bitmapword) * BITS_PER_BYTE);
+ new125->base.isset[0] = (bitmapword) (((uint64) 1 << class32_max.fanout) - 1);
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
+ int count = n32->base.n.count;
+
+ if (insertpos < count)
+ {
+ Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_SHIFT(n32->base.chunks, n32->values,
+ count, insertpos);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_SHIFT(n32->base.chunks, n32->children,
+ count, insertpos);
+#endif
+ }
+
+ n32->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+ n32->values[insertpos] = *value_p;
+#else
+ n32->children[insertpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos;
+ int cnt = 0;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ slotpos = n125->base.slot_idxs[chunk];
+ if (slotpos != RT_INVALID_SLOT_IDX)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n125->values[slotpos] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n125)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE256_TYPE *new256;
+ const uint8 new_kind = RT_NODE_KIND_256;
+ const RT_SIZE_CLASS new_class = RT_CLASS_256;
+
+ /* grow node from 125 to 256 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new256 = (RT_NODE256_TYPE *) newnode;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS && cnt < n125->base.n.count; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+ continue;
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_SET(new256, i, RT_NODE_LEAF_125_GET_VALUE(n125, i));
+#else
+ RT_NODE_INNER_256_SET(new256, i, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+ cnt++;
+ }
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int idx;
+ bitmapword inverse;
+
+ /* get the first word with at least one bit not set */
+ for (idx = 0; idx < RT_BM_IDX(RT_SLOT_IDX_LIMIT); idx++)
+ {
+ if (n125->base.isset[idx] < ~((bitmapword) 0))
+ break;
+ }
+
+ /* To get the first unset bit in X, get the first set bit in ~X */
+ inverse = ~(n125->base.isset[idx]);
+ slotpos = idx * BITS_PER_BITMAPWORD;
+ slotpos += bmw_rightmost_one_pos(inverse);
+ Assert(slotpos < node->fanout);
+
+ /* mark the slot used */
+ n125->base.isset[idx] |= bmw_rightmost_one(inverse);
+ n125->base.slot_idxs[chunk] = slotpos;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ n125->values[slotpos] = *value_p;
+#else
+ n125->children[slotpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ chunk_exists = RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk);
+ Assert(chunk_exists || node->count < RT_NODE_MAX_SLOTS);
+ RT_NODE_LEAF_256_SET(n256, chunk, *value_p);
+#else
+ Assert(node->count < RT_NODE_MAX_SLOTS);
+ RT_NODE_INNER_256_SET(n256, chunk, child);
+#endif
+ break;
+ }
+ }
+
+ /* Update statistics */
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!chunk_exists)
+ node->count++;
+#else
+ node->count++;
+#endif
+
+ /*
+ * Done. Finally, verify the chunk and value is inserted or replaced
+ * properly in the node.
+ */
+ RT_VERIFY_NODE(node);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ return chunk_exists;
+#else
+ return;
+#endif
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_iter_impl.h b/src/include/lib/radixtree_iter_impl.h
new file mode 100644
index 0000000000..98c78eb237
--- /dev/null
+++ b/src/include/lib/radixtree_iter_impl.h
@@ -0,0 +1,153 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_iter_impl.h
+ * Common implementation for iteration in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_iter_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ bool found = false;
+ uint8 key_chunk;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_VALUE_TYPE value;
+
+ Assert(RT_NODE_IS_LEAF(node_iter->node));
+#else
+ RT_PTR_LOCAL child = NULL;
+
+ Assert(!RT_NODE_IS_LEAF(node_iter->node));
+#endif
+
+#ifdef RT_SHMEM
+ Assert(iter->tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ switch (node_iter->node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node_iter->node;
+
+ node_iter->current_idx++;
+ if (node_iter->current_idx >= n3->base.n.count)
+ break;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = n3->values[node_iter->current_idx];
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, n3->children[node_iter->current_idx]);
+#endif
+ key_chunk = n3->base.chunks[node_iter->current_idx];
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node_iter->node;
+
+ node_iter->current_idx++;
+ if (node_iter->current_idx >= n32->base.n.count)
+ break;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ value = n32->values[node_iter->current_idx];
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, n32->children[node_iter->current_idx]);
+#endif
+ key_chunk = n32->base.chunks[node_iter->current_idx];
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node_iter->node;
+ int i;
+
+ for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (RT_NODE_125_IS_CHUNK_USED((RT_NODE_BASE_125 *) n125, i))
+ break;
+ }
+
+ if (i >= RT_NODE_MAX_SLOTS)
+ break;
+
+ node_iter->current_idx = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = RT_NODE_LEAF_125_GET_VALUE(n125, i);
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+ key_chunk = i;
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node_iter->node;
+ int i;
+
+ for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ {
+#ifdef RT_NODE_LEVEL_LEAF
+ if (RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+#else
+ if (RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+#endif
+ break;
+ }
+
+ if (i >= RT_NODE_MAX_SLOTS)
+ break;
+
+ node_iter->current_idx = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = RT_NODE_LEAF_256_GET_VALUE(n256, i);
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+#endif
+ key_chunk = i;
+ found = true;
+ break;
+ }
+ }
+
+ if (found)
+ {
+ RT_ITER_UPDATE_KEY(iter, key_chunk, node_iter->node->shift);
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = value;
+#endif
+ }
+
+#ifdef RT_NODE_LEVEL_LEAF
+ return found;
+#else
+ return child;
+#endif
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_search_impl.h b/src/include/lib/radixtree_search_impl.h
new file mode 100644
index 0000000000..a8925c75d0
--- /dev/null
+++ b/src/include/lib/radixtree_search_impl.h
@@ -0,0 +1,138 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_search_impl.h
+ * Common implementation for search in leaf and inner nodes, plus
+ * update for inner nodes only.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_search_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(value_p != NULL);
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+#ifndef RT_ACTION_UPDATE
+ Assert(child_p != NULL);
+#endif
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+#ifdef RT_ACTION_UPDATE
+ Assert(idx >= 0);
+ n3->children[idx] = new_child;
+#else
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n3->values[idx];
+#else
+ *child_p = n3->children[idx];
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+#ifdef RT_ACTION_UPDATE
+ Assert(idx >= 0);
+ n32->children[idx] = new_child;
+#else
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n32->values[idx];
+#else
+ *child_p = n32->children[idx];
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+
+#ifdef RT_ACTION_UPDATE
+ Assert(slotpos != RT_INVALID_SLOT_IDX);
+ n125->children[slotpos] = new_child;
+#else
+ if (slotpos == RT_INVALID_SLOT_IDX)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_125_GET_VALUE(n125, chunk);
+#else
+ *child_p = RT_NODE_INNER_125_GET_CHILD(n125, chunk);
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_ACTION_UPDATE
+ RT_NODE_INNER_256_SET(n256, chunk, new_child);
+#else
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_256_GET_VALUE(n256, chunk);
+#else
+ *child_p = RT_NODE_INNER_256_GET_CHILD(n256, chunk);
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ }
+
+#ifdef RT_ACTION_UPDATE
+ return;
+#else
+ return true;
+#endif /* RT_ACTION_UPDATE */
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/utils/dsa.h b/src/include/utils/dsa.h
index 3ce4ee300a..2af215484f 100644
--- a/src/include/utils/dsa.h
+++ b/src/include/utils/dsa.h
@@ -121,6 +121,7 @@ extern dsa_handle dsa_get_handle(dsa_area *area);
extern dsa_pointer dsa_allocate_extended(dsa_area *area, size_t size, int flags);
extern void dsa_free(dsa_area *area, dsa_pointer dp);
extern void *dsa_get_address(dsa_area *area, dsa_pointer dp);
+extern size_t dsa_get_total_size(dsa_area *area);
extern void dsa_trim(dsa_area *area);
extern void dsa_dump(dsa_area *area);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index c629cbe383..9659eb85d7 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -28,6 +28,7 @@ SUBDIRS = \
test_pg_db_role_setting \
test_pg_dump \
test_predtest \
+ test_radixtree \
test_rbtree \
test_regex \
test_rls_hooks \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 1baa6b558d..232cbdac80 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -24,6 +24,7 @@ subdir('test_parser')
subdir('test_pg_db_role_setting')
subdir('test_pg_dump')
subdir('test_predtest')
+subdir('test_radixtree')
subdir('test_rbtree')
subdir('test_regex')
subdir('test_rls_hooks')
diff --git a/src/test/modules/test_radixtree/.gitignore b/src/test/modules/test_radixtree/.gitignore
new file mode 100644
index 0000000000..5dcb3ff972
--- /dev/null
+++ b/src/test/modules/test_radixtree/.gitignore
@@ -0,0 +1,4 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
diff --git a/src/test/modules/test_radixtree/Makefile b/src/test/modules/test_radixtree/Makefile
new file mode 100644
index 0000000000..da06b93da3
--- /dev/null
+++ b/src/test/modules/test_radixtree/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_radixtree/Makefile
+
+MODULE_big = test_radixtree
+OBJS = \
+ $(WIN32RES) \
+ test_radixtree.o
+PGFILEDESC = "test_radixtree - test code for src/backend/lib/radixtree.c"
+
+EXTENSION = test_radixtree
+DATA = test_radixtree--1.0.sql
+
+REGRESS = test_radixtree
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_radixtree
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_radixtree/README b/src/test/modules/test_radixtree/README
new file mode 100644
index 0000000000..a8b271869a
--- /dev/null
+++ b/src/test/modules/test_radixtree/README
@@ -0,0 +1,7 @@
+test_integerset contains unit tests for testing the integer set implementation
+in src/backend/lib/integerset.c.
+
+The tests verify the correctness of the implementation, but they can also be
+used as a micro-benchmark. If you set the 'intset_test_stats' flag in
+test_integerset.c, the tests will print extra information about execution time
+and memory usage.
diff --git a/src/test/modules/test_radixtree/expected/test_radixtree.out b/src/test/modules/test_radixtree/expected/test_radixtree.out
new file mode 100644
index 0000000000..ce645cb8b5
--- /dev/null
+++ b/src/test/modules/test_radixtree/expected/test_radixtree.out
@@ -0,0 +1,36 @@
+CREATE EXTENSION test_radixtree;
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
+NOTICE: testing basic operations with leaf node 4
+NOTICE: testing basic operations with inner node 4
+NOTICE: testing basic operations with leaf node 32
+NOTICE: testing basic operations with inner node 32
+NOTICE: testing basic operations with leaf node 125
+NOTICE: testing basic operations with inner node 125
+NOTICE: testing basic operations with leaf node 256
+NOTICE: testing basic operations with inner node 256
+NOTICE: testing radix tree node types with shift "0"
+NOTICE: testing radix tree node types with shift "8"
+NOTICE: testing radix tree node types with shift "16"
+NOTICE: testing radix tree node types with shift "24"
+NOTICE: testing radix tree node types with shift "32"
+NOTICE: testing radix tree node types with shift "40"
+NOTICE: testing radix tree node types with shift "48"
+NOTICE: testing radix tree node types with shift "56"
+NOTICE: testing radix tree with pattern "all ones"
+NOTICE: testing radix tree with pattern "alternating bits"
+NOTICE: testing radix tree with pattern "clusters of ten"
+NOTICE: testing radix tree with pattern "clusters of hundred"
+NOTICE: testing radix tree with pattern "one-every-64k"
+NOTICE: testing radix tree with pattern "sparse"
+NOTICE: testing radix tree with pattern "single values, distance > 2^32"
+NOTICE: testing radix tree with pattern "clusters, distance > 2^32"
+NOTICE: testing radix tree with pattern "clusters, distance > 2^60"
+ test_radixtree
+----------------
+
+(1 row)
+
diff --git a/src/test/modules/test_radixtree/meson.build b/src/test/modules/test_radixtree/meson.build
new file mode 100644
index 0000000000..6add06bbdb
--- /dev/null
+++ b/src/test/modules/test_radixtree/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_radixtree_sources = files(
+ 'test_radixtree.c',
+)
+
+if host_system == 'windows'
+ test_radixtree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_radixtree',
+ '--FILEDESC', 'test_radixtree - test code for src/include//lib/radixtree.h',])
+endif
+
+test_radixtree = shared_module('test_radixtree',
+ test_radixtree_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += test_radixtree
+
+install_data(
+ 'test_radixtree.control',
+ 'test_radixtree--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'test_radixtree',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'test_radixtree',
+ ],
+ },
+}
diff --git a/src/test/modules/test_radixtree/sql/test_radixtree.sql b/src/test/modules/test_radixtree/sql/test_radixtree.sql
new file mode 100644
index 0000000000..41ece5e9f5
--- /dev/null
+++ b/src/test/modules/test_radixtree/sql/test_radixtree.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_radixtree;
+
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
diff --git a/src/test/modules/test_radixtree/test_radixtree--1.0.sql b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
new file mode 100644
index 0000000000..074a5a7ea7
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_radixtree/test_radixtree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_radixtree" to load this file. \quit
+
+CREATE FUNCTION test_radixtree()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
new file mode 100644
index 0000000000..f944945db9
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -0,0 +1,674 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_radixtree.c
+ * Test radixtree set data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_radixtree/test_radixtree.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "storage/lwlock.h"
+#include "utils/memutils.h"
+#include "utils/timestamp.h"
+
+#define UINT64_HEX_FORMAT "%" INT64_MODIFIER "X"
+
+/*
+ * The tests pass with uint32, but build with warnings because the string
+ * format expects uint64.
+ */
+typedef uint64 TestValueType;
+
+/*
+ * If you enable this, the "pattern" tests will print information about
+ * how long populating, probing, and iterating the test set takes, and
+ * how much memory the test set consumed. That can be used as
+ * micro-benchmark of various operations and input patterns (you might
+ * want to increase the number of values used in each of the test, if
+ * you do that, to reduce noise).
+ *
+ * The information is printed to the server's stderr, mostly because
+ * that's where MemoryContextStats() output goes.
+ */
+static const bool rt_test_stats = false;
+
+static int rt_node_kind_fanouts[] = {
+ 0,
+ 4, /* RT_NODE_KIND_4 */
+ 32, /* RT_NODE_KIND_32 */
+ 125, /* RT_NODE_KIND_125 */
+ 256 /* RT_NODE_KIND_256 */
+};
+/*
+ * A struct to define a pattern of integers, for use with the test_pattern()
+ * function.
+ */
+typedef struct
+{
+ char *test_name; /* short name of the test, for humans */
+ char *pattern_str; /* a bit pattern */
+ uint64 spacing; /* pattern repeats at this interval */
+ uint64 num_values; /* number of integers to set in total */
+} test_spec;
+
+/* Test patterns borrowed from test_integerset.c */
+static const test_spec test_specs[] = {
+ {
+ "all ones", "1111111111",
+ 10, 1000000
+ },
+ {
+ "alternating bits", "0101010101",
+ 10, 1000000
+ },
+ {
+ "clusters of ten", "1111111111",
+ 10000, 1000000
+ },
+ {
+ "clusters of hundred",
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111",
+ 10000, 1000000
+ },
+ {
+ "one-every-64k", "1",
+ 65536, 1000000
+ },
+ {
+ "sparse", "100000000000000000000000000000001",
+ 10000000, 1000000
+ },
+ {
+ "single values, distance > 2^32", "1",
+ UINT64CONST(10000000000), 100000
+ },
+ {
+ "clusters, distance > 2^32", "10101010",
+ UINT64CONST(10000000000), 1000000
+ },
+ {
+ "clusters, distance > 2^60", "10101010",
+ UINT64CONST(2000000000000000000),
+ 23 /* can't be much higher than this, or we
+ * overflow uint64 */
+ }
+};
+
+/* define the radix tree implementation to test */
+#define RT_PREFIX rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE TestValueType
+// WIP: compiles with warnings because rt_attach is defined but not used
+// #define RT_SHMEM
+#include "lib/radixtree.h"
+
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+ return tree->ctl->num_keys;
+}
+
+PG_MODULE_MAGIC;
+
+PG_FUNCTION_INFO_V1(test_radixtree);
+
+static void
+test_empty(void)
+{
+ rt_radix_tree *radixtree;
+ rt_iter *iter;
+ TestValueType dummy;
+ uint64 key;
+ TestValueType val;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+ dsa = dsa_create(tranche_id);
+
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ if (rt_search(radixtree, 0, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_search(radixtree, 1, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_search(radixtree, PG_UINT64_MAX, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_delete(radixtree, 0))
+ elog(ERROR, "rt_delete on empty tree returned true");
+
+ if (rt_num_entries(radixtree) != 0)
+ elog(ERROR, "rt_num_entries on empty tree return non-zero");
+
+ iter = rt_begin_iterate(radixtree);
+
+ if (rt_iterate_next(iter, &key, &val))
+ elog(ERROR, "rt_itereate_next on empty tree returned true");
+
+ rt_end_iterate(iter);
+
+ rt_free(radixtree);
+
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+static void
+test_basic(int children, bool test_inner)
+{
+ rt_radix_tree *radixtree;
+ uint64 *keys;
+ int shift = test_inner ? 8 : 0;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing basic operations with %s node %d",
+ test_inner ? "inner" : "leaf", children);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ /* prepare keys in order like 1, 32, 2, 31, 2, ... */
+ keys = palloc(sizeof(uint64) * children);
+ for (int i = 0; i < children; i++)
+ {
+ if (i % 2 == 0)
+ keys[i] = (uint64) ((i / 2) + 1) << shift;
+ else
+ keys[i] = (uint64) (children - (i / 2)) << shift;
+ }
+
+ /* insert keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (rt_set(radixtree, keys[i], (TestValueType*) &keys[i]))
+ elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+ }
+
+ /* look up keys */
+ for (int i = 0; i < children; i++)
+ {
+ TestValueType value;
+
+ if (!rt_search(radixtree, keys[i], &value))
+ elog(ERROR, "could not find key 0x" UINT64_HEX_FORMAT, keys[i]);
+ if (value != (TestValueType) keys[i])
+ elog(ERROR, "rt_search returned 0x" UINT64_HEX_FORMAT ", expected " UINT64_HEX_FORMAT,
+ value, (TestValueType) keys[i]);
+ }
+
+ /* update keys */
+ for (int i = 0; i < children; i++)
+ {
+ TestValueType update = keys[i] + 1;
+ if (!rt_set(radixtree, keys[i], (TestValueType*) &update))
+ elog(ERROR, "could not update key 0x" UINT64_HEX_FORMAT, keys[i]);
+ }
+
+ /* repeat deleting and inserting keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (!rt_delete(radixtree, keys[i]))
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, keys[i]);
+ if (rt_set(radixtree, keys[i], (TestValueType*) &keys[i]))
+ elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+ }
+
+ pfree(keys);
+ rt_free(radixtree);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Check if keys from start to end with the shift exist in the tree.
+ */
+static void
+check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end,
+ int incr)
+{
+ for (int i = start; i < end; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ TestValueType val;
+
+ if (!rt_search(radixtree, key, &val))
+ elog(ERROR, "key 0x" UINT64_HEX_FORMAT " is not found on node-%d",
+ key, end);
+ if (val != (TestValueType) key)
+ elog(ERROR, "rt_search with key 0x" UINT64_HEX_FORMAT " returns 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+ key, val, key);
+ }
+}
+
+static void
+test_node_types_insert(rt_radix_tree *radixtree, uint8 shift, bool insert_asc)
+{
+ uint64 num_entries;
+ int ninserted = 0;
+ int start = insert_asc ? 0 : 256;
+ int incr = insert_asc ? 1 : -1;
+ int end = insert_asc ? 256 : 0;
+ int node_kind_idx = 1;
+
+ for (int i = start; i != end; i += incr)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_set(radixtree, key, (TestValueType*) &key);
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " is found", key);
+
+ /*
+ * After filling all slots in each node type, check if the values
+ * are stored properly.
+ */
+ if (ninserted == rt_node_kind_fanouts[node_kind_idx] - 1)
+ {
+ int check_start = insert_asc
+ ? rt_node_kind_fanouts[node_kind_idx - 1]
+ : rt_node_kind_fanouts[node_kind_idx];
+ int check_end = insert_asc
+ ? rt_node_kind_fanouts[node_kind_idx]
+ : rt_node_kind_fanouts[node_kind_idx - 1];
+
+ check_search_on_node(radixtree, shift, check_start, check_end, incr);
+ node_kind_idx++;
+ }
+
+ ninserted++;
+ }
+
+ num_entries = rt_num_entries(radixtree);
+
+ if (num_entries != 256)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+static void
+test_node_types_delete(rt_radix_tree *radixtree, uint8 shift)
+{
+ uint64 num_entries;
+
+ for (int i = 0; i < 256; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_delete(radixtree, key);
+
+ if (!found)
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, key);
+ }
+
+ num_entries = rt_num_entries(radixtree);
+
+ /* The tree must be empty */
+ if (num_entries != 0)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+/*
+ * Test for inserting and deleting key-value pairs to each node type at the given shift
+ * level.
+ */
+static void
+test_node_types(uint8 shift)
+{
+ rt_radix_tree *radixtree;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing radix tree node types with shift \"%d\"", shift);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ /*
+ * Insert and search entries for every node type at the 'shift' level,
+ * then delete all entries to make it empty, and insert and search entries
+ * again.
+ */
+ test_node_types_insert(radixtree, shift, true);
+ test_node_types_delete(radixtree, shift);
+ test_node_types_insert(radixtree, shift, false);
+
+ rt_free(radixtree);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Test with a repeating pattern, defined by the 'spec'.
+ */
+static void
+test_pattern(const test_spec * spec)
+{
+ rt_radix_tree *radixtree;
+ rt_iter *iter;
+ MemoryContext radixtree_ctx;
+ TimestampTz starttime;
+ TimestampTz endtime;
+ uint64 n;
+ uint64 last_int;
+ uint64 ndeleted;
+ uint64 nbefore;
+ uint64 nafter;
+ int patternlen;
+ uint64 *pattern_values;
+ uint64 pattern_num_values;
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing radix tree with pattern \"%s\"", spec->test_name);
+ if (rt_test_stats)
+ fprintf(stderr, "-----\ntesting radix tree with pattern \"%s\"\n", spec->test_name);
+
+ /* Pre-process the pattern, creating an array of integers from it. */
+ patternlen = strlen(spec->pattern_str);
+ pattern_values = palloc(patternlen * sizeof(uint64));
+ pattern_num_values = 0;
+ for (int i = 0; i < patternlen; i++)
+ {
+ if (spec->pattern_str[i] == '1')
+ pattern_values[pattern_num_values++] = i;
+ }
+
+ /*
+ * Allocate the radix tree.
+ *
+ * Allocate it in a separate memory context, so that we can print its
+ * memory usage easily.
+ */
+ radixtree_ctx = AllocSetContextCreate(CurrentMemoryContext,
+ "radixtree test",
+ ALLOCSET_SMALL_SIZES);
+ MemoryContextSetIdentifier(radixtree_ctx, spec->test_name);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(radixtree_ctx, dsa, tranche_id);
+#else
+ radixtree = rt_create(radixtree_ctx);
+#endif
+
+
+ /*
+ * Add values to the set.
+ */
+ starttime = GetCurrentTimestamp();
+
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ uint64 x = 0;
+
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ bool found;
+
+ x = last_int + pattern_values[i];
+
+ found = rt_set(radixtree, x, (TestValueType*) &x);
+
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " found", x);
+
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+
+ endtime = GetCurrentTimestamp();
+
+ if (rt_test_stats)
+ fprintf(stderr, "added " UINT64_FORMAT " values in %d ms\n",
+ spec->num_values, (int) (endtime - starttime) / 1000);
+
+ /*
+ * Print stats on the amount of memory used.
+ *
+ * We print the usage reported by rt_memory_usage(), as well as the stats
+ * from the memory context. They should be in the same ballpark, but it's
+ * hard to automate testing that, so if you're making changes to the
+ * implementation, just observe that manually.
+ */
+ if (rt_test_stats)
+ {
+ uint64 mem_usage;
+
+ /*
+ * Also print memory usage as reported by rt_memory_usage(). It
+ * should be in the same ballpark as the usage reported by
+ * MemoryContextStats().
+ */
+ mem_usage = rt_memory_usage(radixtree);
+ fprintf(stderr, "rt_memory_usage() reported " UINT64_FORMAT " (%0.2f bytes / integer)\n",
+ mem_usage, (double) mem_usage / spec->num_values);
+
+ MemoryContextStats(radixtree_ctx);
+ }
+
+ /* Check that rt_num_entries works */
+ n = rt_num_entries(radixtree);
+ if (n != spec->num_values)
+ elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT, n, spec->num_values);
+
+ /*
+ * Test random-access probes with rt_search()
+ */
+ starttime = GetCurrentTimestamp();
+
+ for (n = 0; n < 100000; n++)
+ {
+ bool found;
+ bool expected;
+ uint64 x;
+ TestValueType v;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit values that are actually in the set.
+ */
+ x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+ /* Do we expect this value to be present in the set? */
+ if (x >= last_int)
+ expected = false;
+ else
+ {
+ uint64 idx = x % spec->spacing;
+
+ if (idx >= patternlen)
+ expected = false;
+ else if (spec->pattern_str[idx] == '1')
+ expected = true;
+ else
+ expected = false;
+ }
+
+ /* Is it present according to rt_search() ? */
+ found = rt_search(radixtree, x, &v);
+
+ if (found != expected)
+ elog(ERROR, "mismatch at 0x" UINT64_HEX_FORMAT ": %d vs %d", x, found, expected);
+ if (found && (v != (TestValueType) x))
+ elog(ERROR, "found 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+ v, x);
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "probed " UINT64_FORMAT " values in %d ms\n",
+ n, (int) (endtime - starttime) / 1000);
+
+ /*
+ * Test iterator
+ */
+ starttime = GetCurrentTimestamp();
+
+ iter = rt_begin_iterate(radixtree);
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ uint64 expected = last_int + pattern_values[i];
+ uint64 x;
+ TestValueType val;
+
+ if (!rt_iterate_next(iter, &x, &val))
+ break;
+
+ if (x != expected)
+ elog(ERROR,
+ "iterate returned wrong key; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d",
+ x, expected, i);
+ if (val != (TestValueType) expected)
+ elog(ERROR,
+ "iterate returned wrong value; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d", x, expected, i);
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "iterated " UINT64_FORMAT " values in %d ms\n",
+ n, (int) (endtime - starttime) / 1000);
+
+ rt_end_iterate(iter);
+
+ if (n < spec->num_values)
+ elog(ERROR, "iterator stopped short after " UINT64_FORMAT " entries, expected " UINT64_FORMAT, n, spec->num_values);
+ if (n > spec->num_values)
+ elog(ERROR, "iterator returned " UINT64_FORMAT " entries, " UINT64_FORMAT " was expected", n, spec->num_values);
+
+ /*
+ * Test random-access probes with rt_delete()
+ */
+ starttime = GetCurrentTimestamp();
+
+ nbefore = rt_num_entries(radixtree);
+ ndeleted = 0;
+ for (n = 0; n < 1; n++)
+ {
+ bool found;
+ uint64 x;
+ TestValueType v;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit values that are actually in the set.
+ */
+ x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+ /* Is it present according to rt_search() ? */
+ found = rt_search(radixtree, x, &v);
+
+ if (!found)
+ continue;
+
+ /* If the key is found, delete it and check again */
+ if (!rt_delete(radixtree, x))
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, x);
+ if (rt_search(radixtree, x, &v))
+ elog(ERROR, "found deleted key 0x" UINT64_HEX_FORMAT, x);
+ if (rt_delete(radixtree, x))
+ elog(ERROR, "deleted already-deleted key 0x" UINT64_HEX_FORMAT, x);
+
+ ndeleted++;
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "deleted " UINT64_FORMAT " values in %d ms\n",
+ ndeleted, (int) (endtime - starttime) / 1000);
+
+ nafter = rt_num_entries(radixtree);
+
+ /* Check that rt_num_entries works */
+ if ((nbefore - ndeleted) != nafter)
+ elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT "after " UINT64_FORMAT " deletion",
+ nafter, (nbefore - ndeleted), ndeleted);
+
+ rt_free(radixtree);
+ MemoryContextDelete(radixtree_ctx);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+Datum
+test_radixtree(PG_FUNCTION_ARGS)
+{
+ test_empty();
+
+ for (int i = 1; i < lengthof(rt_node_kind_fanouts); i++)
+ {
+ test_basic(rt_node_kind_fanouts[i], false);
+ test_basic(rt_node_kind_fanouts[i], true);
+ }
+
+ for (int shift = 0; shift <= (64 - 8); shift += 8)
+ test_node_types(shift);
+
+ /* Test different test patterns, with lots of entries */
+ for (int i = 0; i < lengthof(test_specs); i++)
+ test_pattern(&test_specs[i]);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_radixtree/test_radixtree.control b/src/test/modules/test_radixtree/test_radixtree.control
new file mode 100644
index 0000000000..e53f2a3e0c
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.control
@@ -0,0 +1,4 @@
+comment = 'Test code for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/test_radixtree'
+relocatable = true
diff --git a/src/tools/pginclude/cpluspluscheck b/src/tools/pginclude/cpluspluscheck
index e52fe9f509..09fa6e7432 100755
--- a/src/tools/pginclude/cpluspluscheck
+++ b/src/tools/pginclude/cpluspluscheck
@@ -101,6 +101,12 @@ do
test "$f" = src/include/nodes/nodetags.h && continue
test "$f" = src/backend/nodes/nodetags.h && continue
+ # radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+ test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+ test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+ test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+ test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
# These files are not meant to be included standalone, because
# they contain lists that might have multiple use-cases.
test "$f" = src/include/access/rmgrlist.h && continue
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index abbba7aa63..d4d2f1da03 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -96,6 +96,12 @@ do
test "$f" = src/include/nodes/nodetags.h && continue
test "$f" = src/backend/nodes/nodetags.h && continue
+ # radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+ test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+ test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+ test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+ test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
# These files are not meant to be included standalone, because
# they contain lists that might have multiple use-cases.
test "$f" = src/include/access/rmgrlist.h && continue
--
2.39.1
[text/x-patch] v27-0004-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch (35.5K, ../../CAFBsxsESfmZpWwk3XxWkpdg_DjawH2vizKW91fyawQ+wVFkOiQ@mail.gmail.com/7-v27-0004-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch)
download | inline diff:
From 1be520c83274bc3a2f068689e665c254c8e3c04e Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 23 Dec 2022 23:50:39 +0900
Subject: [PATCH v27 4/9] Add TIDStore, to store sets of TIDs (ItemPointerData)
efficiently.
The TIDStore is designed to store large sets of TIDs efficiently, and
is backed by the radix tree. A TID is encoded into 64-bit key and
value and inserted to the radix tree.
The TIDStore is not used for anything yet, aside from the test code,
but the follow up patch integrates the TIDStore with lazy vacuum,
reducing lazy vacuum memory usage and lifting the 1GB limit on its
size, by storing the list of dead TIDs more efficiently.
This includes a unit test module, in src/test/modules/test_tidstore.
---
doc/src/sgml/monitoring.sgml | 4 +
src/backend/access/common/Makefile | 1 +
src/backend/access/common/meson.build | 1 +
src/backend/access/common/tidstore.c | 685 ++++++++++++++++++
src/backend/storage/lmgr/lwlock.c | 2 +
src/include/access/tidstore.h | 49 ++
src/include/storage/lwlock.h | 1 +
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_tidstore/Makefile | 23 +
.../test_tidstore/expected/test_tidstore.out | 13 +
src/test/modules/test_tidstore/meson.build | 35 +
.../test_tidstore/sql/test_tidstore.sql | 7 +
.../test_tidstore/test_tidstore--1.0.sql | 8 +
.../modules/test_tidstore/test_tidstore.c | 195 +++++
.../test_tidstore/test_tidstore.control | 4 +
16 files changed, 1030 insertions(+)
create mode 100644 src/backend/access/common/tidstore.c
create mode 100644 src/include/access/tidstore.h
create mode 100644 src/test/modules/test_tidstore/Makefile
create mode 100644 src/test/modules/test_tidstore/expected/test_tidstore.out
create mode 100644 src/test/modules/test_tidstore/meson.build
create mode 100644 src/test/modules/test_tidstore/sql/test_tidstore.sql
create mode 100644 src/test/modules/test_tidstore/test_tidstore--1.0.sql
create mode 100644 src/test/modules/test_tidstore/test_tidstore.c
create mode 100644 src/test/modules/test_tidstore/test_tidstore.control
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index b246ddc634..e44387d2c1 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2192,6 +2192,10 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<entry>Waiting to access a shared TID bitmap during a parallel bitmap
index scan.</entry>
</row>
+ <row>
+ <entry><literal>SharedTidStore</literal></entry>
+ <entry>Waiting to access a shared TID store.</entry>
+ </row>
<row>
<entry><literal>SharedTupleStore</literal></entry>
<entry>Waiting to access a shared tuple store during parallel
diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile
index b9aff0ccfd..67b8cc6108 100644
--- a/src/backend/access/common/Makefile
+++ b/src/backend/access/common/Makefile
@@ -27,6 +27,7 @@ OBJS = \
syncscan.o \
toast_compression.o \
toast_internals.o \
+ tidstore.o \
tupconvert.o \
tupdesc.o
diff --git a/src/backend/access/common/meson.build b/src/backend/access/common/meson.build
index f5ac17b498..fce19c09ce 100644
--- a/src/backend/access/common/meson.build
+++ b/src/backend/access/common/meson.build
@@ -15,6 +15,7 @@ backend_sources += files(
'syncscan.c',
'toast_compression.c',
'toast_internals.c',
+ 'tidstore.c',
'tupconvert.c',
'tupdesc.c',
)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
new file mode 100644
index 0000000000..ff8e66936e
--- /dev/null
+++ b/src/backend/access/common/tidstore.c
@@ -0,0 +1,685 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.c
+ * Tid (ItemPointerData) storage implementation.
+ *
+ * This module provides a in-memory data structure to store Tids (ItemPointer).
+ * Internally, a tid is encoded as a pair of 64-bit key and 64-bit value, and
+ * stored in the radix tree.
+ *
+ * A TidStore can be shared among parallel worker processes by passing DSA area
+ * to tidstore_create(). Other backends can attach to the shared TidStore by
+ * tidstore_attach().
+ *
+ * Regarding the concurrency, it basically relies on the concurrency support in
+ * the radix tree, but we acquires the lock on a TidStore in some cases, for
+ * example, when to reset the store and when to access the number tids in the
+ * store (num_tids).
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/access/common/tidstore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/tidstore.h"
+#include "miscadmin.h"
+#include "port/pg_bitutils.h"
+#include "storage/lwlock.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/*
+ * For encoding purposes, tids are represented as a pair of 64-bit key and
+ * 64-bit value. First, we construct 64-bit unsigned integer by combining
+ * the block number and the offset number. The number of bits used for the
+ * offset number is specified by max_offsets in tidstore_create(). We are
+ * frugal with the bits, because smaller keys could help keeping the radix
+ * tree shallow.
+ *
+ * For example, a tid of heap with 8kB blocks uses the lowest 9 bits for
+ * the offset number and uses the next 32 bits for the block number. That
+ * is, only 41 bits are used:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ *
+ * X = bits used for offset number
+ * Y = bits used for block number
+ * u = unused bit
+ * (high on the left, low on the right)
+ *
+ * 9 bits are enough for the offset number, because MaxHeapTuplesPerPage < 2^9
+ * on 8kB blocks.
+ *
+ * The 64-bit value is the bitmap representation of the lowest 6 bits
+ * (TIDSTORE_VALUE_NBITS) of the integer, and the rest 35 bits are used
+ * as the key:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ * |----| value
+ * |---------------------------------------------| key
+ *
+ * The maximum height of the radix tree is 5 in this case.
+ *
+ * If the number of bits for offset number fits in a 64-bit value, we don't
+ * encode tids but directly use the block number and the offset number as key
+ * and value, respectively.
+ */
+#define TIDSTORE_VALUE_NBITS 6 /* log(64, 2) */
+
+/* A magic value used to identify our TidStores. */
+#define TIDSTORE_MAGIC 0x826f6a10
+
+#define RT_PREFIX local_rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE uint64
+#include "lib/radixtree.h"
+
+#define RT_PREFIX shared_rt
+#define RT_SHMEM
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE uint64
+#include "lib/radixtree.h"
+
+/* The control object for a TidStore */
+typedef struct TidStoreControl
+{
+ /* the number of tids in the store */
+ int64 num_tids;
+
+ /* These values are never changed after creation */
+ size_t max_bytes; /* the maximum bytes a TidStore can use */
+ int max_offset; /* the maximum offset number */
+ int offset_nbits; /* the number of bits required for max_offset */
+ int offset_key_nbits; /* the number of bits of a offset number
+ * used for the key */
+
+ /* The below fields are used only in shared case */
+
+ uint32 magic;
+ LWLock lock;
+
+ /* handles for TidStore and radix tree */
+ tidstore_handle handle;
+ shared_rt_handle tree_handle;
+} TidStoreControl;
+
+/* Per-backend state for a TidStore */
+struct TidStore
+{
+ /*
+ * Control object. This is allocated in DSA area 'area' in the shared
+ * case, otherwise in backend-local memory.
+ */
+ TidStoreControl *control;
+
+ /* Storage for Tids. Use either one depending on TidStoreIsShared() */
+ union
+ {
+ local_rt_radix_tree *local;
+ shared_rt_radix_tree *shared;
+ } tree;
+
+ /* DSA area for TidStore if used */
+ dsa_area *area;
+};
+#define TidStoreIsShared(ts) ((ts)->area != NULL)
+
+/* Iterator for TidStore */
+typedef struct TidStoreIter
+{
+ TidStore *ts;
+
+ /* iterator of radix tree. Use either one depending on TidStoreIsShared() */
+ union
+ {
+ shared_rt_iter *shared;
+ local_rt_iter *local;
+ } tree_iter;
+
+ /* we returned all tids? */
+ bool finished;
+
+ /* save for the next iteration */
+ uint64 next_key;
+ uint64 next_val;
+
+ /* output for the caller */
+ TidStoreIterResult result;
+} TidStoreIter;
+
+static void tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val);
+static inline BlockNumber key_get_blkno(TidStore *ts, uint64 key);
+static inline uint64 encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint32 *off);
+static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off);
+
+/*
+ * Create a TidStore. The returned object is allocated in backend-local memory.
+ * The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
+ */
+TidStore *
+tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
+{
+ TidStore *ts;
+
+ ts = palloc0(sizeof(TidStore));
+
+ /*
+ * Create the radix tree for the main storage.
+ *
+ * Memory consumption depends on the number of stored tids, but also on the
+ * distribution of them, how the radix tree stores, and the memory management
+ * that backed the radix tree. The maximum bytes that a TidStore can
+ * use is specified by the max_bytes in tidstore_create(). We want the total
+ * amount of memory consumption by a TidStore not to exceed the max_bytes.
+ *
+ * In local TidStore cases, the radix tree uses slab allocators for each kind
+ * of node class. The most memory consuming case while adding Tids associated
+ * with one page (i.e. during tidstore_add_tids()) is that we allocate a new
+ * slab block for a new radix tree node, which is approximately 70kB. Therefore,
+ * we deduct 70kB from the max_bytes.
+ *
+ * In shared cases, DSA allocates the memory segments big enough to follow
+ * a geometric series that approximately doubles the total DSA size (see
+ * make_new_segment() in dsa.c). We simulated the how DSA increases segment
+ * size and the simulation revealed, the 75% threshold for the maximum bytes
+ * perfectly works in case where the max_bytes is a power-of-2, and the 60%
+ * threshold works for other cases.
+ */
+ if (area != NULL)
+ {
+ dsa_pointer dp;
+ float ratio = ((max_bytes & (max_bytes - 1)) == 0) ? 0.75 : 0.6;
+
+ ts->tree.shared = shared_rt_create(CurrentMemoryContext, area,
+ LWTRANCHE_SHARED_TIDSTORE);
+
+ dp = dsa_allocate0(area, sizeof(TidStoreControl));
+ ts->control = (TidStoreControl *) dsa_get_address(area, dp);
+ ts->control->max_bytes = (uint64) (max_bytes * ratio);
+ ts->area = area;
+
+ ts->control->magic = TIDSTORE_MAGIC;
+ LWLockInitialize(&ts->control->lock, LWTRANCHE_SHARED_TIDSTORE);
+ ts->control->handle = dp;
+ ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+ }
+ else
+ {
+ ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+ ts->control = (TidStoreControl *) palloc0(sizeof(TidStoreControl));
+ ts->control->max_bytes = max_bytes - (70 * 1024);
+ }
+
+ ts->control->max_offset = max_offset;
+ ts->control->offset_nbits = pg_ceil_log2_32(max_offset);
+
+ if (ts->control->offset_nbits < TIDSTORE_VALUE_NBITS)
+ ts->control->offset_nbits = TIDSTORE_VALUE_NBITS;
+
+ /*
+ * We use tid encoding if the number of bits for the offset number doesn't
+ * fix in a value, uint64.
+ */
+ ts->control->offset_key_nbits =
+ ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
+
+ return ts;
+}
+
+/*
+ * Attach to the shared TidStore using a handle. The returned object is
+ * allocated in backend-local memory using the CurrentMemoryContext.
+ */
+TidStore *
+tidstore_attach(dsa_area *area, tidstore_handle handle)
+{
+ TidStore *ts;
+ dsa_pointer control;
+
+ Assert(area != NULL);
+ Assert(DsaPointerIsValid(handle));
+
+ /* create per-backend state */
+ ts = palloc0(sizeof(TidStore));
+
+ /* Find the control object in shared memory */
+ control = handle;
+
+ /* Set up the TidStore */
+ ts->control = (TidStoreControl *) dsa_get_address(area, control);
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ ts->tree.shared = shared_rt_attach(area, ts->control->tree_handle);
+ ts->area = area;
+
+ return ts;
+}
+
+/*
+ * Detach from a TidStore. This detaches from radix tree and frees the
+ * backend-local resources. The radix tree will continue to exist until
+ * it is either explicitly destroyed, or the area that backs it is returned
+ * to the operating system.
+ */
+void
+tidstore_detach(TidStore *ts)
+{
+ Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+ shared_rt_detach(ts->tree.shared);
+ pfree(ts);
+}
+
+/*
+ * Destroy a TidStore, returning all memory.
+ *
+ * TODO: The caller must be certain that no other backend will attempt to
+ * access the TidStore before calling this function. Other backend must
+ * explicitly call tidstore_detach to free up backend-local memory associated
+ * with the TidStore. The backend that calls tidstore_destroy must not call
+ * tidstore_detach.
+ */
+void
+tidstore_destroy(TidStore *ts)
+{
+ if (TidStoreIsShared(ts))
+ {
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ /*
+ * Vandalize the control block to help catch programming error where
+ * other backends access the memory formerly occupied by this radix
+ * tree.
+ */
+ ts->control->magic = 0;
+ dsa_free(ts->area, ts->control->handle);
+ shared_rt_free(ts->tree.shared);
+ }
+ else
+ {
+ pfree(ts->control);
+ local_rt_free(ts->tree.local);
+ }
+
+ pfree(ts);
+}
+
+/*
+ * Forget all collected Tids. It's similar to tidstore_destroy but we don't free
+ * entire TidStore but recreate only the radix tree storage.
+ */
+void
+tidstore_reset(TidStore *ts)
+{
+ if (TidStoreIsShared(ts))
+ {
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+ /*
+ * Free the radix tree and return allocated DSA segments to
+ * the operating system.
+ */
+ shared_rt_free(ts->tree.shared);
+ dsa_trim(ts->area);
+
+ /* Recreate the radix tree */
+ ts->tree.shared = shared_rt_create(CurrentMemoryContext, ts->area,
+ LWTRANCHE_SHARED_TIDSTORE);
+
+ /* update the radix tree handle as we recreated it */
+ ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
+
+ LWLockRelease(&ts->control->lock);
+ }
+ else
+ {
+ local_rt_free(ts->tree.local);
+ ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
+ }
+}
+
+/* Add Tids on a block to TidStore */
+void
+tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets)
+{
+ uint64 *values;
+ uint64 key;
+ uint64 prev_key;
+ uint64 off_bitmap = 0;
+ int idx;
+ const uint64 key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
+ const int nkeys = UINT64CONST(1) << ts->control->offset_key_nbits;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ values = palloc(sizeof(uint64) * nkeys);
+ key = prev_key = key_base;
+
+ for (int i = 0; i < num_offsets; i++)
+ {
+ uint32 off;
+
+ /* encode the tid to key and val */
+ key = encode_key_off(ts, blkno, offsets[i], &off);
+
+ /* make sure we scanned the line pointer array in order */
+ Assert(key >= prev_key);
+
+ if (key > prev_key)
+ {
+ idx = prev_key - key_base;
+ Assert(idx >= 0 && idx < nkeys);
+
+ /* write out offset bitmap for this key */
+ values[idx] = off_bitmap;
+
+ /* zero out any gaps up to the current key */
+ for (int empty_idx = idx + 1; empty_idx < key - key_base; empty_idx++)
+ values[empty_idx] = 0;
+
+ /* reset for current key -- the current offset will be handled below */
+ off_bitmap = 0;
+ prev_key = key;
+ }
+
+ off_bitmap |= UINT64CONST(1) << off;
+ }
+
+ /* save the final index for later */
+ idx = key - key_base;
+ /* write out last offset bitmap */
+ values[idx] = off_bitmap;
+
+ if (TidStoreIsShared(ts))
+ LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+ /* insert the calculated key-values to the tree */
+ for (int i = 0; i <= idx; i++)
+ {
+ if (values[i])
+ {
+ key = key_base + i;
+
+ if (TidStoreIsShared(ts))
+ shared_rt_set(ts->tree.shared, key, &values[i]);
+ else
+ local_rt_set(ts->tree.local, key, &values[i]);
+ }
+ }
+
+ /* update statistics */
+ ts->control->num_tids += num_offsets;
+
+ if (TidStoreIsShared(ts))
+ LWLockRelease(&ts->control->lock);
+
+ pfree(values);
+}
+
+/* Return true if the given tid is present in the TidStore */
+bool
+tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
+{
+ uint64 key;
+ uint64 val = 0;
+ uint32 off;
+ bool found;
+
+ key = tid_to_key_off(ts, tid, &off);
+
+ if (TidStoreIsShared(ts))
+ found = shared_rt_search(ts->tree.shared, key, &val);
+ else
+ found = local_rt_search(ts->tree.local, key, &val);
+
+ if (!found)
+ return false;
+
+ return (val & (UINT64CONST(1) << off)) != 0;
+}
+
+/*
+ * Prepare to iterate through a TidStore. Since the radix tree is locked during the
+ * iteration, so tidstore_end_iterate() needs to called when finished.
+ *
+ * Concurrent updates during the iteration will be blocked when inserting a
+ * key-value to the radix tree.
+ */
+TidStoreIter *
+tidstore_begin_iterate(TidStore *ts)
+{
+ TidStoreIter *iter;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ iter = palloc0(sizeof(TidStoreIter));
+ iter->ts = ts;
+
+ iter->result.blkno = InvalidBlockNumber;
+ iter->result.offsets = palloc(sizeof(OffsetNumber) * ts->control->max_offset);
+
+ if (TidStoreIsShared(ts))
+ iter->tree_iter.shared = shared_rt_begin_iterate(ts->tree.shared);
+ else
+ iter->tree_iter.local = local_rt_begin_iterate(ts->tree.local);
+
+ /* If the TidStore is empty, there is no business */
+ if (tidstore_num_tids(ts) == 0)
+ iter->finished = true;
+
+ return iter;
+}
+
+static inline bool
+tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
+{
+ if (TidStoreIsShared(iter->ts))
+ return shared_rt_iterate_next(iter->tree_iter.shared, key, val);
+
+ return local_rt_iterate_next(iter->tree_iter.local, key, val);
+}
+
+/*
+ * Scan the TidStore and return a pointer to TidStoreIterResult that has tids
+ * in one block. We return the block numbers in ascending order and the offset
+ * numbers in each result is also sorted in ascending order.
+ */
+TidStoreIterResult *
+tidstore_iterate_next(TidStoreIter *iter)
+{
+ uint64 key;
+ uint64 val;
+ TidStoreIterResult *result = &(iter->result);
+
+ if (iter->finished)
+ return NULL;
+
+ if (BlockNumberIsValid(result->blkno))
+ {
+ /* Process the previously collected key-value */
+ result->num_offsets = 0;
+ tidstore_iter_extract_tids(iter, iter->next_key, iter->next_val);
+ }
+
+ while (tidstore_iter_kv(iter, &key, &val))
+ {
+ BlockNumber blkno;
+
+ blkno = key_get_blkno(iter->ts, key);
+
+ if (BlockNumberIsValid(result->blkno) && result->blkno != blkno)
+ {
+ /*
+ * We got a key-value pair for a different block. So return the
+ * collected tids, and remember the key-value for the next iteration.
+ */
+ iter->next_key = key;
+ iter->next_val = val;
+ return result;
+ }
+
+ /* Collect tids extracted from the key-value pair */
+ tidstore_iter_extract_tids(iter, key, val);
+ }
+
+ iter->finished = true;
+ return result;
+}
+
+/*
+ * Finish an iteration over TidStore. This needs to be called after finishing
+ * or when existing an iteration.
+ */
+void
+tidstore_end_iterate(TidStoreIter *iter)
+{
+ if (TidStoreIsShared(iter->ts))
+ shared_rt_end_iterate(iter->tree_iter.shared);
+ else
+ local_rt_end_iterate(iter->tree_iter.local);
+
+ pfree(iter->result.offsets);
+ pfree(iter);
+}
+
+/* Return the number of tids we collected so far */
+int64
+tidstore_num_tids(TidStore *ts)
+{
+ uint64 num_tids;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ if (!TidStoreIsShared(ts))
+ return ts->control->num_tids;
+
+ LWLockAcquire(&ts->control->lock, LW_SHARED);
+ num_tids = ts->control->num_tids;
+ LWLockRelease(&ts->control->lock);
+
+ return num_tids;
+}
+
+/* Return true if the current memory usage of TidStore exceeds the limit */
+bool
+tidstore_is_full(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ return (tidstore_memory_usage(ts) > ts->control->max_bytes);
+}
+
+/* Return the maximum memory TidStore can use */
+size_t
+tidstore_max_memory(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ return ts->control->max_bytes;
+}
+
+/* Return the memory usage of TidStore */
+size_t
+tidstore_memory_usage(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ /*
+ * In the shared case, TidStoreControl and radix_tree are backed by the
+ * same DSA area and rt_memory_usage() returns the value including both.
+ * So we don't need to add the size of TidStoreControl separately.
+ */
+ if (TidStoreIsShared(ts))
+ return sizeof(TidStore) + shared_rt_memory_usage(ts->tree.shared);
+
+ return sizeof(TidStore) + sizeof(TidStore) + local_rt_memory_usage(ts->tree.local);
+}
+
+/*
+ * Get a handle that can be used by other processes to attach to this TidStore
+ */
+tidstore_handle
+tidstore_get_handle(TidStore *ts)
+{
+ Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+ return ts->control->handle;
+}
+
+/* Extract tids from the given key-value pair */
+static void
+tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
+{
+ TidStoreIterResult *result = (&iter->result);
+
+ for (int i = 0; i < sizeof(uint64) * BITS_PER_BYTE; i++)
+ {
+ uint64 tid_i;
+ OffsetNumber off;
+
+ if ((val & (UINT64CONST(1) << i)) == 0)
+ continue;
+
+ tid_i = key << TIDSTORE_VALUE_NBITS;
+ tid_i |= i;
+
+ off = tid_i & ((UINT64CONST(1) << iter->ts->control->offset_nbits) - 1);
+
+ Assert(result->num_offsets < iter->ts->control->max_offset);
+ result->offsets[result->num_offsets++] = off;
+ }
+
+ result->blkno = key_get_blkno(iter->ts, key);
+}
+
+/* Get block number from the given key */
+static inline BlockNumber
+key_get_blkno(TidStore *ts, uint64 key)
+{
+ return (BlockNumber) (key >> ts->control->offset_key_nbits);
+}
+
+/* Encode a tid to key and offset */
+static inline uint64
+tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off)
+{
+ uint32 offset = ItemPointerGetOffsetNumber(tid);
+ BlockNumber block = ItemPointerGetBlockNumber(tid);
+
+ return encode_key_off(ts, block, offset, off);
+}
+
+/* encode a block and offset to a key and partial offset */
+static inline uint64
+encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint32 *off)
+{
+ uint64 key;
+ uint64 tid_i;
+
+ tid_i = offset | ((uint64) block << ts->control->offset_nbits);
+
+ *off = tid_i & ((UINT64CONST(1) << TIDSTORE_VALUE_NBITS) - 1);
+ key = tid_i >> TIDSTORE_VALUE_NBITS;
+ Assert(*off < (sizeof(uint64) * BITS_PER_BYTE));
+
+ return key;
+}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index d2ec396045..55b3a04097 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -176,6 +176,8 @@ static const char *const BuiltinTrancheNames[] = {
"SharedTupleStore",
/* LWTRANCHE_SHARED_TIDBITMAP: */
"SharedTidBitmap",
+ /* LWTRANCHE_SHARED_TIDSTORE: */
+ "SharedTidStore",
/* LWTRANCHE_PARALLEL_APPEND: */
"ParallelAppend",
/* LWTRANCHE_PER_XACT_PREDICATE_LIST: */
diff --git a/src/include/access/tidstore.h b/src/include/access/tidstore.h
new file mode 100644
index 0000000000..a35a52124a
--- /dev/null
+++ b/src/include/access/tidstore.h
@@ -0,0 +1,49 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.h
+ * Tid storage.
+ *
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/tidstore.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef TIDSTORE_H
+#define TIDSTORE_H
+
+#include "storage/itemptr.h"
+#include "utils/dsa.h"
+
+typedef dsa_pointer tidstore_handle;
+
+typedef struct TidStore TidStore;
+typedef struct TidStoreIter TidStoreIter;
+
+typedef struct TidStoreIterResult
+{
+ BlockNumber blkno;
+ OffsetNumber *offsets;
+ int num_offsets;
+} TidStoreIterResult;
+
+extern TidStore *tidstore_create(size_t max_bytes, int max_offset, dsa_area *dsa);
+extern TidStore *tidstore_attach(dsa_area *dsa, dsa_pointer handle);
+extern void tidstore_detach(TidStore *ts);
+extern void tidstore_destroy(TidStore *ts);
+extern void tidstore_reset(TidStore *ts);
+extern void tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets);
+extern bool tidstore_lookup_tid(TidStore *ts, ItemPointer tid);
+extern TidStoreIter * tidstore_begin_iterate(TidStore *ts);
+extern TidStoreIterResult *tidstore_iterate_next(TidStoreIter *iter);
+extern void tidstore_end_iterate(TidStoreIter *iter);
+extern int64 tidstore_num_tids(TidStore *ts);
+extern bool tidstore_is_full(TidStore *ts);
+extern size_t tidstore_max_memory(TidStore *ts);
+extern size_t tidstore_memory_usage(TidStore *ts);
+extern tidstore_handle tidstore_get_handle(TidStore *ts);
+
+#endif /* TIDSTORE_H */
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index d2c7afb8f4..07002fdfbe 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -199,6 +199,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_PER_SESSION_RECORD_TYPMOD,
LWTRANCHE_SHARED_TUPLESTORE,
LWTRANCHE_SHARED_TIDBITMAP,
+ LWTRANCHE_SHARED_TIDSTORE,
LWTRANCHE_PARALLEL_APPEND,
LWTRANCHE_PER_XACT_PREDICATE_LIST,
LWTRANCHE_PGSTATS_DSA,
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 9659eb85d7..bddc16ada7 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
test_rls_hooks \
test_shm_mq \
test_slru \
+ test_tidstore \
unsafe_tests \
worker_spi
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 232cbdac80..c0d5645ad8 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -30,5 +30,6 @@ subdir('test_regex')
subdir('test_rls_hooks')
subdir('test_shm_mq')
subdir('test_slru')
+subdir('test_tidstore')
subdir('unsafe_tests')
subdir('worker_spi')
diff --git a/src/test/modules/test_tidstore/Makefile b/src/test/modules/test_tidstore/Makefile
new file mode 100644
index 0000000000..dab107d70c
--- /dev/null
+++ b/src/test/modules/test_tidstore/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_tidstore/Makefile
+
+MODULE_big = test_tidstore
+OBJS = \
+ $(WIN32RES) \
+ test_tidstore.o
+PGFILEDESC = "test_tidstore - test code for src/backend/access/common/tidstore.c"
+
+EXTENSION = test_tidstore
+DATA = test_tidstore--1.0.sql
+
+REGRESS = test_tidstore
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_tidstore
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_tidstore/expected/test_tidstore.out b/src/test/modules/test_tidstore/expected/test_tidstore.out
new file mode 100644
index 0000000000..7ff2f9af87
--- /dev/null
+++ b/src/test/modules/test_tidstore/expected/test_tidstore.out
@@ -0,0 +1,13 @@
+CREATE EXTENSION test_tidstore;
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
+NOTICE: testing empty tidstore
+NOTICE: testing basic operations
+ test_tidstore
+---------------
+
+(1 row)
+
diff --git a/src/test/modules/test_tidstore/meson.build b/src/test/modules/test_tidstore/meson.build
new file mode 100644
index 0000000000..31f2da7b61
--- /dev/null
+++ b/src/test/modules/test_tidstore/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_tidstore_sources = files(
+ 'test_tidstore.c',
+)
+
+if host_system == 'windows'
+ test_tidstore_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_tidstore',
+ '--FILEDESC', 'test_tidstore - test code for src/backend/access/common/tidstore.c',])
+endif
+
+test_tidstore = shared_module('test_tidstore',
+ test_tidstore_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += test_tidstore
+
+install_data(
+ 'test_tidstore.control',
+ 'test_tidstore--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'test_tidstore',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'test_tidstore',
+ ],
+ },
+}
diff --git a/src/test/modules/test_tidstore/sql/test_tidstore.sql b/src/test/modules/test_tidstore/sql/test_tidstore.sql
new file mode 100644
index 0000000000..03aea31815
--- /dev/null
+++ b/src/test/modules/test_tidstore/sql/test_tidstore.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_tidstore;
+
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
diff --git a/src/test/modules/test_tidstore/test_tidstore--1.0.sql b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
new file mode 100644
index 0000000000..47e9149900
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_tidstore/test_tidstore--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_tidstore" to load this file. \quit
+
+CREATE FUNCTION test_tidstore()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_tidstore/test_tidstore.c b/src/test/modules/test_tidstore/test_tidstore.c
new file mode 100644
index 0000000000..9b849ae8e8
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.c
@@ -0,0 +1,195 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_tidstore.c
+ * Test TidStore data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_tidstore/test_tidstore.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/tidstore.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "utils/memutils.h"
+
+PG_MODULE_MAGIC;
+
+#define TEST_TIDSTORE_MAX_BYTES (2 * 1024 * 1024L) /* 2MB */
+
+PG_FUNCTION_INFO_V1(test_tidstore);
+
+static void
+check_tid(TidStore *ts, BlockNumber blkno, OffsetNumber off, bool expect)
+{
+ ItemPointerData tid;
+ bool found;
+
+ ItemPointerSet(&tid, blkno, off);
+
+ found = tidstore_lookup_tid(ts, &tid);
+
+ if (found != expect)
+ elog(ERROR, "lookup TID (%u, %u) returned %d, expected %d",
+ blkno, off, found, expect);
+}
+
+static void
+test_basic(int max_offset)
+{
+#define TEST_TIDSTORE_NUM_BLOCKS 5
+#define TEST_TIDSTORE_NUM_OFFSETS 5
+
+ TidStore *ts;
+ TidStoreIter *iter;
+ TidStoreIterResult *iter_result;
+ BlockNumber blks[TEST_TIDSTORE_NUM_BLOCKS] = {
+ 0, MaxBlockNumber, MaxBlockNumber - 1, 1, MaxBlockNumber / 2,
+ };
+ BlockNumber blks_sorted[TEST_TIDSTORE_NUM_BLOCKS] = {
+ 0, 1, MaxBlockNumber / 2, MaxBlockNumber - 1, MaxBlockNumber
+ };
+ OffsetNumber offs[TEST_TIDSTORE_NUM_OFFSETS];
+ int blk_idx;
+
+ /* prepare the offset array */
+ offs[0] = FirstOffsetNumber;
+ offs[1] = FirstOffsetNumber + 1;
+ offs[2] = max_offset / 2;
+ offs[3] = max_offset - 1;
+ offs[4] = max_offset;
+
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
+
+ /* add tids */
+ for (int i = 0; i < TEST_TIDSTORE_NUM_BLOCKS; i++)
+ tidstore_add_tids(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
+
+ /* lookup test */
+ for (OffsetNumber off = FirstOffsetNumber ; off < max_offset; off++)
+ {
+ bool expect = false;
+ for (int i = 0; i < TEST_TIDSTORE_NUM_OFFSETS; i++)
+ {
+ if (offs[i] == off)
+ {
+ expect = true;
+ break;
+ }
+ }
+
+ check_tid(ts, 0, off, expect);
+ check_tid(ts, 2, off, false);
+ check_tid(ts, MaxBlockNumber - 2, off, false);
+ check_tid(ts, MaxBlockNumber, off, expect);
+ }
+
+ /* test the number of tids */
+ if (tidstore_num_tids(ts) != (TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS))
+ elog(ERROR, "tidstore_num_tids returned " UINT64_FORMAT ", expected %d",
+ tidstore_num_tids(ts),
+ TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS);
+
+ /* iteration test */
+ iter = tidstore_begin_iterate(ts);
+ blk_idx = 0;
+ while ((iter_result = tidstore_iterate_next(iter)) != NULL)
+ {
+ /* check the returned block number */
+ if (blks_sorted[blk_idx] != iter_result->blkno)
+ elog(ERROR, "tidstore_iterate_next returned block number %u, expected %u",
+ iter_result->blkno, blks_sorted[blk_idx]);
+
+ /* check the returned offset numbers */
+ if (TEST_TIDSTORE_NUM_OFFSETS != iter_result->num_offsets)
+ elog(ERROR, "tidstore_iterate_next returned %u offsets, expected %u",
+ iter_result->num_offsets, TEST_TIDSTORE_NUM_OFFSETS);
+
+ for (int i = 0; i < iter_result->num_offsets; i++)
+ {
+ if (offs[i] != iter_result->offsets[i])
+ elog(ERROR, "tidstore_iterate_next returned offset number %u on block %u, expected %u",
+ iter_result->offsets[i], iter_result->blkno, offs[i]);
+ }
+
+ blk_idx++;
+ }
+
+ if (blk_idx != TEST_TIDSTORE_NUM_BLOCKS)
+ elog(ERROR, "tidstore_iterate_next returned %d blocks, expected %d",
+ blk_idx, TEST_TIDSTORE_NUM_BLOCKS);
+
+ /* remove all tids */
+ tidstore_reset(ts);
+
+ /* test the number of tids */
+ if (tidstore_num_tids(ts) != 0)
+ elog(ERROR, "tidstore_num_tids on empty store returned non-zero");
+
+ /* lookup test for empty store */
+ for (OffsetNumber off = FirstOffsetNumber ; off < MaxHeapTuplesPerPage;
+ off++)
+ {
+ check_tid(ts, 0, off, false);
+ check_tid(ts, 2, off, false);
+ check_tid(ts, MaxBlockNumber - 2, off, false);
+ check_tid(ts, MaxBlockNumber, off, false);
+ }
+
+ tidstore_destroy(ts);
+}
+
+static void
+test_empty(void)
+{
+ TidStore *ts;
+ TidStoreIter *iter;
+ ItemPointerData tid;
+
+ elog(NOTICE, "testing empty tidstore");
+
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
+
+ ItemPointerSet(&tid, 0, FirstOffsetNumber);
+ if (tidstore_lookup_tid(ts, &tid))
+ elog(ERROR, "tidstore_lookup_tid for (0,1) on empty store returned true");
+
+ ItemPointerSet(&tid, MaxBlockNumber, MaxOffsetNumber);
+ if (tidstore_lookup_tid(ts, &tid))
+ elog(ERROR, "tidstore_lookup_tid for (%u,%u) on empty store returned true",
+ MaxBlockNumber, MaxOffsetNumber);
+
+ if (tidstore_num_tids(ts) != 0)
+ elog(ERROR, "tidstore_num_entries on empty store returned non-zero");
+
+ if (tidstore_is_full(ts))
+ elog(ERROR, "tidstore_is_full on empty store returned true");
+
+ iter = tidstore_begin_iterate(ts);
+
+ if (tidstore_iterate_next(iter) != NULL)
+ elog(ERROR, "tidstore_iterate_next on empty store returned TIDs");
+
+ tidstore_end_iterate(iter);
+
+ tidstore_destroy(ts);
+}
+
+Datum
+test_tidstore(PG_FUNCTION_ARGS)
+{
+ test_empty();
+
+ elog(NOTICE, "testing basic operations");
+ test_basic(MaxHeapTuplesPerPage);
+ test_basic(10);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_tidstore/test_tidstore.control b/src/test/modules/test_tidstore/test_tidstore.control
new file mode 100644
index 0000000000..9b6bd4638f
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.control
@@ -0,0 +1,4 @@
+comment = 'Test code for tidstore'
+default_version = '1.0'
+module_pathname = '$libdir/test_tidstore'
+relocatable = true
--
2.39.1
[text/x-patch] v27-0006-Tool-for-measuring-radix-tree-and-tidstore-perfo.patch (23.9K, ../../CAFBsxsESfmZpWwk3XxWkpdg_DjawH2vizKW91fyawQ+wVFkOiQ@mail.gmail.com/8-v27-0006-Tool-for-measuring-radix-tree-and-tidstore-perfo.patch)
download | inline diff:
From b0515a40b3aa4709047c7b70b9c0cadded979d15 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 16 Sep 2022 11:57:03 +0900
Subject: [PATCH v27 6/9] Tool for measuring radix tree and tidstore
performance
Includes Meson support, but commented out to avoid warnings
XXX: Not for commit
---
contrib/bench_radix_tree/Makefile | 21 +
.../bench_radix_tree--1.0.sql | 87 +++
contrib/bench_radix_tree/bench_radix_tree.c | 717 ++++++++++++++++++
.../bench_radix_tree/bench_radix_tree.control | 6 +
contrib/bench_radix_tree/expected/bench.out | 13 +
contrib/bench_radix_tree/meson.build | 33 +
contrib/bench_radix_tree/sql/bench.sql | 16 +
contrib/meson.build | 1 +
8 files changed, 894 insertions(+)
create mode 100644 contrib/bench_radix_tree/Makefile
create mode 100644 contrib/bench_radix_tree/bench_radix_tree--1.0.sql
create mode 100644 contrib/bench_radix_tree/bench_radix_tree.c
create mode 100644 contrib/bench_radix_tree/bench_radix_tree.control
create mode 100644 contrib/bench_radix_tree/expected/bench.out
create mode 100644 contrib/bench_radix_tree/meson.build
create mode 100644 contrib/bench_radix_tree/sql/bench.sql
diff --git a/contrib/bench_radix_tree/Makefile b/contrib/bench_radix_tree/Makefile
new file mode 100644
index 0000000000..952bb0ceae
--- /dev/null
+++ b/contrib/bench_radix_tree/Makefile
@@ -0,0 +1,21 @@
+# contrib/bench_radix_tree/Makefile
+
+MODULE_big = bench_radix_tree
+OBJS = \
+ bench_radix_tree.o
+
+EXTENSION = bench_radix_tree
+DATA = bench_radix_tree--1.0.sql
+
+REGRESS = bench_fixed_height
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/bench_radix_tree
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
new file mode 100644
index 0000000000..fbf51c1086
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
@@ -0,0 +1,87 @@
+/* contrib/bench_radix_tree/bench_radix_tree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION bench_radix_tree" to load this file. \quit
+
+create function bench_shuffle_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_seq_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_load_random_int(
+cnt int8,
+OUT mem_allocated int8,
+OUT load_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_search_random_nodes(
+cnt int8,
+filter_str text DEFAULT NULL,
+OUT mem_allocated int8,
+OUT load_ms int8,
+OUT search_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C VOLATILE PARALLEL UNSAFE;
+
+create function bench_fixed_height_search(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT rt_search_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_node128_load(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_sparseload_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_tidstore_load(
+minblk int4,
+maxblk int4,
+OUT mem_allocated int8,
+OUT load_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
new file mode 100644
index 0000000000..b5ad75364c
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -0,0 +1,717 @@
+/*-------------------------------------------------------------------------
+ *
+ * bench_radix_tree.c
+ *
+ * Copyright (c) 2016-2022, PostgreSQL Global Development Group
+ *
+ * contrib/bench_radix_tree/bench_radix_tree.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/tidstore.h"
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "funcapi.h"
+#include "lib/radixtree.h"
+#include <math.h>
+#include "miscadmin.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/timestamp.h"
+
+PG_MODULE_MAGIC;
+
+/* run benchmark also for binary-search case? */
+/* #define MEASURE_BINARY_SEARCH 1 */
+
+#define TIDS_PER_BLOCK_FOR_LOAD 30
+#define TIDS_PER_BLOCK_FOR_LOOKUP 50
+
+//#define RT_DEBUG
+#define RT_PREFIX rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE uint64
+// WIP: compiles with warnings because rt_attach is defined but not used
+// #define RT_SHMEM
+#include "lib/radixtree.h"
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+ return tree->ctl->num_keys;
+}
+
+
+PG_FUNCTION_INFO_V1(bench_seq_search);
+PG_FUNCTION_INFO_V1(bench_shuffle_search);
+PG_FUNCTION_INFO_V1(bench_load_random_int);
+PG_FUNCTION_INFO_V1(bench_fixed_height_search);
+PG_FUNCTION_INFO_V1(bench_search_random_nodes);
+PG_FUNCTION_INFO_V1(bench_node128_load);
+PG_FUNCTION_INFO_V1(bench_tidstore_load);
+
+static uint64
+tid_to_key_off(ItemPointer tid, uint32 *off)
+{
+ uint64 upper;
+ uint32 shift = pg_ceil_log2_32(MaxHeapTuplesPerPage);
+ int64 tid_i;
+
+ Assert(ItemPointerGetOffsetNumber(tid) < MaxHeapTuplesPerPage);
+
+ tid_i = ItemPointerGetOffsetNumber(tid);
+ tid_i |= (uint64) ItemPointerGetBlockNumber(tid) << shift;
+
+ /* log(sizeof(uint64) * BITS_PER_BYTE, 2) = log(64, 2) = 6 */
+ *off = tid_i & ((1 << 6) - 1);
+ upper = tid_i >> 6;
+ Assert(*off < (sizeof(uint64) * BITS_PER_BYTE));
+
+ Assert(*off < 64);
+
+ return upper;
+}
+
+static int
+shuffle_randrange(pg_prng_state *state, int lower, int upper)
+{
+ return (int) floor(pg_prng_double(state) * ((upper - lower) + 0.999999)) + lower;
+}
+
+/* Naive Fisher-Yates implementation*/
+static void
+shuffle_itemptrs(ItemPointer itemptr, uint64 nitems)
+{
+ /* reproducability */
+ pg_prng_state state;
+
+ pg_prng_seed(&state, 0);
+
+ for (int i = 0; i < nitems - 1; i++)
+ {
+ int j = shuffle_randrange(&state, i, nitems - 1);
+ ItemPointerData t = itemptr[j];
+
+ itemptr[j] = itemptr[i];
+ itemptr[i] = t;
+ }
+}
+
+static ItemPointer
+generate_tids(BlockNumber minblk, BlockNumber maxblk, int ntids_per_blk, uint64 *ntids_p,
+ bool random_block)
+{
+ ItemPointer tids;
+ uint64 maxitems;
+ uint64 ntids = 0;
+ pg_prng_state state;
+
+ maxitems = (maxblk - minblk + 1) * ntids_per_blk;
+ tids = MemoryContextAllocHuge(TopTransactionContext,
+ sizeof(ItemPointerData) * maxitems);
+
+ if (random_block)
+ pg_prng_seed(&state, 0x9E3779B185EBCA87);
+
+ for (BlockNumber blk = minblk; blk < maxblk; blk++)
+ {
+ if (random_block && !pg_prng_bool(&state))
+ continue;
+
+ for (OffsetNumber off = FirstOffsetNumber;
+ off <= ntids_per_blk; off++)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ ItemPointerSetBlockNumber(&(tids[ntids]), blk);
+ ItemPointerSetOffsetNumber(&(tids[ntids]), off);
+
+ ntids++;
+ }
+ }
+
+ *ntids_p = ntids;
+ return tids;
+}
+
+#ifdef MEASURE_BINARY_SEARCH
+static int
+vac_cmp_itemptr(const void *left, const void *right)
+{
+ BlockNumber lblk,
+ rblk;
+ OffsetNumber loff,
+ roff;
+
+ lblk = ItemPointerGetBlockNumber((ItemPointer) left);
+ rblk = ItemPointerGetBlockNumber((ItemPointer) right);
+
+ if (lblk < rblk)
+ return -1;
+ if (lblk > rblk)
+ return 1;
+
+ loff = ItemPointerGetOffsetNumber((ItemPointer) left);
+ roff = ItemPointerGetOffsetNumber((ItemPointer) right);
+
+ if (loff < roff)
+ return -1;
+ if (loff > roff)
+ return 1;
+
+ return 0;
+}
+#endif
+
+Datum
+bench_tidstore_load(PG_FUNCTION_ARGS)
+{
+ BlockNumber minblk = PG_GETARG_INT32(0);
+ BlockNumber maxblk = PG_GETARG_INT32(1);
+ TidStore *ts;
+ OffsetNumber *offs;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_ms;
+ TupleDesc tupdesc;
+ Datum values[2];
+ bool nulls[2] = {false};
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ offs = palloc(sizeof(OffsetNumber) * TIDS_PER_BLOCK_FOR_LOAD);
+ for (int i = 0; i < TIDS_PER_BLOCK_FOR_LOAD; i++)
+ offs[i] = i + 1; /* FirstOffsetNumber is 1 */
+
+ ts = tidstore_create(1 * 1024L * 1024L * 1024L, MaxHeapTuplesPerPage, NULL);
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* load tids */
+ start_time = GetCurrentTimestamp();
+ for (BlockNumber blkno = minblk; blkno < maxblk; blkno++)
+ tidstore_add_tids(ts, blkno, offs, TIDS_PER_BLOCK_FOR_LOAD);
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_ms = secs * 1000 + usecs / 1000;
+
+ values[0] = Int64GetDatum(tidstore_memory_usage(ts));
+ values[1] = Int64GetDatum(load_ms);
+
+ tidstore_destroy(ts);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+static Datum
+bench_search(FunctionCallInfo fcinfo, bool shuffle)
+{
+ BlockNumber minblk = PG_GETARG_INT32(0);
+ BlockNumber maxblk = PG_GETARG_INT32(1);
+ bool random_block = PG_GETARG_BOOL(2);
+ rt_radix_tree *rt = NULL;
+ uint64 ntids;
+ uint64 key;
+ uint64 last_key = PG_UINT64_MAX;
+ uint64 val = 0;
+ ItemPointer tids;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_load_ms,
+ rt_search_ms;
+ Datum values[7];
+ bool nulls[7];
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ tids = generate_tids(minblk, maxblk, TIDS_PER_BLOCK_FOR_LOAD, &ntids, random_block);
+
+ /* measure the load time of the radix tree */
+ rt = rt_create(CurrentMemoryContext);
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ uint32 off;
+
+ CHECK_FOR_INTERRUPTS();
+
+ key = tid_to_key_off(tid, &off);
+
+ if (last_key != PG_UINT64_MAX && last_key != key)
+ {
+ rt_set(rt, last_key, &val);
+ val = 0;
+ }
+
+ last_key = key;
+ val |= (uint64) 1 << off;
+ }
+ if (last_key != PG_UINT64_MAX)
+ rt_set(rt, last_key, &val);
+
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_load_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+ if (shuffle)
+ shuffle_itemptrs(tids, ntids);
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* meaure the serach time of the radix tree */
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ uint32 off;
+ volatile bool ret; /* prevent calling rt_search from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ key = tid_to_key_off(tid, &off);
+
+ ret = rt_search(rt, key, &val);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_search_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int64GetDatum(rt_num_entries(rt));
+ values[1] = Int64GetDatum(rt_memory_usage(rt));
+ values[2] = Int64GetDatum(sizeof(ItemPointerData) * ntids);
+ values[3] = Int64GetDatum(rt_load_ms);
+ nulls[4] = true; /* ar_load_ms */
+ values[5] = Int64GetDatum(rt_search_ms);
+ nulls[6] = true; /* ar_search_ms */
+
+#ifdef MEASURE_BINARY_SEARCH
+ {
+ ItemPointer itemptrs = NULL;
+
+ int64 ar_load_ms,
+ ar_search_ms;
+
+ /* measure the load time of the array */
+ itemptrs = MemoryContextAllocHuge(CurrentMemoryContext,
+ sizeof(ItemPointerData) * ntids);
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointerSetBlockNumber(&(itemptrs[i]),
+ ItemPointerGetBlockNumber(&(tids[i])));
+ ItemPointerSetOffsetNumber(&(itemptrs[i]),
+ ItemPointerGetOffsetNumber(&(tids[i])));
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ ar_load_ms = secs * 1000 + usecs / 1000;
+
+ /* next, measure the serach time of the array */
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ volatile bool ret; /* prevent calling bsearch from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ ret = bsearch((void *) tid,
+ (void *) itemptrs,
+ ntids,
+ sizeof(ItemPointerData),
+ vac_cmp_itemptr);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ ar_search_ms = secs * 1000 + usecs / 1000;
+
+ /* set the result */
+ nulls[4] = false;
+ values[4] = Int64GetDatum(ar_load_ms);
+ nulls[6] = false;
+ values[6] = Int64GetDatum(ar_search_ms);
+ }
+#endif
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_seq_search(PG_FUNCTION_ARGS)
+{
+ return bench_search(fcinfo, false);
+}
+
+Datum
+bench_shuffle_search(PG_FUNCTION_ARGS)
+{
+ return bench_search(fcinfo, true);
+}
+
+Datum
+bench_load_random_int(PG_FUNCTION_ARGS)
+{
+ uint64 cnt = (uint64) PG_GETARG_INT64(0);
+ rt_radix_tree *rt;
+ pg_prng_state state;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_time_ms;
+ Datum values[2];
+ bool nulls[2];
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ pg_prng_seed(&state, 0);
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 key = pg_prng_uint64(&state);
+
+ rt_set(rt, key, &key);
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_time_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int64GetDatum(rt_memory_usage(rt));
+ values[1] = Int64GetDatum(load_time_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+/* copy of splitmix64() */
+static uint64
+hash64(uint64 x)
+{
+ x ^= x >> 30;
+ x *= UINT64CONST(0xbf58476d1ce4e5b9);
+ x ^= x >> 27;
+ x *= UINT64CONST(0x94d049bb133111eb);
+ x ^= x >> 31;
+ return x;
+}
+
+/* attempts to have a relatively even population of node kinds */
+Datum
+bench_search_random_nodes(PG_FUNCTION_ARGS)
+{
+ uint64 cnt = (uint64) PG_GETARG_INT64(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_time_ms;
+ int64 search_time_ms;
+ Datum values[3] = {0};
+ bool nulls[3] = {0};
+ /* from trial and error */
+ uint64 filter = (((uint64) 0x7F<<32) | (0x07<<24) | (0xFF<<16) | 0xFF);
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (!PG_ARGISNULL(1))
+ {
+ char *filter_str = text_to_cstring(PG_GETARG_TEXT_P(1));
+
+ if (sscanf(filter_str, "0x%lX", &filter) == 0)
+ elog(ERROR, "invalid filter string %s", filter_str);
+ }
+ elog(NOTICE, "bench with filter 0x%lX", filter);
+
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 hash = hash64(i);
+ uint64 key = hash & filter;
+
+ rt_set(rt, key, &key);
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_time_ms = secs * 1000 + usecs / 1000;
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 hash = hash64(i);
+ uint64 key = hash & filter;
+ uint64 val;
+ volatile bool ret; /* prevent calling rt_search from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ ret = rt_search(rt, key, &val);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ search_time_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ values[0] = Int64GetDatum(rt_memory_usage(rt));
+ values[1] = Int64GetDatum(load_time_ms);
+ values[2] = Int64GetDatum(search_time_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_fixed_height_search(PG_FUNCTION_ARGS)
+{
+ int fanout = PG_GETARG_INT32(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_load_ms,
+ rt_search_ms;
+ Datum values[5];
+ bool nulls[5];
+
+ /* test boundary between vector and iteration */
+ const int n_keys = 5 * 16 * 16 * 16 * 16;
+ uint64 r,
+ h,
+ i,
+ j,
+ k;
+ uint64 key_id;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+
+ key_id = 0;
+
+ /*
+ * lower nodes have limited fanout, the top is only limited by
+ * bits-per-byte
+ */
+ for (r = 1;; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ for (i = 1; i <= fanout; i++)
+ {
+ for (j = 1; j <= fanout; j++)
+ {
+ for (k = 1; k <= fanout; k++)
+ {
+ uint64 key;
+
+ key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+ CHECK_FOR_INTERRUPTS();
+
+ key_id++;
+ if (key_id > n_keys)
+ goto finish_set;
+
+ rt_set(rt, key, &key_id);
+ }
+ }
+ }
+ }
+ }
+finish_set:
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_load_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ /* meaure the search time of the radix tree */
+ start_time = GetCurrentTimestamp();
+
+ key_id = 0;
+ for (r = 1;; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ for (i = 1; i <= fanout; i++)
+ {
+ for (j = 1; j <= fanout; j++)
+ {
+ for (k = 1; k <= fanout; k++)
+ {
+ uint64 key,
+ val;
+
+ key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+ CHECK_FOR_INTERRUPTS();
+
+ key_id++;
+ if (key_id > n_keys)
+ goto finish_search;
+
+ rt_search(rt, key, &val);
+ }
+ }
+ }
+ }
+ }
+finish_search:
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_search_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int32GetDatum(fanout);
+ values[1] = Int64GetDatum(rt_num_entries(rt));
+ values[2] = Int64GetDatum(rt_memory_usage(rt));
+ values[3] = Int64GetDatum(rt_load_ms);
+ values[4] = Int64GetDatum(rt_search_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_node128_load(PG_FUNCTION_ARGS)
+{
+ int fanout = PG_GETARG_INT32(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_sparseload_ms;
+ Datum values[5];
+ bool nulls[5];
+
+ uint64 r,
+ h;
+ uint64 key_id;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ rt = rt_create(CurrentMemoryContext);
+
+ key_id = 0;
+
+ for (r = 1; r <= fanout; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (h);
+
+ key_id++;
+ rt_set(rt, key, &key_id);
+ }
+ }
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ /* measure sparse deletion and re-loading */
+ start_time = GetCurrentTimestamp();
+
+ for (int t = 0; t<10000; t++)
+ {
+ /* delete one key in each leaf */
+ for (r = 1; r <= fanout; r++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (fanout);
+
+ rt_delete(rt, key);
+ }
+
+ /* add them all back */
+ key_id = 0;
+ for (r = 1; r <= fanout; r++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (fanout);
+
+ key_id++;
+ rt_set(rt, key, &key_id);
+ }
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_sparseload_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int32GetDatum(fanout);
+ values[1] = Int64GetDatum(rt_num_entries(rt));
+ values[2] = Int64GetDatum(rt_memory_usage(rt));
+ values[3] = Int64GetDatum(rt_sparseload_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
diff --git a/contrib/bench_radix_tree/bench_radix_tree.control b/contrib/bench_radix_tree/bench_radix_tree.control
new file mode 100644
index 0000000000..1d988e6c9a
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.control
@@ -0,0 +1,6 @@
+# bench_radix_tree extension
+comment = 'benchmark suits for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/bench_radix_tree'
+relocatable = true
+trusted = true
diff --git a/contrib/bench_radix_tree/expected/bench.out b/contrib/bench_radix_tree/expected/bench.out
new file mode 100644
index 0000000000..60c303892e
--- /dev/null
+++ b/contrib/bench_radix_tree/expected/bench.out
@@ -0,0 +1,13 @@
+create extension bench_radix_tree;
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/bench_radix_tree/meson.build b/contrib/bench_radix_tree/meson.build
new file mode 100644
index 0000000000..332c1ae7df
--- /dev/null
+++ b/contrib/bench_radix_tree/meson.build
@@ -0,0 +1,33 @@
+bench_radix_tree_sources = files(
+ 'bench_radix_tree.c',
+)
+
+if host_system == 'windows'
+ bench_radix_tree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'bench_radix_tree',
+ '--FILEDESC', 'bench_radix_tree - performance test code for radix tree',])
+endif
+
+bench_radix_tree = shared_module('bench_radix_tree',
+ bench_radix_tree_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += bench_radix_tree
+
+install_data(
+ 'bench_radix_tree.control',
+ 'bench_radix_tree--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'bench_radix_tree',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'bench_radix_tree',
+ ],
+ },
+}
diff --git a/contrib/bench_radix_tree/sql/bench.sql b/contrib/bench_radix_tree/sql/bench.sql
new file mode 100644
index 0000000000..a46018c9d4
--- /dev/null
+++ b/contrib/bench_radix_tree/sql/bench.sql
@@ -0,0 +1,16 @@
+create extension bench_radix_tree;
+
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/meson.build b/contrib/meson.build
index bd4a57c43c..52253de793 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,6 +12,7 @@ subdir('amcheck')
subdir('auth_delay')
subdir('auto_explain')
subdir('basic_archive')
+#subdir('bench_radix_tree')
subdir('bloom')
subdir('basebackup_to_shell')
subdir('bool_plperl')
--
2.39.1
[text/x-patch] v27-0008-Measure-iteration-of-tidstore.patch (3.7K, ../../CAFBsxsESfmZpWwk3XxWkpdg_DjawH2vizKW91fyawQ+wVFkOiQ@mail.gmail.com/9-v27-0008-Measure-iteration-of-tidstore.patch)
download | inline diff:
From 72bb462b1dab005cbc2aff265baedbaaee62cb2b Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 14 Feb 2023 17:02:53 +0700
Subject: [PATCH v27 8/9] Measure iteration of tidstore
---
.../bench_radix_tree--1.0.sql | 3 +-
contrib/bench_radix_tree/bench_radix_tree.c | 40 ++++++++++++++++---
contrib/meson.build | 2 +-
3 files changed, 38 insertions(+), 7 deletions(-)
diff --git a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
index fbf51c1086..ad66265e23 100644
--- a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
+++ b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
@@ -80,7 +80,8 @@ create function bench_tidstore_load(
minblk int4,
maxblk int4,
OUT mem_allocated int8,
-OUT load_ms int8
+OUT load_ms int8,
+OUT iter_ms int8
)
returns record
as 'MODULE_PATHNAME'
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
index b5ad75364c..6e5149e2c4 100644
--- a/contrib/bench_radix_tree/bench_radix_tree.c
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -176,15 +176,18 @@ bench_tidstore_load(PG_FUNCTION_ARGS)
BlockNumber minblk = PG_GETARG_INT32(0);
BlockNumber maxblk = PG_GETARG_INT32(1);
TidStore *ts;
+ TidStoreIter *iter;
+ TidStoreIterResult *result;
OffsetNumber *offs;
TimestampTz start_time,
end_time;
long secs;
int usecs;
int64 load_ms;
+ int64 iter_ms;
TupleDesc tupdesc;
- Datum values[2];
- bool nulls[2] = {false};
+ Datum values[3];
+ bool nulls[3] = {false};
/* Build a tuple descriptor for our result type */
if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
@@ -196,9 +199,6 @@ bench_tidstore_load(PG_FUNCTION_ARGS)
ts = tidstore_create(1 * 1024L * 1024L * 1024L, MaxHeapTuplesPerPage, NULL);
- elog(NOTICE, "sleeping for 2 seconds...");
- pg_usleep(2 * 1000000L);
-
/* load tids */
start_time = GetCurrentTimestamp();
for (BlockNumber blkno = minblk; blkno < maxblk; blkno++)
@@ -207,8 +207,22 @@ bench_tidstore_load(PG_FUNCTION_ARGS)
TimestampDifference(start_time, end_time, &secs, &usecs);
load_ms = secs * 1000 + usecs / 1000;
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* iterate through tids */
+ iter = tidstore_begin_iterate(ts);
+ start_time = GetCurrentTimestamp();
+ while ((result = tidstore_iterate_next(iter)) != NULL)
+ ;
+ tidstore_end_iterate(iter);
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ iter_ms = secs * 1000 + usecs / 1000;
+
values[0] = Int64GetDatum(tidstore_memory_usage(ts));
values[1] = Int64GetDatum(load_ms);
+ values[2] = Int64GetDatum(iter_ms);
tidstore_destroy(ts);
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
@@ -715,3 +729,19 @@ bench_node128_load(PG_FUNCTION_ARGS)
rt_free(rt);
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
}
+
+/* to silence warnings about unused iter functions */
+static void pg_attribute_unused()
+stub_iter()
+{
+ rt_radix_tree *rt;
+ rt_iter *iter;
+ uint64 key = 1;
+ uint64 value = 1;
+
+ rt = rt_create(CurrentMemoryContext);
+
+ iter = rt_begin_iterate(rt);
+ rt_iterate_next(iter, &key, &value);
+ rt_end_iterate(iter);
+}
\ No newline at end of file
diff --git a/contrib/meson.build b/contrib/meson.build
index 52253de793..421d469f8c 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,7 +12,7 @@ subdir('amcheck')
subdir('auth_delay')
subdir('auto_explain')
subdir('basic_archive')
-#subdir('bench_radix_tree')
+subdir('bench_radix_tree')
subdir('bloom')
subdir('basebackup_to_shell')
subdir('bool_plperl')
--
2.39.1
[text/x-patch] v27-0007-Prevent-inlining-of-interface-functions-for-shme.patch (750B, ../../CAFBsxsESfmZpWwk3XxWkpdg_DjawH2vizKW91fyawQ+wVFkOiQ@mail.gmail.com/10-v27-0007-Prevent-inlining-of-interface-functions-for-shme.patch)
download | inline diff:
From 54ab02eb2188382185436059ff6e7ad95d970c5d Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 14 Feb 2023 17:00:31 +0700
Subject: [PATCH v27 7/9] Prevent inlining of interface functions for shmem
---
src/backend/access/common/tidstore.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index ad8c0866e2..d1b4675ea4 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -84,7 +84,7 @@
#define RT_PREFIX shared_rt
#define RT_SHMEM
-#define RT_SCOPE static
+#define RT_SCOPE static pg_noinline
#define RT_DECLARE
#define RT_DEFINE
#define RT_VALUE_TYPE uint64
--
2.39.1
[text/x-patch] v27-0009-Speed-up-tidstore_iter_extract_tids.patch (1.3K, ../../CAFBsxsESfmZpWwk3XxWkpdg_DjawH2vizKW91fyawQ+wVFkOiQ@mail.gmail.com/11-v27-0009-Speed-up-tidstore_iter_extract_tids.patch)
download | inline diff:
From 8ccc66211973bcc44a6bad45c05302ca743c1489 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 14 Feb 2023 17:53:37 +0700
Subject: [PATCH v27 9/9] Speed up tidstore_iter_extract_tids()
---
src/backend/access/common/tidstore.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index d1b4675ea4..5a897c01f7 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -632,21 +632,21 @@ tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
{
TidStoreIterResult *result = (&iter->result);
- for (int i = 0; i < sizeof(uint64) * BITS_PER_BYTE; i++)
+ while (val)
{
uint64 tid_i;
OffsetNumber off;
- if ((val & (UINT64CONST(1) << i)) == 0)
- continue;
-
tid_i = key << TIDSTORE_VALUE_NBITS;
- tid_i |= i;
+ tid_i |= pg_rightmost_one_pos64(val);
off = tid_i & ((UINT64CONST(1) << iter->ts->control->offset_nbits) - 1);
Assert(result->num_offsets < iter->ts->control->max_offset);
result->offsets[result->num_offsets++] = off;
+
+ /* unset the rightmost bit */
+ val &= ~pg_rightmost_one64(val);
}
result->blkno = key_get_blkno(iter->ts, key);
--
2.39.1
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-02-14 12:36 ` John Naylor <[email protected]>
1 sibling, 0 replies; 78+ messages in thread
From: John Naylor @ 2023-02-14 12:36 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
The benchmark module shouldn't have been un-commented-out, so attached a
revert of that.
--
John Naylor
EDB: http://www.enterprisedb.com
Attachments:
[text/x-patch] v28-0002-Move-some-bitmap-logic-out-of-bitmapset.c.patch (6.1K, ../../CAFBsxsF4QAOzSPbQvekFn56WJJuYoZM8m=FqbKyb3dwhB+oyWg@mail.gmail.com/3-v28-0002-Move-some-bitmap-logic-out-of-bitmapset.c.patch)
download | inline diff:
From 149a49f51f7a16b7c1eb762e704f1ec476ecb65a Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 6 Dec 2022 13:39:41 +0700
Subject: [PATCH v28 02/10] Move some bitmap logic out of bitmapset.c
Add pg_rightmost_one32/64 functions. This functionality was previously
private to bitmapset.c as the RIGHTMOST_ONE macro. It has practical
use in other contexts, so move to pg_bitutils.h.
Also move the logic for selecting appropriate pg_bitutils functions
based on word size to bitmapset.h for wider visibility and add
appropriate selection for bmw_rightmost_one().
Since the previous macro relied on casting to signedbitmapword,
and the new functions do not, remove that typedef.
Design input and review by Tom Lane
Discussion: https://www.postgresql.org/message-id/CAFBsxsFW2JjTo58jtDB%2B3sZhxMx3t-3evew8%3DAcr%2BGGhC%2BkFaA%40mail.gmail.com
---
src/backend/nodes/bitmapset.c | 36 ++------------------------------
src/include/nodes/bitmapset.h | 16 ++++++++++++--
src/include/port/pg_bitutils.h | 31 +++++++++++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 -
4 files changed, 47 insertions(+), 37 deletions(-)
diff --git a/src/backend/nodes/bitmapset.c b/src/backend/nodes/bitmapset.c
index 98660524ad..fcd8e2ccbc 100644
--- a/src/backend/nodes/bitmapset.c
+++ b/src/backend/nodes/bitmapset.c
@@ -32,39 +32,7 @@
#define BITMAPSET_SIZE(nwords) \
(offsetof(Bitmapset, words) + (nwords) * sizeof(bitmapword))
-/*----------
- * This is a well-known cute trick for isolating the rightmost one-bit
- * in a word. It assumes two's complement arithmetic. Consider any
- * nonzero value, and focus attention on the rightmost one. The value is
- * then something like
- * xxxxxx10000
- * where x's are unspecified bits. The two's complement negative is formed
- * by inverting all the bits and adding one. Inversion gives
- * yyyyyy01111
- * where each y is the inverse of the corresponding x. Incrementing gives
- * yyyyyy10000
- * and then ANDing with the original value gives
- * 00000010000
- * This works for all cases except original value = zero, where of course
- * we get zero.
- *----------
- */
-#define RIGHTMOST_ONE(x) ((signedbitmapword) (x) & -((signedbitmapword) (x)))
-
-#define HAS_MULTIPLE_ONES(x) ((bitmapword) RIGHTMOST_ONE(x) != (x))
-
-/* Select appropriate bit-twiddling functions for bitmap word size */
-#if BITS_PER_BITMAPWORD == 32
-#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w)
-#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w)
-#define bmw_popcount(w) pg_popcount32(w)
-#elif BITS_PER_BITMAPWORD == 64
-#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w)
-#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w)
-#define bmw_popcount(w) pg_popcount64(w)
-#else
-#error "invalid BITS_PER_BITMAPWORD"
-#endif
+#define HAS_MULTIPLE_ONES(x) (bmw_rightmost_one(x) != (x))
/*
@@ -1013,7 +981,7 @@ bms_first_member(Bitmapset *a)
{
int result;
- w = RIGHTMOST_ONE(w);
+ w = bmw_rightmost_one(w);
a->words[wordnum] &= ~w;
result = wordnum * BITS_PER_BITMAPWORD;
diff --git a/src/include/nodes/bitmapset.h b/src/include/nodes/bitmapset.h
index 3d2225e1ae..5f9a511b4a 100644
--- a/src/include/nodes/bitmapset.h
+++ b/src/include/nodes/bitmapset.h
@@ -38,13 +38,11 @@ struct List;
#define BITS_PER_BITMAPWORD 64
typedef uint64 bitmapword; /* must be an unsigned type */
-typedef int64 signedbitmapword; /* must be the matching signed type */
#else
#define BITS_PER_BITMAPWORD 32
typedef uint32 bitmapword; /* must be an unsigned type */
-typedef int32 signedbitmapword; /* must be the matching signed type */
#endif
@@ -75,6 +73,20 @@ typedef enum
BMS_MULTIPLE /* >1 member */
} BMS_Membership;
+/* Select appropriate bit-twiddling functions for bitmap word size */
+#if BITS_PER_BITMAPWORD == 32
+#define bmw_rightmost_one(w) pg_rightmost_one32(w)
+#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w)
+#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w)
+#define bmw_popcount(w) pg_popcount32(w)
+#elif BITS_PER_BITMAPWORD == 64
+#define bmw_rightmost_one(w) pg_rightmost_one64(w)
+#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w)
+#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w)
+#define bmw_popcount(w) pg_popcount64(w)
+#else
+#error "invalid BITS_PER_BITMAPWORD"
+#endif
/*
* function prototypes in nodes/bitmapset.c
diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h
index a49a9c03d9..7235ad25ee 100644
--- a/src/include/port/pg_bitutils.h
+++ b/src/include/port/pg_bitutils.h
@@ -17,6 +17,37 @@ extern PGDLLIMPORT const uint8 pg_leftmost_one_pos[256];
extern PGDLLIMPORT const uint8 pg_rightmost_one_pos[256];
extern PGDLLIMPORT const uint8 pg_number_of_ones[256];
+/*----------
+ * This is a well-known cute trick for isolating the rightmost one-bit
+ * in a word. It assumes two's complement arithmetic. Consider any
+ * nonzero value, and focus attention on the rightmost one. The value is
+ * then something like
+ * xxxxxx10000
+ * where x's are unspecified bits. The two's complement negative is formed
+ * by inverting all the bits and adding one. Inversion gives
+ * yyyyyy01111
+ * where each y is the inverse of the corresponding x. Incrementing gives
+ * yyyyyy10000
+ * and then ANDing with the original value gives
+ * 00000010000
+ * This works for all cases except original value = zero, where of course
+ * we get zero.
+ *----------
+ */
+static inline uint32
+pg_rightmost_one32(uint32 word)
+{
+ int32 result = (int32) word & -((int32) word);
+ return (uint32) result;
+}
+
+static inline uint64
+pg_rightmost_one64(uint64 word)
+{
+ int64 result = (int64) word & -((int64) word);
+ return (uint64) result;
+}
+
/*
* pg_leftmost_one_pos32
* Returns the position of the most significant set bit in "word",
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 36d1dc0117..a0c60feade 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3669,7 +3669,6 @@ shmem_request_hook_type
shmem_startup_hook_type
sig_atomic_t
sigjmp_buf
-signedbitmapword
sigset_t
size_t
slist_head
--
2.39.1
[text/x-patch] v28-0001-Introduce-helper-SIMD-functions-for-small-byte-a.patch (2.9K, ../../CAFBsxsF4QAOzSPbQvekFn56WJJuYoZM8m=FqbKyb3dwhB+oyWg@mail.gmail.com/4-v28-0001-Introduce-helper-SIMD-functions-for-small-byte-a.patch)
download | inline diff:
From d577ef9d9755e7ca4d3722c1a044381a81d66244 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 24 Oct 2022 14:07:09 +0900
Subject: [PATCH v28 01/10] Introduce helper SIMD functions for small byte
arrays
vector8_min - helper for emulating ">=" semantics
vector8_highbit_mask - used to turn the result of a vector
comparison into a bitmask
Masahiko Sawada
Reviewed by Nathan Bossart, additional adjustments by me
Discussion: https://www.postgresql.org/message-id/CAD21AoDap240WDDdUDE0JMpCmuMMnGajrKrkCRxM7zn9Xk3JRA%40mail.gmail.com
---
src/include/port/simd.h | 47 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)
diff --git a/src/include/port/simd.h b/src/include/port/simd.h
index c836360d4b..350e2caaea 100644
--- a/src/include/port/simd.h
+++ b/src/include/port/simd.h
@@ -79,6 +79,7 @@ static inline bool vector8_has_le(const Vector8 v, const uint8 c);
static inline bool vector8_is_highbit_set(const Vector8 v);
#ifndef USE_NO_SIMD
static inline bool vector32_is_highbit_set(const Vector32 v);
+static inline uint32 vector8_highbit_mask(const Vector8 v);
#endif
/* arithmetic operations */
@@ -96,6 +97,7 @@ static inline Vector8 vector8_ssub(const Vector8 v1, const Vector8 v2);
*/
#ifndef USE_NO_SIMD
static inline Vector8 vector8_eq(const Vector8 v1, const Vector8 v2);
+static inline Vector8 vector8_min(const Vector8 v1, const Vector8 v2);
static inline Vector32 vector32_eq(const Vector32 v1, const Vector32 v2);
#endif
@@ -299,6 +301,36 @@ vector32_is_highbit_set(const Vector32 v)
}
#endif /* ! USE_NO_SIMD */
+/*
+ * Return a bitmask formed from the high-bit of each element.
+ */
+#ifndef USE_NO_SIMD
+static inline uint32
+vector8_highbit_mask(const Vector8 v)
+{
+#ifdef USE_SSE2
+ return (uint32) _mm_movemask_epi8(v);
+#elif defined(USE_NEON)
+ /*
+ * Note: There is a faster way to do this, but it returns a uint64 and
+ * and if the caller wanted to extract the bit position using CTZ,
+ * it would have to divide that result by 4.
+ */
+ static const uint8 mask[16] = {
+ 1 << 0, 1 << 1, 1 << 2, 1 << 3,
+ 1 << 4, 1 << 5, 1 << 6, 1 << 7,
+ 1 << 0, 1 << 1, 1 << 2, 1 << 3,
+ 1 << 4, 1 << 5, 1 << 6, 1 << 7,
+ };
+
+ uint8x16_t masked = vandq_u8(vld1q_u8(mask), (uint8x16_t) vshrq_n_s8(v, 7));
+ uint8x16_t maskedhi = vextq_u8(masked, masked, 8);
+
+ return (uint32) vaddvq_u16((uint16x8_t) vzip1q_u8(masked, maskedhi));
+#endif
+}
+#endif /* ! USE_NO_SIMD */
+
/*
* Return the bitwise OR of the inputs
*/
@@ -372,4 +404,19 @@ vector32_eq(const Vector32 v1, const Vector32 v2)
}
#endif /* ! USE_NO_SIMD */
+/*
+ * Given two vectors, return a vector with the minimum element of each.
+ */
+#ifndef USE_NO_SIMD
+static inline Vector8
+vector8_min(const Vector8 v1, const Vector8 v2)
+{
+#ifdef USE_SSE2
+ return _mm_min_epu8(v1, v2);
+#elif defined(USE_NEON)
+ return vminq_u8(v1, v2);
+#endif
+}
+#endif /* ! USE_NO_SIMD */
+
#endif /* SIMD_H */
--
2.39.1
[text/x-patch] v28-0005-Do-bitmap-conversion-in-one-place-rather-than-fo.patch (3.8K, ../../CAFBsxsF4QAOzSPbQvekFn56WJJuYoZM8m=FqbKyb3dwhB+oyWg@mail.gmail.com/5-v28-0005-Do-bitmap-conversion-in-one-place-rather-than-fo.patch)
download | inline diff:
From dba9497b5b587da873fbb2de89570ec8b36d604b Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Sun, 12 Feb 2023 15:17:40 +0700
Subject: [PATCH v28 05/10] Do bitmap conversion in one place rather than
forcing callers to do it
---
src/backend/access/common/tidstore.c | 31 +++++++++++++++-------------
1 file changed, 17 insertions(+), 14 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index ff8e66936e..ad8c0866e2 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -70,6 +70,7 @@
* and value, respectively.
*/
#define TIDSTORE_VALUE_NBITS 6 /* log(64, 2) */
+#define TIDSTORE_OFFSET_MASK ((1 << TIDSTORE_VALUE_NBITS) - 1)
/* A magic value used to identify our TidStores. */
#define TIDSTORE_MAGIC 0x826f6a10
@@ -158,8 +159,8 @@ typedef struct TidStoreIter
static void tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val);
static inline BlockNumber key_get_blkno(TidStore *ts, uint64 key);
-static inline uint64 encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint32 *off);
-static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off);
+static inline uint64 encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit);
+static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit);
/*
* Create a TidStore. The returned object is allocated in backend-local memory.
@@ -376,10 +377,10 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
for (int i = 0; i < num_offsets; i++)
{
- uint32 off;
+ uint64 off_bit;
/* encode the tid to key and val */
- key = encode_key_off(ts, blkno, offsets[i], &off);
+ key = encode_key_off(ts, blkno, offsets[i], &off_bit);
/* make sure we scanned the line pointer array in order */
Assert(key >= prev_key);
@@ -401,7 +402,7 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
prev_key = key;
}
- off_bitmap |= UINT64CONST(1) << off;
+ off_bitmap |= off_bit;
}
/* save the final index for later */
@@ -441,10 +442,10 @@ tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
{
uint64 key;
uint64 val = 0;
- uint32 off;
+ uint64 off_bit;
bool found;
- key = tid_to_key_off(ts, tid, &off);
+ key = tid_to_key_off(ts, tid, &off_bit);
if (TidStoreIsShared(ts))
found = shared_rt_search(ts->tree.shared, key, &val);
@@ -454,7 +455,7 @@ tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
if (!found)
return false;
- return (val & (UINT64CONST(1) << off)) != 0;
+ return (val & off_bit) != 0;
}
/*
@@ -660,26 +661,28 @@ key_get_blkno(TidStore *ts, uint64 key)
/* Encode a tid to key and offset */
static inline uint64
-tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off)
+tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit)
{
uint32 offset = ItemPointerGetOffsetNumber(tid);
BlockNumber block = ItemPointerGetBlockNumber(tid);
- return encode_key_off(ts, block, offset, off);
+ return encode_key_off(ts, block, offset, off_bit);
}
/* encode a block and offset to a key and partial offset */
static inline uint64
-encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint32 *off)
+encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit)
{
uint64 key;
uint64 tid_i;
+ uint32 off_lower;
- tid_i = offset | ((uint64) block << ts->control->offset_nbits);
+ off_lower = offset & TIDSTORE_OFFSET_MASK;
+ Assert(off_lower < (sizeof(uint64) * BITS_PER_BYTE));
- *off = tid_i & ((UINT64CONST(1) << TIDSTORE_VALUE_NBITS) - 1);
+ *off_bit = UINT64CONST(1) << off_lower;
+ tid_i = offset | ((uint64) block << ts->control->offset_nbits);
key = tid_i >> TIDSTORE_VALUE_NBITS;
- Assert(*off < (sizeof(uint64) * BITS_PER_BYTE));
return key;
}
--
2.39.1
[text/x-patch] v28-0004-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch (35.5K, ../../CAFBsxsF4QAOzSPbQvekFn56WJJuYoZM8m=FqbKyb3dwhB+oyWg@mail.gmail.com/6-v28-0004-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch)
download | inline diff:
From 1be520c83274bc3a2f068689e665c254c8e3c04e Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 23 Dec 2022 23:50:39 +0900
Subject: [PATCH v28 04/10] Add TIDStore, to store sets of TIDs
(ItemPointerData) efficiently.
The TIDStore is designed to store large sets of TIDs efficiently, and
is backed by the radix tree. A TID is encoded into 64-bit key and
value and inserted to the radix tree.
The TIDStore is not used for anything yet, aside from the test code,
but the follow up patch integrates the TIDStore with lazy vacuum,
reducing lazy vacuum memory usage and lifting the 1GB limit on its
size, by storing the list of dead TIDs more efficiently.
This includes a unit test module, in src/test/modules/test_tidstore.
---
doc/src/sgml/monitoring.sgml | 4 +
src/backend/access/common/Makefile | 1 +
src/backend/access/common/meson.build | 1 +
src/backend/access/common/tidstore.c | 685 ++++++++++++++++++
src/backend/storage/lmgr/lwlock.c | 2 +
src/include/access/tidstore.h | 49 ++
src/include/storage/lwlock.h | 1 +
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_tidstore/Makefile | 23 +
.../test_tidstore/expected/test_tidstore.out | 13 +
src/test/modules/test_tidstore/meson.build | 35 +
.../test_tidstore/sql/test_tidstore.sql | 7 +
.../test_tidstore/test_tidstore--1.0.sql | 8 +
.../modules/test_tidstore/test_tidstore.c | 195 +++++
.../test_tidstore/test_tidstore.control | 4 +
16 files changed, 1030 insertions(+)
create mode 100644 src/backend/access/common/tidstore.c
create mode 100644 src/include/access/tidstore.h
create mode 100644 src/test/modules/test_tidstore/Makefile
create mode 100644 src/test/modules/test_tidstore/expected/test_tidstore.out
create mode 100644 src/test/modules/test_tidstore/meson.build
create mode 100644 src/test/modules/test_tidstore/sql/test_tidstore.sql
create mode 100644 src/test/modules/test_tidstore/test_tidstore--1.0.sql
create mode 100644 src/test/modules/test_tidstore/test_tidstore.c
create mode 100644 src/test/modules/test_tidstore/test_tidstore.control
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index b246ddc634..e44387d2c1 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2192,6 +2192,10 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<entry>Waiting to access a shared TID bitmap during a parallel bitmap
index scan.</entry>
</row>
+ <row>
+ <entry><literal>SharedTidStore</literal></entry>
+ <entry>Waiting to access a shared TID store.</entry>
+ </row>
<row>
<entry><literal>SharedTupleStore</literal></entry>
<entry>Waiting to access a shared tuple store during parallel
diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile
index b9aff0ccfd..67b8cc6108 100644
--- a/src/backend/access/common/Makefile
+++ b/src/backend/access/common/Makefile
@@ -27,6 +27,7 @@ OBJS = \
syncscan.o \
toast_compression.o \
toast_internals.o \
+ tidstore.o \
tupconvert.o \
tupdesc.o
diff --git a/src/backend/access/common/meson.build b/src/backend/access/common/meson.build
index f5ac17b498..fce19c09ce 100644
--- a/src/backend/access/common/meson.build
+++ b/src/backend/access/common/meson.build
@@ -15,6 +15,7 @@ backend_sources += files(
'syncscan.c',
'toast_compression.c',
'toast_internals.c',
+ 'tidstore.c',
'tupconvert.c',
'tupdesc.c',
)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
new file mode 100644
index 0000000000..ff8e66936e
--- /dev/null
+++ b/src/backend/access/common/tidstore.c
@@ -0,0 +1,685 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.c
+ * Tid (ItemPointerData) storage implementation.
+ *
+ * This module provides a in-memory data structure to store Tids (ItemPointer).
+ * Internally, a tid is encoded as a pair of 64-bit key and 64-bit value, and
+ * stored in the radix tree.
+ *
+ * A TidStore can be shared among parallel worker processes by passing DSA area
+ * to tidstore_create(). Other backends can attach to the shared TidStore by
+ * tidstore_attach().
+ *
+ * Regarding the concurrency, it basically relies on the concurrency support in
+ * the radix tree, but we acquires the lock on a TidStore in some cases, for
+ * example, when to reset the store and when to access the number tids in the
+ * store (num_tids).
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/access/common/tidstore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/tidstore.h"
+#include "miscadmin.h"
+#include "port/pg_bitutils.h"
+#include "storage/lwlock.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/*
+ * For encoding purposes, tids are represented as a pair of 64-bit key and
+ * 64-bit value. First, we construct 64-bit unsigned integer by combining
+ * the block number and the offset number. The number of bits used for the
+ * offset number is specified by max_offsets in tidstore_create(). We are
+ * frugal with the bits, because smaller keys could help keeping the radix
+ * tree shallow.
+ *
+ * For example, a tid of heap with 8kB blocks uses the lowest 9 bits for
+ * the offset number and uses the next 32 bits for the block number. That
+ * is, only 41 bits are used:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ *
+ * X = bits used for offset number
+ * Y = bits used for block number
+ * u = unused bit
+ * (high on the left, low on the right)
+ *
+ * 9 bits are enough for the offset number, because MaxHeapTuplesPerPage < 2^9
+ * on 8kB blocks.
+ *
+ * The 64-bit value is the bitmap representation of the lowest 6 bits
+ * (TIDSTORE_VALUE_NBITS) of the integer, and the rest 35 bits are used
+ * as the key:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ * |----| value
+ * |---------------------------------------------| key
+ *
+ * The maximum height of the radix tree is 5 in this case.
+ *
+ * If the number of bits for offset number fits in a 64-bit value, we don't
+ * encode tids but directly use the block number and the offset number as key
+ * and value, respectively.
+ */
+#define TIDSTORE_VALUE_NBITS 6 /* log(64, 2) */
+
+/* A magic value used to identify our TidStores. */
+#define TIDSTORE_MAGIC 0x826f6a10
+
+#define RT_PREFIX local_rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE uint64
+#include "lib/radixtree.h"
+
+#define RT_PREFIX shared_rt
+#define RT_SHMEM
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE uint64
+#include "lib/radixtree.h"
+
+/* The control object for a TidStore */
+typedef struct TidStoreControl
+{
+ /* the number of tids in the store */
+ int64 num_tids;
+
+ /* These values are never changed after creation */
+ size_t max_bytes; /* the maximum bytes a TidStore can use */
+ int max_offset; /* the maximum offset number */
+ int offset_nbits; /* the number of bits required for max_offset */
+ int offset_key_nbits; /* the number of bits of a offset number
+ * used for the key */
+
+ /* The below fields are used only in shared case */
+
+ uint32 magic;
+ LWLock lock;
+
+ /* handles for TidStore and radix tree */
+ tidstore_handle handle;
+ shared_rt_handle tree_handle;
+} TidStoreControl;
+
+/* Per-backend state for a TidStore */
+struct TidStore
+{
+ /*
+ * Control object. This is allocated in DSA area 'area' in the shared
+ * case, otherwise in backend-local memory.
+ */
+ TidStoreControl *control;
+
+ /* Storage for Tids. Use either one depending on TidStoreIsShared() */
+ union
+ {
+ local_rt_radix_tree *local;
+ shared_rt_radix_tree *shared;
+ } tree;
+
+ /* DSA area for TidStore if used */
+ dsa_area *area;
+};
+#define TidStoreIsShared(ts) ((ts)->area != NULL)
+
+/* Iterator for TidStore */
+typedef struct TidStoreIter
+{
+ TidStore *ts;
+
+ /* iterator of radix tree. Use either one depending on TidStoreIsShared() */
+ union
+ {
+ shared_rt_iter *shared;
+ local_rt_iter *local;
+ } tree_iter;
+
+ /* we returned all tids? */
+ bool finished;
+
+ /* save for the next iteration */
+ uint64 next_key;
+ uint64 next_val;
+
+ /* output for the caller */
+ TidStoreIterResult result;
+} TidStoreIter;
+
+static void tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val);
+static inline BlockNumber key_get_blkno(TidStore *ts, uint64 key);
+static inline uint64 encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint32 *off);
+static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off);
+
+/*
+ * Create a TidStore. The returned object is allocated in backend-local memory.
+ * The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
+ */
+TidStore *
+tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
+{
+ TidStore *ts;
+
+ ts = palloc0(sizeof(TidStore));
+
+ /*
+ * Create the radix tree for the main storage.
+ *
+ * Memory consumption depends on the number of stored tids, but also on the
+ * distribution of them, how the radix tree stores, and the memory management
+ * that backed the radix tree. The maximum bytes that a TidStore can
+ * use is specified by the max_bytes in tidstore_create(). We want the total
+ * amount of memory consumption by a TidStore not to exceed the max_bytes.
+ *
+ * In local TidStore cases, the radix tree uses slab allocators for each kind
+ * of node class. The most memory consuming case while adding Tids associated
+ * with one page (i.e. during tidstore_add_tids()) is that we allocate a new
+ * slab block for a new radix tree node, which is approximately 70kB. Therefore,
+ * we deduct 70kB from the max_bytes.
+ *
+ * In shared cases, DSA allocates the memory segments big enough to follow
+ * a geometric series that approximately doubles the total DSA size (see
+ * make_new_segment() in dsa.c). We simulated the how DSA increases segment
+ * size and the simulation revealed, the 75% threshold for the maximum bytes
+ * perfectly works in case where the max_bytes is a power-of-2, and the 60%
+ * threshold works for other cases.
+ */
+ if (area != NULL)
+ {
+ dsa_pointer dp;
+ float ratio = ((max_bytes & (max_bytes - 1)) == 0) ? 0.75 : 0.6;
+
+ ts->tree.shared = shared_rt_create(CurrentMemoryContext, area,
+ LWTRANCHE_SHARED_TIDSTORE);
+
+ dp = dsa_allocate0(area, sizeof(TidStoreControl));
+ ts->control = (TidStoreControl *) dsa_get_address(area, dp);
+ ts->control->max_bytes = (uint64) (max_bytes * ratio);
+ ts->area = area;
+
+ ts->control->magic = TIDSTORE_MAGIC;
+ LWLockInitialize(&ts->control->lock, LWTRANCHE_SHARED_TIDSTORE);
+ ts->control->handle = dp;
+ ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+ }
+ else
+ {
+ ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+ ts->control = (TidStoreControl *) palloc0(sizeof(TidStoreControl));
+ ts->control->max_bytes = max_bytes - (70 * 1024);
+ }
+
+ ts->control->max_offset = max_offset;
+ ts->control->offset_nbits = pg_ceil_log2_32(max_offset);
+
+ if (ts->control->offset_nbits < TIDSTORE_VALUE_NBITS)
+ ts->control->offset_nbits = TIDSTORE_VALUE_NBITS;
+
+ /*
+ * We use tid encoding if the number of bits for the offset number doesn't
+ * fix in a value, uint64.
+ */
+ ts->control->offset_key_nbits =
+ ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
+
+ return ts;
+}
+
+/*
+ * Attach to the shared TidStore using a handle. The returned object is
+ * allocated in backend-local memory using the CurrentMemoryContext.
+ */
+TidStore *
+tidstore_attach(dsa_area *area, tidstore_handle handle)
+{
+ TidStore *ts;
+ dsa_pointer control;
+
+ Assert(area != NULL);
+ Assert(DsaPointerIsValid(handle));
+
+ /* create per-backend state */
+ ts = palloc0(sizeof(TidStore));
+
+ /* Find the control object in shared memory */
+ control = handle;
+
+ /* Set up the TidStore */
+ ts->control = (TidStoreControl *) dsa_get_address(area, control);
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ ts->tree.shared = shared_rt_attach(area, ts->control->tree_handle);
+ ts->area = area;
+
+ return ts;
+}
+
+/*
+ * Detach from a TidStore. This detaches from radix tree and frees the
+ * backend-local resources. The radix tree will continue to exist until
+ * it is either explicitly destroyed, or the area that backs it is returned
+ * to the operating system.
+ */
+void
+tidstore_detach(TidStore *ts)
+{
+ Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+ shared_rt_detach(ts->tree.shared);
+ pfree(ts);
+}
+
+/*
+ * Destroy a TidStore, returning all memory.
+ *
+ * TODO: The caller must be certain that no other backend will attempt to
+ * access the TidStore before calling this function. Other backend must
+ * explicitly call tidstore_detach to free up backend-local memory associated
+ * with the TidStore. The backend that calls tidstore_destroy must not call
+ * tidstore_detach.
+ */
+void
+tidstore_destroy(TidStore *ts)
+{
+ if (TidStoreIsShared(ts))
+ {
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ /*
+ * Vandalize the control block to help catch programming error where
+ * other backends access the memory formerly occupied by this radix
+ * tree.
+ */
+ ts->control->magic = 0;
+ dsa_free(ts->area, ts->control->handle);
+ shared_rt_free(ts->tree.shared);
+ }
+ else
+ {
+ pfree(ts->control);
+ local_rt_free(ts->tree.local);
+ }
+
+ pfree(ts);
+}
+
+/*
+ * Forget all collected Tids. It's similar to tidstore_destroy but we don't free
+ * entire TidStore but recreate only the radix tree storage.
+ */
+void
+tidstore_reset(TidStore *ts)
+{
+ if (TidStoreIsShared(ts))
+ {
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+ /*
+ * Free the radix tree and return allocated DSA segments to
+ * the operating system.
+ */
+ shared_rt_free(ts->tree.shared);
+ dsa_trim(ts->area);
+
+ /* Recreate the radix tree */
+ ts->tree.shared = shared_rt_create(CurrentMemoryContext, ts->area,
+ LWTRANCHE_SHARED_TIDSTORE);
+
+ /* update the radix tree handle as we recreated it */
+ ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
+
+ LWLockRelease(&ts->control->lock);
+ }
+ else
+ {
+ local_rt_free(ts->tree.local);
+ ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
+ }
+}
+
+/* Add Tids on a block to TidStore */
+void
+tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets)
+{
+ uint64 *values;
+ uint64 key;
+ uint64 prev_key;
+ uint64 off_bitmap = 0;
+ int idx;
+ const uint64 key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
+ const int nkeys = UINT64CONST(1) << ts->control->offset_key_nbits;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ values = palloc(sizeof(uint64) * nkeys);
+ key = prev_key = key_base;
+
+ for (int i = 0; i < num_offsets; i++)
+ {
+ uint32 off;
+
+ /* encode the tid to key and val */
+ key = encode_key_off(ts, blkno, offsets[i], &off);
+
+ /* make sure we scanned the line pointer array in order */
+ Assert(key >= prev_key);
+
+ if (key > prev_key)
+ {
+ idx = prev_key - key_base;
+ Assert(idx >= 0 && idx < nkeys);
+
+ /* write out offset bitmap for this key */
+ values[idx] = off_bitmap;
+
+ /* zero out any gaps up to the current key */
+ for (int empty_idx = idx + 1; empty_idx < key - key_base; empty_idx++)
+ values[empty_idx] = 0;
+
+ /* reset for current key -- the current offset will be handled below */
+ off_bitmap = 0;
+ prev_key = key;
+ }
+
+ off_bitmap |= UINT64CONST(1) << off;
+ }
+
+ /* save the final index for later */
+ idx = key - key_base;
+ /* write out last offset bitmap */
+ values[idx] = off_bitmap;
+
+ if (TidStoreIsShared(ts))
+ LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+ /* insert the calculated key-values to the tree */
+ for (int i = 0; i <= idx; i++)
+ {
+ if (values[i])
+ {
+ key = key_base + i;
+
+ if (TidStoreIsShared(ts))
+ shared_rt_set(ts->tree.shared, key, &values[i]);
+ else
+ local_rt_set(ts->tree.local, key, &values[i]);
+ }
+ }
+
+ /* update statistics */
+ ts->control->num_tids += num_offsets;
+
+ if (TidStoreIsShared(ts))
+ LWLockRelease(&ts->control->lock);
+
+ pfree(values);
+}
+
+/* Return true if the given tid is present in the TidStore */
+bool
+tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
+{
+ uint64 key;
+ uint64 val = 0;
+ uint32 off;
+ bool found;
+
+ key = tid_to_key_off(ts, tid, &off);
+
+ if (TidStoreIsShared(ts))
+ found = shared_rt_search(ts->tree.shared, key, &val);
+ else
+ found = local_rt_search(ts->tree.local, key, &val);
+
+ if (!found)
+ return false;
+
+ return (val & (UINT64CONST(1) << off)) != 0;
+}
+
+/*
+ * Prepare to iterate through a TidStore. Since the radix tree is locked during the
+ * iteration, so tidstore_end_iterate() needs to called when finished.
+ *
+ * Concurrent updates during the iteration will be blocked when inserting a
+ * key-value to the radix tree.
+ */
+TidStoreIter *
+tidstore_begin_iterate(TidStore *ts)
+{
+ TidStoreIter *iter;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ iter = palloc0(sizeof(TidStoreIter));
+ iter->ts = ts;
+
+ iter->result.blkno = InvalidBlockNumber;
+ iter->result.offsets = palloc(sizeof(OffsetNumber) * ts->control->max_offset);
+
+ if (TidStoreIsShared(ts))
+ iter->tree_iter.shared = shared_rt_begin_iterate(ts->tree.shared);
+ else
+ iter->tree_iter.local = local_rt_begin_iterate(ts->tree.local);
+
+ /* If the TidStore is empty, there is no business */
+ if (tidstore_num_tids(ts) == 0)
+ iter->finished = true;
+
+ return iter;
+}
+
+static inline bool
+tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
+{
+ if (TidStoreIsShared(iter->ts))
+ return shared_rt_iterate_next(iter->tree_iter.shared, key, val);
+
+ return local_rt_iterate_next(iter->tree_iter.local, key, val);
+}
+
+/*
+ * Scan the TidStore and return a pointer to TidStoreIterResult that has tids
+ * in one block. We return the block numbers in ascending order and the offset
+ * numbers in each result is also sorted in ascending order.
+ */
+TidStoreIterResult *
+tidstore_iterate_next(TidStoreIter *iter)
+{
+ uint64 key;
+ uint64 val;
+ TidStoreIterResult *result = &(iter->result);
+
+ if (iter->finished)
+ return NULL;
+
+ if (BlockNumberIsValid(result->blkno))
+ {
+ /* Process the previously collected key-value */
+ result->num_offsets = 0;
+ tidstore_iter_extract_tids(iter, iter->next_key, iter->next_val);
+ }
+
+ while (tidstore_iter_kv(iter, &key, &val))
+ {
+ BlockNumber blkno;
+
+ blkno = key_get_blkno(iter->ts, key);
+
+ if (BlockNumberIsValid(result->blkno) && result->blkno != blkno)
+ {
+ /*
+ * We got a key-value pair for a different block. So return the
+ * collected tids, and remember the key-value for the next iteration.
+ */
+ iter->next_key = key;
+ iter->next_val = val;
+ return result;
+ }
+
+ /* Collect tids extracted from the key-value pair */
+ tidstore_iter_extract_tids(iter, key, val);
+ }
+
+ iter->finished = true;
+ return result;
+}
+
+/*
+ * Finish an iteration over TidStore. This needs to be called after finishing
+ * or when existing an iteration.
+ */
+void
+tidstore_end_iterate(TidStoreIter *iter)
+{
+ if (TidStoreIsShared(iter->ts))
+ shared_rt_end_iterate(iter->tree_iter.shared);
+ else
+ local_rt_end_iterate(iter->tree_iter.local);
+
+ pfree(iter->result.offsets);
+ pfree(iter);
+}
+
+/* Return the number of tids we collected so far */
+int64
+tidstore_num_tids(TidStore *ts)
+{
+ uint64 num_tids;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ if (!TidStoreIsShared(ts))
+ return ts->control->num_tids;
+
+ LWLockAcquire(&ts->control->lock, LW_SHARED);
+ num_tids = ts->control->num_tids;
+ LWLockRelease(&ts->control->lock);
+
+ return num_tids;
+}
+
+/* Return true if the current memory usage of TidStore exceeds the limit */
+bool
+tidstore_is_full(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ return (tidstore_memory_usage(ts) > ts->control->max_bytes);
+}
+
+/* Return the maximum memory TidStore can use */
+size_t
+tidstore_max_memory(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ return ts->control->max_bytes;
+}
+
+/* Return the memory usage of TidStore */
+size_t
+tidstore_memory_usage(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ /*
+ * In the shared case, TidStoreControl and radix_tree are backed by the
+ * same DSA area and rt_memory_usage() returns the value including both.
+ * So we don't need to add the size of TidStoreControl separately.
+ */
+ if (TidStoreIsShared(ts))
+ return sizeof(TidStore) + shared_rt_memory_usage(ts->tree.shared);
+
+ return sizeof(TidStore) + sizeof(TidStore) + local_rt_memory_usage(ts->tree.local);
+}
+
+/*
+ * Get a handle that can be used by other processes to attach to this TidStore
+ */
+tidstore_handle
+tidstore_get_handle(TidStore *ts)
+{
+ Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+ return ts->control->handle;
+}
+
+/* Extract tids from the given key-value pair */
+static void
+tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
+{
+ TidStoreIterResult *result = (&iter->result);
+
+ for (int i = 0; i < sizeof(uint64) * BITS_PER_BYTE; i++)
+ {
+ uint64 tid_i;
+ OffsetNumber off;
+
+ if ((val & (UINT64CONST(1) << i)) == 0)
+ continue;
+
+ tid_i = key << TIDSTORE_VALUE_NBITS;
+ tid_i |= i;
+
+ off = tid_i & ((UINT64CONST(1) << iter->ts->control->offset_nbits) - 1);
+
+ Assert(result->num_offsets < iter->ts->control->max_offset);
+ result->offsets[result->num_offsets++] = off;
+ }
+
+ result->blkno = key_get_blkno(iter->ts, key);
+}
+
+/* Get block number from the given key */
+static inline BlockNumber
+key_get_blkno(TidStore *ts, uint64 key)
+{
+ return (BlockNumber) (key >> ts->control->offset_key_nbits);
+}
+
+/* Encode a tid to key and offset */
+static inline uint64
+tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off)
+{
+ uint32 offset = ItemPointerGetOffsetNumber(tid);
+ BlockNumber block = ItemPointerGetBlockNumber(tid);
+
+ return encode_key_off(ts, block, offset, off);
+}
+
+/* encode a block and offset to a key and partial offset */
+static inline uint64
+encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint32 *off)
+{
+ uint64 key;
+ uint64 tid_i;
+
+ tid_i = offset | ((uint64) block << ts->control->offset_nbits);
+
+ *off = tid_i & ((UINT64CONST(1) << TIDSTORE_VALUE_NBITS) - 1);
+ key = tid_i >> TIDSTORE_VALUE_NBITS;
+ Assert(*off < (sizeof(uint64) * BITS_PER_BYTE));
+
+ return key;
+}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index d2ec396045..55b3a04097 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -176,6 +176,8 @@ static const char *const BuiltinTrancheNames[] = {
"SharedTupleStore",
/* LWTRANCHE_SHARED_TIDBITMAP: */
"SharedTidBitmap",
+ /* LWTRANCHE_SHARED_TIDSTORE: */
+ "SharedTidStore",
/* LWTRANCHE_PARALLEL_APPEND: */
"ParallelAppend",
/* LWTRANCHE_PER_XACT_PREDICATE_LIST: */
diff --git a/src/include/access/tidstore.h b/src/include/access/tidstore.h
new file mode 100644
index 0000000000..a35a52124a
--- /dev/null
+++ b/src/include/access/tidstore.h
@@ -0,0 +1,49 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.h
+ * Tid storage.
+ *
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/tidstore.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef TIDSTORE_H
+#define TIDSTORE_H
+
+#include "storage/itemptr.h"
+#include "utils/dsa.h"
+
+typedef dsa_pointer tidstore_handle;
+
+typedef struct TidStore TidStore;
+typedef struct TidStoreIter TidStoreIter;
+
+typedef struct TidStoreIterResult
+{
+ BlockNumber blkno;
+ OffsetNumber *offsets;
+ int num_offsets;
+} TidStoreIterResult;
+
+extern TidStore *tidstore_create(size_t max_bytes, int max_offset, dsa_area *dsa);
+extern TidStore *tidstore_attach(dsa_area *dsa, dsa_pointer handle);
+extern void tidstore_detach(TidStore *ts);
+extern void tidstore_destroy(TidStore *ts);
+extern void tidstore_reset(TidStore *ts);
+extern void tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets);
+extern bool tidstore_lookup_tid(TidStore *ts, ItemPointer tid);
+extern TidStoreIter * tidstore_begin_iterate(TidStore *ts);
+extern TidStoreIterResult *tidstore_iterate_next(TidStoreIter *iter);
+extern void tidstore_end_iterate(TidStoreIter *iter);
+extern int64 tidstore_num_tids(TidStore *ts);
+extern bool tidstore_is_full(TidStore *ts);
+extern size_t tidstore_max_memory(TidStore *ts);
+extern size_t tidstore_memory_usage(TidStore *ts);
+extern tidstore_handle tidstore_get_handle(TidStore *ts);
+
+#endif /* TIDSTORE_H */
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index d2c7afb8f4..07002fdfbe 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -199,6 +199,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_PER_SESSION_RECORD_TYPMOD,
LWTRANCHE_SHARED_TUPLESTORE,
LWTRANCHE_SHARED_TIDBITMAP,
+ LWTRANCHE_SHARED_TIDSTORE,
LWTRANCHE_PARALLEL_APPEND,
LWTRANCHE_PER_XACT_PREDICATE_LIST,
LWTRANCHE_PGSTATS_DSA,
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 9659eb85d7..bddc16ada7 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
test_rls_hooks \
test_shm_mq \
test_slru \
+ test_tidstore \
unsafe_tests \
worker_spi
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 232cbdac80..c0d5645ad8 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -30,5 +30,6 @@ subdir('test_regex')
subdir('test_rls_hooks')
subdir('test_shm_mq')
subdir('test_slru')
+subdir('test_tidstore')
subdir('unsafe_tests')
subdir('worker_spi')
diff --git a/src/test/modules/test_tidstore/Makefile b/src/test/modules/test_tidstore/Makefile
new file mode 100644
index 0000000000..dab107d70c
--- /dev/null
+++ b/src/test/modules/test_tidstore/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_tidstore/Makefile
+
+MODULE_big = test_tidstore
+OBJS = \
+ $(WIN32RES) \
+ test_tidstore.o
+PGFILEDESC = "test_tidstore - test code for src/backend/access/common/tidstore.c"
+
+EXTENSION = test_tidstore
+DATA = test_tidstore--1.0.sql
+
+REGRESS = test_tidstore
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_tidstore
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_tidstore/expected/test_tidstore.out b/src/test/modules/test_tidstore/expected/test_tidstore.out
new file mode 100644
index 0000000000..7ff2f9af87
--- /dev/null
+++ b/src/test/modules/test_tidstore/expected/test_tidstore.out
@@ -0,0 +1,13 @@
+CREATE EXTENSION test_tidstore;
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
+NOTICE: testing empty tidstore
+NOTICE: testing basic operations
+ test_tidstore
+---------------
+
+(1 row)
+
diff --git a/src/test/modules/test_tidstore/meson.build b/src/test/modules/test_tidstore/meson.build
new file mode 100644
index 0000000000..31f2da7b61
--- /dev/null
+++ b/src/test/modules/test_tidstore/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_tidstore_sources = files(
+ 'test_tidstore.c',
+)
+
+if host_system == 'windows'
+ test_tidstore_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_tidstore',
+ '--FILEDESC', 'test_tidstore - test code for src/backend/access/common/tidstore.c',])
+endif
+
+test_tidstore = shared_module('test_tidstore',
+ test_tidstore_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += test_tidstore
+
+install_data(
+ 'test_tidstore.control',
+ 'test_tidstore--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'test_tidstore',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'test_tidstore',
+ ],
+ },
+}
diff --git a/src/test/modules/test_tidstore/sql/test_tidstore.sql b/src/test/modules/test_tidstore/sql/test_tidstore.sql
new file mode 100644
index 0000000000..03aea31815
--- /dev/null
+++ b/src/test/modules/test_tidstore/sql/test_tidstore.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_tidstore;
+
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
diff --git a/src/test/modules/test_tidstore/test_tidstore--1.0.sql b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
new file mode 100644
index 0000000000..47e9149900
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_tidstore/test_tidstore--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_tidstore" to load this file. \quit
+
+CREATE FUNCTION test_tidstore()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_tidstore/test_tidstore.c b/src/test/modules/test_tidstore/test_tidstore.c
new file mode 100644
index 0000000000..9b849ae8e8
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.c
@@ -0,0 +1,195 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_tidstore.c
+ * Test TidStore data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_tidstore/test_tidstore.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/tidstore.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "utils/memutils.h"
+
+PG_MODULE_MAGIC;
+
+#define TEST_TIDSTORE_MAX_BYTES (2 * 1024 * 1024L) /* 2MB */
+
+PG_FUNCTION_INFO_V1(test_tidstore);
+
+static void
+check_tid(TidStore *ts, BlockNumber blkno, OffsetNumber off, bool expect)
+{
+ ItemPointerData tid;
+ bool found;
+
+ ItemPointerSet(&tid, blkno, off);
+
+ found = tidstore_lookup_tid(ts, &tid);
+
+ if (found != expect)
+ elog(ERROR, "lookup TID (%u, %u) returned %d, expected %d",
+ blkno, off, found, expect);
+}
+
+static void
+test_basic(int max_offset)
+{
+#define TEST_TIDSTORE_NUM_BLOCKS 5
+#define TEST_TIDSTORE_NUM_OFFSETS 5
+
+ TidStore *ts;
+ TidStoreIter *iter;
+ TidStoreIterResult *iter_result;
+ BlockNumber blks[TEST_TIDSTORE_NUM_BLOCKS] = {
+ 0, MaxBlockNumber, MaxBlockNumber - 1, 1, MaxBlockNumber / 2,
+ };
+ BlockNumber blks_sorted[TEST_TIDSTORE_NUM_BLOCKS] = {
+ 0, 1, MaxBlockNumber / 2, MaxBlockNumber - 1, MaxBlockNumber
+ };
+ OffsetNumber offs[TEST_TIDSTORE_NUM_OFFSETS];
+ int blk_idx;
+
+ /* prepare the offset array */
+ offs[0] = FirstOffsetNumber;
+ offs[1] = FirstOffsetNumber + 1;
+ offs[2] = max_offset / 2;
+ offs[3] = max_offset - 1;
+ offs[4] = max_offset;
+
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
+
+ /* add tids */
+ for (int i = 0; i < TEST_TIDSTORE_NUM_BLOCKS; i++)
+ tidstore_add_tids(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
+
+ /* lookup test */
+ for (OffsetNumber off = FirstOffsetNumber ; off < max_offset; off++)
+ {
+ bool expect = false;
+ for (int i = 0; i < TEST_TIDSTORE_NUM_OFFSETS; i++)
+ {
+ if (offs[i] == off)
+ {
+ expect = true;
+ break;
+ }
+ }
+
+ check_tid(ts, 0, off, expect);
+ check_tid(ts, 2, off, false);
+ check_tid(ts, MaxBlockNumber - 2, off, false);
+ check_tid(ts, MaxBlockNumber, off, expect);
+ }
+
+ /* test the number of tids */
+ if (tidstore_num_tids(ts) != (TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS))
+ elog(ERROR, "tidstore_num_tids returned " UINT64_FORMAT ", expected %d",
+ tidstore_num_tids(ts),
+ TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS);
+
+ /* iteration test */
+ iter = tidstore_begin_iterate(ts);
+ blk_idx = 0;
+ while ((iter_result = tidstore_iterate_next(iter)) != NULL)
+ {
+ /* check the returned block number */
+ if (blks_sorted[blk_idx] != iter_result->blkno)
+ elog(ERROR, "tidstore_iterate_next returned block number %u, expected %u",
+ iter_result->blkno, blks_sorted[blk_idx]);
+
+ /* check the returned offset numbers */
+ if (TEST_TIDSTORE_NUM_OFFSETS != iter_result->num_offsets)
+ elog(ERROR, "tidstore_iterate_next returned %u offsets, expected %u",
+ iter_result->num_offsets, TEST_TIDSTORE_NUM_OFFSETS);
+
+ for (int i = 0; i < iter_result->num_offsets; i++)
+ {
+ if (offs[i] != iter_result->offsets[i])
+ elog(ERROR, "tidstore_iterate_next returned offset number %u on block %u, expected %u",
+ iter_result->offsets[i], iter_result->blkno, offs[i]);
+ }
+
+ blk_idx++;
+ }
+
+ if (blk_idx != TEST_TIDSTORE_NUM_BLOCKS)
+ elog(ERROR, "tidstore_iterate_next returned %d blocks, expected %d",
+ blk_idx, TEST_TIDSTORE_NUM_BLOCKS);
+
+ /* remove all tids */
+ tidstore_reset(ts);
+
+ /* test the number of tids */
+ if (tidstore_num_tids(ts) != 0)
+ elog(ERROR, "tidstore_num_tids on empty store returned non-zero");
+
+ /* lookup test for empty store */
+ for (OffsetNumber off = FirstOffsetNumber ; off < MaxHeapTuplesPerPage;
+ off++)
+ {
+ check_tid(ts, 0, off, false);
+ check_tid(ts, 2, off, false);
+ check_tid(ts, MaxBlockNumber - 2, off, false);
+ check_tid(ts, MaxBlockNumber, off, false);
+ }
+
+ tidstore_destroy(ts);
+}
+
+static void
+test_empty(void)
+{
+ TidStore *ts;
+ TidStoreIter *iter;
+ ItemPointerData tid;
+
+ elog(NOTICE, "testing empty tidstore");
+
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
+
+ ItemPointerSet(&tid, 0, FirstOffsetNumber);
+ if (tidstore_lookup_tid(ts, &tid))
+ elog(ERROR, "tidstore_lookup_tid for (0,1) on empty store returned true");
+
+ ItemPointerSet(&tid, MaxBlockNumber, MaxOffsetNumber);
+ if (tidstore_lookup_tid(ts, &tid))
+ elog(ERROR, "tidstore_lookup_tid for (%u,%u) on empty store returned true",
+ MaxBlockNumber, MaxOffsetNumber);
+
+ if (tidstore_num_tids(ts) != 0)
+ elog(ERROR, "tidstore_num_entries on empty store returned non-zero");
+
+ if (tidstore_is_full(ts))
+ elog(ERROR, "tidstore_is_full on empty store returned true");
+
+ iter = tidstore_begin_iterate(ts);
+
+ if (tidstore_iterate_next(iter) != NULL)
+ elog(ERROR, "tidstore_iterate_next on empty store returned TIDs");
+
+ tidstore_end_iterate(iter);
+
+ tidstore_destroy(ts);
+}
+
+Datum
+test_tidstore(PG_FUNCTION_ARGS)
+{
+ test_empty();
+
+ elog(NOTICE, "testing basic operations");
+ test_basic(MaxHeapTuplesPerPage);
+ test_basic(10);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_tidstore/test_tidstore.control b/src/test/modules/test_tidstore/test_tidstore.control
new file mode 100644
index 0000000000..9b6bd4638f
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.control
@@ -0,0 +1,4 @@
+comment = 'Test code for tidstore'
+default_version = '1.0'
+module_pathname = '$libdir/test_tidstore'
+relocatable = true
--
2.39.1
[text/x-patch] v28-0003-Add-radixtree-template.patch (116.8K, ../../CAFBsxsF4QAOzSPbQvekFn56WJJuYoZM8m=FqbKyb3dwhB+oyWg@mail.gmail.com/7-v28-0003-Add-radixtree-template.patch)
download | inline diff:
From bf9d659187537b250683af321b0167d69c7fb18a Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 14 Sep 2022 12:38:51 +0000
Subject: [PATCH v28 03/10] Add radixtree template
WIP: commit message based on template comments
---
src/backend/utils/mmgr/dsa.c | 12 +
src/include/lib/radixtree.h | 2516 +++++++++++++++++
src/include/lib/radixtree_delete_impl.h | 122 +
src/include/lib/radixtree_insert_impl.h | 328 +++
src/include/lib/radixtree_iter_impl.h | 153 +
src/include/lib/radixtree_search_impl.h | 138 +
src/include/utils/dsa.h | 1 +
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_radixtree/.gitignore | 4 +
src/test/modules/test_radixtree/Makefile | 23 +
src/test/modules/test_radixtree/README | 7 +
.../expected/test_radixtree.out | 36 +
src/test/modules/test_radixtree/meson.build | 35 +
.../test_radixtree/sql/test_radixtree.sql | 7 +
.../test_radixtree/test_radixtree--1.0.sql | 8 +
.../modules/test_radixtree/test_radixtree.c | 674 +++++
.../test_radixtree/test_radixtree.control | 4 +
src/tools/pginclude/cpluspluscheck | 6 +
src/tools/pginclude/headerscheck | 6 +
20 files changed, 4082 insertions(+)
create mode 100644 src/include/lib/radixtree.h
create mode 100644 src/include/lib/radixtree_delete_impl.h
create mode 100644 src/include/lib/radixtree_insert_impl.h
create mode 100644 src/include/lib/radixtree_iter_impl.h
create mode 100644 src/include/lib/radixtree_search_impl.h
create mode 100644 src/test/modules/test_radixtree/.gitignore
create mode 100644 src/test/modules/test_radixtree/Makefile
create mode 100644 src/test/modules/test_radixtree/README
create mode 100644 src/test/modules/test_radixtree/expected/test_radixtree.out
create mode 100644 src/test/modules/test_radixtree/meson.build
create mode 100644 src/test/modules/test_radixtree/sql/test_radixtree.sql
create mode 100644 src/test/modules/test_radixtree/test_radixtree--1.0.sql
create mode 100644 src/test/modules/test_radixtree/test_radixtree.c
create mode 100644 src/test/modules/test_radixtree/test_radixtree.control
diff --git a/src/backend/utils/mmgr/dsa.c b/src/backend/utils/mmgr/dsa.c
index f5a62061a3..80555aefff 100644
--- a/src/backend/utils/mmgr/dsa.c
+++ b/src/backend/utils/mmgr/dsa.c
@@ -1024,6 +1024,18 @@ dsa_set_size_limit(dsa_area *area, size_t limit)
LWLockRelease(DSA_AREA_LOCK(area));
}
+size_t
+dsa_get_total_size(dsa_area *area)
+{
+ size_t size;
+
+ LWLockAcquire(DSA_AREA_LOCK(area), LW_SHARED);
+ size = area->control->total_segment_size;
+ LWLockRelease(DSA_AREA_LOCK(area));
+
+ return size;
+}
+
/*
* Aggressively free all spare memory in the hope of returning DSM segments to
* the operating system.
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
new file mode 100644
index 0000000000..1cdb995e54
--- /dev/null
+++ b/src/include/lib/radixtree.h
@@ -0,0 +1,2516 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree.h
+ * Template for adaptive radix tree.
+ *
+ * This module employs the idea from the paper "The Adaptive Radix Tree: ARTful
+ * Indexing for Main-Memory Databases" by Viktor Leis, Alfons Kemper, and Thomas
+ * Neumann, 2013. The radix tree uses adaptive node sizes, a small number of node
+ * types, each with a different numbers of elements. Depending on the number of
+ * children, the appropriate node type is used.
+ *
+ * WIP: notes about traditional radix tree trading off span vs height...
+ *
+ * There are two kinds of nodes, inner nodes and leaves. Inner nodes
+ * map partial keys to child pointers.
+ *
+ * The ART paper mentions three ways to implement leaves:
+ *
+ * "- Single-value leaves: The values are stored using an addi-
+ * tional leaf node type which stores one value.
+ * - Multi-value leaves: The values are stored in one of four
+ * different leaf node types, which mirror the structure of
+ * inner nodes, but contain values instead of pointers.
+ * - Combined pointer/value slots: If values fit into point-
+ * ers, no separate node types are necessary. Instead, each
+ * pointer storage location in an inner node can either
+ * store a pointer or a value."
+ *
+ * We chose "multi-value leaves" to avoid the additional pointer traversal
+ * required by "single-value leaves"
+ *
+ * For simplicity, the key is assumed to be 64-bit unsigned integer. The
+ * tree doesn't need to contain paths where the highest bytes of all keys
+ * are zero. That way, the tree's height adapts to the distribution of keys.
+ *
+ * TODO: In the future it might be worthwhile to offer configurability of
+ * leaf implementation for different use cases. Single-values leaves would
+ * give more flexibility in key type, including variable-length keys.
+ *
+ * There are some optimizations not yet implemented, particularly path
+ * compression and lazy path expansion.
+ *
+ * To handle concurrency, we use a single reader-writer lock for the radix
+ * tree. The radix tree is exclusively locked during write operations such
+ * as RT_SET() and RT_DELETE(), and shared locked during read operations
+ * such as RT_SEARCH(). An iteration also holds the shared lock on the radix
+ * tree until it is completed.
+ *
+ * TODO: The current locking mechanism is not optimized for high concurrency
+ * with mixed read-write workloads. In the future it might be worthwhile
+ * to replace it with the Optimistic Lock Coupling or ROWEX mentioned in
+ * the paper "The ART of Practical Synchronization" by the same authors as
+ * the ART paper, 2016.
+ *
+ * WIP: the radix tree nodes don't shrink.
+ *
+ * To generate a radix tree and associated functions for a use case several
+ * macros have to be #define'ed before this file is included. Including
+ * the file #undef's all those, so a new radix tree can be generated
+ * afterwards.
+ * The relevant parameters are:
+ * - RT_PREFIX - prefix for all symbol names generated. A prefix of 'foo'
+ * will result in radix tree type 'foo_radix_tree' and functions like
+ * 'foo_create'/'foo_free' and so forth.
+ * - RT_DECLARE - if defined function prototypes and type declarations are
+ * generated
+ * - RT_DEFINE - if defined function definitions are generated
+ * - RT_SCOPE - in which scope (e.g. extern, static inline) do function
+ * declarations reside
+ * - RT_VALUE_TYPE - the type of the value.
+ *
+ * Optional parameters:
+ * - RT_SHMEM - if defined, the radix tree is created in the DSA area
+ * so that multiple processes can access it simultaneously.
+ * - RT_DEBUG - if defined add stats tracking and debugging functions
+ *
+ * Interface
+ * ---------
+ *
+ * RT_CREATE - Create a new, empty radix tree
+ * RT_FREE - Free the radix tree
+ * RT_SEARCH - Search a key-value pair
+ * RT_SET - Set a key-value pair
+ * RT_BEGIN_ITERATE - Begin iterating through all key-value pairs
+ * RT_ITERATE_NEXT - Return next key-value pair, if any
+ * RT_END_ITER - End iteration
+ * RT_MEMORY_USAGE - Get the memory usage
+ *
+ * Interface for Shared Memory
+ * ---------
+ *
+ * RT_ATTACH - Attach to the radix tree
+ * RT_DETACH - Detach from the radix tree
+ * RT_GET_HANDLE - Return the handle of the radix tree
+ *
+ * Optional Interface
+ * ---------
+ *
+ * RT_DELETE - Delete a key-value pair. Declared/define if RT_USE_DELETE is defined
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/include/lib/radixtree.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/stringinfo.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "port/pg_bitutils.h"
+#include "port/simd.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/* helpers */
+#define RT_MAKE_PREFIX(a) CppConcat(a,_)
+#define RT_MAKE_NAME(name) RT_MAKE_NAME_(RT_MAKE_PREFIX(RT_PREFIX),name)
+#define RT_MAKE_NAME_(a,b) CppConcat(a,b)
+
+/* function declarations */
+#define RT_CREATE RT_MAKE_NAME(create)
+#define RT_FREE RT_MAKE_NAME(free)
+#define RT_SEARCH RT_MAKE_NAME(search)
+#ifdef RT_SHMEM
+#define RT_ATTACH RT_MAKE_NAME(attach)
+#define RT_DETACH RT_MAKE_NAME(detach)
+#define RT_GET_HANDLE RT_MAKE_NAME(get_handle)
+#endif
+#define RT_SET RT_MAKE_NAME(set)
+#define RT_BEGIN_ITERATE RT_MAKE_NAME(begin_iterate)
+#define RT_ITERATE_NEXT RT_MAKE_NAME(iterate_next)
+#define RT_END_ITERATE RT_MAKE_NAME(end_iterate)
+#ifdef RT_USE_DELETE
+#define RT_DELETE RT_MAKE_NAME(delete)
+#endif
+#define RT_MEMORY_USAGE RT_MAKE_NAME(memory_usage)
+#ifdef RT_DEBUG
+#define RT_DUMP RT_MAKE_NAME(dump)
+#define RT_DUMP_NODE RT_MAKE_NAME(dump_node)
+#define RT_DUMP_SEARCH RT_MAKE_NAME(dump_search)
+#define RT_STATS RT_MAKE_NAME(stats)
+#endif
+
+/* internal helper functions (no externally visible prototypes) */
+#define RT_NEW_ROOT RT_MAKE_NAME(new_root)
+#define RT_ALLOC_NODE RT_MAKE_NAME(alloc_node)
+#define RT_INIT_NODE RT_MAKE_NAME(init_node)
+#define RT_FREE_NODE RT_MAKE_NAME(free_node)
+#define RT_FREE_RECURSE RT_MAKE_NAME(free_recurse)
+#define RT_EXTEND RT_MAKE_NAME(extend)
+#define RT_SET_EXTEND RT_MAKE_NAME(set_extend)
+#define RT_SWITCH_NODE_KIND RT_MAKE_NAME(grow_node_kind)
+#define RT_COPY_NODE RT_MAKE_NAME(copy_node)
+#define RT_REPLACE_NODE RT_MAKE_NAME(replace_node)
+#define RT_PTR_GET_LOCAL RT_MAKE_NAME(ptr_get_local)
+#define RT_PTR_ALLOC_IS_VALID RT_MAKE_NAME(ptr_stored_is_valid)
+#define RT_NODE_3_SEARCH_EQ RT_MAKE_NAME(node_3_search_eq)
+#define RT_NODE_32_SEARCH_EQ RT_MAKE_NAME(node_32_search_eq)
+#define RT_NODE_3_GET_INSERTPOS RT_MAKE_NAME(node_3_get_insertpos)
+#define RT_NODE_32_GET_INSERTPOS RT_MAKE_NAME(node_32_get_insertpos)
+#define RT_CHUNK_CHILDREN_ARRAY_SHIFT RT_MAKE_NAME(chunk_children_array_shift)
+#define RT_CHUNK_VALUES_ARRAY_SHIFT RT_MAKE_NAME(chunk_values_array_shift)
+#define RT_CHUNK_CHILDREN_ARRAY_DELETE RT_MAKE_NAME(chunk_children_array_delete)
+#define RT_CHUNK_VALUES_ARRAY_DELETE RT_MAKE_NAME(chunk_values_array_delete)
+#define RT_CHUNK_CHILDREN_ARRAY_COPY RT_MAKE_NAME(chunk_children_array_copy)
+#define RT_CHUNK_VALUES_ARRAY_COPY RT_MAKE_NAME(chunk_values_array_copy)
+#define RT_NODE_125_IS_CHUNK_USED RT_MAKE_NAME(node_125_is_chunk_used)
+#define RT_NODE_INNER_125_GET_CHILD RT_MAKE_NAME(node_inner_125_get_child)
+#define RT_NODE_LEAF_125_GET_VALUE RT_MAKE_NAME(node_leaf_125_get_value)
+#define RT_NODE_INNER_256_IS_CHUNK_USED RT_MAKE_NAME(node_inner_256_is_chunk_used)
+#define RT_NODE_LEAF_256_IS_CHUNK_USED RT_MAKE_NAME(node_leaf_256_is_chunk_used)
+#define RT_NODE_INNER_256_GET_CHILD RT_MAKE_NAME(node_inner_256_get_child)
+#define RT_NODE_LEAF_256_GET_VALUE RT_MAKE_NAME(node_leaf_256_get_value)
+#define RT_NODE_INNER_256_SET RT_MAKE_NAME(node_inner_256_set)
+#define RT_NODE_LEAF_256_SET RT_MAKE_NAME(node_leaf_256_set)
+#define RT_NODE_INNER_256_DELETE RT_MAKE_NAME(node_inner_256_delete)
+#define RT_NODE_LEAF_256_DELETE RT_MAKE_NAME(node_leaf_256_delete)
+#define RT_KEY_GET_SHIFT RT_MAKE_NAME(key_get_shift)
+#define RT_SHIFT_GET_MAX_VAL RT_MAKE_NAME(shift_get_max_val)
+#define RT_NODE_SEARCH_INNER RT_MAKE_NAME(node_search_inner)
+#define RT_NODE_SEARCH_LEAF RT_MAKE_NAME(node_search_leaf)
+#define RT_NODE_UPDATE_INNER RT_MAKE_NAME(node_update_inner)
+#define RT_NODE_DELETE_INNER RT_MAKE_NAME(node_delete_inner)
+#define RT_NODE_DELETE_LEAF RT_MAKE_NAME(node_delete_leaf)
+#define RT_NODE_INSERT_INNER RT_MAKE_NAME(node_insert_inner)
+#define RT_NODE_INSERT_LEAF RT_MAKE_NAME(node_insert_leaf)
+#define RT_NODE_INNER_ITERATE_NEXT RT_MAKE_NAME(node_inner_iterate_next)
+#define RT_NODE_LEAF_ITERATE_NEXT RT_MAKE_NAME(node_leaf_iterate_next)
+#define RT_UPDATE_ITER_STACK RT_MAKE_NAME(update_iter_stack)
+#define RT_ITER_UPDATE_KEY RT_MAKE_NAME(iter_update_key)
+#define RT_VERIFY_NODE RT_MAKE_NAME(verify_node)
+
+/* type declarations */
+#define RT_RADIX_TREE RT_MAKE_NAME(radix_tree)
+#define RT_RADIX_TREE_CONTROL RT_MAKE_NAME(radix_tree_control)
+#define RT_ITER RT_MAKE_NAME(iter)
+#ifdef RT_SHMEM
+#define RT_HANDLE RT_MAKE_NAME(handle)
+#endif
+#define RT_NODE RT_MAKE_NAME(node)
+#define RT_NODE_ITER RT_MAKE_NAME(node_iter)
+#define RT_NODE_BASE_3 RT_MAKE_NAME(node_base_3)
+#define RT_NODE_BASE_32 RT_MAKE_NAME(node_base_32)
+#define RT_NODE_BASE_125 RT_MAKE_NAME(node_base_125)
+#define RT_NODE_BASE_256 RT_MAKE_NAME(node_base_256)
+#define RT_NODE_INNER_3 RT_MAKE_NAME(node_inner_3)
+#define RT_NODE_INNER_32 RT_MAKE_NAME(node_inner_32)
+#define RT_NODE_INNER_125 RT_MAKE_NAME(node_inner_125)
+#define RT_NODE_INNER_256 RT_MAKE_NAME(node_inner_256)
+#define RT_NODE_LEAF_3 RT_MAKE_NAME(node_leaf_3)
+#define RT_NODE_LEAF_32 RT_MAKE_NAME(node_leaf_32)
+#define RT_NODE_LEAF_125 RT_MAKE_NAME(node_leaf_125)
+#define RT_NODE_LEAF_256 RT_MAKE_NAME(node_leaf_256)
+#define RT_SIZE_CLASS RT_MAKE_NAME(size_class)
+#define RT_SIZE_CLASS_ELEM RT_MAKE_NAME(size_class_elem)
+#define RT_SIZE_CLASS_INFO RT_MAKE_NAME(size_class_info)
+#define RT_CLASS_3 RT_MAKE_NAME(class_3)
+#define RT_CLASS_32_MIN RT_MAKE_NAME(class_32_min)
+#define RT_CLASS_32_MAX RT_MAKE_NAME(class_32_max)
+#define RT_CLASS_125 RT_MAKE_NAME(class_125)
+#define RT_CLASS_256 RT_MAKE_NAME(class_256)
+
+/* generate forward declarations necessary to use the radix tree */
+#ifdef RT_DECLARE
+
+typedef struct RT_RADIX_TREE RT_RADIX_TREE;
+typedef struct RT_ITER RT_ITER;
+
+#ifdef RT_SHMEM
+typedef dsa_pointer RT_HANDLE;
+#endif
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id);
+RT_SCOPE RT_RADIX_TREE * RT_ATTACH(dsa_area *dsa, dsa_pointer dp);
+RT_SCOPE void RT_DETACH(RT_RADIX_TREE *tree);
+RT_SCOPE RT_HANDLE RT_GET_HANDLE(RT_RADIX_TREE *tree);
+#else
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx);
+#endif
+RT_SCOPE void RT_FREE(RT_RADIX_TREE *tree);
+
+RT_SCOPE bool RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p);
+RT_SCOPE bool RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p);
+#ifdef RT_USE_DELETE
+RT_SCOPE bool RT_DELETE(RT_RADIX_TREE *tree, uint64 key);
+#endif
+
+RT_SCOPE RT_ITER * RT_BEGIN_ITERATE(RT_RADIX_TREE *tree);
+RT_SCOPE bool RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p);
+RT_SCOPE void RT_END_ITERATE(RT_ITER *iter);
+
+RT_SCOPE uint64 RT_MEMORY_USAGE(RT_RADIX_TREE *tree);
+
+#ifdef RT_DEBUG
+RT_SCOPE void RT_DUMP(RT_RADIX_TREE *tree);
+RT_SCOPE void RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key);
+RT_SCOPE void RT_STATS(RT_RADIX_TREE *tree);
+#endif
+
+#endif /* RT_DECLARE */
+
+
+/* generate implementation of the radix tree */
+#ifdef RT_DEFINE
+
+/* The number of bits encoded in one tree level */
+#define RT_NODE_SPAN BITS_PER_BYTE
+
+/* The number of maximum slots in the node */
+#define RT_NODE_MAX_SLOTS (1 << RT_NODE_SPAN)
+
+/* Mask for extracting a chunk from the key */
+#define RT_CHUNK_MASK ((1 << RT_NODE_SPAN) - 1)
+
+/* Maximum shift the radix tree uses */
+#define RT_MAX_SHIFT RT_KEY_GET_SHIFT(UINT64_MAX)
+
+/* Tree level the radix tree uses */
+#define RT_MAX_LEVEL ((sizeof(uint64) * BITS_PER_BYTE) / RT_NODE_SPAN)
+
+/*
+ * Number of bits necessary for isset array in the slot-index node.
+ * Since bitmapword can be 64 bits, the only values that make sense
+ * here are 64 and 128.
+ */
+#define RT_SLOT_IDX_LIMIT (RT_NODE_MAX_SLOTS / 2)
+
+/* Invalid index used in node-125 */
+#define RT_INVALID_SLOT_IDX 0xFF
+
+/* Get a chunk from the key */
+#define RT_GET_KEY_CHUNK(key, shift) ((uint8) (((key) >> (shift)) & RT_CHUNK_MASK))
+
+/* For accessing bitmaps */
+#define RT_BM_IDX(x) ((x) / BITS_PER_BITMAPWORD)
+#define RT_BM_BIT(x) ((x) % BITS_PER_BITMAPWORD)
+
+/*
+ * Node kinds
+ *
+ * The different node kinds are what make the tree "adaptive".
+ *
+ * Each node kind is associated with a different datatype and different
+ * search/set/delete/iterate algorithms adapted for its size. The largest
+ * kind, node256 is basically the same as a traditional radix tree,
+ * and would be most wasteful of memory when sparsely populated. The
+ * smaller nodes expend some additional CPU time to enable a smaller
+ * memory footprint.
+ *
+ * XXX There are 4 node kinds, and this should never be increased,
+ * for several reasons:
+ * 1. With 5 or more kinds, gcc tends to use a jump table for switch
+ * statements.
+ * 2. The 4 kinds can be represented with 2 bits, so we have the option
+ * in the future to tag the node pointer with the kind, even on
+ * platforms with 32-bit pointers. This might speed up node traversal
+ * in trees with highly random node kinds.
+ * 3. We can have multiple size classes per node kind.
+ */
+#define RT_NODE_KIND_3 0x00
+#define RT_NODE_KIND_32 0x01
+#define RT_NODE_KIND_125 0x02
+#define RT_NODE_KIND_256 0x03
+#define RT_NODE_KIND_COUNT 4
+
+/*
+ * Calculate the slab blocksize so that we can allocate at least 32 chunks
+ * from the block.
+ */
+#define RT_SLAB_BLOCK_SIZE(size) \
+ Max((SLAB_DEFAULT_BLOCK_SIZE / (size)) * (size), (size) * 32)
+
+/* Common type for all nodes types */
+typedef struct RT_NODE
+{
+ /*
+ * Number of children. We use uint16 to be able to indicate 256 children
+ * at the fanout of 8.
+ */
+ uint16 count;
+
+ /*
+ * Max capacity for the current size class. Storing this in the
+ * node enables multiple size classes per node kind.
+ * Technically, kinds with a single size class don't need this, so we could
+ * keep this in the individual base types, but the code is simpler this way.
+ * Note: node256 is unique in that it cannot possibly have more than a
+ * single size class, so for that kind we store zero, and uint8 is
+ * sufficient for other kinds.
+ */
+ uint8 fanout;
+
+ /*
+ * Shift indicates which part of the key space is represented by this
+ * node. That is, the key is shifted by 'shift' and the lowest
+ * RT_NODE_SPAN bits are then represented in chunk.
+ */
+ uint8 shift;
+
+ /* Node kind, one per search/set algorithm */
+ uint8 kind;
+} RT_NODE;
+
+
+#define RT_PTR_LOCAL RT_NODE *
+
+#ifdef RT_SHMEM
+#define RT_PTR_ALLOC dsa_pointer
+#else
+#define RT_PTR_ALLOC RT_PTR_LOCAL
+#endif
+
+
+#ifdef RT_SHMEM
+#define RT_INVALID_PTR_ALLOC InvalidDsaPointer
+#else
+#define RT_INVALID_PTR_ALLOC NULL
+#endif
+
+#ifdef RT_SHMEM
+#define RT_LOCK_EXCLUSIVE(tree) LWLockAcquire(&tree->ctl->lock, LW_EXCLUSIVE)
+#define RT_LOCK_SHARED(tree) LWLockAcquire(&tree->ctl->lock, LW_SHARED)
+#define RT_UNLOCK(tree) LWLockRelease(&tree->ctl->lock);
+#else
+#define RT_LOCK_EXCLUSIVE(tree) ((void) 0)
+#define RT_LOCK_SHARED(tree) ((void) 0)
+#define RT_UNLOCK(tree) ((void) 0)
+#endif
+
+/*
+ * Inner nodes and leaf nodes have analogous structure. To distinguish
+ * them at runtime, we take advantage of the fact that the key chunk
+ * is accessed by shifting: Inner tree nodes (shift > 0), store the
+ * pointer to its child node in the slot. In leaf nodes (shift == 0),
+ * the slot contains the value corresponding to the key.
+ */
+#define RT_NODE_IS_LEAF(n) (((RT_PTR_LOCAL) (n))->shift == 0)
+
+#define RT_NODE_MUST_GROW(node) \
+ ((node)->base.n.count == (node)->base.n.fanout)
+
+/*
+ * Base type of each node kinds for leaf and inner nodes.
+ * The base types must be a be able to accommodate the largest size
+ * class for variable-sized node kinds.
+ */
+typedef struct RT_NODE_BASE_3
+{
+ RT_NODE n;
+
+ /* 3 children, for key chunks */
+ uint8 chunks[3];
+} RT_NODE_BASE_3;
+
+typedef struct RT_NODE_BASE_32
+{
+ RT_NODE n;
+
+ /* 32 children, for key chunks */
+ uint8 chunks[32];
+} RT_NODE_BASE_32;
+
+/*
+ * node-125 uses slot_idx array, an array of RT_NODE_MAX_SLOTS length
+ * to store indexes into a second array that contains the values (or
+ * child pointers).
+ */
+typedef struct RT_NODE_BASE_125
+{
+ RT_NODE n;
+
+ /* The index of slots for each fanout */
+ uint8 slot_idxs[RT_NODE_MAX_SLOTS];
+
+ /* bitmap to track which slots are in use */
+ bitmapword isset[RT_BM_IDX(RT_SLOT_IDX_LIMIT)];
+} RT_NODE_BASE_125;
+
+typedef struct RT_NODE_BASE_256
+{
+ RT_NODE n;
+} RT_NODE_BASE_256;
+
+/*
+ * Inner and leaf nodes.
+ *
+ * These are separate because the value type might be different than
+ * something fitting into a pointer-width type.
+ */
+typedef struct RT_NODE_INNER_3
+{
+ RT_NODE_BASE_3 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_3;
+
+typedef struct RT_NODE_LEAF_3
+{
+ RT_NODE_BASE_3 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_3;
+
+typedef struct RT_NODE_INNER_32
+{
+ RT_NODE_BASE_32 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_32;
+
+typedef struct RT_NODE_LEAF_32
+{
+ RT_NODE_BASE_32 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_32;
+
+typedef struct RT_NODE_INNER_125
+{
+ RT_NODE_BASE_125 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_125;
+
+typedef struct RT_NODE_LEAF_125
+{
+ RT_NODE_BASE_125 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_125;
+
+/*
+ * node-256 is the largest node type. This node has an array
+ * for directly storing values (or child pointers in inner nodes).
+ * Unlike other node kinds, it's array size is by definition
+ * fixed.
+ */
+typedef struct RT_NODE_INNER_256
+{
+ RT_NODE_BASE_256 base;
+
+ /* Slots for 256 children */
+ RT_PTR_ALLOC children[RT_NODE_MAX_SLOTS];
+} RT_NODE_INNER_256;
+
+typedef struct RT_NODE_LEAF_256
+{
+ RT_NODE_BASE_256 base;
+
+ /*
+ * Unlike with inner256, zero is a valid value here, so we use a
+ * bitmap to track which slots are in use.
+ */
+ bitmapword isset[RT_BM_IDX(RT_NODE_MAX_SLOTS)];
+
+ /* Slots for 256 values */
+ RT_VALUE_TYPE values[RT_NODE_MAX_SLOTS];
+} RT_NODE_LEAF_256;
+
+/*
+ * Node size classes
+ *
+ * Nodes of different kinds necessarily belong to different size classes.
+ * The main innovation in our implementation compared to the ART paper
+ * is decoupling the notion of size class from kind.
+ *
+ * The size classes within a given node kind have the same underlying
+ * type, but a variable number of children/values. This is possible
+ * because the base type contains small fixed data structures that
+ * work the same way regardless of how full the node is. We store the
+ * node's allocated capacity in the "fanout" member of RT_NODE, to allow
+ * runtime introspection.
+ *
+ * Growing from one node kind to another requires special code for each
+ * case, but growing from one size class to another within the same kind
+ * is basically just allocate + memcpy.
+ *
+ * The size classes have been chosen so that inner nodes on platforms
+ * with 64-bit pointers (and leaf nodes when using a 64-bit key) are
+ * equal to or slightly smaller than some DSA size class.
+ */
+typedef enum RT_SIZE_CLASS
+{
+ RT_CLASS_3 = 0,
+ RT_CLASS_32_MIN,
+ RT_CLASS_32_MAX,
+ RT_CLASS_125,
+ RT_CLASS_256
+} RT_SIZE_CLASS;
+
+/* Information for each size class */
+typedef struct RT_SIZE_CLASS_ELEM
+{
+ const char *name;
+ int fanout;
+
+ /* slab chunk size */
+ Size inner_size;
+ Size leaf_size;
+} RT_SIZE_CLASS_ELEM;
+
+static const RT_SIZE_CLASS_ELEM RT_SIZE_CLASS_INFO[] = {
+ [RT_CLASS_3] = {
+ .name = "radix tree node 3",
+ .fanout = 3,
+ .inner_size = sizeof(RT_NODE_INNER_3) + 3 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_3) + 3 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_32_MIN] = {
+ .name = "radix tree node 15",
+ .fanout = 15,
+ .inner_size = sizeof(RT_NODE_INNER_32) + 15 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_32) + 15 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_32_MAX] = {
+ .name = "radix tree node 32",
+ .fanout = 32,
+ .inner_size = sizeof(RT_NODE_INNER_32) + 32 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_32) + 32 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_125] = {
+ .name = "radix tree node 125",
+ .fanout = 125,
+ .inner_size = sizeof(RT_NODE_INNER_125) + 125 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_125) + 125 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_256] = {
+ .name = "radix tree node 256",
+ .fanout = 256,
+ .inner_size = sizeof(RT_NODE_INNER_256),
+ .leaf_size = sizeof(RT_NODE_LEAF_256),
+ },
+};
+
+#define RT_SIZE_CLASS_COUNT lengthof(RT_SIZE_CLASS_INFO)
+
+#ifdef RT_SHMEM
+/* A magic value used to identify our radix tree */
+#define RT_RADIX_TREE_MAGIC 0x54A48167
+#endif
+
+/* Contains the actual tree and ancillary info */
+// WIP: this name is a bit strange
+typedef struct RT_RADIX_TREE_CONTROL
+{
+#ifdef RT_SHMEM
+ RT_HANDLE handle;
+ uint32 magic;
+ LWLock lock;
+#endif
+
+ RT_PTR_ALLOC root;
+ uint64 max_val;
+ uint64 num_keys;
+
+ /* statistics */
+#ifdef RT_DEBUG
+ int32 cnt[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE_CONTROL;
+
+/* Entry point for allocating and accessing the tree */
+typedef struct RT_RADIX_TREE
+{
+ MemoryContext context;
+
+ /* pointing to either local memory or DSA */
+ RT_RADIX_TREE_CONTROL *ctl;
+
+#ifdef RT_SHMEM
+ dsa_area *dsa;
+#else
+ MemoryContextData *inner_slabs[RT_SIZE_CLASS_COUNT];
+ MemoryContextData *leaf_slabs[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE;
+
+/*
+ * Iteration support.
+ *
+ * Iterating the radix tree returns each pair of key and value in the ascending
+ * order of the key. To support this, the we iterate nodes of each level.
+ *
+ * RT_NODE_ITER struct is used to track the iteration within a node.
+ *
+ * RT_ITER is the struct for iteration of the radix tree, and uses RT_NODE_ITER
+ * in order to track the iteration of each level. During iteration, we also
+ * construct the key whenever updating the node iteration information, e.g., when
+ * advancing the current index within the node or when moving to the next node
+ * at the same level.
+ *
+ * XXX: Currently we allow only one process to do iteration. Therefore, rt_node_iter
+ * has the local pointers to nodes, rather than RT_PTR_ALLOC.
+ * We need either a safeguard to disallow other processes to begin the iteration
+ * while one process is doing or to allow multiple processes to do the iteration.
+ */
+typedef struct RT_NODE_ITER
+{
+ RT_PTR_LOCAL node; /* current node being iterated */
+ int current_idx; /* current position. -1 for initial value */
+} RT_NODE_ITER;
+
+typedef struct RT_ITER
+{
+ RT_RADIX_TREE *tree;
+
+ /* Track the iteration on nodes of each level */
+ RT_NODE_ITER stack[RT_MAX_LEVEL];
+ int stack_len;
+
+ /* The key is constructed during iteration */
+ uint64 key;
+} RT_ITER;
+
+
+static void RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_PTR_ALLOC child);
+static bool RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_VALUE_TYPE *value_p);
+
+/* verification (available only with assertion) */
+static void RT_VERIFY_NODE(RT_PTR_LOCAL node);
+
+/* Get the local address of an allocated node */
+static inline RT_PTR_LOCAL
+RT_PTR_GET_LOCAL(RT_RADIX_TREE *tree, RT_PTR_ALLOC node)
+{
+#ifdef RT_SHMEM
+ return dsa_get_address(tree->dsa, (dsa_pointer) node);
+#else
+ return node;
+#endif
+}
+
+static inline bool
+RT_PTR_ALLOC_IS_VALID(RT_PTR_ALLOC ptr)
+{
+#ifdef RT_SHMEM
+ return DsaPointerIsValid(ptr);
+#else
+ return PointerIsValid(ptr);
+#endif
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_3_SEARCH_EQ(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+ int idx = -1;
+
+ for (int i = 0; i < node->n.count; i++)
+ {
+ if (node->chunks[i] == chunk)
+ {
+ idx = i;
+ break;
+ }
+ }
+
+ return idx;
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_3_GET_INSERTPOS(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+ int idx;
+
+ for (idx = 0; idx < node->n.count; idx++)
+ {
+ if (node->chunks[idx] >= chunk)
+ break;
+ }
+
+ return idx;
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_32_SEARCH_EQ(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+ int count = node->n.count;
+#ifndef USE_NO_SIMD
+ Vector8 spread_chunk;
+ Vector8 haystack1;
+ Vector8 haystack2;
+ Vector8 cmp1;
+ Vector8 cmp2;
+ uint32 bitfield;
+ int index_simd = -1;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+ int index = -1;
+
+ for (int i = 0; i < count; i++)
+ {
+ if (node->chunks[i] == chunk)
+ {
+ index = i;
+ break;
+ }
+ }
+#endif
+
+#ifndef USE_NO_SIMD
+ /* replicate the search key */
+ spread_chunk = vector8_broadcast(chunk);
+
+ /* compare to all 32 keys stored in the node */
+ vector8_load(&haystack1, &node->chunks[0]);
+ vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+ cmp1 = vector8_eq(spread_chunk, haystack1);
+ cmp2 = vector8_eq(spread_chunk, haystack2);
+
+ /* convert comparison to a bitfield */
+ bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+
+ /* mask off invalid entries */
+ bitfield &= ((UINT64CONST(1) << count) - 1);
+
+ /* convert bitfield to index by counting trailing zeros */
+ if (bitfield)
+ index_simd = pg_rightmost_one_pos32(bitfield);
+
+ Assert(index_simd == index);
+ return index_simd;
+#else
+ return index;
+#endif
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_32_GET_INSERTPOS(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+ int count = node->n.count;
+#ifndef USE_NO_SIMD
+ Vector8 spread_chunk;
+ Vector8 haystack1;
+ Vector8 haystack2;
+ Vector8 cmp1;
+ Vector8 cmp2;
+ Vector8 min1;
+ Vector8 min2;
+ uint32 bitfield;
+ int index_simd;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+ int index;
+
+ for (index = 0; index < count; index++)
+ {
+ /*
+ * This is coded with '>=' to match what we can do with SIMD,
+ * with an assert to keep us honest.
+ */
+ if (node->chunks[index] >= chunk)
+ {
+ Assert(node->chunks[index] != chunk);
+ break;
+ }
+ }
+#endif
+
+#ifndef USE_NO_SIMD
+ /*
+ * This is a bit more complicated than RT_NODE_32_SEARCH_EQ(), because
+ * no unsigned uint8 comparison instruction exists, at least for SSE2. So
+ * we need to play some trickery using vector8_min() to effectively get
+ * >=. There'll never be any equal elements in current uses, but that's
+ * what we get here...
+ */
+ spread_chunk = vector8_broadcast(chunk);
+ vector8_load(&haystack1, &node->chunks[0]);
+ vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+ min1 = vector8_min(spread_chunk, haystack1);
+ min2 = vector8_min(spread_chunk, haystack2);
+ cmp1 = vector8_eq(spread_chunk, min1);
+ cmp2 = vector8_eq(spread_chunk, min2);
+ bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+ bitfield &= ((UINT64CONST(1) << count) - 1);
+
+ if (bitfield)
+ index_simd = pg_rightmost_one_pos32(bitfield);
+ else
+ index_simd = count;
+
+ Assert(index_simd == index);
+ return index_simd;
+#else
+ return index;
+#endif
+}
+
+
+/*
+ * Functions to manipulate both chunks array and children/values array.
+ * These are used for node-3 and node-32.
+ */
+
+/* Shift the elements right at 'idx' by one */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_SHIFT(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+ memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+ memmove(&(children[idx + 1]), &(children[idx]), sizeof(RT_PTR_ALLOC) * (count - idx));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_SHIFT(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+ memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+ memmove(&(values[idx + 1]), &(values[idx]), sizeof(RT_VALUE_TYPE) * (count - idx));
+}
+
+/* Delete the element at 'idx' */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_DELETE(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+ memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+ memmove(&(children[idx]), &(children[idx + 1]), sizeof(RT_PTR_ALLOC) * (count - idx - 1));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_DELETE(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+ memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+ memmove(&(values[idx]), &(values[idx + 1]), sizeof(RT_VALUE_TYPE) * (count - idx - 1));
+}
+
+/* Copy both chunks and children/values arrays */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_COPY(uint8 *src_chunks, RT_PTR_ALLOC *src_children,
+ uint8 *dst_chunks, RT_PTR_ALLOC *dst_children)
+{
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const Size chunk_size = sizeof(uint8) * fanout;
+ const Size children_size = sizeof(RT_PTR_ALLOC) * fanout;
+
+ memcpy(dst_chunks, src_chunks, chunk_size);
+ memcpy(dst_children, src_children, children_size);
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_COPY(uint8 *src_chunks, RT_VALUE_TYPE *src_values,
+ uint8 *dst_chunks, RT_VALUE_TYPE *dst_values)
+{
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const Size chunk_size = sizeof(uint8) * fanout;
+ const Size values_size = sizeof(RT_VALUE_TYPE) * fanout;
+
+ memcpy(dst_chunks, src_chunks, chunk_size);
+ memcpy(dst_values, src_values, values_size);
+}
+
+/* Functions to manipulate inner and leaf node-125 */
+
+/* Does the given chunk in the node has the value? */
+static inline bool
+RT_NODE_125_IS_CHUNK_USED(RT_NODE_BASE_125 *node, uint8 chunk)
+{
+ return node->slot_idxs[chunk] != RT_INVALID_SLOT_IDX;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_125_GET_CHILD(RT_NODE_INNER_125 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ return node->children[node->base.slot_idxs[chunk]];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_125_GET_VALUE(RT_NODE_LEAF_125 *node, uint8 chunk)
+{
+ Assert(RT_NODE_IS_LEAF(node));
+ Assert(((RT_NODE_BASE_125 *) node)->slot_idxs[chunk] != RT_INVALID_SLOT_IDX);
+ return node->values[node->base.slot_idxs[chunk]];
+}
+
+/* Functions to manipulate inner and leaf node-256 */
+
+/* Return true if the slot corresponding to the given chunk is in use */
+static inline bool
+RT_NODE_INNER_256_IS_CHUNK_USED(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ return node->children[chunk] != RT_INVALID_PTR_ALLOC;
+}
+
+static inline bool
+RT_NODE_LEAF_256_IS_CHUNK_USED(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ return (node->isset[idx] & ((bitmapword) 1 << bitnum)) != 0;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_256_GET_CHILD(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ Assert(RT_NODE_INNER_256_IS_CHUNK_USED(node, chunk));
+ return node->children[chunk];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_256_GET_VALUE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ Assert(RT_NODE_IS_LEAF(node));
+ Assert(RT_NODE_LEAF_256_IS_CHUNK_USED(node, chunk));
+ return node->values[chunk];
+}
+
+/* Set the child in the node-256 */
+static inline void
+RT_NODE_INNER_256_SET(RT_NODE_INNER_256 *node, uint8 chunk, RT_PTR_ALLOC child)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ node->children[chunk] = child;
+}
+
+/* Set the value in the node-256 */
+static inline void
+RT_NODE_LEAF_256_SET(RT_NODE_LEAF_256 *node, uint8 chunk, RT_VALUE_TYPE value)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ node->isset[idx] |= ((bitmapword) 1 << bitnum);
+ node->values[chunk] = value;
+}
+
+/* Set the slot at the given chunk position */
+static inline void
+RT_NODE_INNER_256_DELETE(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ node->children[chunk] = RT_INVALID_PTR_ALLOC;
+}
+
+static inline void
+RT_NODE_LEAF_256_DELETE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ node->isset[idx] &= ~((bitmapword) 1 << bitnum);
+}
+
+/*
+ * Return the largest shift that will allowing storing the given key.
+ */
+static inline int
+RT_KEY_GET_SHIFT(uint64 key)
+{
+ if (key == 0)
+ return 0;
+ else
+ return (pg_leftmost_one_pos64(key) / RT_NODE_SPAN) * RT_NODE_SPAN;
+}
+
+/*
+ * Return the max value that can be stored in the tree with the given shift.
+ */
+static uint64
+RT_SHIFT_GET_MAX_VAL(int shift)
+{
+ if (shift == RT_MAX_SHIFT)
+ return UINT64_MAX;
+
+ return (UINT64CONST(1) << (shift + RT_NODE_SPAN)) - 1;
+}
+
+/*
+ * Allocate a new node with the given node kind.
+ */
+static RT_PTR_ALLOC
+RT_ALLOC_NODE(RT_RADIX_TREE *tree, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+ RT_PTR_ALLOC allocnode;
+ size_t allocsize;
+
+ if (is_leaf)
+ allocsize = RT_SIZE_CLASS_INFO[size_class].leaf_size;
+ else
+ allocsize = RT_SIZE_CLASS_INFO[size_class].inner_size;
+
+#ifdef RT_SHMEM
+ allocnode = dsa_allocate(tree->dsa, allocsize);
+#else
+ if (is_leaf)
+ allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->leaf_slabs[size_class],
+ allocsize);
+ else
+ allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->inner_slabs[size_class],
+ allocsize);
+#endif
+
+#ifdef RT_DEBUG
+ /* update the statistics */
+ tree->ctl->cnt[size_class]++;
+#endif
+
+ return allocnode;
+}
+
+/* Initialize the node contents */
+static inline void
+RT_INIT_NODE(RT_PTR_LOCAL node, uint8 kind, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+ if (is_leaf)
+ MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].leaf_size);
+ else
+ MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].inner_size);
+
+ node->kind = kind;
+
+ if (kind == RT_NODE_KIND_256)
+ /* See comment for the RT_NODE type */
+ Assert(node->fanout == 0);
+ else
+ node->fanout = RT_SIZE_CLASS_INFO[size_class].fanout;
+
+ /* Initialize slot_idxs to invalid values */
+ if (kind == RT_NODE_KIND_125)
+ {
+ RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+
+ memset(n125->slot_idxs, RT_INVALID_SLOT_IDX, sizeof(n125->slot_idxs));
+ }
+}
+
+/*
+ * Create a new node as the root. Subordinate nodes will be created during
+ * the insertion.
+ */
+static pg_noinline void
+RT_NEW_ROOT(RT_RADIX_TREE *tree, uint64 key)
+{
+ int shift = RT_KEY_GET_SHIFT(key);
+ bool is_leaf = shift == 0;
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(newnode, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ newnode->shift = shift;
+ tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(shift);
+ tree->ctl->root = allocnode;
+}
+
+static inline void
+RT_COPY_NODE(RT_PTR_LOCAL newnode, RT_PTR_LOCAL oldnode)
+{
+ newnode->shift = oldnode->shift;
+ newnode->count = oldnode->count;
+}
+
+/*
+ * Given a new allocated node and an old node, initalize the new
+ * node with the necessary fields and return its local pointer.
+ */
+static inline RT_PTR_LOCAL
+RT_SWITCH_NODE_KIND(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, RT_PTR_LOCAL node,
+ uint8 new_kind, uint8 new_class, bool is_leaf)
+{
+ RT_PTR_LOCAL newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(newnode, new_kind, new_class, is_leaf);
+ RT_COPY_NODE(newnode, node);
+
+ return newnode;
+}
+
+/* Free the given node */
+static void
+RT_FREE_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode)
+{
+ /* If we're deleting the root node, make the tree empty */
+ if (tree->ctl->root == allocnode)
+ {
+ tree->ctl->root = RT_INVALID_PTR_ALLOC;
+ tree->ctl->max_val = 0;
+ }
+
+#ifdef RT_DEBUG
+ {
+ int i;
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+ /* update the statistics */
+ for (i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ if (node->fanout == RT_SIZE_CLASS_INFO[i].fanout)
+ break;
+ }
+
+ /* fanout of node256 is intentionally 0 */
+ if (i == RT_SIZE_CLASS_COUNT)
+ i = RT_CLASS_256;
+
+ tree->ctl->cnt[i]--;
+ Assert(tree->ctl->cnt[i] >= 0);
+ }
+#endif
+
+#ifdef RT_SHMEM
+ dsa_free(tree->dsa, allocnode);
+#else
+ pfree(allocnode);
+#endif
+}
+
+/* Update the parent's pointer when growing a node */
+static inline void
+RT_NODE_UPDATE_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC new_child)
+{
+#define RT_ACTION_UPDATE
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+#undef RT_ACTION_UPDATE
+}
+
+/*
+ * Replace old_child with new_child, and free the old one.
+ */
+static inline void
+RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
+ RT_PTR_ALLOC stored_old_child, RT_PTR_LOCAL old_child,
+ RT_PTR_ALLOC new_child, uint64 key)
+{
+#ifdef USE_ASSERT_CHECKING
+ RT_PTR_LOCAL new = RT_PTR_GET_LOCAL(tree, new_child);
+
+ Assert(old_child->shift == new->shift);
+ Assert(old_child->count == new->count);
+#endif
+
+ if (parent == old_child)
+ {
+ /* Replace the root node with the new larger node */
+ tree->ctl->root = new_child;
+ }
+ else
+ RT_NODE_UPDATE_INNER(parent, key, new_child);
+
+ RT_FREE_NODE(tree, stored_old_child);
+}
+
+/*
+ * The radix tree doesn't have sufficient height. Extend the radix tree so
+ * it can store the key.
+ */
+static pg_noinline void
+RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
+{
+ int target_shift;
+ RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+ int shift = root->shift + RT_NODE_SPAN;
+
+ target_shift = RT_KEY_GET_SHIFT(key);
+
+ /* Grow tree from 'shift' to 'target_shift' */
+ while (shift <= target_shift)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL node;
+ RT_NODE_INNER_3 *n3;
+
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, true);
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(node, RT_NODE_KIND_3, RT_CLASS_3, true);
+ node->shift = shift;
+ node->count = 1;
+
+ n3 = (RT_NODE_INNER_3 *) node;
+ n3->base.chunks[0] = 0;
+ n3->children[0] = tree->ctl->root;
+
+ /* Update the root */
+ tree->ctl->root = allocnode;
+
+ shift += RT_NODE_SPAN;
+ }
+
+ tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(target_shift);
+}
+
+/*
+ * The radix tree doesn't have inner and leaf nodes for given key-value pair.
+ * Insert inner and leaf nodes from 'node' to bottom.
+ */
+static pg_noinline void
+RT_SET_EXTEND(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
+ RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node)
+{
+ int shift = node->shift;
+
+ Assert(RT_PTR_GET_LOCAL(tree, stored_node) == node);
+
+ while (shift >= RT_NODE_SPAN)
+ {
+ RT_PTR_ALLOC allocchild;
+ RT_PTR_LOCAL newchild;
+ int newshift = shift - RT_NODE_SPAN;
+ bool is_leaf = newshift == 0;
+
+ allocchild = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ newchild = RT_PTR_GET_LOCAL(tree, allocchild);
+ RT_INIT_NODE(newchild, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ newchild->shift = newshift;
+ RT_NODE_INSERT_INNER(tree, parent, stored_node, node, key, allocchild);
+
+ parent = node;
+ node = newchild;
+ stored_node = allocchild;
+ shift -= RT_NODE_SPAN;
+ }
+
+ RT_NODE_INSERT_LEAF(tree, parent, stored_node, node, key, value_p);
+ tree->ctl->num_keys++;
+}
+
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the child
+ * pointer is set to child_p.
+ */
+static inline bool
+RT_NODE_SEARCH_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC *child_p)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the pointer
+ * to the value is set to value_p.
+ */
+static inline bool
+RT_NODE_SEARCH_LEAF(RT_PTR_LOCAL node, uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_INNER(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_LEAF(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+#endif
+
+/*
+ * Insert "child" into "node".
+ *
+ * "parent" is the parent of "node", so the grandparent of the child.
+ * If the node we're inserting into needs to grow, we update the parent's
+ * child pointer with the pointer to the new larger node.
+ */
+static void
+RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_PTR_ALLOC child)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/* Like RT_NODE_INSERT_INNER, but for leaf nodes */
+static bool
+RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Create the radix tree in the given memory context and return it.
+ */
+RT_SCOPE RT_RADIX_TREE *
+#ifdef RT_SHMEM
+RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id)
+#else
+RT_CREATE(MemoryContext ctx)
+#endif
+{
+ RT_RADIX_TREE *tree;
+ MemoryContext old_ctx;
+#ifdef RT_SHMEM
+ dsa_pointer dp;
+#endif
+
+ old_ctx = MemoryContextSwitchTo(ctx);
+
+ tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+ tree->context = ctx;
+
+#ifdef RT_SHMEM
+ tree->dsa = dsa;
+ dp = dsa_allocate0(dsa, sizeof(RT_RADIX_TREE_CONTROL));
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, dp);
+ tree->ctl->handle = dp;
+ tree->ctl->magic = RT_RADIX_TREE_MAGIC;
+ LWLockInitialize(&tree->ctl->lock, tranche_id);
+#else
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) palloc0(sizeof(RT_RADIX_TREE_CONTROL));
+
+ /* Create a slab context for each size class */
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ RT_SIZE_CLASS_ELEM size_class = RT_SIZE_CLASS_INFO[i];
+ size_t inner_blocksize = RT_SLAB_BLOCK_SIZE(size_class.inner_size);
+ size_t leaf_blocksize = RT_SLAB_BLOCK_SIZE(size_class.leaf_size);
+
+ tree->inner_slabs[i] = SlabContextCreate(ctx,
+ size_class.name,
+ inner_blocksize,
+ size_class.inner_size);
+ tree->leaf_slabs[i] = SlabContextCreate(ctx,
+ size_class.name,
+ leaf_blocksize,
+ size_class.leaf_size);
+ }
+#endif
+
+ tree->ctl->root = RT_INVALID_PTR_ALLOC;
+
+ MemoryContextSwitchTo(old_ctx);
+
+ return tree;
+}
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE *
+RT_ATTACH(dsa_area *dsa, RT_HANDLE handle)
+{
+ RT_RADIX_TREE *tree;
+ dsa_pointer control;
+
+ tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+
+ /* Find the control object in shard memory */
+ control = handle;
+
+ tree->dsa = dsa;
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, control);
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+ return tree;
+}
+
+RT_SCOPE void
+RT_DETACH(RT_RADIX_TREE *tree)
+{
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ pfree(tree);
+}
+
+RT_SCOPE RT_HANDLE
+RT_GET_HANDLE(RT_RADIX_TREE *tree)
+{
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ return tree->ctl->handle;
+}
+
+/*
+ * Recursively free all nodes allocated to the DSA area.
+ */
+static void
+RT_FREE_RECURSE(RT_RADIX_TREE *tree, RT_PTR_ALLOC ptr)
+{
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, ptr);
+
+ check_stack_depth();
+ CHECK_FOR_INTERRUPTS();
+
+ /* The leaf node doesn't have child pointers */
+ if (RT_NODE_IS_LEAF(node))
+ {
+ dsa_free(tree->dsa, ptr);
+ return;
+ }
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+ for (int i = 0; i < n3->base.n.count; i++)
+ RT_FREE_RECURSE(tree, n3->children[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+ for (int i = 0; i < n32->base.n.count; i++)
+ RT_FREE_RECURSE(tree, n32->children[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) node;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+ continue;
+
+ RT_FREE_RECURSE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+ }
+
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ RT_FREE_RECURSE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+ }
+
+ break;
+ }
+ }
+
+ /* Free the inner node */
+ dsa_free(tree->dsa, ptr);
+}
+#endif
+
+/*
+ * Free the given radix tree.
+ */
+RT_SCOPE void
+RT_FREE(RT_RADIX_TREE *tree)
+{
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+ /* Free all memory used for radix tree nodes */
+ if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ RT_FREE_RECURSE(tree, tree->ctl->root);
+
+ /*
+ * Vandalize the control block to help catch programming error where
+ * other backends access the memory formerly occupied by this radix tree.
+ */
+ tree->ctl->magic = 0;
+ dsa_free(tree->dsa, tree->ctl->handle);
+#else
+ pfree(tree->ctl);
+
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ MemoryContextDelete(tree->inner_slabs[i]);
+ MemoryContextDelete(tree->leaf_slabs[i]);
+ }
+#endif
+
+ pfree(tree);
+}
+
+/*
+ * Set key to value. If the entry already exists, we update its value to 'value'
+ * and return true. Returns false if entry doesn't yet exist.
+ */
+RT_SCOPE bool
+RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+ int shift;
+ bool updated;
+ RT_PTR_LOCAL parent;
+ RT_PTR_ALLOC stored_child;
+ RT_PTR_LOCAL child;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ RT_LOCK_EXCLUSIVE(tree);
+
+ /* Empty tree, create the root */
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ RT_NEW_ROOT(tree, key);
+
+ /* Extend the tree if necessary */
+ if (key > tree->ctl->max_val)
+ RT_EXTEND(tree, key);
+
+ stored_child = tree->ctl->root;
+ parent = RT_PTR_GET_LOCAL(tree, stored_child);
+ shift = parent->shift;
+
+ /* Descend the tree until we reach a leaf node */
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC new_child = RT_INVALID_PTR_ALLOC;;
+
+ child = RT_PTR_GET_LOCAL(tree, stored_child);
+
+ if (RT_NODE_IS_LEAF(child))
+ break;
+
+ if (!RT_NODE_SEARCH_INNER(child, key, &new_child))
+ {
+ RT_SET_EXTEND(tree, key, value_p, parent, stored_child, child);
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ parent = child;
+ stored_child = new_child;
+ shift -= RT_NODE_SPAN;
+ }
+
+ updated = RT_NODE_INSERT_LEAF(tree, parent, stored_child, child, key, value_p);
+
+ /* Update the statistics */
+ if (!updated)
+ tree->ctl->num_keys++;
+
+ RT_UNLOCK(tree);
+ return updated;
+}
+
+/*
+ * Search the given key in the radix tree. Return true if there is the key,
+ * otherwise return false. On success, we set the value to *value_p so it must
+ * not be NULL.
+ */
+RT_SCOPE bool
+RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+ RT_PTR_LOCAL node;
+ int shift;
+ bool found;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+ Assert(value_p != NULL);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ node = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+ shift = node->shift;
+
+ /* Descend the tree until a leaf node */
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+ if (RT_NODE_IS_LEAF(node))
+ break;
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ node = RT_PTR_GET_LOCAL(tree, child);
+ shift -= RT_NODE_SPAN;
+ }
+
+ found = RT_NODE_SEARCH_LEAF(node, key, value_p);
+
+ RT_UNLOCK(tree);
+ return found;
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Delete the given key from the radix tree. Return true if the key is found (and
+ * deleted), otherwise do nothing and return false.
+ */
+RT_SCOPE bool
+RT_DELETE(RT_RADIX_TREE *tree, uint64 key)
+{
+ RT_PTR_LOCAL node;
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_ALLOC stack[RT_MAX_LEVEL] = {0};
+ int shift;
+ int level;
+ bool deleted;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ RT_LOCK_EXCLUSIVE(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ /*
+ * Descend the tree to search the key while building a stack of nodes we
+ * visited.
+ */
+ allocnode = tree->ctl->root;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift = node->shift;
+ level = -1;
+ while (shift > 0)
+ {
+ RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+ /* Push the current node to the stack */
+ stack[++level] = allocnode;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ allocnode = child;
+ shift -= RT_NODE_SPAN;
+ }
+
+ /* Delete the key from the leaf node if exists */
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ deleted = RT_NODE_DELETE_LEAF(node, key);
+
+ if (!deleted)
+ {
+ /* no key is found in the leaf node */
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ /* Found the key to delete. Update the statistics */
+ tree->ctl->num_keys--;
+
+ /*
+ * Return if the leaf node still has keys and we don't need to delete the
+ * node.
+ */
+ if (node->count > 0)
+ {
+ RT_UNLOCK(tree);
+ return true;
+ }
+
+ /* Free the empty leaf node */
+ RT_FREE_NODE(tree, allocnode);
+
+ /* Delete the key in inner nodes recursively */
+ while (level >= 0)
+ {
+ allocnode = stack[level--];
+
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ deleted = RT_NODE_DELETE_INNER(node, key);
+ Assert(deleted);
+
+ /* If the node didn't become empty, we stop deleting the key */
+ if (node->count > 0)
+ break;
+
+ /* The node became empty */
+ RT_FREE_NODE(tree, allocnode);
+ }
+
+ RT_UNLOCK(tree);
+ return true;
+}
+#endif
+
+static inline void
+RT_ITER_UPDATE_KEY(RT_ITER *iter, uint8 chunk, uint8 shift)
+{
+ iter->key &= ~(((uint64) RT_CHUNK_MASK) << shift);
+ iter->key |= (((uint64) chunk) << shift);
+}
+
+/*
+ * Advance the slot in the inner node. Return the child if exists, otherwise
+ * null.
+ */
+static inline RT_PTR_LOCAL
+RT_NODE_INNER_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Advance the slot in the leaf node. On success, return true and the value
+ * is set to value_p, otherwise return false.
+ */
+static inline bool
+RT_NODE_LEAF_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter,
+ RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Update each node_iter for inner nodes in the iterator node stack.
+ */
+static void
+RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
+{
+ int level = from;
+ RT_PTR_LOCAL node = from_node;
+
+ for (;;)
+ {
+ RT_NODE_ITER *node_iter = &(iter->stack[level--]);
+
+ node_iter->node = node;
+ node_iter->current_idx = -1;
+
+ /* We don't advance the leaf node iterator here */
+ if (RT_NODE_IS_LEAF(node))
+ return;
+
+ /* Advance to the next slot in the inner node */
+ node = RT_NODE_INNER_ITERATE_NEXT(iter, node_iter);
+
+ /* We must find the first children in the node */
+ Assert(node);
+ }
+}
+
+/*
+ * Create and return the iterator for the given radix tree.
+ *
+ * The radix tree is locked in shared mode during the iteration, so
+ * RT_END_ITERATE needs to be called when finished to release the lock.
+ */
+RT_SCOPE RT_ITER *
+RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
+{
+ MemoryContext old_ctx;
+ RT_ITER *iter;
+ RT_PTR_LOCAL root;
+ int top_level;
+
+ old_ctx = MemoryContextSwitchTo(tree->context);
+
+ iter = (RT_ITER *) palloc0(sizeof(RT_ITER));
+ iter->tree = tree;
+
+ RT_LOCK_SHARED(tree);
+
+ /* empty tree */
+ if (!iter->tree->ctl->root)
+ return iter;
+
+ root = RT_PTR_GET_LOCAL(tree, iter->tree->ctl->root);
+ top_level = root->shift / RT_NODE_SPAN;
+ iter->stack_len = top_level;
+
+ /*
+ * Descend to the left most leaf node from the root. The key is being
+ * constructed while descending to the leaf.
+ */
+ RT_UPDATE_ITER_STACK(iter, root, top_level);
+
+ MemoryContextSwitchTo(old_ctx);
+
+ return iter;
+}
+
+/*
+ * Return true with setting key_p and value_p if there is next key. Otherwise
+ * return false.
+ */
+RT_SCOPE bool
+RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p)
+{
+ /* Empty tree */
+ if (!iter->tree->ctl->root)
+ return false;
+
+ for (;;)
+ {
+ RT_PTR_LOCAL child = NULL;
+ RT_VALUE_TYPE value;
+ int level;
+ bool found;
+
+ /* Advance the leaf node iterator to get next key-value pair */
+ found = RT_NODE_LEAF_ITERATE_NEXT(iter, &(iter->stack[0]), &value);
+
+ if (found)
+ {
+ *key_p = iter->key;
+ *value_p = value;
+ return true;
+ }
+
+ /*
+ * We've visited all values in the leaf node, so advance inner node
+ * iterators from the level=1 until we find the next child node.
+ */
+ for (level = 1; level <= iter->stack_len; level++)
+ {
+ child = RT_NODE_INNER_ITERATE_NEXT(iter, &(iter->stack[level]));
+
+ if (child)
+ break;
+ }
+
+ /* the iteration finished */
+ if (!child)
+ return false;
+
+ /*
+ * Set the node to the node iterator and update the iterator stack
+ * from this node.
+ */
+ RT_UPDATE_ITER_STACK(iter, child, level - 1);
+
+ /* Node iterators are updated, so try again from the leaf */
+ }
+
+ return false;
+}
+
+/*
+ * Terminate the iteration and release the lock.
+ *
+ * This function needs to be called after finishing or when exiting an
+ * iteration.
+ */
+RT_SCOPE void
+RT_END_ITERATE(RT_ITER *iter)
+{
+#ifdef RT_SHMEM
+ Assert(LWLockHeldByMe(&iter->tree->ctl->lock));
+#endif
+
+ RT_UNLOCK(iter->tree);
+ pfree(iter);
+}
+
+/*
+ * Return the statistics of the amount of memory used by the radix tree.
+ */
+RT_SCOPE uint64
+RT_MEMORY_USAGE(RT_RADIX_TREE *tree)
+{
+ Size total = 0;
+
+ RT_LOCK_SHARED(tree);
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ total = dsa_get_total_size(tree->dsa);
+#else
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ total += MemoryContextMemAllocated(tree->inner_slabs[i], true);
+ total += MemoryContextMemAllocated(tree->leaf_slabs[i], true);
+ }
+#endif
+
+ RT_UNLOCK(tree);
+ return total;
+}
+
+/*
+ * Verify the radix tree node.
+ */
+static void
+RT_VERIFY_NODE(RT_PTR_LOCAL node)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(node->count >= 0);
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE_BASE_3 *n3 = (RT_NODE_BASE_3 *) node;
+
+ for (int i = 1; i < n3->n.count; i++)
+ Assert(n3->chunks[i - 1] < n3->chunks[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE_BASE_32 *n32 = (RT_NODE_BASE_32 *) node;
+
+ for (int i = 1; i < n32->n.count; i++)
+ Assert(n32->chunks[i - 1] < n32->chunks[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+ int cnt = 0;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ uint8 slot = n125->slot_idxs[i];
+ int idx = RT_BM_IDX(slot);
+ int bitnum = RT_BM_BIT(slot);
+
+ if (!RT_NODE_125_IS_CHUNK_USED(n125, i))
+ continue;
+
+ /* Check if the corresponding slot is used */
+ Assert(slot < node->fanout);
+ Assert((n125->isset[idx] & ((bitmapword) 1 << bitnum)) != 0);
+
+ cnt++;
+ }
+
+ Assert(n125->n.count == cnt);
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+ int cnt = 0;
+
+ for (int i = 0; i < RT_BM_IDX(RT_NODE_MAX_SLOTS); i++)
+ cnt += bmw_popcount(n256->isset[i]);
+
+ /* Check if the number of used chunk matches */
+ Assert(n256->base.n.count == cnt);
+
+ break;
+ }
+ }
+ }
+#endif
+}
+
+/***************** DEBUG FUNCTIONS *****************/
+#ifdef RT_DEBUG
+
+#define RT_UINT64_FORMAT_HEX "%" INT64_MODIFIER "X"
+
+RT_SCOPE void
+RT_STATS(RT_RADIX_TREE *tree)
+{
+ RT_LOCK_SHARED(tree);
+
+ fprintf(stderr, "max_val = " UINT64_FORMAT "\n", tree->ctl->max_val);
+ fprintf(stderr, "num_keys = " UINT64_FORMAT "\n", tree->ctl->num_keys);
+
+#ifdef RT_SHMEM
+ fprintf(stderr, "handle = " UINT64_FORMAT "\n", tree->ctl->handle);
+#endif
+
+ if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+
+ fprintf(stderr, "height = %d, n3 = %u, n15 = %u, n32 = %u, n125 = %u, n256 = %u\n",
+ root->shift / RT_NODE_SPAN,
+ tree->ctl->cnt[RT_CLASS_3],
+ tree->ctl->cnt[RT_CLASS_32_MIN],
+ tree->ctl->cnt[RT_CLASS_32_MAX],
+ tree->ctl->cnt[RT_CLASS_125],
+ tree->ctl->cnt[RT_CLASS_256]);
+ }
+
+ RT_UNLOCK(tree);
+}
+
+static void
+RT_DUMP_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, int level,
+ bool recurse, StringInfo buf)
+{
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+ StringInfoData spaces;
+
+ initStringInfo(&spaces);
+ appendStringInfoSpaces(&spaces, (level * 4) + 1);
+
+ appendStringInfo(buf, "%s%s[%s] kind %d, fanout %d, count %u, shift %u:\n",
+ spaces.data,
+ level == 0 ? "" : "-> ",
+ RT_NODE_IS_LEAF(node) ? "LEAF" : "INNR",
+ (node->kind == RT_NODE_KIND_3) ? 3 :
+ (node->kind == RT_NODE_KIND_32) ? 32 :
+ (node->kind == RT_NODE_KIND_125) ? 125 : 256,
+ node->fanout == 0 ? 256 : node->fanout,
+ node->count, node->shift);
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ for (int i = 0; i < node->count; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_3 *n3 = (RT_NODE_LEAF_3 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+ spaces.data, i, n3->base.chunks[i]);
+ }
+ else
+ {
+ RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X",
+ spaces.data, i, n3->base.chunks[i]);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, n3->children[i], level + 1,
+ recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ for (int i = 0; i < node->count; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_32 *n32 = (RT_NODE_LEAF_32 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+ spaces.data, i, n32->base.chunks[i]);
+ }
+ else
+ {
+ RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X",
+ spaces.data, i, n32->base.chunks[i]);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, n32->children[i], level + 1,
+ recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_BASE_125 *b125 = (RT_NODE_BASE_125 *) node;
+ char *sep = "";
+
+ appendStringInfo(buf, "%sslot_idxs: ", spaces.data);
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+ continue;
+
+ appendStringInfo(buf, "%s[%d]=%d ",
+ sep, i, b125->slot_idxs[i]);
+ sep = ",";
+ }
+
+ appendStringInfo(buf, "\n%sisset-bitmap: ", spaces.data);
+ for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+ appendStringInfo(buf, "%X ", ((uint8 *) b125->isset)[i]);
+ appendStringInfo(buf, "\n");
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+ continue;
+
+ if (RT_NODE_IS_LEAF(node))
+ appendStringInfo(buf, "%schunk 0x%X\n",
+ spaces.data, i);
+ else
+ {
+ RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) b125;
+
+ appendStringInfo(buf, "%schunk 0x%X",
+ spaces.data, i);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i),
+ level + 1, recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+ appendStringInfo(buf, "%sisset-bitmap: ", spaces.data);
+ for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+ appendStringInfo(buf, "%X ", ((uint8 *) n256->isset)[i]);
+ appendStringInfo(buf, "\n");
+ }
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ appendStringInfo(buf, "%schunk 0x%X\n",
+ spaces.data, i);
+ }
+ else
+ {
+ RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ appendStringInfo(buf, "%schunk 0x%X",
+ spaces.data, i);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i),
+ level + 1, recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ }
+}
+
+RT_SCOPE void
+RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key)
+{
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL node;
+ StringInfoData buf;
+ int shift;
+ int level = 0;
+
+ RT_STATS(tree);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "empty tree\n");
+ return;
+ }
+
+ if (key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "key " UINT64_FORMAT "(0x" RT_UINT64_FORMAT_HEX ") is larger than max val\n",
+ key, key);
+ return;
+ }
+
+ initStringInfo(&buf);
+ allocnode = tree->ctl->root;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift = node->shift;
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC child;
+
+ RT_DUMP_NODE(tree, allocnode, level, false, &buf);
+
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_VALUE_TYPE dummy;
+
+ /* We reached at a leaf node, find the corresponding slot */
+ RT_NODE_SEARCH_LEAF(node, key, &dummy);
+
+ break;
+ }
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ break;
+
+ allocnode = child;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift -= RT_NODE_SPAN;
+ level++;
+ }
+ RT_UNLOCK(tree);
+
+ fprintf(stderr, "%s", buf.data);
+}
+
+RT_SCOPE void
+RT_DUMP(RT_RADIX_TREE *tree)
+{
+ StringInfoData buf;
+
+ RT_STATS(tree);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "empty tree\n");
+ return;
+ }
+
+ initStringInfo(&buf);
+
+ RT_DUMP_NODE(tree, tree->ctl->root, 0, true, &buf);
+ RT_UNLOCK(tree);
+
+ fprintf(stderr, "%s",buf.data);
+}
+#endif
+
+#endif /* RT_DEFINE */
+
+
+/* undefine external parameters, so next radix tree can be defined */
+#undef RT_PREFIX
+#undef RT_SCOPE
+#undef RT_DECLARE
+#undef RT_DEFINE
+#undef RT_VALUE_TYPE
+
+/* locally declared macros */
+#undef RT_MAKE_PREFIX
+#undef RT_MAKE_NAME
+#undef RT_MAKE_NAME_
+#undef RT_NODE_SPAN
+#undef RT_NODE_MAX_SLOTS
+#undef RT_CHUNK_MASK
+#undef RT_MAX_SHIFT
+#undef RT_MAX_LEVEL
+#undef RT_GET_KEY_CHUNK
+#undef RT_BM_IDX
+#undef RT_BM_BIT
+#undef RT_LOCK_EXCLUSIVE
+#undef RT_LOCK_SHARED
+#undef RT_UNLOCK
+#undef RT_NODE_IS_LEAF
+#undef RT_NODE_MUST_GROW
+#undef RT_NODE_KIND_COUNT
+#undef RT_SIZE_CLASS_COUNT
+#undef RT_SLOT_IDX_LIMIT
+#undef RT_INVALID_SLOT_IDX
+#undef RT_SLAB_BLOCK_SIZE
+#undef RT_RADIX_TREE_MAGIC
+#undef RT_UINT64_FORMAT_HEX
+
+/* type declarations */
+#undef RT_RADIX_TREE
+#undef RT_RADIX_TREE_CONTROL
+#undef RT_PTR_LOCAL
+#undef RT_PTR_ALLOC
+#undef RT_INVALID_PTR_ALLOC
+#undef RT_HANDLE
+#undef RT_ITER
+#undef RT_NODE
+#undef RT_NODE_ITER
+#undef RT_NODE_KIND_3
+#undef RT_NODE_KIND_32
+#undef RT_NODE_KIND_125
+#undef RT_NODE_KIND_256
+#undef RT_NODE_BASE_3
+#undef RT_NODE_BASE_32
+#undef RT_NODE_BASE_125
+#undef RT_NODE_BASE_256
+#undef RT_NODE_INNER_3
+#undef RT_NODE_INNER_32
+#undef RT_NODE_INNER_125
+#undef RT_NODE_INNER_256
+#undef RT_NODE_LEAF_3
+#undef RT_NODE_LEAF_32
+#undef RT_NODE_LEAF_125
+#undef RT_NODE_LEAF_256
+#undef RT_SIZE_CLASS
+#undef RT_SIZE_CLASS_ELEM
+#undef RT_SIZE_CLASS_INFO
+#undef RT_CLASS_3
+#undef RT_CLASS_32_MIN
+#undef RT_CLASS_32_MAX
+#undef RT_CLASS_125
+#undef RT_CLASS_256
+
+/* function declarations */
+#undef RT_CREATE
+#undef RT_FREE
+#undef RT_ATTACH
+#undef RT_DETACH
+#undef RT_GET_HANDLE
+#undef RT_SEARCH
+#undef RT_SET
+#undef RT_BEGIN_ITERATE
+#undef RT_ITERATE_NEXT
+#undef RT_END_ITERATE
+#undef RT_USE_DELETE
+#undef RT_DELETE
+#undef RT_MEMORY_USAGE
+#undef RT_DUMP
+#undef RT_DUMP_NODE
+#undef RT_DUMP_SEARCH
+#undef RT_STATS
+
+/* internal helper functions */
+#undef RT_NEW_ROOT
+#undef RT_ALLOC_NODE
+#undef RT_INIT_NODE
+#undef RT_FREE_NODE
+#undef RT_FREE_RECURSE
+#undef RT_EXTEND
+#undef RT_SET_EXTEND
+#undef RT_SWITCH_NODE_KIND
+#undef RT_COPY_NODE
+#undef RT_REPLACE_NODE
+#undef RT_PTR_GET_LOCAL
+#undef RT_PTR_ALLOC_IS_VALID
+#undef RT_NODE_3_SEARCH_EQ
+#undef RT_NODE_32_SEARCH_EQ
+#undef RT_NODE_3_GET_INSERTPOS
+#undef RT_NODE_32_GET_INSERTPOS
+#undef RT_CHUNK_CHILDREN_ARRAY_SHIFT
+#undef RT_CHUNK_VALUES_ARRAY_SHIFT
+#undef RT_CHUNK_CHILDREN_ARRAY_DELETE
+#undef RT_CHUNK_VALUES_ARRAY_DELETE
+#undef RT_CHUNK_CHILDREN_ARRAY_COPY
+#undef RT_CHUNK_VALUES_ARRAY_COPY
+#undef RT_NODE_125_IS_CHUNK_USED
+#undef RT_NODE_INNER_125_GET_CHILD
+#undef RT_NODE_LEAF_125_GET_VALUE
+#undef RT_NODE_INNER_256_IS_CHUNK_USED
+#undef RT_NODE_LEAF_256_IS_CHUNK_USED
+#undef RT_NODE_INNER_256_GET_CHILD
+#undef RT_NODE_LEAF_256_GET_VALUE
+#undef RT_NODE_INNER_256_SET
+#undef RT_NODE_LEAF_256_SET
+#undef RT_NODE_INNER_256_DELETE
+#undef RT_NODE_LEAF_256_DELETE
+#undef RT_KEY_GET_SHIFT
+#undef RT_SHIFT_GET_MAX_VAL
+#undef RT_NODE_SEARCH_INNER
+#undef RT_NODE_SEARCH_LEAF
+#undef RT_NODE_UPDATE_INNER
+#undef RT_NODE_DELETE_INNER
+#undef RT_NODE_DELETE_LEAF
+#undef RT_NODE_INSERT_INNER
+#undef RT_NODE_INSERT_LEAF
+#undef RT_NODE_INNER_ITERATE_NEXT
+#undef RT_NODE_LEAF_ITERATE_NEXT
+#undef RT_UPDATE_ITER_STACK
+#undef RT_ITER_UPDATE_KEY
+#undef RT_VERIFY_NODE
+
+#undef RT_DEBUG
diff --git a/src/include/lib/radixtree_delete_impl.h b/src/include/lib/radixtree_delete_impl.h
new file mode 100644
index 0000000000..5f6dda1f12
--- /dev/null
+++ b/src/include/lib/radixtree_delete_impl.h
@@ -0,0 +1,122 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_delete_impl.h
+ * Common implementation for deletion in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ * TODO: Shrink nodes when deletion would allow them to fit in a smaller
+ * size class.
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_delete_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_DELETE(n3->base.chunks, n3->values,
+ n3->base.n.count, idx);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_DELETE(n3->base.chunks, n3->children,
+ n3->base.n.count, idx);
+#endif
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_DELETE(n32->base.chunks, n32->values,
+ n32->base.n.count, idx);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_DELETE(n32->base.chunks, n32->children,
+ n32->base.n.count, idx);
+#endif
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+ int idx;
+ int bitnum;
+
+ if (slotpos == RT_INVALID_SLOT_IDX)
+ return false;
+
+ idx = RT_BM_IDX(slotpos);
+ bitnum = RT_BM_BIT(slotpos);
+ n125->base.isset[idx] &= ~((bitmapword) 1 << bitnum);
+ n125->base.slot_idxs[chunk] = RT_INVALID_SLOT_IDX;
+
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_DELETE(n256, chunk);
+#else
+ RT_NODE_INNER_256_DELETE(n256, chunk);
+#endif
+ break;
+ }
+ }
+
+ /* update statistics */
+ node->count--;
+
+ return true;
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
new file mode 100644
index 0000000000..d56e58dcac
--- /dev/null
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -0,0 +1,328 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_insert_impl.h
+ * Common implementation for insertion in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_insert_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ const bool is_leaf = true;
+ bool chunk_exists = false;
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+ const bool is_leaf = false;
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ int idx = RT_NODE_3_SEARCH_EQ(&n3->base, chunk);
+
+ if (idx != -1)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n3->values[idx] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n3)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE32_TYPE *new32;
+ const uint8 new_kind = RT_NODE_KIND_32;
+ const RT_SIZE_CLASS new_class = RT_CLASS_32_MIN;
+
+ /* grow node from 3 to 32 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_COPY(n3->base.chunks, n3->values,
+ new32->base.chunks, new32->values);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_COPY(n3->base.chunks, n3->children,
+ new32->base.chunks, new32->children);
+#endif
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
+ int count = n3->base.n.count;
+
+ /* shift chunks and children */
+ if (insertpos < count)
+ {
+ Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_SHIFT(n3->base.chunks, n3->values,
+ count, insertpos);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_SHIFT(n3->base.chunks, n3->children,
+ count, insertpos);
+#endif
+ }
+
+ n3->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+ n3->values[insertpos] = *value_p;
+#else
+ n3->children[insertpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_32:
+ {
+ const RT_SIZE_CLASS_ELEM class32_max = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX];
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ int idx = RT_NODE_32_SEARCH_EQ(&n32->base, chunk);
+
+ if (idx != -1)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n32->values[idx] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n32)) &&
+ n32->base.n.fanout < class32_max.fanout)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ const RT_SIZE_CLASS_ELEM class32_min = RT_SIZE_CLASS_INFO[RT_CLASS_32_MIN];
+ const RT_SIZE_CLASS new_class = RT_CLASS_32_MAX;
+
+ Assert(n32->base.n.fanout == class32_min.fanout);
+
+ /* grow to the next size class of this kind */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ n32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ memcpy(newnode, node, class32_min.leaf_size);
+#else
+ memcpy(newnode, node, class32_min.inner_size);
+#endif
+ newnode->fanout = class32_max.fanout;
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+
+ if (unlikely(RT_NODE_MUST_GROW(n32)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE125_TYPE *new125;
+ const uint8 new_kind = RT_NODE_KIND_125;
+ const RT_SIZE_CLASS new_class = RT_CLASS_125;
+
+ Assert(n32->base.n.fanout == class32_max.fanout);
+
+ /* grow node from 32 to 125 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new125 = (RT_NODE125_TYPE *) newnode;
+
+ for (int i = 0; i < class32_max.fanout; i++)
+ {
+ new125->base.slot_idxs[n32->base.chunks[i]] = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ new125->values[i] = n32->values[i];
+#else
+ new125->children[i] = n32->children[i];
+#endif
+ }
+
+ /*
+ * Since we just copied a dense array, we can set the bits
+ * using a single store, provided the length of that array
+ * is at most the number of bits in a bitmapword.
+ */
+ Assert(class32_max.fanout <= sizeof(bitmapword) * BITS_PER_BYTE);
+ new125->base.isset[0] = (bitmapword) (((uint64) 1 << class32_max.fanout) - 1);
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
+ int count = n32->base.n.count;
+
+ if (insertpos < count)
+ {
+ Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_SHIFT(n32->base.chunks, n32->values,
+ count, insertpos);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_SHIFT(n32->base.chunks, n32->children,
+ count, insertpos);
+#endif
+ }
+
+ n32->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+ n32->values[insertpos] = *value_p;
+#else
+ n32->children[insertpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos;
+ int cnt = 0;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ slotpos = n125->base.slot_idxs[chunk];
+ if (slotpos != RT_INVALID_SLOT_IDX)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n125->values[slotpos] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n125)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE256_TYPE *new256;
+ const uint8 new_kind = RT_NODE_KIND_256;
+ const RT_SIZE_CLASS new_class = RT_CLASS_256;
+
+ /* grow node from 125 to 256 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new256 = (RT_NODE256_TYPE *) newnode;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS && cnt < n125->base.n.count; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+ continue;
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_SET(new256, i, RT_NODE_LEAF_125_GET_VALUE(n125, i));
+#else
+ RT_NODE_INNER_256_SET(new256, i, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+ cnt++;
+ }
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int idx;
+ bitmapword inverse;
+
+ /* get the first word with at least one bit not set */
+ for (idx = 0; idx < RT_BM_IDX(RT_SLOT_IDX_LIMIT); idx++)
+ {
+ if (n125->base.isset[idx] < ~((bitmapword) 0))
+ break;
+ }
+
+ /* To get the first unset bit in X, get the first set bit in ~X */
+ inverse = ~(n125->base.isset[idx]);
+ slotpos = idx * BITS_PER_BITMAPWORD;
+ slotpos += bmw_rightmost_one_pos(inverse);
+ Assert(slotpos < node->fanout);
+
+ /* mark the slot used */
+ n125->base.isset[idx] |= bmw_rightmost_one(inverse);
+ n125->base.slot_idxs[chunk] = slotpos;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ n125->values[slotpos] = *value_p;
+#else
+ n125->children[slotpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ chunk_exists = RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk);
+ Assert(chunk_exists || node->count < RT_NODE_MAX_SLOTS);
+ RT_NODE_LEAF_256_SET(n256, chunk, *value_p);
+#else
+ Assert(node->count < RT_NODE_MAX_SLOTS);
+ RT_NODE_INNER_256_SET(n256, chunk, child);
+#endif
+ break;
+ }
+ }
+
+ /* Update statistics */
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!chunk_exists)
+ node->count++;
+#else
+ node->count++;
+#endif
+
+ /*
+ * Done. Finally, verify the chunk and value is inserted or replaced
+ * properly in the node.
+ */
+ RT_VERIFY_NODE(node);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ return chunk_exists;
+#else
+ return;
+#endif
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_iter_impl.h b/src/include/lib/radixtree_iter_impl.h
new file mode 100644
index 0000000000..98c78eb237
--- /dev/null
+++ b/src/include/lib/radixtree_iter_impl.h
@@ -0,0 +1,153 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_iter_impl.h
+ * Common implementation for iteration in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_iter_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ bool found = false;
+ uint8 key_chunk;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_VALUE_TYPE value;
+
+ Assert(RT_NODE_IS_LEAF(node_iter->node));
+#else
+ RT_PTR_LOCAL child = NULL;
+
+ Assert(!RT_NODE_IS_LEAF(node_iter->node));
+#endif
+
+#ifdef RT_SHMEM
+ Assert(iter->tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ switch (node_iter->node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node_iter->node;
+
+ node_iter->current_idx++;
+ if (node_iter->current_idx >= n3->base.n.count)
+ break;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = n3->values[node_iter->current_idx];
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, n3->children[node_iter->current_idx]);
+#endif
+ key_chunk = n3->base.chunks[node_iter->current_idx];
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node_iter->node;
+
+ node_iter->current_idx++;
+ if (node_iter->current_idx >= n32->base.n.count)
+ break;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ value = n32->values[node_iter->current_idx];
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, n32->children[node_iter->current_idx]);
+#endif
+ key_chunk = n32->base.chunks[node_iter->current_idx];
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node_iter->node;
+ int i;
+
+ for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (RT_NODE_125_IS_CHUNK_USED((RT_NODE_BASE_125 *) n125, i))
+ break;
+ }
+
+ if (i >= RT_NODE_MAX_SLOTS)
+ break;
+
+ node_iter->current_idx = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = RT_NODE_LEAF_125_GET_VALUE(n125, i);
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+ key_chunk = i;
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node_iter->node;
+ int i;
+
+ for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ {
+#ifdef RT_NODE_LEVEL_LEAF
+ if (RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+#else
+ if (RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+#endif
+ break;
+ }
+
+ if (i >= RT_NODE_MAX_SLOTS)
+ break;
+
+ node_iter->current_idx = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = RT_NODE_LEAF_256_GET_VALUE(n256, i);
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+#endif
+ key_chunk = i;
+ found = true;
+ break;
+ }
+ }
+
+ if (found)
+ {
+ RT_ITER_UPDATE_KEY(iter, key_chunk, node_iter->node->shift);
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = value;
+#endif
+ }
+
+#ifdef RT_NODE_LEVEL_LEAF
+ return found;
+#else
+ return child;
+#endif
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_search_impl.h b/src/include/lib/radixtree_search_impl.h
new file mode 100644
index 0000000000..a8925c75d0
--- /dev/null
+++ b/src/include/lib/radixtree_search_impl.h
@@ -0,0 +1,138 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_search_impl.h
+ * Common implementation for search in leaf and inner nodes, plus
+ * update for inner nodes only.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_search_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(value_p != NULL);
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+#ifndef RT_ACTION_UPDATE
+ Assert(child_p != NULL);
+#endif
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+#ifdef RT_ACTION_UPDATE
+ Assert(idx >= 0);
+ n3->children[idx] = new_child;
+#else
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n3->values[idx];
+#else
+ *child_p = n3->children[idx];
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+#ifdef RT_ACTION_UPDATE
+ Assert(idx >= 0);
+ n32->children[idx] = new_child;
+#else
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n32->values[idx];
+#else
+ *child_p = n32->children[idx];
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+
+#ifdef RT_ACTION_UPDATE
+ Assert(slotpos != RT_INVALID_SLOT_IDX);
+ n125->children[slotpos] = new_child;
+#else
+ if (slotpos == RT_INVALID_SLOT_IDX)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_125_GET_VALUE(n125, chunk);
+#else
+ *child_p = RT_NODE_INNER_125_GET_CHILD(n125, chunk);
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_ACTION_UPDATE
+ RT_NODE_INNER_256_SET(n256, chunk, new_child);
+#else
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_256_GET_VALUE(n256, chunk);
+#else
+ *child_p = RT_NODE_INNER_256_GET_CHILD(n256, chunk);
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ }
+
+#ifdef RT_ACTION_UPDATE
+ return;
+#else
+ return true;
+#endif /* RT_ACTION_UPDATE */
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/utils/dsa.h b/src/include/utils/dsa.h
index 3ce4ee300a..2af215484f 100644
--- a/src/include/utils/dsa.h
+++ b/src/include/utils/dsa.h
@@ -121,6 +121,7 @@ extern dsa_handle dsa_get_handle(dsa_area *area);
extern dsa_pointer dsa_allocate_extended(dsa_area *area, size_t size, int flags);
extern void dsa_free(dsa_area *area, dsa_pointer dp);
extern void *dsa_get_address(dsa_area *area, dsa_pointer dp);
+extern size_t dsa_get_total_size(dsa_area *area);
extern void dsa_trim(dsa_area *area);
extern void dsa_dump(dsa_area *area);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index c629cbe383..9659eb85d7 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -28,6 +28,7 @@ SUBDIRS = \
test_pg_db_role_setting \
test_pg_dump \
test_predtest \
+ test_radixtree \
test_rbtree \
test_regex \
test_rls_hooks \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 1baa6b558d..232cbdac80 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -24,6 +24,7 @@ subdir('test_parser')
subdir('test_pg_db_role_setting')
subdir('test_pg_dump')
subdir('test_predtest')
+subdir('test_radixtree')
subdir('test_rbtree')
subdir('test_regex')
subdir('test_rls_hooks')
diff --git a/src/test/modules/test_radixtree/.gitignore b/src/test/modules/test_radixtree/.gitignore
new file mode 100644
index 0000000000..5dcb3ff972
--- /dev/null
+++ b/src/test/modules/test_radixtree/.gitignore
@@ -0,0 +1,4 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
diff --git a/src/test/modules/test_radixtree/Makefile b/src/test/modules/test_radixtree/Makefile
new file mode 100644
index 0000000000..da06b93da3
--- /dev/null
+++ b/src/test/modules/test_radixtree/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_radixtree/Makefile
+
+MODULE_big = test_radixtree
+OBJS = \
+ $(WIN32RES) \
+ test_radixtree.o
+PGFILEDESC = "test_radixtree - test code for src/backend/lib/radixtree.c"
+
+EXTENSION = test_radixtree
+DATA = test_radixtree--1.0.sql
+
+REGRESS = test_radixtree
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_radixtree
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_radixtree/README b/src/test/modules/test_radixtree/README
new file mode 100644
index 0000000000..a8b271869a
--- /dev/null
+++ b/src/test/modules/test_radixtree/README
@@ -0,0 +1,7 @@
+test_integerset contains unit tests for testing the integer set implementation
+in src/backend/lib/integerset.c.
+
+The tests verify the correctness of the implementation, but they can also be
+used as a micro-benchmark. If you set the 'intset_test_stats' flag in
+test_integerset.c, the tests will print extra information about execution time
+and memory usage.
diff --git a/src/test/modules/test_radixtree/expected/test_radixtree.out b/src/test/modules/test_radixtree/expected/test_radixtree.out
new file mode 100644
index 0000000000..ce645cb8b5
--- /dev/null
+++ b/src/test/modules/test_radixtree/expected/test_radixtree.out
@@ -0,0 +1,36 @@
+CREATE EXTENSION test_radixtree;
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
+NOTICE: testing basic operations with leaf node 4
+NOTICE: testing basic operations with inner node 4
+NOTICE: testing basic operations with leaf node 32
+NOTICE: testing basic operations with inner node 32
+NOTICE: testing basic operations with leaf node 125
+NOTICE: testing basic operations with inner node 125
+NOTICE: testing basic operations with leaf node 256
+NOTICE: testing basic operations with inner node 256
+NOTICE: testing radix tree node types with shift "0"
+NOTICE: testing radix tree node types with shift "8"
+NOTICE: testing radix tree node types with shift "16"
+NOTICE: testing radix tree node types with shift "24"
+NOTICE: testing radix tree node types with shift "32"
+NOTICE: testing radix tree node types with shift "40"
+NOTICE: testing radix tree node types with shift "48"
+NOTICE: testing radix tree node types with shift "56"
+NOTICE: testing radix tree with pattern "all ones"
+NOTICE: testing radix tree with pattern "alternating bits"
+NOTICE: testing radix tree with pattern "clusters of ten"
+NOTICE: testing radix tree with pattern "clusters of hundred"
+NOTICE: testing radix tree with pattern "one-every-64k"
+NOTICE: testing radix tree with pattern "sparse"
+NOTICE: testing radix tree with pattern "single values, distance > 2^32"
+NOTICE: testing radix tree with pattern "clusters, distance > 2^32"
+NOTICE: testing radix tree with pattern "clusters, distance > 2^60"
+ test_radixtree
+----------------
+
+(1 row)
+
diff --git a/src/test/modules/test_radixtree/meson.build b/src/test/modules/test_radixtree/meson.build
new file mode 100644
index 0000000000..6add06bbdb
--- /dev/null
+++ b/src/test/modules/test_radixtree/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_radixtree_sources = files(
+ 'test_radixtree.c',
+)
+
+if host_system == 'windows'
+ test_radixtree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_radixtree',
+ '--FILEDESC', 'test_radixtree - test code for src/include//lib/radixtree.h',])
+endif
+
+test_radixtree = shared_module('test_radixtree',
+ test_radixtree_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += test_radixtree
+
+install_data(
+ 'test_radixtree.control',
+ 'test_radixtree--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'test_radixtree',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'test_radixtree',
+ ],
+ },
+}
diff --git a/src/test/modules/test_radixtree/sql/test_radixtree.sql b/src/test/modules/test_radixtree/sql/test_radixtree.sql
new file mode 100644
index 0000000000..41ece5e9f5
--- /dev/null
+++ b/src/test/modules/test_radixtree/sql/test_radixtree.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_radixtree;
+
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
diff --git a/src/test/modules/test_radixtree/test_radixtree--1.0.sql b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
new file mode 100644
index 0000000000..074a5a7ea7
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_radixtree/test_radixtree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_radixtree" to load this file. \quit
+
+CREATE FUNCTION test_radixtree()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
new file mode 100644
index 0000000000..f944945db9
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -0,0 +1,674 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_radixtree.c
+ * Test radixtree set data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_radixtree/test_radixtree.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "storage/lwlock.h"
+#include "utils/memutils.h"
+#include "utils/timestamp.h"
+
+#define UINT64_HEX_FORMAT "%" INT64_MODIFIER "X"
+
+/*
+ * The tests pass with uint32, but build with warnings because the string
+ * format expects uint64.
+ */
+typedef uint64 TestValueType;
+
+/*
+ * If you enable this, the "pattern" tests will print information about
+ * how long populating, probing, and iterating the test set takes, and
+ * how much memory the test set consumed. That can be used as
+ * micro-benchmark of various operations and input patterns (you might
+ * want to increase the number of values used in each of the test, if
+ * you do that, to reduce noise).
+ *
+ * The information is printed to the server's stderr, mostly because
+ * that's where MemoryContextStats() output goes.
+ */
+static const bool rt_test_stats = false;
+
+static int rt_node_kind_fanouts[] = {
+ 0,
+ 4, /* RT_NODE_KIND_4 */
+ 32, /* RT_NODE_KIND_32 */
+ 125, /* RT_NODE_KIND_125 */
+ 256 /* RT_NODE_KIND_256 */
+};
+/*
+ * A struct to define a pattern of integers, for use with the test_pattern()
+ * function.
+ */
+typedef struct
+{
+ char *test_name; /* short name of the test, for humans */
+ char *pattern_str; /* a bit pattern */
+ uint64 spacing; /* pattern repeats at this interval */
+ uint64 num_values; /* number of integers to set in total */
+} test_spec;
+
+/* Test patterns borrowed from test_integerset.c */
+static const test_spec test_specs[] = {
+ {
+ "all ones", "1111111111",
+ 10, 1000000
+ },
+ {
+ "alternating bits", "0101010101",
+ 10, 1000000
+ },
+ {
+ "clusters of ten", "1111111111",
+ 10000, 1000000
+ },
+ {
+ "clusters of hundred",
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111",
+ 10000, 1000000
+ },
+ {
+ "one-every-64k", "1",
+ 65536, 1000000
+ },
+ {
+ "sparse", "100000000000000000000000000000001",
+ 10000000, 1000000
+ },
+ {
+ "single values, distance > 2^32", "1",
+ UINT64CONST(10000000000), 100000
+ },
+ {
+ "clusters, distance > 2^32", "10101010",
+ UINT64CONST(10000000000), 1000000
+ },
+ {
+ "clusters, distance > 2^60", "10101010",
+ UINT64CONST(2000000000000000000),
+ 23 /* can't be much higher than this, or we
+ * overflow uint64 */
+ }
+};
+
+/* define the radix tree implementation to test */
+#define RT_PREFIX rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE TestValueType
+// WIP: compiles with warnings because rt_attach is defined but not used
+// #define RT_SHMEM
+#include "lib/radixtree.h"
+
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+ return tree->ctl->num_keys;
+}
+
+PG_MODULE_MAGIC;
+
+PG_FUNCTION_INFO_V1(test_radixtree);
+
+static void
+test_empty(void)
+{
+ rt_radix_tree *radixtree;
+ rt_iter *iter;
+ TestValueType dummy;
+ uint64 key;
+ TestValueType val;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+ dsa = dsa_create(tranche_id);
+
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ if (rt_search(radixtree, 0, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_search(radixtree, 1, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_search(radixtree, PG_UINT64_MAX, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_delete(radixtree, 0))
+ elog(ERROR, "rt_delete on empty tree returned true");
+
+ if (rt_num_entries(radixtree) != 0)
+ elog(ERROR, "rt_num_entries on empty tree return non-zero");
+
+ iter = rt_begin_iterate(radixtree);
+
+ if (rt_iterate_next(iter, &key, &val))
+ elog(ERROR, "rt_itereate_next on empty tree returned true");
+
+ rt_end_iterate(iter);
+
+ rt_free(radixtree);
+
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+static void
+test_basic(int children, bool test_inner)
+{
+ rt_radix_tree *radixtree;
+ uint64 *keys;
+ int shift = test_inner ? 8 : 0;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing basic operations with %s node %d",
+ test_inner ? "inner" : "leaf", children);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ /* prepare keys in order like 1, 32, 2, 31, 2, ... */
+ keys = palloc(sizeof(uint64) * children);
+ for (int i = 0; i < children; i++)
+ {
+ if (i % 2 == 0)
+ keys[i] = (uint64) ((i / 2) + 1) << shift;
+ else
+ keys[i] = (uint64) (children - (i / 2)) << shift;
+ }
+
+ /* insert keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (rt_set(radixtree, keys[i], (TestValueType*) &keys[i]))
+ elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+ }
+
+ /* look up keys */
+ for (int i = 0; i < children; i++)
+ {
+ TestValueType value;
+
+ if (!rt_search(radixtree, keys[i], &value))
+ elog(ERROR, "could not find key 0x" UINT64_HEX_FORMAT, keys[i]);
+ if (value != (TestValueType) keys[i])
+ elog(ERROR, "rt_search returned 0x" UINT64_HEX_FORMAT ", expected " UINT64_HEX_FORMAT,
+ value, (TestValueType) keys[i]);
+ }
+
+ /* update keys */
+ for (int i = 0; i < children; i++)
+ {
+ TestValueType update = keys[i] + 1;
+ if (!rt_set(radixtree, keys[i], (TestValueType*) &update))
+ elog(ERROR, "could not update key 0x" UINT64_HEX_FORMAT, keys[i]);
+ }
+
+ /* repeat deleting and inserting keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (!rt_delete(radixtree, keys[i]))
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, keys[i]);
+ if (rt_set(radixtree, keys[i], (TestValueType*) &keys[i]))
+ elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+ }
+
+ pfree(keys);
+ rt_free(radixtree);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Check if keys from start to end with the shift exist in the tree.
+ */
+static void
+check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end,
+ int incr)
+{
+ for (int i = start; i < end; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ TestValueType val;
+
+ if (!rt_search(radixtree, key, &val))
+ elog(ERROR, "key 0x" UINT64_HEX_FORMAT " is not found on node-%d",
+ key, end);
+ if (val != (TestValueType) key)
+ elog(ERROR, "rt_search with key 0x" UINT64_HEX_FORMAT " returns 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+ key, val, key);
+ }
+}
+
+static void
+test_node_types_insert(rt_radix_tree *radixtree, uint8 shift, bool insert_asc)
+{
+ uint64 num_entries;
+ int ninserted = 0;
+ int start = insert_asc ? 0 : 256;
+ int incr = insert_asc ? 1 : -1;
+ int end = insert_asc ? 256 : 0;
+ int node_kind_idx = 1;
+
+ for (int i = start; i != end; i += incr)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_set(radixtree, key, (TestValueType*) &key);
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " is found", key);
+
+ /*
+ * After filling all slots in each node type, check if the values
+ * are stored properly.
+ */
+ if (ninserted == rt_node_kind_fanouts[node_kind_idx] - 1)
+ {
+ int check_start = insert_asc
+ ? rt_node_kind_fanouts[node_kind_idx - 1]
+ : rt_node_kind_fanouts[node_kind_idx];
+ int check_end = insert_asc
+ ? rt_node_kind_fanouts[node_kind_idx]
+ : rt_node_kind_fanouts[node_kind_idx - 1];
+
+ check_search_on_node(radixtree, shift, check_start, check_end, incr);
+ node_kind_idx++;
+ }
+
+ ninserted++;
+ }
+
+ num_entries = rt_num_entries(radixtree);
+
+ if (num_entries != 256)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+static void
+test_node_types_delete(rt_radix_tree *radixtree, uint8 shift)
+{
+ uint64 num_entries;
+
+ for (int i = 0; i < 256; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_delete(radixtree, key);
+
+ if (!found)
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, key);
+ }
+
+ num_entries = rt_num_entries(radixtree);
+
+ /* The tree must be empty */
+ if (num_entries != 0)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+/*
+ * Test for inserting and deleting key-value pairs to each node type at the given shift
+ * level.
+ */
+static void
+test_node_types(uint8 shift)
+{
+ rt_radix_tree *radixtree;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing radix tree node types with shift \"%d\"", shift);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ /*
+ * Insert and search entries for every node type at the 'shift' level,
+ * then delete all entries to make it empty, and insert and search entries
+ * again.
+ */
+ test_node_types_insert(radixtree, shift, true);
+ test_node_types_delete(radixtree, shift);
+ test_node_types_insert(radixtree, shift, false);
+
+ rt_free(radixtree);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Test with a repeating pattern, defined by the 'spec'.
+ */
+static void
+test_pattern(const test_spec * spec)
+{
+ rt_radix_tree *radixtree;
+ rt_iter *iter;
+ MemoryContext radixtree_ctx;
+ TimestampTz starttime;
+ TimestampTz endtime;
+ uint64 n;
+ uint64 last_int;
+ uint64 ndeleted;
+ uint64 nbefore;
+ uint64 nafter;
+ int patternlen;
+ uint64 *pattern_values;
+ uint64 pattern_num_values;
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing radix tree with pattern \"%s\"", spec->test_name);
+ if (rt_test_stats)
+ fprintf(stderr, "-----\ntesting radix tree with pattern \"%s\"\n", spec->test_name);
+
+ /* Pre-process the pattern, creating an array of integers from it. */
+ patternlen = strlen(spec->pattern_str);
+ pattern_values = palloc(patternlen * sizeof(uint64));
+ pattern_num_values = 0;
+ for (int i = 0; i < patternlen; i++)
+ {
+ if (spec->pattern_str[i] == '1')
+ pattern_values[pattern_num_values++] = i;
+ }
+
+ /*
+ * Allocate the radix tree.
+ *
+ * Allocate it in a separate memory context, so that we can print its
+ * memory usage easily.
+ */
+ radixtree_ctx = AllocSetContextCreate(CurrentMemoryContext,
+ "radixtree test",
+ ALLOCSET_SMALL_SIZES);
+ MemoryContextSetIdentifier(radixtree_ctx, spec->test_name);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(radixtree_ctx, dsa, tranche_id);
+#else
+ radixtree = rt_create(radixtree_ctx);
+#endif
+
+
+ /*
+ * Add values to the set.
+ */
+ starttime = GetCurrentTimestamp();
+
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ uint64 x = 0;
+
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ bool found;
+
+ x = last_int + pattern_values[i];
+
+ found = rt_set(radixtree, x, (TestValueType*) &x);
+
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " found", x);
+
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+
+ endtime = GetCurrentTimestamp();
+
+ if (rt_test_stats)
+ fprintf(stderr, "added " UINT64_FORMAT " values in %d ms\n",
+ spec->num_values, (int) (endtime - starttime) / 1000);
+
+ /*
+ * Print stats on the amount of memory used.
+ *
+ * We print the usage reported by rt_memory_usage(), as well as the stats
+ * from the memory context. They should be in the same ballpark, but it's
+ * hard to automate testing that, so if you're making changes to the
+ * implementation, just observe that manually.
+ */
+ if (rt_test_stats)
+ {
+ uint64 mem_usage;
+
+ /*
+ * Also print memory usage as reported by rt_memory_usage(). It
+ * should be in the same ballpark as the usage reported by
+ * MemoryContextStats().
+ */
+ mem_usage = rt_memory_usage(radixtree);
+ fprintf(stderr, "rt_memory_usage() reported " UINT64_FORMAT " (%0.2f bytes / integer)\n",
+ mem_usage, (double) mem_usage / spec->num_values);
+
+ MemoryContextStats(radixtree_ctx);
+ }
+
+ /* Check that rt_num_entries works */
+ n = rt_num_entries(radixtree);
+ if (n != spec->num_values)
+ elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT, n, spec->num_values);
+
+ /*
+ * Test random-access probes with rt_search()
+ */
+ starttime = GetCurrentTimestamp();
+
+ for (n = 0; n < 100000; n++)
+ {
+ bool found;
+ bool expected;
+ uint64 x;
+ TestValueType v;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit values that are actually in the set.
+ */
+ x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+ /* Do we expect this value to be present in the set? */
+ if (x >= last_int)
+ expected = false;
+ else
+ {
+ uint64 idx = x % spec->spacing;
+
+ if (idx >= patternlen)
+ expected = false;
+ else if (spec->pattern_str[idx] == '1')
+ expected = true;
+ else
+ expected = false;
+ }
+
+ /* Is it present according to rt_search() ? */
+ found = rt_search(radixtree, x, &v);
+
+ if (found != expected)
+ elog(ERROR, "mismatch at 0x" UINT64_HEX_FORMAT ": %d vs %d", x, found, expected);
+ if (found && (v != (TestValueType) x))
+ elog(ERROR, "found 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+ v, x);
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "probed " UINT64_FORMAT " values in %d ms\n",
+ n, (int) (endtime - starttime) / 1000);
+
+ /*
+ * Test iterator
+ */
+ starttime = GetCurrentTimestamp();
+
+ iter = rt_begin_iterate(radixtree);
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ uint64 expected = last_int + pattern_values[i];
+ uint64 x;
+ TestValueType val;
+
+ if (!rt_iterate_next(iter, &x, &val))
+ break;
+
+ if (x != expected)
+ elog(ERROR,
+ "iterate returned wrong key; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d",
+ x, expected, i);
+ if (val != (TestValueType) expected)
+ elog(ERROR,
+ "iterate returned wrong value; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d", x, expected, i);
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "iterated " UINT64_FORMAT " values in %d ms\n",
+ n, (int) (endtime - starttime) / 1000);
+
+ rt_end_iterate(iter);
+
+ if (n < spec->num_values)
+ elog(ERROR, "iterator stopped short after " UINT64_FORMAT " entries, expected " UINT64_FORMAT, n, spec->num_values);
+ if (n > spec->num_values)
+ elog(ERROR, "iterator returned " UINT64_FORMAT " entries, " UINT64_FORMAT " was expected", n, spec->num_values);
+
+ /*
+ * Test random-access probes with rt_delete()
+ */
+ starttime = GetCurrentTimestamp();
+
+ nbefore = rt_num_entries(radixtree);
+ ndeleted = 0;
+ for (n = 0; n < 1; n++)
+ {
+ bool found;
+ uint64 x;
+ TestValueType v;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit values that are actually in the set.
+ */
+ x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+ /* Is it present according to rt_search() ? */
+ found = rt_search(radixtree, x, &v);
+
+ if (!found)
+ continue;
+
+ /* If the key is found, delete it and check again */
+ if (!rt_delete(radixtree, x))
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, x);
+ if (rt_search(radixtree, x, &v))
+ elog(ERROR, "found deleted key 0x" UINT64_HEX_FORMAT, x);
+ if (rt_delete(radixtree, x))
+ elog(ERROR, "deleted already-deleted key 0x" UINT64_HEX_FORMAT, x);
+
+ ndeleted++;
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "deleted " UINT64_FORMAT " values in %d ms\n",
+ ndeleted, (int) (endtime - starttime) / 1000);
+
+ nafter = rt_num_entries(radixtree);
+
+ /* Check that rt_num_entries works */
+ if ((nbefore - ndeleted) != nafter)
+ elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT "after " UINT64_FORMAT " deletion",
+ nafter, (nbefore - ndeleted), ndeleted);
+
+ rt_free(radixtree);
+ MemoryContextDelete(radixtree_ctx);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+Datum
+test_radixtree(PG_FUNCTION_ARGS)
+{
+ test_empty();
+
+ for (int i = 1; i < lengthof(rt_node_kind_fanouts); i++)
+ {
+ test_basic(rt_node_kind_fanouts[i], false);
+ test_basic(rt_node_kind_fanouts[i], true);
+ }
+
+ for (int shift = 0; shift <= (64 - 8); shift += 8)
+ test_node_types(shift);
+
+ /* Test different test patterns, with lots of entries */
+ for (int i = 0; i < lengthof(test_specs); i++)
+ test_pattern(&test_specs[i]);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_radixtree/test_radixtree.control b/src/test/modules/test_radixtree/test_radixtree.control
new file mode 100644
index 0000000000..e53f2a3e0c
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.control
@@ -0,0 +1,4 @@
+comment = 'Test code for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/test_radixtree'
+relocatable = true
diff --git a/src/tools/pginclude/cpluspluscheck b/src/tools/pginclude/cpluspluscheck
index e52fe9f509..09fa6e7432 100755
--- a/src/tools/pginclude/cpluspluscheck
+++ b/src/tools/pginclude/cpluspluscheck
@@ -101,6 +101,12 @@ do
test "$f" = src/include/nodes/nodetags.h && continue
test "$f" = src/backend/nodes/nodetags.h && continue
+ # radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+ test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+ test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+ test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+ test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
# These files are not meant to be included standalone, because
# they contain lists that might have multiple use-cases.
test "$f" = src/include/access/rmgrlist.h && continue
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index abbba7aa63..d4d2f1da03 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -96,6 +96,12 @@ do
test "$f" = src/include/nodes/nodetags.h && continue
test "$f" = src/backend/nodes/nodetags.h && continue
+ # radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+ test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+ test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+ test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+ test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
# These files are not meant to be included standalone, because
# they contain lists that might have multiple use-cases.
test "$f" = src/include/access/rmgrlist.h && continue
--
2.39.1
[text/x-patch] v28-0006-Tool-for-measuring-radix-tree-and-tidstore-perfo.patch (23.9K, ../../CAFBsxsF4QAOzSPbQvekFn56WJJuYoZM8m=FqbKyb3dwhB+oyWg@mail.gmail.com/8-v28-0006-Tool-for-measuring-radix-tree-and-tidstore-perfo.patch)
download | inline diff:
From b0515a40b3aa4709047c7b70b9c0cadded979d15 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 16 Sep 2022 11:57:03 +0900
Subject: [PATCH v28 06/10] Tool for measuring radix tree and tidstore
performance
Includes Meson support, but commented out to avoid warnings
XXX: Not for commit
---
contrib/bench_radix_tree/Makefile | 21 +
.../bench_radix_tree--1.0.sql | 87 +++
contrib/bench_radix_tree/bench_radix_tree.c | 717 ++++++++++++++++++
.../bench_radix_tree/bench_radix_tree.control | 6 +
contrib/bench_radix_tree/expected/bench.out | 13 +
contrib/bench_radix_tree/meson.build | 33 +
contrib/bench_radix_tree/sql/bench.sql | 16 +
contrib/meson.build | 1 +
8 files changed, 894 insertions(+)
create mode 100644 contrib/bench_radix_tree/Makefile
create mode 100644 contrib/bench_radix_tree/bench_radix_tree--1.0.sql
create mode 100644 contrib/bench_radix_tree/bench_radix_tree.c
create mode 100644 contrib/bench_radix_tree/bench_radix_tree.control
create mode 100644 contrib/bench_radix_tree/expected/bench.out
create mode 100644 contrib/bench_radix_tree/meson.build
create mode 100644 contrib/bench_radix_tree/sql/bench.sql
diff --git a/contrib/bench_radix_tree/Makefile b/contrib/bench_radix_tree/Makefile
new file mode 100644
index 0000000000..952bb0ceae
--- /dev/null
+++ b/contrib/bench_radix_tree/Makefile
@@ -0,0 +1,21 @@
+# contrib/bench_radix_tree/Makefile
+
+MODULE_big = bench_radix_tree
+OBJS = \
+ bench_radix_tree.o
+
+EXTENSION = bench_radix_tree
+DATA = bench_radix_tree--1.0.sql
+
+REGRESS = bench_fixed_height
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/bench_radix_tree
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
new file mode 100644
index 0000000000..fbf51c1086
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
@@ -0,0 +1,87 @@
+/* contrib/bench_radix_tree/bench_radix_tree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION bench_radix_tree" to load this file. \quit
+
+create function bench_shuffle_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_seq_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_load_random_int(
+cnt int8,
+OUT mem_allocated int8,
+OUT load_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_search_random_nodes(
+cnt int8,
+filter_str text DEFAULT NULL,
+OUT mem_allocated int8,
+OUT load_ms int8,
+OUT search_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C VOLATILE PARALLEL UNSAFE;
+
+create function bench_fixed_height_search(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT rt_search_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_node128_load(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_sparseload_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_tidstore_load(
+minblk int4,
+maxblk int4,
+OUT mem_allocated int8,
+OUT load_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
new file mode 100644
index 0000000000..b5ad75364c
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -0,0 +1,717 @@
+/*-------------------------------------------------------------------------
+ *
+ * bench_radix_tree.c
+ *
+ * Copyright (c) 2016-2022, PostgreSQL Global Development Group
+ *
+ * contrib/bench_radix_tree/bench_radix_tree.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/tidstore.h"
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "funcapi.h"
+#include "lib/radixtree.h"
+#include <math.h>
+#include "miscadmin.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/timestamp.h"
+
+PG_MODULE_MAGIC;
+
+/* run benchmark also for binary-search case? */
+/* #define MEASURE_BINARY_SEARCH 1 */
+
+#define TIDS_PER_BLOCK_FOR_LOAD 30
+#define TIDS_PER_BLOCK_FOR_LOOKUP 50
+
+//#define RT_DEBUG
+#define RT_PREFIX rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE uint64
+// WIP: compiles with warnings because rt_attach is defined but not used
+// #define RT_SHMEM
+#include "lib/radixtree.h"
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+ return tree->ctl->num_keys;
+}
+
+
+PG_FUNCTION_INFO_V1(bench_seq_search);
+PG_FUNCTION_INFO_V1(bench_shuffle_search);
+PG_FUNCTION_INFO_V1(bench_load_random_int);
+PG_FUNCTION_INFO_V1(bench_fixed_height_search);
+PG_FUNCTION_INFO_V1(bench_search_random_nodes);
+PG_FUNCTION_INFO_V1(bench_node128_load);
+PG_FUNCTION_INFO_V1(bench_tidstore_load);
+
+static uint64
+tid_to_key_off(ItemPointer tid, uint32 *off)
+{
+ uint64 upper;
+ uint32 shift = pg_ceil_log2_32(MaxHeapTuplesPerPage);
+ int64 tid_i;
+
+ Assert(ItemPointerGetOffsetNumber(tid) < MaxHeapTuplesPerPage);
+
+ tid_i = ItemPointerGetOffsetNumber(tid);
+ tid_i |= (uint64) ItemPointerGetBlockNumber(tid) << shift;
+
+ /* log(sizeof(uint64) * BITS_PER_BYTE, 2) = log(64, 2) = 6 */
+ *off = tid_i & ((1 << 6) - 1);
+ upper = tid_i >> 6;
+ Assert(*off < (sizeof(uint64) * BITS_PER_BYTE));
+
+ Assert(*off < 64);
+
+ return upper;
+}
+
+static int
+shuffle_randrange(pg_prng_state *state, int lower, int upper)
+{
+ return (int) floor(pg_prng_double(state) * ((upper - lower) + 0.999999)) + lower;
+}
+
+/* Naive Fisher-Yates implementation*/
+static void
+shuffle_itemptrs(ItemPointer itemptr, uint64 nitems)
+{
+ /* reproducability */
+ pg_prng_state state;
+
+ pg_prng_seed(&state, 0);
+
+ for (int i = 0; i < nitems - 1; i++)
+ {
+ int j = shuffle_randrange(&state, i, nitems - 1);
+ ItemPointerData t = itemptr[j];
+
+ itemptr[j] = itemptr[i];
+ itemptr[i] = t;
+ }
+}
+
+static ItemPointer
+generate_tids(BlockNumber minblk, BlockNumber maxblk, int ntids_per_blk, uint64 *ntids_p,
+ bool random_block)
+{
+ ItemPointer tids;
+ uint64 maxitems;
+ uint64 ntids = 0;
+ pg_prng_state state;
+
+ maxitems = (maxblk - minblk + 1) * ntids_per_blk;
+ tids = MemoryContextAllocHuge(TopTransactionContext,
+ sizeof(ItemPointerData) * maxitems);
+
+ if (random_block)
+ pg_prng_seed(&state, 0x9E3779B185EBCA87);
+
+ for (BlockNumber blk = minblk; blk < maxblk; blk++)
+ {
+ if (random_block && !pg_prng_bool(&state))
+ continue;
+
+ for (OffsetNumber off = FirstOffsetNumber;
+ off <= ntids_per_blk; off++)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ ItemPointerSetBlockNumber(&(tids[ntids]), blk);
+ ItemPointerSetOffsetNumber(&(tids[ntids]), off);
+
+ ntids++;
+ }
+ }
+
+ *ntids_p = ntids;
+ return tids;
+}
+
+#ifdef MEASURE_BINARY_SEARCH
+static int
+vac_cmp_itemptr(const void *left, const void *right)
+{
+ BlockNumber lblk,
+ rblk;
+ OffsetNumber loff,
+ roff;
+
+ lblk = ItemPointerGetBlockNumber((ItemPointer) left);
+ rblk = ItemPointerGetBlockNumber((ItemPointer) right);
+
+ if (lblk < rblk)
+ return -1;
+ if (lblk > rblk)
+ return 1;
+
+ loff = ItemPointerGetOffsetNumber((ItemPointer) left);
+ roff = ItemPointerGetOffsetNumber((ItemPointer) right);
+
+ if (loff < roff)
+ return -1;
+ if (loff > roff)
+ return 1;
+
+ return 0;
+}
+#endif
+
+Datum
+bench_tidstore_load(PG_FUNCTION_ARGS)
+{
+ BlockNumber minblk = PG_GETARG_INT32(0);
+ BlockNumber maxblk = PG_GETARG_INT32(1);
+ TidStore *ts;
+ OffsetNumber *offs;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_ms;
+ TupleDesc tupdesc;
+ Datum values[2];
+ bool nulls[2] = {false};
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ offs = palloc(sizeof(OffsetNumber) * TIDS_PER_BLOCK_FOR_LOAD);
+ for (int i = 0; i < TIDS_PER_BLOCK_FOR_LOAD; i++)
+ offs[i] = i + 1; /* FirstOffsetNumber is 1 */
+
+ ts = tidstore_create(1 * 1024L * 1024L * 1024L, MaxHeapTuplesPerPage, NULL);
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* load tids */
+ start_time = GetCurrentTimestamp();
+ for (BlockNumber blkno = minblk; blkno < maxblk; blkno++)
+ tidstore_add_tids(ts, blkno, offs, TIDS_PER_BLOCK_FOR_LOAD);
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_ms = secs * 1000 + usecs / 1000;
+
+ values[0] = Int64GetDatum(tidstore_memory_usage(ts));
+ values[1] = Int64GetDatum(load_ms);
+
+ tidstore_destroy(ts);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+static Datum
+bench_search(FunctionCallInfo fcinfo, bool shuffle)
+{
+ BlockNumber minblk = PG_GETARG_INT32(0);
+ BlockNumber maxblk = PG_GETARG_INT32(1);
+ bool random_block = PG_GETARG_BOOL(2);
+ rt_radix_tree *rt = NULL;
+ uint64 ntids;
+ uint64 key;
+ uint64 last_key = PG_UINT64_MAX;
+ uint64 val = 0;
+ ItemPointer tids;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_load_ms,
+ rt_search_ms;
+ Datum values[7];
+ bool nulls[7];
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ tids = generate_tids(minblk, maxblk, TIDS_PER_BLOCK_FOR_LOAD, &ntids, random_block);
+
+ /* measure the load time of the radix tree */
+ rt = rt_create(CurrentMemoryContext);
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ uint32 off;
+
+ CHECK_FOR_INTERRUPTS();
+
+ key = tid_to_key_off(tid, &off);
+
+ if (last_key != PG_UINT64_MAX && last_key != key)
+ {
+ rt_set(rt, last_key, &val);
+ val = 0;
+ }
+
+ last_key = key;
+ val |= (uint64) 1 << off;
+ }
+ if (last_key != PG_UINT64_MAX)
+ rt_set(rt, last_key, &val);
+
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_load_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+ if (shuffle)
+ shuffle_itemptrs(tids, ntids);
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* meaure the serach time of the radix tree */
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ uint32 off;
+ volatile bool ret; /* prevent calling rt_search from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ key = tid_to_key_off(tid, &off);
+
+ ret = rt_search(rt, key, &val);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_search_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int64GetDatum(rt_num_entries(rt));
+ values[1] = Int64GetDatum(rt_memory_usage(rt));
+ values[2] = Int64GetDatum(sizeof(ItemPointerData) * ntids);
+ values[3] = Int64GetDatum(rt_load_ms);
+ nulls[4] = true; /* ar_load_ms */
+ values[5] = Int64GetDatum(rt_search_ms);
+ nulls[6] = true; /* ar_search_ms */
+
+#ifdef MEASURE_BINARY_SEARCH
+ {
+ ItemPointer itemptrs = NULL;
+
+ int64 ar_load_ms,
+ ar_search_ms;
+
+ /* measure the load time of the array */
+ itemptrs = MemoryContextAllocHuge(CurrentMemoryContext,
+ sizeof(ItemPointerData) * ntids);
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointerSetBlockNumber(&(itemptrs[i]),
+ ItemPointerGetBlockNumber(&(tids[i])));
+ ItemPointerSetOffsetNumber(&(itemptrs[i]),
+ ItemPointerGetOffsetNumber(&(tids[i])));
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ ar_load_ms = secs * 1000 + usecs / 1000;
+
+ /* next, measure the serach time of the array */
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ volatile bool ret; /* prevent calling bsearch from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ ret = bsearch((void *) tid,
+ (void *) itemptrs,
+ ntids,
+ sizeof(ItemPointerData),
+ vac_cmp_itemptr);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ ar_search_ms = secs * 1000 + usecs / 1000;
+
+ /* set the result */
+ nulls[4] = false;
+ values[4] = Int64GetDatum(ar_load_ms);
+ nulls[6] = false;
+ values[6] = Int64GetDatum(ar_search_ms);
+ }
+#endif
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_seq_search(PG_FUNCTION_ARGS)
+{
+ return bench_search(fcinfo, false);
+}
+
+Datum
+bench_shuffle_search(PG_FUNCTION_ARGS)
+{
+ return bench_search(fcinfo, true);
+}
+
+Datum
+bench_load_random_int(PG_FUNCTION_ARGS)
+{
+ uint64 cnt = (uint64) PG_GETARG_INT64(0);
+ rt_radix_tree *rt;
+ pg_prng_state state;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_time_ms;
+ Datum values[2];
+ bool nulls[2];
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ pg_prng_seed(&state, 0);
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 key = pg_prng_uint64(&state);
+
+ rt_set(rt, key, &key);
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_time_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int64GetDatum(rt_memory_usage(rt));
+ values[1] = Int64GetDatum(load_time_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+/* copy of splitmix64() */
+static uint64
+hash64(uint64 x)
+{
+ x ^= x >> 30;
+ x *= UINT64CONST(0xbf58476d1ce4e5b9);
+ x ^= x >> 27;
+ x *= UINT64CONST(0x94d049bb133111eb);
+ x ^= x >> 31;
+ return x;
+}
+
+/* attempts to have a relatively even population of node kinds */
+Datum
+bench_search_random_nodes(PG_FUNCTION_ARGS)
+{
+ uint64 cnt = (uint64) PG_GETARG_INT64(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_time_ms;
+ int64 search_time_ms;
+ Datum values[3] = {0};
+ bool nulls[3] = {0};
+ /* from trial and error */
+ uint64 filter = (((uint64) 0x7F<<32) | (0x07<<24) | (0xFF<<16) | 0xFF);
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (!PG_ARGISNULL(1))
+ {
+ char *filter_str = text_to_cstring(PG_GETARG_TEXT_P(1));
+
+ if (sscanf(filter_str, "0x%lX", &filter) == 0)
+ elog(ERROR, "invalid filter string %s", filter_str);
+ }
+ elog(NOTICE, "bench with filter 0x%lX", filter);
+
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 hash = hash64(i);
+ uint64 key = hash & filter;
+
+ rt_set(rt, key, &key);
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_time_ms = secs * 1000 + usecs / 1000;
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 hash = hash64(i);
+ uint64 key = hash & filter;
+ uint64 val;
+ volatile bool ret; /* prevent calling rt_search from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ ret = rt_search(rt, key, &val);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ search_time_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ values[0] = Int64GetDatum(rt_memory_usage(rt));
+ values[1] = Int64GetDatum(load_time_ms);
+ values[2] = Int64GetDatum(search_time_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_fixed_height_search(PG_FUNCTION_ARGS)
+{
+ int fanout = PG_GETARG_INT32(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_load_ms,
+ rt_search_ms;
+ Datum values[5];
+ bool nulls[5];
+
+ /* test boundary between vector and iteration */
+ const int n_keys = 5 * 16 * 16 * 16 * 16;
+ uint64 r,
+ h,
+ i,
+ j,
+ k;
+ uint64 key_id;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+
+ key_id = 0;
+
+ /*
+ * lower nodes have limited fanout, the top is only limited by
+ * bits-per-byte
+ */
+ for (r = 1;; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ for (i = 1; i <= fanout; i++)
+ {
+ for (j = 1; j <= fanout; j++)
+ {
+ for (k = 1; k <= fanout; k++)
+ {
+ uint64 key;
+
+ key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+ CHECK_FOR_INTERRUPTS();
+
+ key_id++;
+ if (key_id > n_keys)
+ goto finish_set;
+
+ rt_set(rt, key, &key_id);
+ }
+ }
+ }
+ }
+ }
+finish_set:
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_load_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ /* meaure the search time of the radix tree */
+ start_time = GetCurrentTimestamp();
+
+ key_id = 0;
+ for (r = 1;; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ for (i = 1; i <= fanout; i++)
+ {
+ for (j = 1; j <= fanout; j++)
+ {
+ for (k = 1; k <= fanout; k++)
+ {
+ uint64 key,
+ val;
+
+ key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+ CHECK_FOR_INTERRUPTS();
+
+ key_id++;
+ if (key_id > n_keys)
+ goto finish_search;
+
+ rt_search(rt, key, &val);
+ }
+ }
+ }
+ }
+ }
+finish_search:
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_search_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int32GetDatum(fanout);
+ values[1] = Int64GetDatum(rt_num_entries(rt));
+ values[2] = Int64GetDatum(rt_memory_usage(rt));
+ values[3] = Int64GetDatum(rt_load_ms);
+ values[4] = Int64GetDatum(rt_search_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_node128_load(PG_FUNCTION_ARGS)
+{
+ int fanout = PG_GETARG_INT32(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_sparseload_ms;
+ Datum values[5];
+ bool nulls[5];
+
+ uint64 r,
+ h;
+ uint64 key_id;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ rt = rt_create(CurrentMemoryContext);
+
+ key_id = 0;
+
+ for (r = 1; r <= fanout; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (h);
+
+ key_id++;
+ rt_set(rt, key, &key_id);
+ }
+ }
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ /* measure sparse deletion and re-loading */
+ start_time = GetCurrentTimestamp();
+
+ for (int t = 0; t<10000; t++)
+ {
+ /* delete one key in each leaf */
+ for (r = 1; r <= fanout; r++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (fanout);
+
+ rt_delete(rt, key);
+ }
+
+ /* add them all back */
+ key_id = 0;
+ for (r = 1; r <= fanout; r++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (fanout);
+
+ key_id++;
+ rt_set(rt, key, &key_id);
+ }
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_sparseload_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int32GetDatum(fanout);
+ values[1] = Int64GetDatum(rt_num_entries(rt));
+ values[2] = Int64GetDatum(rt_memory_usage(rt));
+ values[3] = Int64GetDatum(rt_sparseload_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
diff --git a/contrib/bench_radix_tree/bench_radix_tree.control b/contrib/bench_radix_tree/bench_radix_tree.control
new file mode 100644
index 0000000000..1d988e6c9a
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.control
@@ -0,0 +1,6 @@
+# bench_radix_tree extension
+comment = 'benchmark suits for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/bench_radix_tree'
+relocatable = true
+trusted = true
diff --git a/contrib/bench_radix_tree/expected/bench.out b/contrib/bench_radix_tree/expected/bench.out
new file mode 100644
index 0000000000..60c303892e
--- /dev/null
+++ b/contrib/bench_radix_tree/expected/bench.out
@@ -0,0 +1,13 @@
+create extension bench_radix_tree;
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/bench_radix_tree/meson.build b/contrib/bench_radix_tree/meson.build
new file mode 100644
index 0000000000..332c1ae7df
--- /dev/null
+++ b/contrib/bench_radix_tree/meson.build
@@ -0,0 +1,33 @@
+bench_radix_tree_sources = files(
+ 'bench_radix_tree.c',
+)
+
+if host_system == 'windows'
+ bench_radix_tree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'bench_radix_tree',
+ '--FILEDESC', 'bench_radix_tree - performance test code for radix tree',])
+endif
+
+bench_radix_tree = shared_module('bench_radix_tree',
+ bench_radix_tree_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += bench_radix_tree
+
+install_data(
+ 'bench_radix_tree.control',
+ 'bench_radix_tree--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'bench_radix_tree',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'bench_radix_tree',
+ ],
+ },
+}
diff --git a/contrib/bench_radix_tree/sql/bench.sql b/contrib/bench_radix_tree/sql/bench.sql
new file mode 100644
index 0000000000..a46018c9d4
--- /dev/null
+++ b/contrib/bench_radix_tree/sql/bench.sql
@@ -0,0 +1,16 @@
+create extension bench_radix_tree;
+
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/meson.build b/contrib/meson.build
index bd4a57c43c..52253de793 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,6 +12,7 @@ subdir('amcheck')
subdir('auth_delay')
subdir('auto_explain')
subdir('basic_archive')
+#subdir('bench_radix_tree')
subdir('bloom')
subdir('basebackup_to_shell')
subdir('bool_plperl')
--
2.39.1
[text/x-patch] v28-0007-Prevent-inlining-of-interface-functions-for-shme.patch (752B, ../../CAFBsxsF4QAOzSPbQvekFn56WJJuYoZM8m=FqbKyb3dwhB+oyWg@mail.gmail.com/9-v28-0007-Prevent-inlining-of-interface-functions-for-shme.patch)
download | inline diff:
From 54ab02eb2188382185436059ff6e7ad95d970c5d Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 14 Feb 2023 17:00:31 +0700
Subject: [PATCH v28 07/10] Prevent inlining of interface functions for shmem
---
src/backend/access/common/tidstore.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index ad8c0866e2..d1b4675ea4 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -84,7 +84,7 @@
#define RT_PREFIX shared_rt
#define RT_SHMEM
-#define RT_SCOPE static
+#define RT_SCOPE static pg_noinline
#define RT_DECLARE
#define RT_DEFINE
#define RT_VALUE_TYPE uint64
--
2.39.1
[text/x-patch] v28-0009-Speed-up-tidstore_iter_extract_tids.patch (1.3K, ../../CAFBsxsF4QAOzSPbQvekFn56WJJuYoZM8m=FqbKyb3dwhB+oyWg@mail.gmail.com/10-v28-0009-Speed-up-tidstore_iter_extract_tids.patch)
download | inline diff:
From 8ccc66211973bcc44a6bad45c05302ca743c1489 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 14 Feb 2023 17:53:37 +0700
Subject: [PATCH v28 09/10] Speed up tidstore_iter_extract_tids()
---
src/backend/access/common/tidstore.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index d1b4675ea4..5a897c01f7 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -632,21 +632,21 @@ tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
{
TidStoreIterResult *result = (&iter->result);
- for (int i = 0; i < sizeof(uint64) * BITS_PER_BYTE; i++)
+ while (val)
{
uint64 tid_i;
OffsetNumber off;
- if ((val & (UINT64CONST(1) << i)) == 0)
- continue;
-
tid_i = key << TIDSTORE_VALUE_NBITS;
- tid_i |= i;
+ tid_i |= pg_rightmost_one_pos64(val);
off = tid_i & ((UINT64CONST(1) << iter->ts->control->offset_nbits) - 1);
Assert(result->num_offsets < iter->ts->control->max_offset);
result->offsets[result->num_offsets++] = off;
+
+ /* unset the rightmost bit */
+ val &= ~pg_rightmost_one64(val);
}
result->blkno = key_get_blkno(iter->ts, key);
--
2.39.1
[text/x-patch] v28-0008-Measure-iteration-of-tidstore.patch (3.7K, ../../CAFBsxsF4QAOzSPbQvekFn56WJJuYoZM8m=FqbKyb3dwhB+oyWg@mail.gmail.com/11-v28-0008-Measure-iteration-of-tidstore.patch)
download | inline diff:
From 72bb462b1dab005cbc2aff265baedbaaee62cb2b Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 14 Feb 2023 17:02:53 +0700
Subject: [PATCH v28 08/10] Measure iteration of tidstore
---
.../bench_radix_tree--1.0.sql | 3 +-
contrib/bench_radix_tree/bench_radix_tree.c | 40 ++++++++++++++++---
contrib/meson.build | 2 +-
3 files changed, 38 insertions(+), 7 deletions(-)
diff --git a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
index fbf51c1086..ad66265e23 100644
--- a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
+++ b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
@@ -80,7 +80,8 @@ create function bench_tidstore_load(
minblk int4,
maxblk int4,
OUT mem_allocated int8,
-OUT load_ms int8
+OUT load_ms int8,
+OUT iter_ms int8
)
returns record
as 'MODULE_PATHNAME'
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
index b5ad75364c..6e5149e2c4 100644
--- a/contrib/bench_radix_tree/bench_radix_tree.c
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -176,15 +176,18 @@ bench_tidstore_load(PG_FUNCTION_ARGS)
BlockNumber minblk = PG_GETARG_INT32(0);
BlockNumber maxblk = PG_GETARG_INT32(1);
TidStore *ts;
+ TidStoreIter *iter;
+ TidStoreIterResult *result;
OffsetNumber *offs;
TimestampTz start_time,
end_time;
long secs;
int usecs;
int64 load_ms;
+ int64 iter_ms;
TupleDesc tupdesc;
- Datum values[2];
- bool nulls[2] = {false};
+ Datum values[3];
+ bool nulls[3] = {false};
/* Build a tuple descriptor for our result type */
if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
@@ -196,9 +199,6 @@ bench_tidstore_load(PG_FUNCTION_ARGS)
ts = tidstore_create(1 * 1024L * 1024L * 1024L, MaxHeapTuplesPerPage, NULL);
- elog(NOTICE, "sleeping for 2 seconds...");
- pg_usleep(2 * 1000000L);
-
/* load tids */
start_time = GetCurrentTimestamp();
for (BlockNumber blkno = minblk; blkno < maxblk; blkno++)
@@ -207,8 +207,22 @@ bench_tidstore_load(PG_FUNCTION_ARGS)
TimestampDifference(start_time, end_time, &secs, &usecs);
load_ms = secs * 1000 + usecs / 1000;
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* iterate through tids */
+ iter = tidstore_begin_iterate(ts);
+ start_time = GetCurrentTimestamp();
+ while ((result = tidstore_iterate_next(iter)) != NULL)
+ ;
+ tidstore_end_iterate(iter);
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ iter_ms = secs * 1000 + usecs / 1000;
+
values[0] = Int64GetDatum(tidstore_memory_usage(ts));
values[1] = Int64GetDatum(load_ms);
+ values[2] = Int64GetDatum(iter_ms);
tidstore_destroy(ts);
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
@@ -715,3 +729,19 @@ bench_node128_load(PG_FUNCTION_ARGS)
rt_free(rt);
PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
}
+
+/* to silence warnings about unused iter functions */
+static void pg_attribute_unused()
+stub_iter()
+{
+ rt_radix_tree *rt;
+ rt_iter *iter;
+ uint64 key = 1;
+ uint64 value = 1;
+
+ rt = rt_create(CurrentMemoryContext);
+
+ iter = rt_begin_iterate(rt);
+ rt_iterate_next(iter, &key, &value);
+ rt_end_iterate(iter);
+}
\ No newline at end of file
diff --git a/contrib/meson.build b/contrib/meson.build
index 52253de793..421d469f8c 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,7 +12,7 @@ subdir('amcheck')
subdir('auth_delay')
subdir('auto_explain')
subdir('basic_archive')
-#subdir('bench_radix_tree')
+subdir('bench_radix_tree')
subdir('bloom')
subdir('basebackup_to_shell')
subdir('bool_plperl')
--
2.39.1
[text/x-patch] v28-0010-Revert-building-benchmark-module-for-CI.patch (694B, ../../CAFBsxsF4QAOzSPbQvekFn56WJJuYoZM8m=FqbKyb3dwhB+oyWg@mail.gmail.com/12-v28-0010-Revert-building-benchmark-module-for-CI.patch)
download | inline diff:
From 42ba46f8073ee33bc5df6766f74f4c57587b070a Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 14 Feb 2023 19:31:34 +0700
Subject: [PATCH v28 10/10] Revert building benchmark module for CI
---
contrib/meson.build | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/contrib/meson.build b/contrib/meson.build
index 421d469f8c..52253de793 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,7 +12,7 @@ subdir('amcheck')
subdir('auth_delay')
subdir('auto_explain')
subdir('basic_archive')
-subdir('bench_radix_tree')
+#subdir('bench_radix_tree')
subdir('bloom')
subdir('basebackup_to_shell')
subdir('bool_plperl')
--
2.39.1
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-02-16 03:24 ` Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
1 sibling, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-02-16 03:24 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Tue, Feb 14, 2023 at 8:24 PM John Naylor
<[email protected]> wrote:
>
> On Mon, Feb 13, 2023 at 2:51 PM Masahiko Sawada <[email protected]> wrote:
> >
> > On Sat, Feb 11, 2023 at 2:33 PM John Naylor
> > <[email protected]> wrote:
> > >
> > > I didn't get any closer to radix-tree regression,
> >
> > Me neither. It seems that in v26, inserting chunks into node-32 is
> > slow but needs more analysis. I'll share if I found something
> > interesting.
>
> If that were the case, then the other benchmarks I ran would likely have slowed down as well, but they are the same or faster. There is one microbenchmark I didn't run before: "select * from bench_fixed_height_search(15)" (15 to reduce noise from growing size class, and despite the name it measures load time as well). Trying this now shows no difference: a few runs range 19 to 21ms in each version. That also reinforces that update_inner is fine and that the move to value pointer API didn't regress.
>
> Changing TIDS_PER_BLOCK_FOR_LOAD to 1 to stress the tree more gives (min of 5, perf run separate from measurements):
>
> v15 + v26 store:
>
> mem_allocated | load_ms
> ---------------+---------
> 98202152 | 553
>
> 19.71% postgres postgres [.] tidstore_add_tids
> + 31.47% postgres postgres [.] rt_set
> = 51.18%
>
> 20.62% postgres postgres [.] rt_node_insert_leaf
> 6.05% postgres postgres [.] AllocSetAlloc
> 4.74% postgres postgres [.] AllocSetFree
> 4.62% postgres postgres [.] palloc
> 2.23% postgres postgres [.] SlabAlloc
>
> v26:
>
> mem_allocated | load_ms
> ---------------+---------
> 98202032 | 617
>
> 57.45% postgres postgres [.] tidstore_add_tids
>
> 20.67% postgres postgres [.] local_rt_node_insert_leaf
> 5.99% postgres postgres [.] AllocSetAlloc
> 3.55% postgres postgres [.] palloc
> 3.05% postgres postgres [.] AllocSetFree
> 2.05% postgres postgres [.] SlabAlloc
>
> So it seems the store itself got faster when we removed shared memory paths from the v26 store to test it against v15.
>
> I thought to favor the local memory case in the tidstore by controlling inlining -- it's smaller and will be called much more often, so I tried the following (done in 0007)
>
> #define RT_PREFIX shared_rt
> #define RT_SHMEM
> -#define RT_SCOPE static
> +#define RT_SCOPE static pg_noinline
>
> That brings it down to
>
> mem_allocated | load_ms
> ---------------+---------
> 98202032 | 590
The improvement makes sense to me. I've also done the same test (with
changing TIDS_PER_BLOCK_FOR_LOAD to 1):
w/o 0007 patch:
mem_allocated | load_ms | iter_ms
---------------+---------+---------
98202032 | 334 | 445
(1 row)
w/ 0007 patch:
mem_allocated | load_ms | iter_ms
---------------+---------+---------
98202032 | 316 | 434
(1 row)
On the other hand, with TIDS_PER_BLOCK_FOR_LOAD being 30, the load
performance didn't improve:
w/0 0007 patch:
mem_allocated | load_ms | iter_ms
---------------+---------+---------
98202032 | 601 | 608
(1 row)
w/ 0007 patch:
mem_allocated | load_ms | iter_ms
---------------+---------+---------
98202032 | 610 | 606
(1 row)
That being said, it might be within noise level, so I agree with 0007 patch.
> Perhaps some slowdown is unavoidable, but it would be nice to understand why.
True.
>
> > I can think that something like traversing a HOT chain could visit
> > offsets out of order. But fortunately we prune such collected TIDs
> > before heap vacuum in heap case.
>
> Further, currently we *already* assume we populate the tid array in order (for binary search), so we can just continue assuming that (with an assert added since it's more public in this form). I'm not sure why such basic common sense evaded me a few versions ago...
Right. TidStore is implemented not only for heap, so loading
out-of-order TIDs might be important in the future.
> > > If these are acceptable, I can incorporate them into a later patchset.
> >
> > These are nice improvements! I agree with all changes.
>
> Great, I've squashed these into the tidstore patch (0004). Also added 0005, which is just a simplification.
>
I've attached some small patches to improve the radix tree and tidstrore:
We have the following WIP comment in test_radixtree:
// WIP: compiles with warnings because rt_attach is defined but not used
// #define RT_SHMEM
How about unsetting RT_SCOPE to suppress warnings for unused rt_attach
and friends?
FYI I've briefly tested the TidStore with blocksize = 32kb, and it
seems to work fine.
> I squashed the earlier dead code removal into the radix tree patch.
Thanks!
>
> v27-0008 measures tid store iteration performance and adds a stub function to prevent spurious warnings, so the benchmarking module can always be built.
>
> Getting the list of offsets from the old array for a given block is always trivial, but tidstore_iter_extract_tids() is doing a huge amount of unnecessary work when TIDS_PER_BLOCK_FOR_LOAD is 1, enough to exceed the load time:
>
> mem_allocated | load_ms | iter_ms
> ---------------+---------+---------
> 98202032 | 589 | 915
>
> Fortunately, it's an easy fix, done in 0009.
>
> mem_allocated | load_ms | iter_ms
> ---------------+---------+---------
> 98202032 | 589 | 153
Cool!
>
> I'll soon resume more cosmetic review of the tid store, but this is enough to post.
Thanks!
You removed the vacuum integration patch from v27, is there any reason for that?
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
From f06557689f33d9b11be1083362fcce19665b4014 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Thu, 16 Feb 2023 12:18:22 +0900
Subject: [PATCH v2899 2/2] Small improvements for radixtree and tests.
---
src/include/lib/radixtree.h | 2 +-
src/test/modules/test_radixtree/test_radixtree.c | 13 ++++++++++---
2 files changed, 11 insertions(+), 4 deletions(-)
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index 1cdb995e54..e546bd705c 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -1622,7 +1622,7 @@ RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
/* Descend the tree until we reach a leaf node */
while (shift >= 0)
{
- RT_PTR_ALLOC new_child = RT_INVALID_PTR_ALLOC;;
+ RT_PTR_ALLOC new_child = RT_INVALID_PTR_ALLOC;
child = RT_PTR_GET_LOCAL(tree, stored_child);
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
index f944945db9..afe53382f3 100644
--- a/src/test/modules/test_radixtree/test_radixtree.c
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -107,13 +107,12 @@ static const test_spec test_specs[] = {
/* define the radix tree implementation to test */
#define RT_PREFIX rt
-#define RT_SCOPE static
+#define RT_SCOPE
#define RT_DECLARE
#define RT_DEFINE
#define RT_USE_DELETE
#define RT_VALUE_TYPE TestValueType
-// WIP: compiles with warnings because rt_attach is defined but not used
-// #define RT_SHMEM
+/* #define RT_SHMEM */
#include "lib/radixtree.h"
@@ -142,6 +141,8 @@ test_empty(void)
#ifdef RT_SHMEM
int tranche_id = LWLockNewTrancheId();
dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
dsa = dsa_create(tranche_id);
radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
@@ -188,6 +189,8 @@ test_basic(int children, bool test_inner)
#ifdef RT_SHMEM
int tranche_id = LWLockNewTrancheId();
dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
dsa = dsa_create(tranche_id);
#endif
@@ -358,6 +361,8 @@ test_node_types(uint8 shift)
#ifdef RT_SHMEM
int tranche_id = LWLockNewTrancheId();
dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
dsa = dsa_create(tranche_id);
#endif
@@ -406,6 +411,8 @@ test_pattern(const test_spec * spec)
#ifdef RT_SHMEM
int tranche_id = LWLockNewTrancheId();
dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
dsa = dsa_create(tranche_id);
#endif
--
2.31.1
From f6ed6e18b2281cee96af98a39bdfc453117e6a21 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Thu, 16 Feb 2023 12:17:59 +0900
Subject: [PATCH v2899 1/2] comment update and test the shared tidstore.
---
src/backend/access/common/tidstore.c | 19 +++-------
.../modules/test_tidstore/test_tidstore.c | 37 +++++++++++++++++--
2 files changed, 40 insertions(+), 16 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 015e3dea81..8c05e60d92 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -64,13 +64,9 @@
* |---------------------------------------------| key
*
* The maximum height of the radix tree is 5 in this case.
- *
- * If the number of bits for offset number fits in a 64-bit value, we don't
- * encode tids but directly use the block number and the offset number as key
- * and value, respectively.
*/
#define TIDSTORE_VALUE_NBITS 6 /* log(64, 2) */
-#define TIDSTORE_OFFSET_MASK ((1 << TIDSTORE_VALUE_NBITS) - 1)
+#define TIDSTORE_OFFSET_MASK ((1 << TIDSTORE_VALUE_NBITS) - 1)
/* A magic value used to identify our TidStores. */
#define TIDSTORE_MAGIC 0x826f6a10
@@ -99,9 +95,10 @@ typedef struct TidStoreControl
/* These values are never changed after creation */
size_t max_bytes; /* the maximum bytes a TidStore can use */
int max_offset; /* the maximum offset number */
- int offset_nbits; /* the number of bits required for max_offset */
- int offset_key_nbits; /* the number of bits of a offset number
- * used for the key */
+ int offset_nbits; /* the number of bits required for an offset
+ * number */
+ int offset_key_nbits; /* the number of bits of an offset number
+ * used in a key */
/* The below fields are used only in shared case */
@@ -227,10 +224,6 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
if (ts->control->offset_nbits < TIDSTORE_VALUE_NBITS)
ts->control->offset_nbits = TIDSTORE_VALUE_NBITS;
- /*
- * We use tid encoding if the number of bits for the offset number doesn't
- * fix in a value, uint64.
- */
ts->control->offset_key_nbits =
ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
@@ -379,7 +372,7 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
{
uint64 off_bit;
- /* encode the tid to key and val */
+ /* encode the tid to a key and partial offset */
key = encode_key_off(ts, blkno, offsets[i], &off_bit);
/* make sure we scanned the line pointer array in order */
diff --git a/src/test/modules/test_tidstore/test_tidstore.c b/src/test/modules/test_tidstore/test_tidstore.c
index 9b849ae8e8..9a1217f833 100644
--- a/src/test/modules/test_tidstore/test_tidstore.c
+++ b/src/test/modules/test_tidstore/test_tidstore.c
@@ -18,10 +18,13 @@
#include "miscadmin.h"
#include "storage/block.h"
#include "storage/itemptr.h"
+#include "storage/lwlock.h"
#include "utils/memutils.h"
PG_MODULE_MAGIC;
+/* #define TEST_SHARED_TIDSTORE 1 */
+
#define TEST_TIDSTORE_MAX_BYTES (2 * 1024 * 1024L) /* 2MB */
PG_FUNCTION_INFO_V1(test_tidstore);
@@ -59,6 +62,18 @@ test_basic(int max_offset)
OffsetNumber offs[TEST_TIDSTORE_NUM_OFFSETS];
int blk_idx;
+#ifdef TEST_SHARED_TIDSTORE
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_tidstore");
+ dsa = dsa_create(tranche_id);
+
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, dsa);
+#else
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
+#endif
+
/* prepare the offset array */
offs[0] = FirstOffsetNumber;
offs[1] = FirstOffsetNumber + 1;
@@ -66,8 +81,6 @@ test_basic(int max_offset)
offs[3] = max_offset - 1;
offs[4] = max_offset;
- ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
-
/* add tids */
for (int i = 0; i < TEST_TIDSTORE_NUM_BLOCKS; i++)
tidstore_add_tids(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
@@ -144,6 +157,10 @@ test_basic(int max_offset)
}
tidstore_destroy(ts);
+
+#ifdef TEST_SHARED_TIDSTORE
+ dsa_detach(dsa);
+#endif
}
static void
@@ -153,9 +170,19 @@ test_empty(void)
TidStoreIter *iter;
ItemPointerData tid;
- elog(NOTICE, "testing empty tidstore");
+#ifdef TEST_SHARED_TIDSTORE
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_tidstore");
+ dsa = dsa_create(tranche_id);
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, dsa);
+#else
ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
+#endif
+
+ elog(NOTICE, "testing empty tidstore");
ItemPointerSet(&tid, 0, FirstOffsetNumber);
if (tidstore_lookup_tid(ts, &tid))
@@ -180,6 +207,10 @@ test_empty(void)
tidstore_end_iterate(iter);
tidstore_destroy(ts);
+
+#ifdef TEST_SHARED_TIDSTORE
+ dsa_detach(dsa);
+#endif
}
Datum
--
2.31.1
Attachments:
[text/plain] v2899-0002-Small-improvements-for-radixtree-and-tests.patch.txt (2.5K, ../../CAD21AoD-sOZY5KFTQw1PQDvRExFGhda4SrW8C1tFr7jgJ_6ByQ@mail.gmail.com/2-v2899-0002-Small-improvements-for-radixtree-and-tests.patch.txt)
download | inline diff:
From f06557689f33d9b11be1083362fcce19665b4014 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Thu, 16 Feb 2023 12:18:22 +0900
Subject: [PATCH v2899 2/2] Small improvements for radixtree and tests.
---
src/include/lib/radixtree.h | 2 +-
src/test/modules/test_radixtree/test_radixtree.c | 13 ++++++++++---
2 files changed, 11 insertions(+), 4 deletions(-)
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index 1cdb995e54..e546bd705c 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -1622,7 +1622,7 @@ RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
/* Descend the tree until we reach a leaf node */
while (shift >= 0)
{
- RT_PTR_ALLOC new_child = RT_INVALID_PTR_ALLOC;;
+ RT_PTR_ALLOC new_child = RT_INVALID_PTR_ALLOC;
child = RT_PTR_GET_LOCAL(tree, stored_child);
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
index f944945db9..afe53382f3 100644
--- a/src/test/modules/test_radixtree/test_radixtree.c
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -107,13 +107,12 @@ static const test_spec test_specs[] = {
/* define the radix tree implementation to test */
#define RT_PREFIX rt
-#define RT_SCOPE static
+#define RT_SCOPE
#define RT_DECLARE
#define RT_DEFINE
#define RT_USE_DELETE
#define RT_VALUE_TYPE TestValueType
-// WIP: compiles with warnings because rt_attach is defined but not used
-// #define RT_SHMEM
+/* #define RT_SHMEM */
#include "lib/radixtree.h"
@@ -142,6 +141,8 @@ test_empty(void)
#ifdef RT_SHMEM
int tranche_id = LWLockNewTrancheId();
dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
dsa = dsa_create(tranche_id);
radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
@@ -188,6 +189,8 @@ test_basic(int children, bool test_inner)
#ifdef RT_SHMEM
int tranche_id = LWLockNewTrancheId();
dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
dsa = dsa_create(tranche_id);
#endif
@@ -358,6 +361,8 @@ test_node_types(uint8 shift)
#ifdef RT_SHMEM
int tranche_id = LWLockNewTrancheId();
dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
dsa = dsa_create(tranche_id);
#endif
@@ -406,6 +411,8 @@ test_pattern(const test_spec * spec)
#ifdef RT_SHMEM
int tranche_id = LWLockNewTrancheId();
dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
dsa = dsa_create(tranche_id);
#endif
--
2.31.1
[text/plain] v2899-0001-comment-update-and-test-the-shared-tidstore.patch.txt (4.8K, ../../CAD21AoD-sOZY5KFTQw1PQDvRExFGhda4SrW8C1tFr7jgJ_6ByQ@mail.gmail.com/3-v2899-0001-comment-update-and-test-the-shared-tidstore.patch.txt)
download | inline diff:
From f6ed6e18b2281cee96af98a39bdfc453117e6a21 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Thu, 16 Feb 2023 12:17:59 +0900
Subject: [PATCH v2899 1/2] comment update and test the shared tidstore.
---
src/backend/access/common/tidstore.c | 19 +++-------
.../modules/test_tidstore/test_tidstore.c | 37 +++++++++++++++++--
2 files changed, 40 insertions(+), 16 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 015e3dea81..8c05e60d92 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -64,13 +64,9 @@
* |---------------------------------------------| key
*
* The maximum height of the radix tree is 5 in this case.
- *
- * If the number of bits for offset number fits in a 64-bit value, we don't
- * encode tids but directly use the block number and the offset number as key
- * and value, respectively.
*/
#define TIDSTORE_VALUE_NBITS 6 /* log(64, 2) */
-#define TIDSTORE_OFFSET_MASK ((1 << TIDSTORE_VALUE_NBITS) - 1)
+#define TIDSTORE_OFFSET_MASK ((1 << TIDSTORE_VALUE_NBITS) - 1)
/* A magic value used to identify our TidStores. */
#define TIDSTORE_MAGIC 0x826f6a10
@@ -99,9 +95,10 @@ typedef struct TidStoreControl
/* These values are never changed after creation */
size_t max_bytes; /* the maximum bytes a TidStore can use */
int max_offset; /* the maximum offset number */
- int offset_nbits; /* the number of bits required for max_offset */
- int offset_key_nbits; /* the number of bits of a offset number
- * used for the key */
+ int offset_nbits; /* the number of bits required for an offset
+ * number */
+ int offset_key_nbits; /* the number of bits of an offset number
+ * used in a key */
/* The below fields are used only in shared case */
@@ -227,10 +224,6 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
if (ts->control->offset_nbits < TIDSTORE_VALUE_NBITS)
ts->control->offset_nbits = TIDSTORE_VALUE_NBITS;
- /*
- * We use tid encoding if the number of bits for the offset number doesn't
- * fix in a value, uint64.
- */
ts->control->offset_key_nbits =
ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
@@ -379,7 +372,7 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
{
uint64 off_bit;
- /* encode the tid to key and val */
+ /* encode the tid to a key and partial offset */
key = encode_key_off(ts, blkno, offsets[i], &off_bit);
/* make sure we scanned the line pointer array in order */
diff --git a/src/test/modules/test_tidstore/test_tidstore.c b/src/test/modules/test_tidstore/test_tidstore.c
index 9b849ae8e8..9a1217f833 100644
--- a/src/test/modules/test_tidstore/test_tidstore.c
+++ b/src/test/modules/test_tidstore/test_tidstore.c
@@ -18,10 +18,13 @@
#include "miscadmin.h"
#include "storage/block.h"
#include "storage/itemptr.h"
+#include "storage/lwlock.h"
#include "utils/memutils.h"
PG_MODULE_MAGIC;
+/* #define TEST_SHARED_TIDSTORE 1 */
+
#define TEST_TIDSTORE_MAX_BYTES (2 * 1024 * 1024L) /* 2MB */
PG_FUNCTION_INFO_V1(test_tidstore);
@@ -59,6 +62,18 @@ test_basic(int max_offset)
OffsetNumber offs[TEST_TIDSTORE_NUM_OFFSETS];
int blk_idx;
+#ifdef TEST_SHARED_TIDSTORE
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_tidstore");
+ dsa = dsa_create(tranche_id);
+
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, dsa);
+#else
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
+#endif
+
/* prepare the offset array */
offs[0] = FirstOffsetNumber;
offs[1] = FirstOffsetNumber + 1;
@@ -66,8 +81,6 @@ test_basic(int max_offset)
offs[3] = max_offset - 1;
offs[4] = max_offset;
- ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
-
/* add tids */
for (int i = 0; i < TEST_TIDSTORE_NUM_BLOCKS; i++)
tidstore_add_tids(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
@@ -144,6 +157,10 @@ test_basic(int max_offset)
}
tidstore_destroy(ts);
+
+#ifdef TEST_SHARED_TIDSTORE
+ dsa_detach(dsa);
+#endif
}
static void
@@ -153,9 +170,19 @@ test_empty(void)
TidStoreIter *iter;
ItemPointerData tid;
- elog(NOTICE, "testing empty tidstore");
+#ifdef TEST_SHARED_TIDSTORE
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_tidstore");
+ dsa = dsa_create(tranche_id);
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, dsa);
+#else
ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
+#endif
+
+ elog(NOTICE, "testing empty tidstore");
ItemPointerSet(&tid, 0, FirstOffsetNumber);
if (tidstore_lookup_tid(ts, &tid))
@@ -180,6 +207,10 @@ test_empty(void)
tidstore_end_iterate(iter);
tidstore_destroy(ts);
+
+#ifdef TEST_SHARED_TIDSTORE
+ dsa_detach(dsa);
+#endif
}
Datum
--
2.31.1
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-02-16 09:22 ` John Naylor <[email protected]>
2023-02-16 16:44 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Andres Freund <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
0 siblings, 2 replies; 78+ messages in thread
From: John Naylor @ 2023-02-16 09:22 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Thu, Feb 16, 2023 at 10:24 AM Masahiko Sawada <[email protected]>
wrote:
>
> On Tue, Feb 14, 2023 at 8:24 PM John Naylor
> <[email protected]> wrote:
> > > I can think that something like traversing a HOT chain could visit
> > > offsets out of order. But fortunately we prune such collected TIDs
> > > before heap vacuum in heap case.
> >
> > Further, currently we *already* assume we populate the tid array in
order (for binary search), so we can just continue assuming that (with an
assert added since it's more public in this form). I'm not sure why such
basic common sense evaded me a few versions ago...
>
> Right. TidStore is implemented not only for heap, so loading
> out-of-order TIDs might be important in the future.
That's what I was probably thinking about some weeks ago, but I'm having a
hard time imagining how it would come up, even for something like the
conveyor-belt concept.
> We have the following WIP comment in test_radixtree:
>
> // WIP: compiles with warnings because rt_attach is defined but not used
> // #define RT_SHMEM
>
> How about unsetting RT_SCOPE to suppress warnings for unused rt_attach
> and friends?
Sounds good to me, and the other fixes make sense as well.
> FYI I've briefly tested the TidStore with blocksize = 32kb, and it
> seems to work fine.
That was on my list, so great! How about the other end -- nominally we
allow 512b. (In practice it won't matter, but this would make sure I didn't
mess anything up when forcing all MaxTuplesPerPage to encode.)
> You removed the vacuum integration patch from v27, is there any reason
for that?
Just an oversight.
Now for some general comments on the tid store...
+ * TODO: The caller must be certain that no other backend will attempt to
+ * access the TidStore before calling this function. Other backend must
+ * explicitly call tidstore_detach to free up backend-local memory
associated
+ * with the TidStore. The backend that calls tidstore_destroy must not call
+ * tidstore_detach.
+ */
+void
+tidstore_destroy(TidStore *ts)
Do we need to do anything for this todo?
It might help readability to have a concept of "off_upper/off_lower", just
so we can describe things more clearly. The key is block + off_upper, and
the value is a bitmap of all the off_lower bits. I hinted at that in my
addition of encode_key_off(). Along those lines, maybe
s/TIDSTORE_OFFSET_MASK/TIDSTORE_OFFSET_LOWER_MASK/. Actually, I'm not even
sure the TIDSTORE_ prefix is valuable for these local macros.
The word "value" as a variable name is pretty generic in this context, and
it might be better to call it the off_lower_bitmap, at least in some
places. The "key" doesn't have a good short term for naming, but in
comments we should make sure we're clear it's "block# + off_upper".
I'm not a fan of the name "tid_i", even as a temp variable -- maybe
"compressed_tid"?
maybe s/tid_to_key_off/encode_tid/ and s/encode_key_off/encode_block_offset/
It might be worth using typedefs for key and value type. Actually, since
key type is fixed for the foreseeable future, maybe the radix tree template
should define a key typedef?
The term "result" is probably fine within the tidstore, but as a public
name used by vacuum, it's not very descriptive. I don't have a good idea,
though.
Some files in backend/access use CamelCase for public functions, although
it's not consistent. I think doing that for tidstore would help
readability, since they would stand out from rt_* functions and vacuum
functions. It's a matter of taste, though.
I don't understand the control flow in tidstore_iterate_next(), or when
BlockNumberIsValid() is true. If this is the best way to code this, it
needs more commentary.
Some comments on vacuum:
I think we'd better get some real-world testing of this, fairly soon.
I had an idea: If it's not too much effort, it might be worth splitting it
into two parts: one that just adds the store (not caring about its memory
limits or progress reporting etc). During index scan, check both the new
store and the array and log a warning (we don't want to exit or crash,
better to try to investigate while live if possible) if the result doesn't
match. Then perhaps set up an instance and let something like TPC-C run for
a few days. The second patch would just restore the rest of the current
patch. That would help reassure us it's working as designed. Soon I plan to
do some measurements with vacuuming large tables to get some concrete
numbers that the community can get excited about.
We also want to verify that progress reporting works as designed and has no
weird corner cases.
* autovacuum_work_mem) memory space to keep track of dead TIDs. We
initially
...
+ * create a TidStore with the maximum bytes that can be used by the
TidStore.
This kind of implies that we allocate the maximum bytes upfront. I think
this sentence can be removed. We already mentioned in the previous
paragraph that we set an upper bound.
- (errmsg("table \"%s\": removed %lld dead item identifiers in %u pages",
- vacrel->relname, (long long) index, vacuumed_pages)));
+ (errmsg("table \"%s\": removed " UINT64_FORMAT "dead item identifiers in
%u pages",
+ vacrel->relname, tidstore_num_tids(vacrel->dead_items),
+ vacuumed_pages)));
I don't think the format string has to change, since num_tids was changed
back to int64 in an earlier patch version?
- * the memory space for storing dead items allocated in the DSM segment.
We
[a lot of whitespace adjustment]
+ * the shared TidStore. We launch parallel worker processes at the start of
The old comment still seems mostly ok? Maybe just s/DSM segment/DSA area/
or something else minor.
- /* Estimate size for dead_items -- PARALLEL_VACUUM_KEY_DEAD_ITEMS */
- est_dead_items_len = vac_max_items_to_alloc_size(max_items);
- shm_toc_estimate_chunk(&pcxt->estimator, est_dead_items_len);
+ /* Estimate size for dead tuple DSA -- PARALLEL_VACUUM_KEY_DSA */
+ shm_toc_estimate_chunk(&pcxt->estimator, dsa_minsize);
If we're starting from the minimum, "estimate" doesn't really describe it
anymore? Maybe "Initial size"?
What does dsa_minimum_size() work out to in practice? 1MB?
Also, I think PARALLEL_VACUUM_KEY_DSA is left over from an earlier patch.
Lastly, on the radix tree:
I find extend, set, and set_extend hard to keep straight when studying the
code. Maybe EXTEND -> EXTEND_UP , SET_EXTEND -> EXTEND_DOWN ?
RT_ITER_UPDATE_KEY is unused, but I somehow didn't notice when turning it
into a template.
+ /*
+ * Set the node to the node iterator and update the iterator stack
+ * from this node.
+ */
+ RT_UPDATE_ITER_STACK(iter, child, level - 1);
+/*
+ * Update each node_iter for inner nodes in the iterator node stack.
+ */
+static void
+RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
These comments don't really help readers unfamiliar with the code. The
iteration coding in general needs clearer description.
In the test:
+ 4, /* RT_NODE_KIND_4 */
The small size was changed to 3 -- if this test needs to know the max size
for each kind (class?), I wonder why it didn't fail. Should it? Maybe we
need symbols for the various fanouts.
I also want to mention now that we better decide soon if we want to support
shrinking of nodes for v16, even if the tidstore never shrinks. We'll need
to do it at some point, but I'm not sure if doing it now would make more
work for future changes targeting highly concurrent workloads. If so, doing
it now would just be wasted work. On the other hand, someone might have a
use that needs deletion before someone else needs concurrency. Just in
case, I have a start of node-shrinking logic, but needs some work because
we need the (local pointer) parent to update to the new smaller node, just
like the growing case.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-02-16 16:44 ` Andres Freund <[email protected]>
2023-02-17 08:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
1 sibling, 1 reply; 78+ messages in thread
From: Andres Freund @ 2023-02-16 16:44 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
Hi,
On 2023-02-16 16:22:56 +0700, John Naylor wrote:
> On Thu, Feb 16, 2023 at 10:24 AM Masahiko Sawada <[email protected]>
> > Right. TidStore is implemented not only for heap, so loading
> > out-of-order TIDs might be important in the future.
>
> That's what I was probably thinking about some weeks ago, but I'm having a
> hard time imagining how it would come up, even for something like the
> conveyor-belt concept.
We really ought to replace the tid bitmap used for bitmap heap scans. The
hashtable we use is a pretty awful data structure for it. And that's not
filled in-order, for example.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 16:44 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Andres Freund <[email protected]>
@ 2023-02-17 08:00 ` John Naylor <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: John Naylor @ 2023-02-17 08:00 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Thu, Feb 16, 2023 at 11:44 PM Andres Freund <[email protected]> wrote:
>
> Hi,
>
> On 2023-02-16 16:22:56 +0700, John Naylor wrote:
> > On Thu, Feb 16, 2023 at 10:24 AM Masahiko Sawada <[email protected]>
> > > Right. TidStore is implemented not only for heap, so loading
> > > out-of-order TIDs might be important in the future.
> >
> > That's what I was probably thinking about some weeks ago, but I'm
having a
> > hard time imagining how it would come up, even for something like the
> > conveyor-belt concept.
>
> We really ought to replace the tid bitmap used for bitmap heap scans. The
> hashtable we use is a pretty awful data structure for it. And that's not
> filled in-order, for example.
I took a brief look at that and agree we should sometime make it work there
as well.
v26 tidstore_add_tids() appears to assume that it's only called once per
blocknumber. While the order of offsets doesn't matter there for a single
block, calling it again with the same block would wipe out the earlier
offsets, IIUC. To do an actual "add tid" where the order doesn't matter, it
seems we would need to (acquire lock if needed), read the current bitmap
and OR in the new bit if it exists, then write it back out.
That sounds slow, so it might still be good for vacuum to call a function
that passes a block and an array of offsets that are assumed ordered (as in
v28), but with a more accurate name, like tidstore_set_block_offsets().
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-02-20 05:56 ` Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
1 sibling, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-02-20 05:56 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Thu, Feb 16, 2023 at 6:23 PM John Naylor
<[email protected]> wrote:
>
> On Thu, Feb 16, 2023 at 10:24 AM Masahiko Sawada <[email protected]> wrote:
> >
> > On Tue, Feb 14, 2023 at 8:24 PM John Naylor
> > <[email protected]> wrote:
>
> > > > I can think that something like traversing a HOT chain could visit
> > > > offsets out of order. But fortunately we prune such collected TIDs
> > > > before heap vacuum in heap case.
> > >
> > > Further, currently we *already* assume we populate the tid array in order (for binary search), so we can just continue assuming that (with an assert added since it's more public in this form). I'm not sure why such basic common sense evaded me a few versions ago...
> >
> > Right. TidStore is implemented not only for heap, so loading
> > out-of-order TIDs might be important in the future.
>
> That's what I was probably thinking about some weeks ago, but I'm having a hard time imagining how it would come up, even for something like the conveyor-belt concept.
>
> > We have the following WIP comment in test_radixtree:
> >
> > // WIP: compiles with warnings because rt_attach is defined but not used
> > // #define RT_SHMEM
> >
> > How about unsetting RT_SCOPE to suppress warnings for unused rt_attach
> > and friends?
>
> Sounds good to me, and the other fixes make sense as well.
Thanks, I merged them.
>
> > FYI I've briefly tested the TidStore with blocksize = 32kb, and it
> > seems to work fine.
>
> That was on my list, so great! How about the other end -- nominally we allow 512b. (In practice it won't matter, but this would make sure I didn't mess anything up when forcing all MaxTuplesPerPage to encode.)
According to the doc, the minimum block size is 1kB. It seems to work
fine with 1kB blocks.
>
> > You removed the vacuum integration patch from v27, is there any reason for that?
>
> Just an oversight.
>
> Now for some general comments on the tid store...
>
> + * TODO: The caller must be certain that no other backend will attempt to
> + * access the TidStore before calling this function. Other backend must
> + * explicitly call tidstore_detach to free up backend-local memory associated
> + * with the TidStore. The backend that calls tidstore_destroy must not call
> + * tidstore_detach.
> + */
> +void
> +tidstore_destroy(TidStore *ts)
>
> Do we need to do anything for this todo?
Since it's practically no problem, I think we can live with it for
now. dshash also has the same todo.
>
> It might help readability to have a concept of "off_upper/off_lower", just so we can describe things more clearly. The key is block + off_upper, and the value is a bitmap of all the off_lower bits. I hinted at that in my addition of encode_key_off(). Along those lines, maybe s/TIDSTORE_OFFSET_MASK/TIDSTORE_OFFSET_LOWER_MASK/. Actually, I'm not even sure the TIDSTORE_ prefix is valuable for these local macros.
>
> The word "value" as a variable name is pretty generic in this context, and it might be better to call it the off_lower_bitmap, at least in some places. The "key" doesn't have a good short term for naming, but in comments we should make sure we're clear it's "block# + off_upper".
>
> I'm not a fan of the name "tid_i", even as a temp variable -- maybe "compressed_tid"?
>
> maybe s/tid_to_key_off/encode_tid/ and s/encode_key_off/encode_block_offset/
>
> It might be worth using typedefs for key and value type. Actually, since key type is fixed for the foreseeable future, maybe the radix tree template should define a key typedef?
>
> The term "result" is probably fine within the tidstore, but as a public name used by vacuum, it's not very descriptive. I don't have a good idea, though.
>
> Some files in backend/access use CamelCase for public functions, although it's not consistent. I think doing that for tidstore would help readability, since they would stand out from rt_* functions and vacuum functions. It's a matter of taste, though.
>
> I don't understand the control flow in tidstore_iterate_next(), or when BlockNumberIsValid() is true. If this is the best way to code this, it needs more commentary.
The attached 0008 patch addressed all above comments on tidstore.
> Some comments on vacuum:
>
> I think we'd better get some real-world testing of this, fairly soon.
>
> I had an idea: If it's not too much effort, it might be worth splitting it into two parts: one that just adds the store (not caring about its memory limits or progress reporting etc). During index scan, check both the new store and the array and log a warning (we don't want to exit or crash, better to try to investigate while live if possible) if the result doesn't match. Then perhaps set up an instance and let something like TPC-C run for a few days. The second patch would just restore the rest of the current patch. That would help reassure us it's working as designed.
Yeah, I did a similar thing in an earlier version of tidstore patch.
Since we're trying to introduce two new components: radix tree and
tidstore, I sometimes find it hard to investigate failures happening
during lazy (parallel) vacuum due to a bug either in tidstore or radix
tree. If there is a bug in lazy vacuum, we cannot even do initdb. So
it might be a good idea to do such checks in USE_ASSERT_CHECKING (or
with another macro say DEBUG_TIDSTORE) builds. For example, TidStore
stores tids to both the radix tree and array, and checks if the
results match when lookup or iteration. It will use more memory but it
would not be a big problem in USE_ASSERT_CHECKING builds. It would
also be great if we can enable such checks on some bf animals.
> Soon I plan to do some measurements with vacuuming large tables to get some concrete numbers that the community can get excited about.
Thanks!
>
> We also want to verify that progress reporting works as designed and has no weird corner cases.
>
> * autovacuum_work_mem) memory space to keep track of dead TIDs. We initially
> ...
> + * create a TidStore with the maximum bytes that can be used by the TidStore.
>
> This kind of implies that we allocate the maximum bytes upfront. I think this sentence can be removed. We already mentioned in the previous paragraph that we set an upper bound.
Agreed.
>
> - (errmsg("table \"%s\": removed %lld dead item identifiers in %u pages",
> - vacrel->relname, (long long) index, vacuumed_pages)));
> + (errmsg("table \"%s\": removed " UINT64_FORMAT "dead item identifiers in %u pages",
> + vacrel->relname, tidstore_num_tids(vacrel->dead_items),
> + vacuumed_pages)));
>
> I don't think the format string has to change, since num_tids was changed back to int64 in an earlier patch version?
I think we need to change the format to INT64_FORMAT.
>
> - * the memory space for storing dead items allocated in the DSM segment. We
> [a lot of whitespace adjustment]
> + * the shared TidStore. We launch parallel worker processes at the start of
>
> The old comment still seems mostly ok? Maybe just s/DSM segment/DSA area/ or something else minor.
>
> - /* Estimate size for dead_items -- PARALLEL_VACUUM_KEY_DEAD_ITEMS */
> - est_dead_items_len = vac_max_items_to_alloc_size(max_items);
> - shm_toc_estimate_chunk(&pcxt->estimator, est_dead_items_len);
> + /* Estimate size for dead tuple DSA -- PARALLEL_VACUUM_KEY_DSA */
> + shm_toc_estimate_chunk(&pcxt->estimator, dsa_minsize);
>
> If we're starting from the minimum, "estimate" doesn't really describe it anymore? Maybe "Initial size"?
> What does dsa_minimum_size() work out to in practice? 1MB?
> Also, I think PARALLEL_VACUUM_KEY_DSA is left over from an earlier patch.
>
Right. The attached 0009 patch addressed comments on vacuum
integration except for the correctness checking.
> Lastly, on the radix tree:
>
> I find extend, set, and set_extend hard to keep straight when studying the code. Maybe EXTEND -> EXTEND_UP , SET_EXTEND -> EXTEND_DOWN ?
>
> RT_ITER_UPDATE_KEY is unused, but I somehow didn't notice when turning it into a template.
It was used in radixtree_iter_impl.h. But I removed it as it was not necessary.
>
> + /*
> + * Set the node to the node iterator and update the iterator stack
> + * from this node.
> + */
> + RT_UPDATE_ITER_STACK(iter, child, level - 1);
>
> +/*
> + * Update each node_iter for inner nodes in the iterator node stack.
> + */
> +static void
> +RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
>
> These comments don't really help readers unfamiliar with the code. The iteration coding in general needs clearer description.
>
I agree with all of the above comments. The attached 0007 patch
addressed comments on the radix tree.
> In the test:
>
> + 4, /* RT_NODE_KIND_4 */
>
> The small size was changed to 3 -- if this test needs to know the max size for each kind (class?), I wonder why it didn't fail. Should it? Maybe we need symbols for the various fanouts.
>
Since this information is used to the number of keys inserted, it
doesn't check the node kind. So we just didn't test node-3. It might
be better to expose and use both RT_SIZE_CLASS and RT_SIZE_CLASS_INFO.
> I also want to mention now that we better decide soon if we want to support shrinking of nodes for v16, even if the tidstore never shrinks. We'll need to do it at some point, but I'm not sure if doing it now would make more work for future changes targeting highly concurrent workloads. If so, doing it now would just be wasted work. On the other hand, someone might have a use that needs deletion before someone else needs concurrency. Just in case, I have a start of node-shrinking logic, but needs some work because we need the (local pointer) parent to update to the new smaller node, just like the growing case.
Thanks, that's also on my todo list. TBH I'm not sure we should
improve the deletion at this stage as there is no use case of deletion
in the core. I'd prefer to focus on improving the quality of the
current radix tree and tidstore now, and I think we can support
node-shrinking once we are confident with the current implementation.
On Fri, Feb 17, 2023 at 5:00 PM John Naylor
<[email protected]> wrote:
>
>That sounds slow, so it might still be good for vacuum to call a function that passes a block and an array of offsets that are assumed ordered (as in v28), but with a more accurate name, like tidstore_set_block_offsets().
tidstore_set_block_offsets() sounds better. I used
TidStoreSetBlockOffsets() in the latest patch set.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] v29-0006-Use-TIDStore-for-storing-dead-tuple-TID-during-l.patch (48.1K, ../../CAD21AoAFiw2DS07nPhiS14eVXpnF-bBmjvab4y+Ct8igU2z_8A@mail.gmail.com/2-v29-0006-Use-TIDStore-for-storing-dead-tuple-TID-during-l.patch)
download | inline diff:
From b9883174cb69d87e6c9fdccb33ae29d5f084cd8e Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Tue, 7 Feb 2023 17:19:29 +0700
Subject: [PATCH v29 06/10] Use TIDStore for storing dead tuple TID during lazy
vacuum
Previously, we used an array of ItemPointerData to store dead tuple
TIDs, which was not space efficient and slow to lookup. Also, we had
the 1GB limit on its size.
Now we use TIDStore to store dead tuple TIDs. Since the TIDStore,
backed by the radix tree, incrementaly allocates the memory, we get
rid of the 1GB limit.
Since we are no longer able to exactly estimate the maximum number of
TIDs can be stored the pg_stat_progress_vacuum shows the progress
information based on the amount of memory in bytes. The column names
are also changed to max_dead_tuple_bytes and num_dead_tuple_bytes.
In addition, since the TIDStore use the radix tree internally, the
minimum amount of memory required by TIDStore is 1MB, the inital DSA
segment size. Due to that, we increase the minimum value of
maintenance_work_mem (also autovacuum_work_mem) from 1MB to 2MB.
XXX: needs to bump catalog version
---
doc/src/sgml/monitoring.sgml | 8 +-
src/backend/access/heap/vacuumlazy.c | 278 ++++++++-------------
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/vacuum.c | 78 +-----
src/backend/commands/vacuumparallel.c | 73 +++---
src/backend/postmaster/autovacuum.c | 6 +-
src/backend/storage/lmgr/lwlock.c | 2 +
src/backend/utils/misc/guc_tables.c | 2 +-
src/include/commands/progress.h | 4 +-
src/include/commands/vacuum.h | 25 +-
src/include/storage/lwlock.h | 1 +
src/test/regress/expected/cluster.out | 2 +-
src/test/regress/expected/create_index.out | 2 +-
src/test/regress/expected/rules.out | 4 +-
src/test/regress/sql/cluster.sql | 2 +-
src/test/regress/sql/create_index.sql | 2 +-
16 files changed, 177 insertions(+), 314 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index e28206e056..1d84e17705 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -7165,10 +7165,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>max_dead_tuples</structfield> <type>bigint</type>
+ <structfield>max_dead_tuple_bytes</structfield> <type>bigint</type>
</para>
<para>
- Number of dead tuples that we can store before needing to perform
+ Amount of dead tuple data that we can store before needing to perform
an index vacuum cycle, based on
<xref linkend="guc-maintenance-work-mem"/>.
</para></entry>
@@ -7176,10 +7176,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>num_dead_tuples</structfield> <type>bigint</type>
+ <structfield>num_dead_tuple_bytes</structfield> <type>bigint</type>
</para>
<para>
- Number of dead tuples collected since the last index vacuum cycle.
+ Amount of dead tuple data collected since the last index vacuum cycle.
</para></entry>
</row>
</tbody>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 8f14cf85f3..b4e40423a8 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3,18 +3,18 @@
* vacuumlazy.c
* Concurrent ("lazy") vacuuming.
*
- * The major space usage for vacuuming is storage for the array of dead TIDs
+ * The major space usage for vacuuming is TidStore, a storage for dead TIDs
* that are to be removed from indexes. We want to ensure we can vacuum even
* the very largest relations with finite memory space usage. To do that, we
- * set upper bounds on the number of TIDs we can keep track of at once.
+ * set upper bounds on the maximum memory that can be used for keeping track
+ * of dead TIDs at once.
*
* We are willing to use at most maintenance_work_mem (or perhaps
* autovacuum_work_mem) memory space to keep track of dead TIDs. We initially
- * allocate an array of TIDs of that size, with an upper limit that depends on
- * table size (this limit ensures we don't allocate a huge area uselessly for
- * vacuuming small tables). If the array threatens to overflow, we must call
- * lazy_vacuum to vacuum indexes (and to vacuum the pages that we've pruned).
- * This frees up the memory space dedicated to storing dead TIDs.
+ * create a TidStore with the maximum bytes that can be used by the TidStore.
+ * If the TidStore is full, we must call lazy_vacuum to vacuum indexes (and to
+ * vacuum the pages that we've pruned). This frees up the memory space dedicated
+ * to storing dead TIDs.
*
* In practice VACUUM will often complete its initial pass over the target
* heap relation without ever running out of space to store TIDs. This means
@@ -40,6 +40,7 @@
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
#include "access/multixact.h"
+#include "access/tidstore.h"
#include "access/transam.h"
#include "access/visibilitymap.h"
#include "access/xact.h"
@@ -188,7 +189,7 @@ typedef struct LVRelState
* lazy_vacuum_heap_rel, which marks the same LP_DEAD line pointers as
* LP_UNUSED during second heap pass.
*/
- VacDeadItems *dead_items; /* TIDs whose index tuples we'll delete */
+ TidStore *dead_items; /* TIDs whose index tuples we'll delete */
BlockNumber rel_pages; /* total number of pages */
BlockNumber scanned_pages; /* # pages examined (not skipped via VM) */
BlockNumber removed_pages; /* # pages removed by relation truncation */
@@ -220,11 +221,14 @@ typedef struct LVRelState
typedef struct LVPagePruneState
{
bool hastup; /* Page prevents rel truncation? */
- bool has_lpdead_items; /* includes existing LP_DEAD items */
+
+ /* collected offsets of LP_DEAD items including existing ones */
+ OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
+ int num_offsets;
/*
* State describes the proper VM bit states to set for the page following
- * pruning and freezing. all_visible implies !has_lpdead_items, but don't
+ * pruning and freezing. all_visible implies num_offsets == 0, but don't
* trust all_frozen result unless all_visible is also set to true.
*/
bool all_visible; /* Every item visible to all? */
@@ -259,8 +263,9 @@ static bool lazy_scan_noprune(LVRelState *vacrel, Buffer buf,
static void lazy_vacuum(LVRelState *vacrel);
static bool lazy_vacuum_all_indexes(LVRelState *vacrel);
static void lazy_vacuum_heap_rel(LVRelState *vacrel);
-static int lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
- Buffer buffer, int index, Buffer vmbuffer);
+static void lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
+ OffsetNumber *offsets, int num_offsets,
+ Buffer buffer, Buffer vmbuffer);
static bool lazy_check_wraparound_failsafe(LVRelState *vacrel);
static void lazy_cleanup_all_indexes(LVRelState *vacrel);
static IndexBulkDeleteResult *lazy_vacuum_one_index(Relation indrel,
@@ -487,11 +492,11 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
}
/*
- * Allocate dead_items array memory using dead_items_alloc. This handles
- * parallel VACUUM initialization as part of allocating shared memory
- * space used for dead_items. (But do a failsafe precheck first, to
- * ensure that parallel VACUUM won't be attempted at all when relfrozenxid
- * is already dangerously old.)
+ * Allocate dead_items memory using dead_items_alloc. This handles parallel
+ * VACUUM initialization as part of allocating shared memory space used for
+ * dead_items. (But do a failsafe precheck first, to ensure that parallel
+ * VACUUM won't be attempted at all when relfrozenxid is already dangerously
+ * old.)
*/
lazy_check_wraparound_failsafe(vacrel);
dead_items_alloc(vacrel, params->nworkers);
@@ -797,7 +802,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* have collected the TIDs whose index tuples need to be removed.
*
* Finally, invokes lazy_vacuum_heap_rel to vacuum heap pages, which
- * largely consists of marking LP_DEAD items (from collected TID array)
+ * largely consists of marking LP_DEAD items (from vacrel->dead_items)
* as LP_UNUSED. This has to happen in a second, final pass over the
* heap, to preserve a basic invariant that all index AMs rely on: no
* extant index tuple can ever be allowed to contain a TID that points to
@@ -825,21 +830,21 @@ lazy_scan_heap(LVRelState *vacrel)
blkno,
next_unskippable_block,
next_fsm_block_to_vacuum = 0;
- VacDeadItems *dead_items = vacrel->dead_items;
+ TidStore *dead_items = vacrel->dead_items;
Buffer vmbuffer = InvalidBuffer;
bool next_unskippable_allvis,
skipping_current_range;
const int initprog_index[] = {
PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_TOTAL_HEAP_BLKS,
- PROGRESS_VACUUM_MAX_DEAD_TUPLES
+ PROGRESS_VACUUM_MAX_DEAD_TUPLE_BYTES
};
int64 initprog_val[3];
/* Report that we're scanning the heap, advertising total # of blocks */
initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
initprog_val[1] = rel_pages;
- initprog_val[2] = dead_items->max_items;
+ initprog_val[2] = tidstore_max_memory(vacrel->dead_items);
pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
/* Set up an initial range of skippable blocks using the visibility map */
@@ -906,8 +911,7 @@ lazy_scan_heap(LVRelState *vacrel)
* dead_items TIDs, pause and do a cycle of vacuuming before we tackle
* this page.
*/
- Assert(dead_items->max_items >= MaxHeapTuplesPerPage);
- if (dead_items->max_items - dead_items->num_items < MaxHeapTuplesPerPage)
+ if (tidstore_is_full(vacrel->dead_items))
{
/*
* Before beginning index vacuuming, we release any pin we may
@@ -969,7 +973,7 @@ lazy_scan_heap(LVRelState *vacrel)
continue;
}
- /* Collect LP_DEAD items in dead_items array, count tuples */
+ /* Collect LP_DEAD items in dead_items, count tuples */
if (lazy_scan_noprune(vacrel, buf, blkno, page, &hastup,
&recordfreespace))
{
@@ -1011,14 +1015,14 @@ lazy_scan_heap(LVRelState *vacrel)
* Prune, freeze, and count tuples.
*
* Accumulates details of remaining LP_DEAD line pointers on page in
- * dead_items array. This includes LP_DEAD line pointers that we
- * pruned ourselves, as well as existing LP_DEAD line pointers that
- * were pruned some time earlier. Also considers freezing XIDs in the
- * tuple headers of remaining items with storage.
+ * dead_items. This includes LP_DEAD line pointers that we pruned
+ * ourselves, as well as existing LP_DEAD line pointers that were pruned
+ * some time earlier. Also considers freezing XIDs in the tuple headers
+ * of remaining items with storage.
*/
lazy_scan_prune(vacrel, buf, blkno, page, &prunestate);
- Assert(!prunestate.all_visible || !prunestate.has_lpdead_items);
+ Assert(!prunestate.all_visible || (prunestate.num_offsets == 0));
/* Remember the location of the last page with nonremovable tuples */
if (prunestate.hastup)
@@ -1034,14 +1038,12 @@ lazy_scan_heap(LVRelState *vacrel)
* performed here can be thought of as the one-pass equivalent of
* a call to lazy_vacuum().
*/
- if (prunestate.has_lpdead_items)
+ if (prunestate.num_offsets > 0)
{
Size freespace;
- lazy_vacuum_heap_page(vacrel, blkno, buf, 0, vmbuffer);
-
- /* Forget the LP_DEAD items that we just vacuumed */
- dead_items->num_items = 0;
+ lazy_vacuum_heap_page(vacrel, blkno, prunestate.deadoffsets,
+ prunestate.num_offsets, buf, vmbuffer);
/*
* Periodically perform FSM vacuuming to make newly-freed
@@ -1078,7 +1080,16 @@ lazy_scan_heap(LVRelState *vacrel)
* with prunestate-driven visibility map and FSM steps (just like
* the two-pass strategy).
*/
- Assert(dead_items->num_items == 0);
+ Assert(tidstore_num_tids(dead_items) == 0);
+ }
+ else if (prunestate.num_offsets > 0)
+ {
+ /* Save details of the LP_DEAD items from the page in dead_items */
+ tidstore_add_tids(dead_items, blkno, prunestate.deadoffsets,
+ prunestate.num_offsets);
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
+ tidstore_memory_usage(dead_items));
}
/*
@@ -1145,7 +1156,7 @@ lazy_scan_heap(LVRelState *vacrel)
* There should never be LP_DEAD items on a page with PD_ALL_VISIBLE
* set, however.
*/
- else if (prunestate.has_lpdead_items && PageIsAllVisible(page))
+ else if ((prunestate.num_offsets > 0) && PageIsAllVisible(page))
{
elog(WARNING, "page containing LP_DEAD items is marked as all-visible in relation \"%s\" page %u",
vacrel->relname, blkno);
@@ -1193,7 +1204,7 @@ lazy_scan_heap(LVRelState *vacrel)
* Final steps for block: drop cleanup lock, record free space in the
* FSM
*/
- if (prunestate.has_lpdead_items && vacrel->do_index_vacuuming)
+ if ((prunestate.num_offsets > 0) && vacrel->do_index_vacuuming)
{
/*
* Wait until lazy_vacuum_heap_rel() to save free space. This
@@ -1249,7 +1260,7 @@ lazy_scan_heap(LVRelState *vacrel)
* Do index vacuuming (call each index's ambulkdelete routine), then do
* related heap vacuuming
*/
- if (dead_items->num_items > 0)
+ if (tidstore_num_tids(dead_items) > 0)
lazy_vacuum(vacrel);
/*
@@ -1524,9 +1535,9 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* The approach we take now is to restart pruning when the race condition is
* detected. This allows heap_page_prune() to prune the tuples inserted by
* the now-aborted transaction. This is a little crude, but it guarantees
- * that any items that make it into the dead_items array are simple LP_DEAD
- * line pointers, and that every remaining item with tuple storage is
- * considered as a candidate for freezing.
+ * that any items that make it into the dead_items are simple LP_DEAD line
+ * pointers, and that every remaining item with tuple storage is considered
+ * as a candidate for freezing.
*/
static void
lazy_scan_prune(LVRelState *vacrel,
@@ -1543,13 +1554,11 @@ lazy_scan_prune(LVRelState *vacrel,
HTSV_Result res;
int tuples_deleted,
tuples_frozen,
- lpdead_items,
live_tuples,
recently_dead_tuples;
int nnewlpdead;
HeapPageFreeze pagefrz;
int64 fpi_before = pgWalUsage.wal_fpi;
- OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
@@ -1571,7 +1580,6 @@ retry:
pagefrz.NoFreezePageRelminMxid = vacrel->NewRelminMxid;
tuples_deleted = 0;
tuples_frozen = 0;
- lpdead_items = 0;
live_tuples = 0;
recently_dead_tuples = 0;
@@ -1580,9 +1588,9 @@ retry:
*
* We count tuples removed by the pruning step as tuples_deleted. Its
* final value can be thought of as the number of tuples that have been
- * deleted from the table. It should not be confused with lpdead_items;
- * lpdead_items's final value can be thought of as the number of tuples
- * that were deleted from indexes.
+ * deleted from the table. It should not be confused with
+ * prunestate->deadoffsets; prunestate->deadoffsets's final value can
+ * be thought of as the number of tuples that were deleted from indexes.
*/
tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest,
InvalidTransactionId, 0, &nnewlpdead,
@@ -1593,7 +1601,7 @@ retry:
* requiring freezing among remaining tuples with storage
*/
prunestate->hastup = false;
- prunestate->has_lpdead_items = false;
+ prunestate->num_offsets = 0;
prunestate->all_visible = true;
prunestate->all_frozen = true;
prunestate->visibility_cutoff_xid = InvalidTransactionId;
@@ -1638,7 +1646,7 @@ retry:
* (This is another case where it's useful to anticipate that any
* LP_DEAD items will become LP_UNUSED during the ongoing VACUUM.)
*/
- deadoffsets[lpdead_items++] = offnum;
+ prunestate->deadoffsets[prunestate->num_offsets++] = offnum;
continue;
}
@@ -1875,7 +1883,7 @@ retry:
*/
#ifdef USE_ASSERT_CHECKING
/* Note that all_frozen value does not matter when !all_visible */
- if (prunestate->all_visible && lpdead_items == 0)
+ if (prunestate->all_visible && prunestate->num_offsets == 0)
{
TransactionId cutoff;
bool all_frozen;
@@ -1888,28 +1896,9 @@ retry:
}
#endif
- /*
- * Now save details of the LP_DEAD items from the page in vacrel
- */
- if (lpdead_items > 0)
+ if (prunestate->num_offsets > 0)
{
- VacDeadItems *dead_items = vacrel->dead_items;
- ItemPointerData tmp;
-
vacrel->lpdead_item_pages++;
- prunestate->has_lpdead_items = true;
-
- ItemPointerSetBlockNumber(&tmp, blkno);
-
- for (int i = 0; i < lpdead_items; i++)
- {
- ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
- dead_items->items[dead_items->num_items++] = tmp;
- }
-
- Assert(dead_items->num_items <= dead_items->max_items);
- pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
- dead_items->num_items);
/*
* It was convenient to ignore LP_DEAD items in all_visible earlier on
@@ -1928,7 +1917,7 @@ retry:
/* Finally, add page-local counts to whole-VACUUM counts */
vacrel->tuples_deleted += tuples_deleted;
vacrel->tuples_frozen += tuples_frozen;
- vacrel->lpdead_items += lpdead_items;
+ vacrel->lpdead_items += prunestate->num_offsets;
vacrel->live_tuples += live_tuples;
vacrel->recently_dead_tuples += recently_dead_tuples;
}
@@ -1940,7 +1929,7 @@ retry:
* lazy_scan_prune, which requires a full cleanup lock. While pruning isn't
* performed here, it's quite possible that an earlier opportunistic pruning
* operation left LP_DEAD items behind. We'll at least collect any such items
- * in the dead_items array for removal from indexes.
+ * in the dead_items for removal from indexes.
*
* For aggressive VACUUM callers, we may return false to indicate that a full
* cleanup lock is required for processing by lazy_scan_prune. This is only
@@ -2099,7 +2088,7 @@ lazy_scan_noprune(LVRelState *vacrel,
vacrel->NewRelfrozenXid = NoFreezePageRelfrozenXid;
vacrel->NewRelminMxid = NoFreezePageRelminMxid;
- /* Save any LP_DEAD items found on the page in dead_items array */
+ /* Save any LP_DEAD items found on the page in dead_items */
if (vacrel->nindexes == 0)
{
/* Using one-pass strategy (since table has no indexes) */
@@ -2129,8 +2118,7 @@ lazy_scan_noprune(LVRelState *vacrel,
}
else
{
- VacDeadItems *dead_items = vacrel->dead_items;
- ItemPointerData tmp;
+ TidStore *dead_items = vacrel->dead_items;
/*
* Page has LP_DEAD items, and so any references/TIDs that remain in
@@ -2139,17 +2127,10 @@ lazy_scan_noprune(LVRelState *vacrel,
*/
vacrel->lpdead_item_pages++;
- ItemPointerSetBlockNumber(&tmp, blkno);
+ tidstore_add_tids(dead_items, blkno, deadoffsets, lpdead_items);
- for (int i = 0; i < lpdead_items; i++)
- {
- ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
- dead_items->items[dead_items->num_items++] = tmp;
- }
-
- Assert(dead_items->num_items <= dead_items->max_items);
- pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
- dead_items->num_items);
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
+ tidstore_memory_usage(dead_items));
vacrel->lpdead_items += lpdead_items;
@@ -2198,7 +2179,7 @@ lazy_vacuum(LVRelState *vacrel)
if (!vacrel->do_index_vacuuming)
{
Assert(!vacrel->do_index_cleanup);
- vacrel->dead_items->num_items = 0;
+ tidstore_reset(vacrel->dead_items);
return;
}
@@ -2227,7 +2208,7 @@ lazy_vacuum(LVRelState *vacrel)
BlockNumber threshold;
Assert(vacrel->num_index_scans == 0);
- Assert(vacrel->lpdead_items == vacrel->dead_items->num_items);
+ Assert(vacrel->lpdead_items == tidstore_num_tids(vacrel->dead_items));
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2254,8 +2235,8 @@ lazy_vacuum(LVRelState *vacrel)
* cases then this may need to be reconsidered.
*/
threshold = (double) vacrel->rel_pages * BYPASS_THRESHOLD_PAGES;
- bypass = (vacrel->lpdead_item_pages < threshold &&
- vacrel->lpdead_items < MAXDEADITEMS(32L * 1024L * 1024L));
+ bypass = (vacrel->lpdead_item_pages < threshold) &&
+ tidstore_memory_usage(vacrel->dead_items) < (32L * 1024L * 1024L);
}
if (bypass)
@@ -2300,7 +2281,7 @@ lazy_vacuum(LVRelState *vacrel)
* Forget the LP_DEAD items that we just vacuumed (or just decided to not
* vacuum)
*/
- vacrel->dead_items->num_items = 0;
+ tidstore_reset(vacrel->dead_items);
}
/*
@@ -2373,7 +2354,7 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
* place).
*/
Assert(vacrel->num_index_scans > 0 ||
- vacrel->dead_items->num_items == vacrel->lpdead_items);
+ tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items);
Assert(allindexes || vacrel->failsafe_active);
/*
@@ -2392,9 +2373,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
/*
* lazy_vacuum_heap_rel() -- second pass over the heap for two pass strategy
*
- * This routine marks LP_DEAD items in vacrel->dead_items array as LP_UNUSED.
- * Pages that never had lazy_scan_prune record LP_DEAD items are not visited
- * at all.
+ * This routine marks LP_DEAD items in vacrel->dead_items as LP_UNUSED. Pages
+ * that never had lazy_scan_prune record LP_DEAD items are not visited at all.
*
* We may also be able to truncate the line pointer array of the heap pages we
* visit. If there is a contiguous group of LP_UNUSED items at the end of the
@@ -2410,10 +2390,11 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
static void
lazy_vacuum_heap_rel(LVRelState *vacrel)
{
- int index = 0;
BlockNumber vacuumed_pages = 0;
Buffer vmbuffer = InvalidBuffer;
LVSavedErrInfo saved_err_info;
+ TidStoreIter *iter;
+ TidStoreIterResult *result;
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2428,7 +2409,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
VACUUM_ERRCB_PHASE_VACUUM_HEAP,
InvalidBlockNumber, InvalidOffsetNumber);
- while (index < vacrel->dead_items->num_items)
+ iter = tidstore_begin_iterate(vacrel->dead_items);
+ while ((result = tidstore_iterate_next(iter)) != NULL)
{
BlockNumber blkno;
Buffer buf;
@@ -2437,7 +2419,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
vacuum_delay_point();
- blkno = ItemPointerGetBlockNumber(&vacrel->dead_items->items[index]);
+ blkno = result->blkno;
vacrel->blkno = blkno;
/*
@@ -2451,7 +2433,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
buf = ReadBufferExtended(vacrel->rel, MAIN_FORKNUM, blkno, RBM_NORMAL,
vacrel->bstrategy);
LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
- index = lazy_vacuum_heap_page(vacrel, blkno, buf, index, vmbuffer);
+ lazy_vacuum_heap_page(vacrel, blkno, result->offsets, result->num_offsets,
+ buf, vmbuffer);
/* Now that we've vacuumed the page, record its available space */
page = BufferGetPage(buf);
@@ -2461,6 +2444,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
vacuumed_pages++;
}
+ tidstore_end_iterate(iter);
vacrel->blkno = InvalidBlockNumber;
if (BufferIsValid(vmbuffer))
@@ -2470,36 +2454,31 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
* We set all LP_DEAD items from the first heap pass to LP_UNUSED during
* the second heap pass. No more, no less.
*/
- Assert(index > 0);
Assert(vacrel->num_index_scans > 1 ||
- (index == vacrel->lpdead_items &&
+ (tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items &&
vacuumed_pages == vacrel->lpdead_item_pages));
ereport(DEBUG2,
- (errmsg("table \"%s\": removed %lld dead item identifiers in %u pages",
- vacrel->relname, (long long) index, vacuumed_pages)));
+ (errmsg("table \"%s\": removed " UINT64_FORMAT "dead item identifiers in %u pages",
+ vacrel->relname, tidstore_num_tids(vacrel->dead_items),
+ vacuumed_pages)));
/* Revert to the previous phase information for error traceback */
restore_vacuum_error_info(vacrel, &saved_err_info);
}
/*
- * lazy_vacuum_heap_page() -- free page's LP_DEAD items listed in the
- * vacrel->dead_items array.
+ * lazy_vacuum_heap_page() -- free page's LP_DEAD items.
*
* Caller must have an exclusive buffer lock on the buffer (though a full
* cleanup lock is also acceptable). vmbuffer must be valid and already have
* a pin on blkno's visibility map page.
- *
- * index is an offset into the vacrel->dead_items array for the first listed
- * LP_DEAD item on the page. The return value is the first index immediately
- * after all LP_DEAD items for the same page in the array.
*/
-static int
-lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
- int index, Buffer vmbuffer)
+static void
+lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
+ OffsetNumber *deadoffsets, int num_offsets, Buffer buffer,
+ Buffer vmbuffer)
{
- VacDeadItems *dead_items = vacrel->dead_items;
Page page = BufferGetPage(buffer);
OffsetNumber unused[MaxHeapTuplesPerPage];
int nunused = 0;
@@ -2518,16 +2497,11 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
START_CRIT_SECTION();
- for (; index < dead_items->num_items; index++)
+ for (int i = 0; i < num_offsets; i++)
{
- BlockNumber tblk;
- OffsetNumber toff;
ItemId itemid;
+ OffsetNumber toff = deadoffsets[i];
- tblk = ItemPointerGetBlockNumber(&dead_items->items[index]);
- if (tblk != blkno)
- break; /* past end of tuples for this block */
- toff = ItemPointerGetOffsetNumber(&dead_items->items[index]);
itemid = PageGetItemId(page, toff);
Assert(ItemIdIsDead(itemid) && !ItemIdHasStorage(itemid));
@@ -2597,7 +2571,6 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
/* Revert to the previous phase information for error traceback */
restore_vacuum_error_info(vacrel, &saved_err_info);
- return index;
}
/*
@@ -2687,8 +2660,8 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
* lazy_vacuum_one_index() -- vacuum index relation.
*
* Delete all the index tuples containing a TID collected in
- * vacrel->dead_items array. Also update running statistics.
- * Exact details depend on index AM's ambulkdelete routine.
+ * vacrel->dead_items. Also update running statistics. Exact
+ * details depend on index AM's ambulkdelete routine.
*
* reltuples is the number of heap tuples to be passed to the
* bulkdelete callback. It's always assumed to be estimated.
@@ -3094,48 +3067,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
}
/*
- * Returns the number of dead TIDs that VACUUM should allocate space to
- * store, given a heap rel of size vacrel->rel_pages, and given current
- * maintenance_work_mem setting (or current autovacuum_work_mem setting,
- * when applicable).
- *
- * See the comments at the head of this file for rationale.
- */
-static int
-dead_items_max_items(LVRelState *vacrel)
-{
- int64 max_items;
- int vac_work_mem = IsAutoVacuumWorkerProcess() &&
- autovacuum_work_mem != -1 ?
- autovacuum_work_mem : maintenance_work_mem;
-
- if (vacrel->nindexes > 0)
- {
- BlockNumber rel_pages = vacrel->rel_pages;
-
- max_items = MAXDEADITEMS(vac_work_mem * 1024L);
- max_items = Min(max_items, INT_MAX);
- max_items = Min(max_items, MAXDEADITEMS(MaxAllocSize));
-
- /* curious coding here to ensure the multiplication can't overflow */
- if ((BlockNumber) (max_items / MaxHeapTuplesPerPage) > rel_pages)
- max_items = rel_pages * MaxHeapTuplesPerPage;
-
- /* stay sane if small maintenance_work_mem */
- max_items = Max(max_items, MaxHeapTuplesPerPage);
- }
- else
- {
- /* One-pass case only stores a single heap page's TIDs at a time */
- max_items = MaxHeapTuplesPerPage;
- }
-
- return (int) max_items;
-}
-
-/*
- * Allocate dead_items (either using palloc, or in dynamic shared memory).
- * Sets dead_items in vacrel for caller.
+ * Allocate a (local or shared) TidStore for storing dead TIDs. Sets dead_items
+ * in vacrel for caller.
*
* Also handles parallel initialization as part of allocating dead_items in
* DSM when required.
@@ -3143,11 +3076,9 @@ dead_items_max_items(LVRelState *vacrel)
static void
dead_items_alloc(LVRelState *vacrel, int nworkers)
{
- VacDeadItems *dead_items;
- int max_items;
-
- max_items = dead_items_max_items(vacrel);
- Assert(max_items >= MaxHeapTuplesPerPage);
+ int vac_work_mem = IsAutoVacuumWorkerProcess() &&
+ autovacuum_work_mem != -1 ?
+ autovacuum_work_mem * 1024L : maintenance_work_mem * 1024L;
/*
* Initialize state for a parallel vacuum. As of now, only one worker can
@@ -3174,7 +3105,7 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
else
vacrel->pvs = parallel_vacuum_init(vacrel->rel, vacrel->indrels,
vacrel->nindexes, nworkers,
- max_items,
+ vac_work_mem, MaxHeapTuplesPerPage,
vacrel->verbose ? INFO : DEBUG2,
vacrel->bstrategy);
@@ -3187,11 +3118,8 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
}
/* Serial VACUUM case */
- dead_items = (VacDeadItems *) palloc(vac_max_items_to_alloc_size(max_items));
- dead_items->max_items = max_items;
- dead_items->num_items = 0;
-
- vacrel->dead_items = dead_items;
+ vacrel->dead_items = tidstore_create(vac_work_mem, MaxHeapTuplesPerPage,
+ NULL);
}
/*
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 34ca0e739f..149d41b41c 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1180,7 +1180,7 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuple_bytes, S.param7 AS dead_tuple_bytes
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index aa79d9de4d..d8e680ca20 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -97,7 +97,6 @@ static bool vacuum_rel(Oid relid, RangeVar *relation, VacuumParams *params,
static double compute_parallel_delay(void);
static VacOptValue get_vacoptval_from_boolean(DefElem *def);
static bool vac_tid_reaped(ItemPointer itemptr, void *state);
-static int vac_cmp_itemptr(const void *left, const void *right);
/*
* Primary entry point for manual VACUUM and ANALYZE commands
@@ -2303,16 +2302,16 @@ get_vacoptval_from_boolean(DefElem *def)
*/
IndexBulkDeleteResult *
vac_bulkdel_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat,
- VacDeadItems *dead_items)
+ TidStore *dead_items)
{
/* Do bulk deletion */
istat = index_bulk_delete(ivinfo, istat, vac_tid_reaped,
(void *) dead_items);
ereport(ivinfo->message_level,
- (errmsg("scanned index \"%s\" to remove %d row versions",
+ (errmsg("scanned index \"%s\" to remove " UINT64_FORMAT " row versions",
RelationGetRelationName(ivinfo->index),
- dead_items->num_items)));
+ tidstore_num_tids(dead_items))));
return istat;
}
@@ -2343,82 +2342,15 @@ vac_cleanup_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat)
return istat;
}
-/*
- * Returns the total required space for VACUUM's dead_items array given a
- * max_items value.
- */
-Size
-vac_max_items_to_alloc_size(int max_items)
-{
- Assert(max_items <= MAXDEADITEMS(MaxAllocSize));
-
- return offsetof(VacDeadItems, items) + sizeof(ItemPointerData) * max_items;
-}
-
/*
* vac_tid_reaped() -- is a particular tid deletable?
*
* This has the right signature to be an IndexBulkDeleteCallback.
- *
- * Assumes dead_items array is sorted (in ascending TID order).
*/
static bool
vac_tid_reaped(ItemPointer itemptr, void *state)
{
- VacDeadItems *dead_items = (VacDeadItems *) state;
- int64 litem,
- ritem,
- item;
- ItemPointer res;
-
- litem = itemptr_encode(&dead_items->items[0]);
- ritem = itemptr_encode(&dead_items->items[dead_items->num_items - 1]);
- item = itemptr_encode(itemptr);
-
- /*
- * Doing a simple bound check before bsearch() is useful to avoid the
- * extra cost of bsearch(), especially if dead items on the heap are
- * concentrated in a certain range. Since this function is called for
- * every index tuple, it pays to be really fast.
- */
- if (item < litem || item > ritem)
- return false;
-
- res = (ItemPointer) bsearch(itemptr,
- dead_items->items,
- dead_items->num_items,
- sizeof(ItemPointerData),
- vac_cmp_itemptr);
-
- return (res != NULL);
-}
-
-/*
- * Comparator routines for use with qsort() and bsearch().
- */
-static int
-vac_cmp_itemptr(const void *left, const void *right)
-{
- BlockNumber lblk,
- rblk;
- OffsetNumber loff,
- roff;
-
- lblk = ItemPointerGetBlockNumber((ItemPointer) left);
- rblk = ItemPointerGetBlockNumber((ItemPointer) right);
-
- if (lblk < rblk)
- return -1;
- if (lblk > rblk)
- return 1;
-
- loff = ItemPointerGetOffsetNumber((ItemPointer) left);
- roff = ItemPointerGetOffsetNumber((ItemPointer) right);
-
- if (loff < roff)
- return -1;
- if (loff > roff)
- return 1;
+ TidStore *dead_items = (TidStore *) state;
- return 0;
+ return tidstore_lookup_tid(dead_items, itemptr);
}
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index bcd40c80a1..d653683693 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -9,12 +9,11 @@
* In a parallel vacuum, we perform both index bulk deletion and index cleanup
* with parallel worker processes. Individual indexes are processed by one
* vacuum process. ParalleVacuumState contains shared information as well as
- * the memory space for storing dead items allocated in the DSM segment. We
- * launch parallel worker processes at the start of parallel index
- * bulk-deletion and index cleanup and once all indexes are processed, the
- * parallel worker processes exit. Each time we process indexes in parallel,
- * the parallel context is re-initialized so that the same DSM can be used for
- * multiple passes of index bulk-deletion and index cleanup.
+ * the shared TidStore. We launch parallel worker processes at the start of
+ * parallel index bulk-deletion and index cleanup and once all indexes are
+ * processed, the parallel worker processes exit. Each time we process indexes
+ * in parallel, the parallel context is re-initialized so that the same DSM can
+ * be used for multiple passes of index bulk-deletion and index cleanup.
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -103,6 +102,9 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /* Handle of the shared TidStore */
+ tidstore_handle dead_items_handle;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -166,7 +168,8 @@ struct ParallelVacuumState
PVIndStats *indstats;
/* Shared dead items space among parallel vacuum workers */
- VacDeadItems *dead_items;
+ TidStore *dead_items;
+ dsa_area *dead_items_area;
/* Points to buffer usage area in DSM */
BufferUsage *buffer_usage;
@@ -222,20 +225,23 @@ static void parallel_vacuum_error_callback(void *arg);
*/
ParallelVacuumState *
parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
- int nrequested_workers, int max_items,
- int elevel, BufferAccessStrategy bstrategy)
+ int nrequested_workers, int vac_work_mem,
+ int max_offset, int elevel,
+ BufferAccessStrategy bstrategy)
{
ParallelVacuumState *pvs;
ParallelContext *pcxt;
PVShared *shared;
- VacDeadItems *dead_items;
+ TidStore *dead_items;
PVIndStats *indstats;
BufferUsage *buffer_usage;
WalUsage *wal_usage;
+ void *area_space;
+ dsa_area *dead_items_dsa;
bool *will_parallel_vacuum;
Size est_indstats_len;
Size est_shared_len;
- Size est_dead_items_len;
+ Size dsa_minsize = dsa_minimum_size();
int nindexes_mwm = 0;
int parallel_workers = 0;
int querylen;
@@ -283,9 +289,8 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_estimate_chunk(&pcxt->estimator, est_shared_len);
shm_toc_estimate_keys(&pcxt->estimator, 1);
- /* Estimate size for dead_items -- PARALLEL_VACUUM_KEY_DEAD_ITEMS */
- est_dead_items_len = vac_max_items_to_alloc_size(max_items);
- shm_toc_estimate_chunk(&pcxt->estimator, est_dead_items_len);
+ /* Estimate size for dead tuple DSA -- PARALLEL_VACUUM_KEY_DSA */
+ shm_toc_estimate_chunk(&pcxt->estimator, dsa_minsize);
shm_toc_estimate_keys(&pcxt->estimator, 1);
/*
@@ -351,6 +356,16 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_INDEX_STATS, indstats);
pvs->indstats = indstats;
+ /* Prepare DSA space for dead items */
+ area_space = shm_toc_allocate(pcxt->toc, dsa_minsize);
+ shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, area_space);
+ dead_items_dsa = dsa_create_in_place(area_space, dsa_minsize,
+ LWTRANCHE_PARALLEL_VACUUM_DSA,
+ pcxt->seg);
+ dead_items = tidstore_create(vac_work_mem, max_offset, dead_items_dsa);
+ pvs->dead_items = dead_items;
+ pvs->dead_items_area = dead_items_dsa;
+
/* Prepare shared information */
shared = (PVShared *) shm_toc_allocate(pcxt->toc, est_shared_len);
MemSet(shared, 0, est_shared_len);
@@ -360,6 +375,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
+ shared->dead_items_handle = tidstore_get_handle(dead_items);
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
@@ -368,15 +384,6 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_SHARED, shared);
pvs->shared = shared;
- /* Prepare the dead_items space */
- dead_items = (VacDeadItems *) shm_toc_allocate(pcxt->toc,
- est_dead_items_len);
- dead_items->max_items = max_items;
- dead_items->num_items = 0;
- MemSet(dead_items->items, 0, sizeof(ItemPointerData) * max_items);
- shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, dead_items);
- pvs->dead_items = dead_items;
-
/*
* Allocate space for each worker's BufferUsage and WalUsage; no need to
* initialize
@@ -434,6 +441,9 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
istats[i] = NULL;
}
+ tidstore_destroy(pvs->dead_items);
+ dsa_detach(pvs->dead_items_area);
+
DestroyParallelContext(pvs->pcxt);
ExitParallelMode();
@@ -442,7 +452,7 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
}
/* Returns the dead items space */
-VacDeadItems *
+TidStore *
parallel_vacuum_get_dead_items(ParallelVacuumState *pvs)
{
return pvs->dead_items;
@@ -940,7 +950,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
Relation *indrels;
PVIndStats *indstats;
PVShared *shared;
- VacDeadItems *dead_items;
+ TidStore *dead_items;
+ void *area_space;
+ dsa_area *dead_items_area;
BufferUsage *buffer_usage;
WalUsage *wal_usage;
int nindexes;
@@ -984,10 +996,10 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
PARALLEL_VACUUM_KEY_INDEX_STATS,
false);
- /* Set dead_items space */
- dead_items = (VacDeadItems *) shm_toc_lookup(toc,
- PARALLEL_VACUUM_KEY_DEAD_ITEMS,
- false);
+ /* Set dead items */
+ area_space = shm_toc_lookup(toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, false);
+ dead_items_area = dsa_attach_in_place(area_space, seg);
+ dead_items = tidstore_attach(dead_items_area, shared->dead_items_handle);
/* Set cost-based vacuum delay */
VacuumCostActive = (VacuumCostDelay > 0);
@@ -1033,6 +1045,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
InstrEndParallelQuery(&buffer_usage[ParallelWorkerNumber],
&wal_usage[ParallelWorkerNumber]);
+ tidstore_detach(pvs.dead_items);
+ dsa_detach(dead_items_area);
+
/* Pop the error context stack */
error_context_stack = errcallback.previous;
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index ff6149a179..a371f6fbba 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -3397,12 +3397,12 @@ check_autovacuum_work_mem(int *newval, void **extra, GucSource source)
return true;
/*
- * We clamp manually-set values to at least 1MB. Since
+ * We clamp manually-set values to at least 2MB. Since
* maintenance_work_mem is always set to at least this value, do the same
* here.
*/
- if (*newval < 1024)
- *newval = 1024;
+ if (*newval < 2048)
+ *newval = 2048;
return true;
}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 55b3a04097..c223a7dc94 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -192,6 +192,8 @@ static const char *const BuiltinTrancheNames[] = {
"LogicalRepLauncherDSA",
/* LWTRANCHE_LAUNCHER_HASH: */
"LogicalRepLauncherHash",
+ /* LWTRANCHE_PARALLEL_VACUUM_DSA: */
+ "ParallelVacuumDSA",
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index b46e3b8c55..27a88b9369 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2312,7 +2312,7 @@ struct config_int ConfigureNamesInt[] =
GUC_UNIT_KB
},
&maintenance_work_mem,
- 65536, 1024, MAX_KILOBYTES,
+ 65536, 2048, MAX_KILOBYTES,
NULL, NULL, NULL
},
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index e5add41352..b209d3cf84 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -23,8 +23,8 @@
#define PROGRESS_VACUUM_HEAP_BLKS_SCANNED 2
#define PROGRESS_VACUUM_HEAP_BLKS_VACUUMED 3
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
-#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
-#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_MAX_DEAD_TUPLE_BYTES 5
+#define PROGRESS_VACUUM_DEAD_TUPLE_BYTES 6
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 689dbb7702..a3ebb169ef 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -17,6 +17,7 @@
#include "access/htup.h"
#include "access/genam.h"
#include "access/parallel.h"
+#include "access/tidstore.h"
#include "catalog/pg_class.h"
#include "catalog/pg_statistic.h"
#include "catalog/pg_type.h"
@@ -276,21 +277,6 @@ struct VacuumCutoffs
MultiXactId MultiXactCutoff;
};
-/*
- * VacDeadItems stores TIDs whose index tuples are deleted by index vacuuming.
- */
-typedef struct VacDeadItems
-{
- int max_items; /* # slots allocated in array */
- int num_items; /* current # of entries */
-
- /* Sorted array of TIDs to delete from indexes */
- ItemPointerData items[FLEXIBLE_ARRAY_MEMBER];
-} VacDeadItems;
-
-#define MAXDEADITEMS(avail_mem) \
- (((avail_mem) - offsetof(VacDeadItems, items)) / sizeof(ItemPointerData))
-
/* GUC parameters */
extern PGDLLIMPORT int default_statistics_target; /* PGDLLIMPORT for PostGIS */
extern PGDLLIMPORT int vacuum_freeze_min_age;
@@ -339,18 +325,17 @@ extern Relation vacuum_open_relation(Oid relid, RangeVar *relation,
LOCKMODE lmode);
extern IndexBulkDeleteResult *vac_bulkdel_one_index(IndexVacuumInfo *ivinfo,
IndexBulkDeleteResult *istat,
- VacDeadItems *dead_items);
+ TidStore *dead_items);
extern IndexBulkDeleteResult *vac_cleanup_one_index(IndexVacuumInfo *ivinfo,
IndexBulkDeleteResult *istat);
-extern Size vac_max_items_to_alloc_size(int max_items);
/* in commands/vacuumparallel.c */
extern ParallelVacuumState *parallel_vacuum_init(Relation rel, Relation *indrels,
int nindexes, int nrequested_workers,
- int max_items, int elevel,
- BufferAccessStrategy bstrategy);
+ int vac_work_mem, int max_offset,
+ int elevel, BufferAccessStrategy bstrategy);
extern void parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats);
-extern VacDeadItems *parallel_vacuum_get_dead_items(ParallelVacuumState *pvs);
+extern TidStore *parallel_vacuum_get_dead_items(ParallelVacuumState *pvs);
extern void parallel_vacuum_bulkdel_all_indexes(ParallelVacuumState *pvs,
long num_table_tuples,
int num_index_scans);
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index 07002fdfbe..537b34b30c 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -207,6 +207,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_PGSTATS_DATA,
LWTRANCHE_LAUNCHER_DSA,
LWTRANCHE_LAUNCHER_HASH,
+ LWTRANCHE_PARALLEL_VACUUM_DSA,
LWTRANCHE_FIRST_USER_DEFINED
} BuiltinTrancheIds;
diff --git a/src/test/regress/expected/cluster.out b/src/test/regress/expected/cluster.out
index 2eec483eaa..e04f50726f 100644
--- a/src/test/regress/expected/cluster.out
+++ b/src/test/regress/expected/cluster.out
@@ -526,7 +526,7 @@ create index cluster_sort on clstr_4 (hundred, thousand, tenthous);
-- ensure we don't use the index in CLUSTER nor the checking SELECTs
set enable_indexscan = off;
-- Use external sort:
-set maintenance_work_mem = '1MB';
+set maintenance_work_mem = '2MB';
cluster clstr_4 using cluster_sort;
select * from
(select hundred, lag(hundred) over () as lhundred,
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 6cd57e3eaa..d1889b9d10 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -1214,7 +1214,7 @@ DROP TABLE unlogged_hash_table;
-- CREATE INDEX hash_ovfl_index ON hash_ovfl_heap USING hash (x int4_ops);
-- Test hash index build tuplesorting. Force hash tuplesort using low
-- maintenance_work_mem setting and fillfactor:
-SET maintenance_work_mem = '1MB';
+SET maintenance_work_mem = '2MB';
CREATE INDEX hash_tuplesort_idx ON tenk1 USING hash (stringu1 name_ops) WITH (fillfactor = 10);
EXPLAIN (COSTS OFF)
SELECT count(*) FROM tenk1 WHERE stringu1 = 'TVAAAA';
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 174b725fff..8fa4e86be8 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2032,8 +2032,8 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param3 AS heap_blks_scanned,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
- s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param6 AS max_dead_tuple_bytes,
+ s.param7 AS dead_tuple_bytes
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_recovery_prefetch| SELECT stats_reset,
diff --git a/src/test/regress/sql/cluster.sql b/src/test/regress/sql/cluster.sql
index a4cfaae807..a4cb5b98a5 100644
--- a/src/test/regress/sql/cluster.sql
+++ b/src/test/regress/sql/cluster.sql
@@ -258,7 +258,7 @@ create index cluster_sort on clstr_4 (hundred, thousand, tenthous);
set enable_indexscan = off;
-- Use external sort:
-set maintenance_work_mem = '1MB';
+set maintenance_work_mem = '2MB';
cluster clstr_4 using cluster_sort;
select * from
(select hundred, lag(hundred) over () as lhundred,
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index a3738833b2..edb5e4b4f3 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -367,7 +367,7 @@ DROP TABLE unlogged_hash_table;
-- Test hash index build tuplesorting. Force hash tuplesort using low
-- maintenance_work_mem setting and fillfactor:
-SET maintenance_work_mem = '1MB';
+SET maintenance_work_mem = '2MB';
CREATE INDEX hash_tuplesort_idx ON tenk1 USING hash (stringu1 name_ops) WITH (fillfactor = 10);
EXPLAIN (COSTS OFF)
SELECT count(*) FROM tenk1 WHERE stringu1 = 'TVAAAA';
--
2.31.1
[application/octet-stream] v29-0007-Review-radix-tree.patch (22.5K, ../../CAD21AoAFiw2DS07nPhiS14eVXpnF-bBmjvab4y+Ct8igU2z_8A@mail.gmail.com/3-v29-0007-Review-radix-tree.patch)
download | inline diff:
From 52e0d50d6e882c0444ccdf15f8afcc1aef3a6987 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 20 Feb 2023 11:28:50 +0900
Subject: [PATCH v29 07/10] Review radix tree.
Mainly improve the iteration codes and comments.
---
src/include/lib/radixtree.h | 169 +++++++++---------
src/include/lib/radixtree_iter_impl.h | 85 ++++-----
.../expected/test_radixtree.out | 6 +-
.../modules/test_radixtree/test_radixtree.c | 103 +++++++----
4 files changed, 197 insertions(+), 166 deletions(-)
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index e546bd705c..8bea606c62 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -83,7 +83,7 @@
* RT_SET - Set a key-value pair
* RT_BEGIN_ITERATE - Begin iterating through all key-value pairs
* RT_ITERATE_NEXT - Return next key-value pair, if any
- * RT_END_ITER - End iteration
+ * RT_END_ITERATE - End iteration
* RT_MEMORY_USAGE - Get the memory usage
*
* Interface for Shared Memory
@@ -152,8 +152,8 @@
#define RT_INIT_NODE RT_MAKE_NAME(init_node)
#define RT_FREE_NODE RT_MAKE_NAME(free_node)
#define RT_FREE_RECURSE RT_MAKE_NAME(free_recurse)
-#define RT_EXTEND RT_MAKE_NAME(extend)
-#define RT_SET_EXTEND RT_MAKE_NAME(set_extend)
+#define RT_EXTEND_UP RT_MAKE_NAME(extend_up)
+#define RT_EXTEND_DOWN RT_MAKE_NAME(extend_down)
#define RT_SWITCH_NODE_KIND RT_MAKE_NAME(grow_node_kind)
#define RT_COPY_NODE RT_MAKE_NAME(copy_node)
#define RT_REPLACE_NODE RT_MAKE_NAME(replace_node)
@@ -191,7 +191,7 @@
#define RT_NODE_INSERT_LEAF RT_MAKE_NAME(node_insert_leaf)
#define RT_NODE_INNER_ITERATE_NEXT RT_MAKE_NAME(node_inner_iterate_next)
#define RT_NODE_LEAF_ITERATE_NEXT RT_MAKE_NAME(node_leaf_iterate_next)
-#define RT_UPDATE_ITER_STACK RT_MAKE_NAME(update_iter_stack)
+#define RT_ITER_SET_NODE_FROM RT_MAKE_NAME(iter_set_node_from)
#define RT_ITER_UPDATE_KEY RT_MAKE_NAME(iter_update_key)
#define RT_VERIFY_NODE RT_MAKE_NAME(verify_node)
@@ -612,7 +612,6 @@ static const RT_SIZE_CLASS_ELEM RT_SIZE_CLASS_INFO[] = {
#endif
/* Contains the actual tree and ancillary info */
-// WIP: this name is a bit strange
typedef struct RT_RADIX_TREE_CONTROL
{
#ifdef RT_SHMEM
@@ -651,36 +650,40 @@ typedef struct RT_RADIX_TREE
* Iteration support.
*
* Iterating the radix tree returns each pair of key and value in the ascending
- * order of the key. To support this, the we iterate nodes of each level.
+ * order of the key.
*
- * RT_NODE_ITER struct is used to track the iteration within a node.
+ * RT_NODE_ITER is the struct for iteration of one radix tree node.
*
* RT_ITER is the struct for iteration of the radix tree, and uses RT_NODE_ITER
- * in order to track the iteration of each level. During iteration, we also
- * construct the key whenever updating the node iteration information, e.g., when
- * advancing the current index within the node or when moving to the next node
- * at the same level.
- *
- * XXX: Currently we allow only one process to do iteration. Therefore, rt_node_iter
- * has the local pointers to nodes, rather than RT_PTR_ALLOC.
- * We need either a safeguard to disallow other processes to begin the iteration
- * while one process is doing or to allow multiple processes to do the iteration.
+ * for each level to track the iteration within the node.
*/
typedef struct RT_NODE_ITER
{
- RT_PTR_LOCAL node; /* current node being iterated */
- int current_idx; /* current position. -1 for initial value */
+ /*
+ * Local pointer to the node we are iterating over.
+ *
+ * Since the radix tree doesn't support the shared iteration among multiple
+ * processes, we use RT_PTR_LOCAL rather than RT_PTR_ALLOC.
+ */
+ RT_PTR_LOCAL node;
+
+ /*
+ * The next index of the chunk array in RT_NODE_KIND_3 and
+ * RT_NODE_KIND_32 nodes, or the next chunk in RT_NODE_KIND_125 and
+ * RT_NODE_KIND_256 nodes. 0 for the initial value.
+ */
+ int idx;
} RT_NODE_ITER;
typedef struct RT_ITER
{
RT_RADIX_TREE *tree;
- /* Track the iteration on nodes of each level */
- RT_NODE_ITER stack[RT_MAX_LEVEL];
- int stack_len;
+ /* Track the nodes for each level. level = 0 is for a leaf node */
+ RT_NODE_ITER node_iters[RT_MAX_LEVEL];
+ int top_level;
- /* The key is constructed during iteration */
+ /* The key constructed during the iteration */
uint64 key;
} RT_ITER;
@@ -1243,7 +1246,7 @@ RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
* it can store the key.
*/
static pg_noinline void
-RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
+RT_EXTEND_UP(RT_RADIX_TREE *tree, uint64 key)
{
int target_shift;
RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
@@ -1282,7 +1285,7 @@ RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
* Insert inner and leaf nodes from 'node' to bottom.
*/
static pg_noinline void
-RT_SET_EXTEND(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
+RT_EXTEND_DOWN(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node)
{
int shift = node->shift;
@@ -1613,7 +1616,7 @@ RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
/* Extend the tree if necessary */
if (key > tree->ctl->max_val)
- RT_EXTEND(tree, key);
+ RT_EXTEND_UP(tree, key);
stored_child = tree->ctl->root;
parent = RT_PTR_GET_LOCAL(tree, stored_child);
@@ -1631,7 +1634,7 @@ RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
if (!RT_NODE_SEARCH_INNER(child, key, &new_child))
{
- RT_SET_EXTEND(tree, key, value_p, parent, stored_child, child);
+ RT_EXTEND_DOWN(tree, key, value_p, parent, stored_child, child);
RT_UNLOCK(tree);
return false;
}
@@ -1805,16 +1808,9 @@ RT_DELETE(RT_RADIX_TREE *tree, uint64 key)
}
#endif
-static inline void
-RT_ITER_UPDATE_KEY(RT_ITER *iter, uint8 chunk, uint8 shift)
-{
- iter->key &= ~(((uint64) RT_CHUNK_MASK) << shift);
- iter->key |= (((uint64) chunk) << shift);
-}
-
/*
- * Advance the slot in the inner node. Return the child if exists, otherwise
- * null.
+ * Scan the inner node and return the next child node if exist, otherwise
+ * return NULL.
*/
static inline RT_PTR_LOCAL
RT_NODE_INNER_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter)
@@ -1825,8 +1821,8 @@ RT_NODE_INNER_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter)
}
/*
- * Advance the slot in the leaf node. On success, return true and the value
- * is set to value_p, otherwise return false.
+ * Scan the leaf node, and return true and the next value is set to value_p
+ * if exists. Otherwise return false.
*/
static inline bool
RT_NODE_LEAF_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter,
@@ -1838,29 +1834,50 @@ RT_NODE_LEAF_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter,
}
/*
- * Update each node_iter for inner nodes in the iterator node stack.
+ * While descending the radix tree from the 'from' node to the bottom, we
+ * set the next node to iterate for each level.
*/
static void
-RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
+RT_ITER_SET_NODE_FROM(RT_ITER *iter, RT_PTR_LOCAL from)
{
- int level = from;
- RT_PTR_LOCAL node = from_node;
+ int level = from->shift / RT_NODE_SPAN;
+ RT_PTR_LOCAL node = from;
for (;;)
{
- RT_NODE_ITER *node_iter = &(iter->stack[level--]);
+ RT_NODE_ITER *node_iter = &(iter->node_iters[level--]);
+
+#ifdef USE_ASSERT_CHECKING
+ if (node_iter->node)
+ {
+ /* We must have finished the iteration on the previous node */
+ if (RT_NODE_IS_LEAF(node_iter->node))
+ {
+ uint64 dummy;
+ Assert(!RT_NODE_LEAF_ITERATE_NEXT(iter, node_iter, &dummy));
+ }
+ else
+ Assert(!RT_NODE_INNER_ITERATE_NEXT(iter, node_iter));
+ }
+#endif
+ /* Set the node to the node iterator of this level */
node_iter->node = node;
- node_iter->current_idx = -1;
+ node_iter->idx = 0;
- /* We don't advance the leaf node iterator here */
if (RT_NODE_IS_LEAF(node))
- return;
+ {
+ /* We will visit the leaf node when RT_ITERATE_NEXT() */
+ break;
+ }
- /* Advance to the next slot in the inner node */
+ /*
+ * Get the first child node from the node, which corresponds to the
+ * lowest chunk within the node.
+ */
node = RT_NODE_INNER_ITERATE_NEXT(iter, node_iter);
- /* We must find the first children in the node */
+ /* The first child must be found */
Assert(node);
}
}
@@ -1874,14 +1891,11 @@ RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
RT_SCOPE RT_ITER *
RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
{
- MemoryContext old_ctx;
RT_ITER *iter;
RT_PTR_LOCAL root;
- int top_level;
- old_ctx = MemoryContextSwitchTo(tree->context);
-
- iter = (RT_ITER *) palloc0(sizeof(RT_ITER));
+ iter = (RT_ITER *) MemoryContextAllocZero(tree->context,
+ sizeof(RT_ITER));
iter->tree = tree;
RT_LOCK_SHARED(tree);
@@ -1891,16 +1905,13 @@ RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
return iter;
root = RT_PTR_GET_LOCAL(tree, iter->tree->ctl->root);
- top_level = root->shift / RT_NODE_SPAN;
- iter->stack_len = top_level;
+ iter->top_level = root->shift / RT_NODE_SPAN;
/*
- * Descend to the left most leaf node from the root. The key is being
- * constructed while descending to the leaf.
+ * Set the next node to iterate for each level from the level of the
+ * root node.
*/
- RT_UPDATE_ITER_STACK(iter, root, top_level);
-
- MemoryContextSwitchTo(old_ctx);
+ RT_ITER_SET_NODE_FROM(iter, root);
return iter;
}
@@ -1912,6 +1923,8 @@ RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
RT_SCOPE bool
RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p)
{
+ Assert(value_p != NULL);
+
/* Empty tree */
if (!iter->tree->ctl->root)
return false;
@@ -1919,43 +1932,38 @@ RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p)
for (;;)
{
RT_PTR_LOCAL child = NULL;
- RT_VALUE_TYPE value;
- int level;
- bool found;
-
- /* Advance the leaf node iterator to get next key-value pair */
- found = RT_NODE_LEAF_ITERATE_NEXT(iter, &(iter->stack[0]), &value);
- if (found)
+ /* Get the next chunk of the leaf node */
+ if (RT_NODE_LEAF_ITERATE_NEXT(iter, &(iter->node_iters[0]), value_p))
{
*key_p = iter->key;
- *value_p = value;
return true;
}
/*
- * We've visited all values in the leaf node, so advance inner node
- * iterators from the level=1 until we find the next child node.
+ * We've visited all values in the leaf node, so advance all inner node
+ * iterators by visiting inner nodes from the level = 1 until we find the
+ * next inner node that has a child node.
*/
- for (level = 1; level <= iter->stack_len; level++)
+ for (int level = 1; level <= iter->top_level; level++)
{
- child = RT_NODE_INNER_ITERATE_NEXT(iter, &(iter->stack[level]));
+ child = RT_NODE_INNER_ITERATE_NEXT(iter, &(iter->node_iters[level]));
if (child)
break;
}
- /* the iteration finished */
+ /* We've visited all nodes, so the iteration finished */
if (!child)
- return false;
+ break;
/*
- * Set the node to the node iterator and update the iterator stack
- * from this node.
+ * Found the new child node. We update the next node to iterate for each
+ * level from the level of this child node.
*/
- RT_UPDATE_ITER_STACK(iter, child, level - 1);
+ RT_ITER_SET_NODE_FROM(iter, child);
- /* Node iterators are updated, so try again from the leaf */
+ /* Find key-value from the leaf node again */
}
return false;
@@ -2470,8 +2478,8 @@ RT_DUMP(RT_RADIX_TREE *tree)
#undef RT_INIT_NODE
#undef RT_FREE_NODE
#undef RT_FREE_RECURSE
-#undef RT_EXTEND
-#undef RT_SET_EXTEND
+#undef RT_EXTEND_UP
+#undef RT_EXTEND_DOWN
#undef RT_SWITCH_NODE_KIND
#undef RT_COPY_NODE
#undef RT_REPLACE_NODE
@@ -2509,8 +2517,7 @@ RT_DUMP(RT_RADIX_TREE *tree)
#undef RT_NODE_INSERT_LEAF
#undef RT_NODE_INNER_ITERATE_NEXT
#undef RT_NODE_LEAF_ITERATE_NEXT
-#undef RT_UPDATE_ITER_STACK
-#undef RT_ITER_UPDATE_KEY
+#undef RT_RT_ITER_SET_NODE_FROM
#undef RT_VERIFY_NODE
#undef RT_DEBUG
diff --git a/src/include/lib/radixtree_iter_impl.h b/src/include/lib/radixtree_iter_impl.h
index 98c78eb237..5c1034768e 100644
--- a/src/include/lib/radixtree_iter_impl.h
+++ b/src/include/lib/radixtree_iter_impl.h
@@ -27,12 +27,10 @@
#error node level must be either inner or leaf
#endif
- bool found = false;
- uint8 key_chunk;
+ uint8 key_chunk = 0;
#ifdef RT_NODE_LEVEL_LEAF
- RT_VALUE_TYPE value;
-
+ Assert(value_p != NULL);
Assert(RT_NODE_IS_LEAF(node_iter->node));
#else
RT_PTR_LOCAL child = NULL;
@@ -50,99 +48,92 @@
{
RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node_iter->node;
- node_iter->current_idx++;
- if (node_iter->current_idx >= n3->base.n.count)
- break;
+ if (node_iter->idx >= n3->base.n.count)
+ return false;
+
#ifdef RT_NODE_LEVEL_LEAF
- value = n3->values[node_iter->current_idx];
+ *value_p = n3->values[node_iter->idx];
#else
- child = RT_PTR_GET_LOCAL(iter->tree, n3->children[node_iter->current_idx]);
+ child = RT_PTR_GET_LOCAL(iter->tree, n3->children[node_iter->idx]);
#endif
- key_chunk = n3->base.chunks[node_iter->current_idx];
- found = true;
+ key_chunk = n3->base.chunks[node_iter->idx];
+ node_iter->idx++;
break;
}
case RT_NODE_KIND_32:
{
RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node_iter->node;
- node_iter->current_idx++;
- if (node_iter->current_idx >= n32->base.n.count)
- break;
+ if (node_iter->idx >= n32->base.n.count)
+ return false;
#ifdef RT_NODE_LEVEL_LEAF
- value = n32->values[node_iter->current_idx];
+ *value_p = n32->values[node_iter->idx];
#else
- child = RT_PTR_GET_LOCAL(iter->tree, n32->children[node_iter->current_idx]);
+ child = RT_PTR_GET_LOCAL(iter->tree, n32->children[node_iter->idx]);
#endif
- key_chunk = n32->base.chunks[node_iter->current_idx];
- found = true;
+ key_chunk = n32->base.chunks[node_iter->idx];
+ node_iter->idx++;
break;
}
case RT_NODE_KIND_125:
{
RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node_iter->node;
- int i;
+ int chunk;
- for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ for (chunk = node_iter->idx; chunk < RT_NODE_MAX_SLOTS; chunk++)
{
- if (RT_NODE_125_IS_CHUNK_USED((RT_NODE_BASE_125 *) n125, i))
+ if (RT_NODE_125_IS_CHUNK_USED((RT_NODE_BASE_125 *) n125, chunk))
break;
}
- if (i >= RT_NODE_MAX_SLOTS)
- break;
+ if (chunk >= RT_NODE_MAX_SLOTS)
+ return false;
- node_iter->current_idx = i;
#ifdef RT_NODE_LEVEL_LEAF
- value = RT_NODE_LEAF_125_GET_VALUE(n125, i);
+ *value_p = RT_NODE_LEAF_125_GET_VALUE(n125, chunk);
#else
- child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_125_GET_CHILD(n125, chunk));
#endif
- key_chunk = i;
- found = true;
+ key_chunk = chunk;
+ node_iter->idx = chunk + 1;
break;
}
case RT_NODE_KIND_256:
{
RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node_iter->node;
- int i;
+ int chunk;
- for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ for (chunk = node_iter->idx; chunk < RT_NODE_MAX_SLOTS; chunk++)
{
#ifdef RT_NODE_LEVEL_LEAF
- if (RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+ if (RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
#else
- if (RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ if (RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
#endif
break;
}
- if (i >= RT_NODE_MAX_SLOTS)
- break;
+ if (chunk >= RT_NODE_MAX_SLOTS)
+ return false;
- node_iter->current_idx = i;
#ifdef RT_NODE_LEVEL_LEAF
- value = RT_NODE_LEAF_256_GET_VALUE(n256, i);
+ *value_p = RT_NODE_LEAF_256_GET_VALUE(n256, chunk);
#else
- child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_256_GET_CHILD(n256, chunk));
#endif
- key_chunk = i;
- found = true;
+ key_chunk = chunk;
+ node_iter->idx = chunk + 1;
break;
}
}
- if (found)
- {
- RT_ITER_UPDATE_KEY(iter, key_chunk, node_iter->node->shift);
-#ifdef RT_NODE_LEVEL_LEAF
- *value_p = value;
-#endif
- }
+ /* Update the part of the key */
+ iter->key &= ~(((uint64) RT_CHUNK_MASK) << node_iter->node->shift);
+ iter->key |= (((uint64) key_chunk) << node_iter->node->shift);
#ifdef RT_NODE_LEVEL_LEAF
- return found;
+ return true;
#else
return child;
#endif
diff --git a/src/test/modules/test_radixtree/expected/test_radixtree.out b/src/test/modules/test_radixtree/expected/test_radixtree.out
index ce645cb8b5..7ad1ce3605 100644
--- a/src/test/modules/test_radixtree/expected/test_radixtree.out
+++ b/src/test/modules/test_radixtree/expected/test_radixtree.out
@@ -4,8 +4,10 @@ CREATE EXTENSION test_radixtree;
-- an error if something fails.
--
SELECT test_radixtree();
-NOTICE: testing basic operations with leaf node 4
-NOTICE: testing basic operations with inner node 4
+NOTICE: testing basic operations with leaf node 3
+NOTICE: testing basic operations with inner node 3
+NOTICE: testing basic operations with leaf node 15
+NOTICE: testing basic operations with inner node 15
NOTICE: testing basic operations with leaf node 32
NOTICE: testing basic operations with inner node 32
NOTICE: testing basic operations with leaf node 125
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
index afe53382f3..5a169854d9 100644
--- a/src/test/modules/test_radixtree/test_radixtree.c
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -43,12 +43,15 @@ typedef uint64 TestValueType;
*/
static const bool rt_test_stats = false;
-static int rt_node_kind_fanouts[] = {
- 0,
- 4, /* RT_NODE_KIND_4 */
- 32, /* RT_NODE_KIND_32 */
- 125, /* RT_NODE_KIND_125 */
- 256 /* RT_NODE_KIND_256 */
+/*
+ * XXX: should we expose and use RT_SIZE_CLASS and RT_SIZE_CLASS_INFO?
+ */
+static int rt_node_class_fanouts[] = {
+ 3, /* RT_CLASS_3 */
+ 15, /* RT_CLASS_32_MIN */
+ 32, /* RT_CLASS_32_MAX */
+ 125, /* RT_CLASS_125 */
+ 256 /* RT_CLASS_256 */
};
/*
* A struct to define a pattern of integers, for use with the test_pattern()
@@ -260,10 +263,9 @@ test_basic(int children, bool test_inner)
* Check if keys from start to end with the shift exist in the tree.
*/
static void
-check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end,
- int incr)
+check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end)
{
- for (int i = start; i < end; i++)
+ for (int i = start; i <= end; i++)
{
uint64 key = ((uint64) i << shift);
TestValueType val;
@@ -277,22 +279,26 @@ check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end,
}
}
+/*
+ * Insert 256 key-value pairs, and check if keys are properly inserted on each
+ * node class.
+ */
+/* Test keys [0, 256) */
+#define NODE_TYPE_TEST_KEY_MIN 0
+#define NODE_TYPE_TEST_KEY_MAX 256
static void
-test_node_types_insert(rt_radix_tree *radixtree, uint8 shift, bool insert_asc)
+test_node_types_insert_asc(rt_radix_tree *radixtree, uint8 shift)
{
- uint64 num_entries;
- int ninserted = 0;
- int start = insert_asc ? 0 : 256;
- int incr = insert_asc ? 1 : -1;
- int end = insert_asc ? 256 : 0;
- int node_kind_idx = 1;
+ uint64 num_entries;
+ int node_class_idx = 0;
+ uint64 key_checked = 0;
- for (int i = start; i != end; i += incr)
+ for (int i = NODE_TYPE_TEST_KEY_MIN; i < NODE_TYPE_TEST_KEY_MAX; i++)
{
uint64 key = ((uint64) i << shift);
bool found;
- found = rt_set(radixtree, key, (TestValueType*) &key);
+ found = rt_set(radixtree, key, (TestValueType *) &key);
if (found)
elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " is found", key);
@@ -300,24 +306,49 @@ test_node_types_insert(rt_radix_tree *radixtree, uint8 shift, bool insert_asc)
* After filling all slots in each node type, check if the values
* are stored properly.
*/
- if (ninserted == rt_node_kind_fanouts[node_kind_idx] - 1)
+ if ((i + 1) == rt_node_class_fanouts[node_class_idx])
{
- int check_start = insert_asc
- ? rt_node_kind_fanouts[node_kind_idx - 1]
- : rt_node_kind_fanouts[node_kind_idx];
- int check_end = insert_asc
- ? rt_node_kind_fanouts[node_kind_idx]
- : rt_node_kind_fanouts[node_kind_idx - 1];
-
- check_search_on_node(radixtree, shift, check_start, check_end, incr);
- node_kind_idx++;
+ check_search_on_node(radixtree, shift, key_checked, i);
+ key_checked = i;
+ node_class_idx++;
}
-
- ninserted++;
}
num_entries = rt_num_entries(radixtree);
+ if (num_entries != 256)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+/*
+ * Similar to test_node_types_insert_asc(), but inserts keys in descending order.
+ */
+static void
+test_node_types_insert_desc(rt_radix_tree *radixtree, uint8 shift)
+{
+ uint64 num_entries;
+ int node_class_idx = 0;
+ uint64 key_checked = NODE_TYPE_TEST_KEY_MAX - 1;
+
+ for (int i = NODE_TYPE_TEST_KEY_MAX - 1; i >= NODE_TYPE_TEST_KEY_MIN; i--)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_set(radixtree, key, (TestValueType *) &key);
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " is found", key);
+ if ((i + 1) == rt_node_class_fanouts[node_class_idx])
+ {
+ check_search_on_node(radixtree, shift, i, key_checked);
+ key_checked = i;
+ node_class_idx++;
+ }
+ }
+
+ num_entries = rt_num_entries(radixtree);
if (num_entries != 256)
elog(ERROR,
"rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
@@ -329,7 +360,7 @@ test_node_types_delete(rt_radix_tree *radixtree, uint8 shift)
{
uint64 num_entries;
- for (int i = 0; i < 256; i++)
+ for (int i = NODE_TYPE_TEST_KEY_MIN; i < NODE_TYPE_TEST_KEY_MAX; i++)
{
uint64 key = ((uint64) i << shift);
bool found;
@@ -379,9 +410,9 @@ test_node_types(uint8 shift)
* then delete all entries to make it empty, and insert and search entries
* again.
*/
- test_node_types_insert(radixtree, shift, true);
+ test_node_types_insert_asc(radixtree, shift);
test_node_types_delete(radixtree, shift);
- test_node_types_insert(radixtree, shift, false);
+ test_node_types_insert_desc(radixtree, shift);
rt_free(radixtree);
#ifdef RT_SHMEM
@@ -664,10 +695,10 @@ test_radixtree(PG_FUNCTION_ARGS)
{
test_empty();
- for (int i = 1; i < lengthof(rt_node_kind_fanouts); i++)
+ for (int i = 0; i < lengthof(rt_node_class_fanouts); i++)
{
- test_basic(rt_node_kind_fanouts[i], false);
- test_basic(rt_node_kind_fanouts[i], true);
+ test_basic(rt_node_class_fanouts[i], false);
+ test_basic(rt_node_class_fanouts[i], true);
}
for (int shift = 0; shift <= (64 - 8); shift += 8)
--
2.31.1
[application/octet-stream] v29-0010-Revert-building-benchmark-module-for-CI.patch (694B, ../../CAD21AoAFiw2DS07nPhiS14eVXpnF-bBmjvab4y+Ct8igU2z_8A@mail.gmail.com/4-v29-0010-Revert-building-benchmark-module-for-CI.patch)
download | inline diff:
From b6a692913ce8c6868996336f4be778eb5f83d02c Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 14 Feb 2023 19:31:34 +0700
Subject: [PATCH v29 10/10] Revert building benchmark module for CI
---
contrib/meson.build | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/contrib/meson.build b/contrib/meson.build
index 421d469f8c..52253de793 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,7 +12,7 @@ subdir('amcheck')
subdir('auth_delay')
subdir('auto_explain')
subdir('basic_archive')
-subdir('bench_radix_tree')
+#subdir('bench_radix_tree')
subdir('bloom')
subdir('basebackup_to_shell')
subdir('bool_plperl')
--
2.31.1
[application/octet-stream] v29-0009-Review-vacuum-integration.patch (12.6K, ../../CAD21AoAFiw2DS07nPhiS14eVXpnF-bBmjvab4y+Ct8igU2z_8A@mail.gmail.com/5-v29-0009-Review-vacuum-integration.patch)
download | inline diff:
From e804119fddce3bc0520bedc70c966470c7db35e9 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 17 Feb 2023 00:04:37 +0900
Subject: [PATCH v29 09/10] Review vacuum integration.
---
src/backend/access/heap/vacuumlazy.c | 61 +++++++++++++--------------
src/backend/commands/vacuum.c | 4 +-
src/backend/commands/vacuumparallel.c | 25 +++++------
3 files changed, 45 insertions(+), 45 deletions(-)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index b4e40423a8..edb9079124 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -10,11 +10,10 @@
* of dead TIDs at once.
*
* We are willing to use at most maintenance_work_mem (or perhaps
- * autovacuum_work_mem) memory space to keep track of dead TIDs. We initially
- * create a TidStore with the maximum bytes that can be used by the TidStore.
- * If the TidStore is full, we must call lazy_vacuum to vacuum indexes (and to
- * vacuum the pages that we've pruned). This frees up the memory space dedicated
- * to storing dead TIDs.
+ * autovacuum_work_mem) memory space to keep track of dead TIDs. If the
+ * TidStore is full, we must call lazy_vacuum to vacuum indexes (and to vacuum
+ * the pages that we've pruned). This frees up the memory space dedicated to
+ * to store dead TIDs.
*
* In practice VACUUM will often complete its initial pass over the target
* heap relation without ever running out of space to store TIDs. This means
@@ -844,7 +843,7 @@ lazy_scan_heap(LVRelState *vacrel)
/* Report that we're scanning the heap, advertising total # of blocks */
initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
initprog_val[1] = rel_pages;
- initprog_val[2] = tidstore_max_memory(vacrel->dead_items);
+ initprog_val[2] = TidStoreMaxMemory(vacrel->dead_items);
pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
/* Set up an initial range of skippable blocks using the visibility map */
@@ -911,7 +910,7 @@ lazy_scan_heap(LVRelState *vacrel)
* dead_items TIDs, pause and do a cycle of vacuuming before we tackle
* this page.
*/
- if (tidstore_is_full(vacrel->dead_items))
+ if (TidStoreIsFull(vacrel->dead_items))
{
/*
* Before beginning index vacuuming, we release any pin we may
@@ -1080,16 +1079,16 @@ lazy_scan_heap(LVRelState *vacrel)
* with prunestate-driven visibility map and FSM steps (just like
* the two-pass strategy).
*/
- Assert(tidstore_num_tids(dead_items) == 0);
+ Assert(TidStoreNumTids(dead_items) == 0);
}
else if (prunestate.num_offsets > 0)
{
/* Save details of the LP_DEAD items from the page in dead_items */
- tidstore_add_tids(dead_items, blkno, prunestate.deadoffsets,
- prunestate.num_offsets);
+ TidStoreSetBlockOffsets(dead_items, blkno, prunestate.deadoffsets,
+ prunestate.num_offsets);
pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
- tidstore_memory_usage(dead_items));
+ TidStoreMemoryUsage(dead_items));
}
/*
@@ -1260,7 +1259,7 @@ lazy_scan_heap(LVRelState *vacrel)
* Do index vacuuming (call each index's ambulkdelete routine), then do
* related heap vacuuming
*/
- if (tidstore_num_tids(dead_items) > 0)
+ if (TidStoreNumTids(dead_items) > 0)
lazy_vacuum(vacrel);
/*
@@ -2127,10 +2126,10 @@ lazy_scan_noprune(LVRelState *vacrel,
*/
vacrel->lpdead_item_pages++;
- tidstore_add_tids(dead_items, blkno, deadoffsets, lpdead_items);
+ TidStoreSetBlockOffsets(dead_items, blkno, deadoffsets, lpdead_items);
pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
- tidstore_memory_usage(dead_items));
+ TidStoreMemoryUsage(dead_items));
vacrel->lpdead_items += lpdead_items;
@@ -2179,7 +2178,7 @@ lazy_vacuum(LVRelState *vacrel)
if (!vacrel->do_index_vacuuming)
{
Assert(!vacrel->do_index_cleanup);
- tidstore_reset(vacrel->dead_items);
+ TidStoreReset(vacrel->dead_items);
return;
}
@@ -2208,7 +2207,7 @@ lazy_vacuum(LVRelState *vacrel)
BlockNumber threshold;
Assert(vacrel->num_index_scans == 0);
- Assert(vacrel->lpdead_items == tidstore_num_tids(vacrel->dead_items));
+ Assert(vacrel->lpdead_items == TidStoreNumTids(vacrel->dead_items));
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2236,7 +2235,7 @@ lazy_vacuum(LVRelState *vacrel)
*/
threshold = (double) vacrel->rel_pages * BYPASS_THRESHOLD_PAGES;
bypass = (vacrel->lpdead_item_pages < threshold) &&
- tidstore_memory_usage(vacrel->dead_items) < (32L * 1024L * 1024L);
+ TidStoreMemoryUsage(vacrel->dead_items) < (32L * 1024L * 1024L);
}
if (bypass)
@@ -2281,7 +2280,7 @@ lazy_vacuum(LVRelState *vacrel)
* Forget the LP_DEAD items that we just vacuumed (or just decided to not
* vacuum)
*/
- tidstore_reset(vacrel->dead_items);
+ TidStoreReset(vacrel->dead_items);
}
/*
@@ -2354,7 +2353,7 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
* place).
*/
Assert(vacrel->num_index_scans > 0 ||
- tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items);
+ TidStoreNumTids(vacrel->dead_items) == vacrel->lpdead_items);
Assert(allindexes || vacrel->failsafe_active);
/*
@@ -2394,7 +2393,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
Buffer vmbuffer = InvalidBuffer;
LVSavedErrInfo saved_err_info;
TidStoreIter *iter;
- TidStoreIterResult *result;
+ TidStoreIterResult *iter_result;
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2409,8 +2408,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
VACUUM_ERRCB_PHASE_VACUUM_HEAP,
InvalidBlockNumber, InvalidOffsetNumber);
- iter = tidstore_begin_iterate(vacrel->dead_items);
- while ((result = tidstore_iterate_next(iter)) != NULL)
+ iter = TidStoreBeginIterate(vacrel->dead_items);
+ while ((iter_result = TidStoreIterateNext(iter)) != NULL)
{
BlockNumber blkno;
Buffer buf;
@@ -2419,7 +2418,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
vacuum_delay_point();
- blkno = result->blkno;
+ blkno = iter_result->blkno;
vacrel->blkno = blkno;
/*
@@ -2433,8 +2432,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
buf = ReadBufferExtended(vacrel->rel, MAIN_FORKNUM, blkno, RBM_NORMAL,
vacrel->bstrategy);
LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
- lazy_vacuum_heap_page(vacrel, blkno, result->offsets, result->num_offsets,
- buf, vmbuffer);
+ lazy_vacuum_heap_page(vacrel, blkno, iter_result->offsets,
+ iter_result->num_offsets, buf, vmbuffer);
/* Now that we've vacuumed the page, record its available space */
page = BufferGetPage(buf);
@@ -2444,7 +2443,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
vacuumed_pages++;
}
- tidstore_end_iterate(iter);
+ TidStoreEndIterate(iter);
vacrel->blkno = InvalidBlockNumber;
if (BufferIsValid(vmbuffer))
@@ -2455,12 +2454,12 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
* the second heap pass. No more, no less.
*/
Assert(vacrel->num_index_scans > 1 ||
- (tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items &&
+ (TidStoreNumTids(vacrel->dead_items) == vacrel->lpdead_items &&
vacuumed_pages == vacrel->lpdead_item_pages));
ereport(DEBUG2,
- (errmsg("table \"%s\": removed " UINT64_FORMAT "dead item identifiers in %u pages",
- vacrel->relname, tidstore_num_tids(vacrel->dead_items),
+ (errmsg("table \"%s\": removed " INT64_FORMAT "dead item identifiers in %u pages",
+ vacrel->relname, TidStoreNumTids(vacrel->dead_items),
vacuumed_pages)));
/* Revert to the previous phase information for error traceback */
@@ -3118,8 +3117,8 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
}
/* Serial VACUUM case */
- vacrel->dead_items = tidstore_create(vac_work_mem, MaxHeapTuplesPerPage,
- NULL);
+ vacrel->dead_items = TidStoreCreate(vac_work_mem, MaxHeapTuplesPerPage,
+ NULL);
}
/*
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index d8e680ca20..5fb30d7e62 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -2311,7 +2311,7 @@ vac_bulkdel_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat,
ereport(ivinfo->message_level,
(errmsg("scanned index \"%s\" to remove " UINT64_FORMAT " row versions",
RelationGetRelationName(ivinfo->index),
- tidstore_num_tids(dead_items))));
+ TidStoreNumTids(dead_items))));
return istat;
}
@@ -2352,5 +2352,5 @@ vac_tid_reaped(ItemPointer itemptr, void *state)
{
TidStore *dead_items = (TidStore *) state;
- return tidstore_lookup_tid(dead_items, itemptr);
+ return TidStoreIsMember(dead_items, itemptr);
}
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index d653683693..9225daf3ab 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -9,11 +9,12 @@
* In a parallel vacuum, we perform both index bulk deletion and index cleanup
* with parallel worker processes. Individual indexes are processed by one
* vacuum process. ParalleVacuumState contains shared information as well as
- * the shared TidStore. We launch parallel worker processes at the start of
- * parallel index bulk-deletion and index cleanup and once all indexes are
- * processed, the parallel worker processes exit. Each time we process indexes
- * in parallel, the parallel context is re-initialized so that the same DSM can
- * be used for multiple passes of index bulk-deletion and index cleanup.
+ * the memory space for storing dead items allocated in the DSA area. We
+ * launch parallel worker processes at the start of parallel index
+ * bulk-deletion and index cleanup and once all indexes are processed, the
+ * parallel worker processes exit. Each time we process indexes in parallel,
+ * the parallel context is re-initialized so that the same DSM can be used for
+ * multiple passes of index bulk-deletion and index cleanup.
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -104,7 +105,7 @@ typedef struct PVShared
pg_atomic_uint32 idx;
/* Handle of the shared TidStore */
- tidstore_handle dead_items_handle;
+ TidStoreHandle dead_items_handle;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -289,7 +290,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_estimate_chunk(&pcxt->estimator, est_shared_len);
shm_toc_estimate_keys(&pcxt->estimator, 1);
- /* Estimate size for dead tuple DSA -- PARALLEL_VACUUM_KEY_DSA */
+ /* Initial size for dead tuple DSA -- PARALLEL_VACUUM_KEY_DEAD_ITEMS */
shm_toc_estimate_chunk(&pcxt->estimator, dsa_minsize);
shm_toc_estimate_keys(&pcxt->estimator, 1);
@@ -362,7 +363,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
dead_items_dsa = dsa_create_in_place(area_space, dsa_minsize,
LWTRANCHE_PARALLEL_VACUUM_DSA,
pcxt->seg);
- dead_items = tidstore_create(vac_work_mem, max_offset, dead_items_dsa);
+ dead_items = TidStoreCreate(vac_work_mem, max_offset, dead_items_dsa);
pvs->dead_items = dead_items;
pvs->dead_items_area = dead_items_dsa;
@@ -375,7 +376,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
- shared->dead_items_handle = tidstore_get_handle(dead_items);
+ shared->dead_items_handle = TidStoreGetHandle(dead_items);
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
@@ -441,7 +442,7 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
istats[i] = NULL;
}
- tidstore_destroy(pvs->dead_items);
+ TidStoreDestroy(pvs->dead_items);
dsa_detach(pvs->dead_items_area);
DestroyParallelContext(pvs->pcxt);
@@ -999,7 +1000,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
/* Set dead items */
area_space = shm_toc_lookup(toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, false);
dead_items_area = dsa_attach_in_place(area_space, seg);
- dead_items = tidstore_attach(dead_items_area, shared->dead_items_handle);
+ dead_items = TidStoreAttach(dead_items_area, shared->dead_items_handle);
/* Set cost-based vacuum delay */
VacuumCostActive = (VacuumCostDelay > 0);
@@ -1045,7 +1046,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
InstrEndParallelQuery(&buffer_usage[ParallelWorkerNumber],
&wal_usage[ParallelWorkerNumber]);
- tidstore_detach(pvs.dead_items);
+ TidStoreDetach(dead_items);
dsa_detach(dead_items_area);
/* Pop the error context stack */
--
2.31.1
[application/octet-stream] v29-0008-Review-TidStore.patch (32.1K, ../../CAD21AoAFiw2DS07nPhiS14eVXpnF-bBmjvab4y+Ct8igU2z_8A@mail.gmail.com/6-v29-0008-Review-TidStore.patch)
download | inline diff:
From fc373e0312e0b3c30bba8bd54286283542d627a2 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Thu, 16 Feb 2023 23:45:39 +0900
Subject: [PATCH v29 08/10] Review TidStore.
---
src/backend/access/common/tidstore.c | 340 +++++++++---------
src/include/access/tidstore.h | 37 +-
.../modules/test_tidstore/test_tidstore.c | 68 ++--
3 files changed, 234 insertions(+), 211 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 8c05e60d92..9360520482 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -3,18 +3,19 @@
* tidstore.c
* Tid (ItemPointerData) storage implementation.
*
- * This module provides a in-memory data structure to store Tids (ItemPointer).
- * Internally, a tid is encoded as a pair of 64-bit key and 64-bit value, and
- * stored in the radix tree.
+ * TidStore is a in-memory data structure to store tids (ItemPointerData).
+ * Internally, a tid is encoded as a pair of 64-bit key and 64-bit value,
+ * and stored in the radix tree.
*
- * A TidStore can be shared among parallel worker processes by passing DSA area
- * to tidstore_create(). Other backends can attach to the shared TidStore by
- * tidstore_attach().
+ * TidStore can be shared among parallel worker processes by passing DSA area
+ * to TidStoreCreate(). Other backends can attach to the shared TidStore by
+ * TidStoreAttach().
*
- * Regarding the concurrency, it basically relies on the concurrency support in
- * the radix tree, but we acquires the lock on a TidStore in some cases, for
- * example, when to reset the store and when to access the number tids in the
- * store (num_tids).
+ * Regarding the concurrency support, we use a single LWLock for the TidStore.
+ * The TidStore is exclusively locked when inserting encoded tids to the
+ * radix tree or when resetting itself. When searching on the TidStore or
+ * doing the iteration, it is not locked but the underlying radix tree is
+ * locked in shared mode.
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -34,16 +35,18 @@
#include "utils/memutils.h"
/*
- * For encoding purposes, tids are represented as a pair of 64-bit key and
- * 64-bit value. First, we construct 64-bit unsigned integer by combining
- * the block number and the offset number. The number of bits used for the
- * offset number is specified by max_offsets in tidstore_create(). We are
- * frugal with the bits, because smaller keys could help keeping the radix
- * tree shallow.
+ * For encoding purposes, a tid is represented as a pair of 64-bit key and
+ * 64-bit value.
*
- * For example, a tid of heap with 8kB blocks uses the lowest 9 bits for
- * the offset number and uses the next 32 bits for the block number. That
- * is, only 41 bits are used:
+ * First, we construct a 64-bit unsigned integer by combining the block
+ * number and the offset number. The number of bits used for the offset number
+ * is specified by max_off in TidStoreCreate(). We are frugal with the bits,
+ * because smaller keys could help keeping the radix tree shallow.
+ *
+ * For example, a tid of heap on a 8kB block uses the lowest 9 bits for
+ * the offset number and uses the next 32 bits for the block number. 9 bits
+ * are enough for the offset number, because MaxHeapTuplesPerPage < 2^9
+ * on 8kB blocks. That is, only 41 bits are used:
*
* uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
*
@@ -52,30 +55,34 @@
* u = unused bit
* (high on the left, low on the right)
*
- * 9 bits are enough for the offset number, because MaxHeapTuplesPerPage < 2^9
- * on 8kB blocks.
- *
- * The 64-bit value is the bitmap representation of the lowest 6 bits
- * (TIDSTORE_VALUE_NBITS) of the integer, and the rest 35 bits are used
- * as the key:
+ * Then, 64-bit value is the bitmap representation of the lowest 6 bits
+ * (LOWER_OFFSET_NBITS) of the integer, and 64-bit key consists of the
+ * upper 3 bits of the offset number and the block number, 35 bits in
+ * total:
*
* uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
* |----| value
- * |---------------------------------------------| key
+ * |--------------------------------------| key
*
* The maximum height of the radix tree is 5 in this case.
+ *
+ * If the number of bits required for offset numbers fits in LOWER_OFFSET_NBITS,
+ * 64-bit value is the bitmap representation of the offset number, and the
+ * 64-bit key is the block number.
*/
-#define TIDSTORE_VALUE_NBITS 6 /* log(64, 2) */
-#define TIDSTORE_OFFSET_MASK ((1 << TIDSTORE_VALUE_NBITS) - 1)
+typedef uint64 tidkey;
+typedef uint64 offsetbm;
+#define LOWER_OFFSET_NBITS 6 /* log(sizeof(offsetbm), 2) */
+#define LOWER_OFFSET_MASK ((1 << LOWER_OFFSET_NBITS) - 1)
-/* A magic value used to identify our TidStores. */
+/* A magic value used to identify our TidStore. */
#define TIDSTORE_MAGIC 0x826f6a10
#define RT_PREFIX local_rt
#define RT_SCOPE static
#define RT_DECLARE
#define RT_DEFINE
-#define RT_VALUE_TYPE uint64
+#define RT_VALUE_TYPE tidkey
#include "lib/radixtree.h"
#define RT_PREFIX shared_rt
@@ -83,7 +90,7 @@
#define RT_SCOPE static
#define RT_DECLARE
#define RT_DEFINE
-#define RT_VALUE_TYPE uint64
+#define RT_VALUE_TYPE tidkey
#include "lib/radixtree.h"
/* The control object for a TidStore */
@@ -94,10 +101,10 @@ typedef struct TidStoreControl
/* These values are never changed after creation */
size_t max_bytes; /* the maximum bytes a TidStore can use */
- int max_offset; /* the maximum offset number */
- int offset_nbits; /* the number of bits required for an offset
- * number */
- int offset_key_nbits; /* the number of bits of an offset number
+ int max_off; /* the maximum offset number */
+ int max_off_nbits; /* the number of bits required for offset
+ * numbers */
+ int upper_off_nbits; /* the number of bits of offset numbers
* used in a key */
/* The below fields are used only in shared case */
@@ -106,7 +113,7 @@ typedef struct TidStoreControl
LWLock lock;
/* handles for TidStore and radix tree */
- tidstore_handle handle;
+ TidStoreHandle handle;
shared_rt_handle tree_handle;
} TidStoreControl;
@@ -147,24 +154,27 @@ typedef struct TidStoreIter
bool finished;
/* save for the next iteration */
- uint64 next_key;
- uint64 next_val;
+ tidkey next_tidkey;
+ offsetbm next_off_bitmap;
- /* output for the caller */
- TidStoreIterResult result;
+ /*
+ * output for the caller. Must be last because variable-size.
+ */
+ TidStoreIterResult output;
} TidStoreIter;
-static void tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val);
-static inline BlockNumber key_get_blkno(TidStore *ts, uint64 key);
-static inline uint64 encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit);
-static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit);
+static void iter_decode_key_off(TidStoreIter *iter, tidkey key, offsetbm off_bitmap);
+static inline BlockNumber key_get_blkno(TidStore *ts, tidkey key);
+static inline tidkey encode_blk_off(TidStore *ts, BlockNumber block,
+ OffsetNumber offset, offsetbm *off_bit);
+static inline tidkey encode_tid(TidStore *ts, ItemPointer tid, offsetbm *off_bit);
/*
* Create a TidStore. The returned object is allocated in backend-local memory.
* The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
*/
TidStore *
-tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
+TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
{
TidStore *ts;
@@ -176,12 +186,12 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
* Memory consumption depends on the number of stored tids, but also on the
* distribution of them, how the radix tree stores, and the memory management
* that backed the radix tree. The maximum bytes that a TidStore can
- * use is specified by the max_bytes in tidstore_create(). We want the total
+ * use is specified by the max_bytes in TidStoreCreate(). We want the total
* amount of memory consumption by a TidStore not to exceed the max_bytes.
*
* In local TidStore cases, the radix tree uses slab allocators for each kind
* of node class. The most memory consuming case while adding Tids associated
- * with one page (i.e. during tidstore_add_tids()) is that we allocate a new
+ * with one page (i.e. during TidStoreSetBlockOffsets()) is that we allocate a new
* slab block for a new radix tree node, which is approximately 70kB. Therefore,
* we deduct 70kB from the max_bytes.
*
@@ -202,7 +212,7 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
dp = dsa_allocate0(area, sizeof(TidStoreControl));
ts->control = (TidStoreControl *) dsa_get_address(area, dp);
- ts->control->max_bytes = (uint64) (max_bytes * ratio);
+ ts->control->max_bytes = (size_t) (max_bytes * ratio);
ts->area = area;
ts->control->magic = TIDSTORE_MAGIC;
@@ -218,14 +228,14 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
ts->control->max_bytes = max_bytes - (70 * 1024);
}
- ts->control->max_offset = max_offset;
- ts->control->offset_nbits = pg_ceil_log2_32(max_offset);
+ ts->control->max_off = max_off;
+ ts->control->max_off_nbits = pg_ceil_log2_32(max_off);
- if (ts->control->offset_nbits < TIDSTORE_VALUE_NBITS)
- ts->control->offset_nbits = TIDSTORE_VALUE_NBITS;
+ if (ts->control->max_off_nbits < LOWER_OFFSET_NBITS)
+ ts->control->max_off_nbits = LOWER_OFFSET_NBITS;
- ts->control->offset_key_nbits =
- ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
+ ts->control->upper_off_nbits =
+ ts->control->max_off_nbits - LOWER_OFFSET_NBITS;
return ts;
}
@@ -235,7 +245,7 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
* allocated in backend-local memory using the CurrentMemoryContext.
*/
TidStore *
-tidstore_attach(dsa_area *area, tidstore_handle handle)
+TidStoreAttach(dsa_area *area, TidStoreHandle handle)
{
TidStore *ts;
dsa_pointer control;
@@ -266,7 +276,7 @@ tidstore_attach(dsa_area *area, tidstore_handle handle)
* to the operating system.
*/
void
-tidstore_detach(TidStore *ts)
+TidStoreDetach(TidStore *ts)
{
Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
@@ -279,12 +289,12 @@ tidstore_detach(TidStore *ts)
*
* TODO: The caller must be certain that no other backend will attempt to
* access the TidStore before calling this function. Other backend must
- * explicitly call tidstore_detach to free up backend-local memory associated
- * with the TidStore. The backend that calls tidstore_destroy must not call
- * tidstore_detach.
+ * explicitly call TidStoreDetach() to free up backend-local memory associated
+ * with the TidStore. The backend that calls TidStoreDestroy() must not call
+ * TidStoreDetach().
*/
void
-tidstore_destroy(TidStore *ts)
+TidStoreDestroy(TidStore *ts)
{
if (TidStoreIsShared(ts))
{
@@ -309,11 +319,11 @@ tidstore_destroy(TidStore *ts)
}
/*
- * Forget all collected Tids. It's similar to tidstore_destroy but we don't free
+ * Forget all collected Tids. It's similar to TidStoreDestroy() but we don't free
* entire TidStore but recreate only the radix tree storage.
*/
void
-tidstore_reset(TidStore *ts)
+TidStoreReset(TidStore *ts)
{
if (TidStoreIsShared(ts))
{
@@ -350,30 +360,34 @@ tidstore_reset(TidStore *ts)
}
}
-/* Add Tids on a block to TidStore */
+/*
+ * Set the given tids on the blkno to TidStore.
+ *
+ * NB: the offset numbers in offsets must be sorted in ascending order.
+ */
void
-tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
- int num_offsets)
+TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets)
{
- uint64 *values;
- uint64 key;
- uint64 prev_key;
- uint64 off_bitmap = 0;
+ offsetbm *bitmaps;
+ tidkey key;
+ tidkey prev_key;
+ offsetbm off_bitmap = 0;
int idx;
- const uint64 key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
- const int nkeys = UINT64CONST(1) << ts->control->offset_key_nbits;
+ const tidkey key_base = ((uint64) blkno) << ts->control->upper_off_nbits;
+ const int nkeys = UINT64CONST(1) << ts->control->upper_off_nbits;
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
- values = palloc(sizeof(uint64) * nkeys);
+ bitmaps = palloc(sizeof(offsetbm) * nkeys);
key = prev_key = key_base;
for (int i = 0; i < num_offsets; i++)
{
- uint64 off_bit;
+ offsetbm off_bit;
/* encode the tid to a key and partial offset */
- key = encode_key_off(ts, blkno, offsets[i], &off_bit);
+ key = encode_blk_off(ts, blkno, offsets[i], &off_bit);
/* make sure we scanned the line pointer array in order */
Assert(key >= prev_key);
@@ -384,11 +398,11 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
Assert(idx >= 0 && idx < nkeys);
/* write out offset bitmap for this key */
- values[idx] = off_bitmap;
+ bitmaps[idx] = off_bitmap;
/* zero out any gaps up to the current key */
for (int empty_idx = idx + 1; empty_idx < key - key_base; empty_idx++)
- values[empty_idx] = 0;
+ bitmaps[empty_idx] = 0;
/* reset for current key -- the current offset will be handled below */
off_bitmap = 0;
@@ -401,7 +415,7 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
/* save the final index for later */
idx = key - key_base;
/* write out last offset bitmap */
- values[idx] = off_bitmap;
+ bitmaps[idx] = off_bitmap;
if (TidStoreIsShared(ts))
LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
@@ -409,14 +423,14 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
/* insert the calculated key-values to the tree */
for (int i = 0; i <= idx; i++)
{
- if (values[i])
+ if (bitmaps[i])
{
key = key_base + i;
if (TidStoreIsShared(ts))
- shared_rt_set(ts->tree.shared, key, &values[i]);
+ shared_rt_set(ts->tree.shared, key, &bitmaps[i]);
else
- local_rt_set(ts->tree.local, key, &values[i]);
+ local_rt_set(ts->tree.local, key, &bitmaps[i]);
}
}
@@ -426,70 +440,70 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
if (TidStoreIsShared(ts))
LWLockRelease(&ts->control->lock);
- pfree(values);
+ pfree(bitmaps);
}
/* Return true if the given tid is present in the TidStore */
bool
-tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
+TidStoreIsMember(TidStore *ts, ItemPointer tid)
{
- uint64 key;
- uint64 val = 0;
- uint64 off_bit;
+ tidkey key;
+ offsetbm off_bitmap = 0;
+ offsetbm off_bit;
bool found;
- key = tid_to_key_off(ts, tid, &off_bit);
+ key = encode_tid(ts, tid, &off_bit);
if (TidStoreIsShared(ts))
- found = shared_rt_search(ts->tree.shared, key, &val);
+ found = shared_rt_search(ts->tree.shared, key, &off_bitmap);
else
- found = local_rt_search(ts->tree.local, key, &val);
+ found = local_rt_search(ts->tree.local, key, &off_bitmap);
if (!found)
return false;
- return (val & off_bit) != 0;
+ return (off_bitmap & off_bit) != 0;
}
/*
- * Prepare to iterate through a TidStore. Since the radix tree is locked during the
- * iteration, so tidstore_end_iterate() needs to called when finished.
+ * Prepare to iterate through a TidStore. Since the radix tree is locked during
+ * the iteration, so TidStoreEndIterate() needs to be called when finished.
+ *
+ * The TidStoreIter struct is created in the caller's memory context.
*
* Concurrent updates during the iteration will be blocked when inserting a
* key-value to the radix tree.
*/
TidStoreIter *
-tidstore_begin_iterate(TidStore *ts)
+TidStoreBeginIterate(TidStore *ts)
{
TidStoreIter *iter;
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
- iter = palloc0(sizeof(TidStoreIter));
+ iter = palloc0(sizeof(TidStoreIter) +
+ sizeof(OffsetNumber) * ts->control->max_off);
iter->ts = ts;
- iter->result.blkno = InvalidBlockNumber;
- iter->result.offsets = palloc(sizeof(OffsetNumber) * ts->control->max_offset);
-
if (TidStoreIsShared(ts))
iter->tree_iter.shared = shared_rt_begin_iterate(ts->tree.shared);
else
iter->tree_iter.local = local_rt_begin_iterate(ts->tree.local);
/* If the TidStore is empty, there is no business */
- if (tidstore_num_tids(ts) == 0)
+ if (TidStoreNumTids(ts) == 0)
iter->finished = true;
return iter;
}
static inline bool
-tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
+tidstore_iter(TidStoreIter *iter, tidkey *key, offsetbm *off_bitmap)
{
if (TidStoreIsShared(iter->ts))
- return shared_rt_iterate_next(iter->tree_iter.shared, key, val);
+ return shared_rt_iterate_next(iter->tree_iter.shared, key, off_bitmap);
- return local_rt_iterate_next(iter->tree_iter.local, key, val);
+ return local_rt_iterate_next(iter->tree_iter.local, key, off_bitmap);
}
/*
@@ -498,45 +512,48 @@ tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
* numbers in each result is also sorted in ascending order.
*/
TidStoreIterResult *
-tidstore_iterate_next(TidStoreIter *iter)
+TidStoreIterateNext(TidStoreIter *iter)
{
- uint64 key;
- uint64 val;
- TidStoreIterResult *result = &(iter->result);
+ tidkey key;
+ offsetbm off_bitmap = 0;
+ TidStoreIterResult *output = &(iter->output);
if (iter->finished)
return NULL;
- if (BlockNumberIsValid(result->blkno))
- {
- /* Process the previously collected key-value */
- result->num_offsets = 0;
- tidstore_iter_extract_tids(iter, iter->next_key, iter->next_val);
- }
+ /* Initialize the outputs */
+ output->blkno = InvalidBlockNumber;
+ output->num_offsets = 0;
- while (tidstore_iter_kv(iter, &key, &val))
- {
- BlockNumber blkno;
+ /*
+ * Decode the key and offset bitmap that are collected in the previous
+ * time, if exists.
+ */
+ if (iter->next_off_bitmap > 0)
+ iter_decode_key_off(iter, iter->next_tidkey, iter->next_off_bitmap);
- blkno = key_get_blkno(iter->ts, key);
+ while (tidstore_iter(iter, &key, &off_bitmap))
+ {
+ BlockNumber blkno = key_get_blkno(iter->ts, key);
- if (BlockNumberIsValid(result->blkno) && result->blkno != blkno)
+ if (BlockNumberIsValid(output->blkno) && output->blkno != blkno)
{
/*
- * We got a key-value pair for a different block. So return the
- * collected tids, and remember the key-value for the next iteration.
+ * We got tids for a different block. We return the collected
+ * tids so far, and remember the key-value for the next
+ * iteration.
*/
- iter->next_key = key;
- iter->next_val = val;
- return result;
+ iter->next_tidkey = key;
+ iter->next_off_bitmap = off_bitmap;
+ return output;
}
- /* Collect tids extracted from the key-value pair */
- tidstore_iter_extract_tids(iter, key, val);
+ /* Collect tids decoded from the key and offset bitmap */
+ iter_decode_key_off(iter, key, off_bitmap);
}
iter->finished = true;
- return result;
+ return output;
}
/*
@@ -544,22 +561,21 @@ tidstore_iterate_next(TidStoreIter *iter)
* or when existing an iteration.
*/
void
-tidstore_end_iterate(TidStoreIter *iter)
+TidStoreEndIterate(TidStoreIter *iter)
{
if (TidStoreIsShared(iter->ts))
shared_rt_end_iterate(iter->tree_iter.shared);
else
local_rt_end_iterate(iter->tree_iter.local);
- pfree(iter->result.offsets);
pfree(iter);
}
/* Return the number of tids we collected so far */
int64
-tidstore_num_tids(TidStore *ts)
+TidStoreNumTids(TidStore *ts)
{
- uint64 num_tids;
+ int64 num_tids;
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
@@ -575,16 +591,16 @@ tidstore_num_tids(TidStore *ts)
/* Return true if the current memory usage of TidStore exceeds the limit */
bool
-tidstore_is_full(TidStore *ts)
+TidStoreIsFull(TidStore *ts)
{
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
- return (tidstore_memory_usage(ts) > ts->control->max_bytes);
+ return (TidStoreMemoryUsage(ts) > ts->control->max_bytes);
}
/* Return the maximum memory TidStore can use */
size_t
-tidstore_max_memory(TidStore *ts)
+TidStoreMaxMemory(TidStore *ts)
{
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
@@ -593,7 +609,7 @@ tidstore_max_memory(TidStore *ts)
/* Return the memory usage of TidStore */
size_t
-tidstore_memory_usage(TidStore *ts)
+TidStoreMemoryUsage(TidStore *ts)
{
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
@@ -611,71 +627,75 @@ tidstore_memory_usage(TidStore *ts)
/*
* Get a handle that can be used by other processes to attach to this TidStore
*/
-tidstore_handle
-tidstore_get_handle(TidStore *ts)
+TidStoreHandle
+TidStoreGetHandle(TidStore *ts)
{
Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
return ts->control->handle;
}
-/* Extract tids from the given key-value pair */
+/*
+ * Decode the key and offset bitmap to tids and store them to the iteration
+ * result.
+ */
static void
-tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
+iter_decode_key_off(TidStoreIter *iter, tidkey key, offsetbm off_bitmap)
{
- TidStoreIterResult *result = (&iter->result);
+ TidStoreIterResult *output = (&iter->output);
- while (val)
+ while (off_bitmap)
{
- uint64 tid_i;
+ uint64 compressed_tid;
OffsetNumber off;
- tid_i = key << TIDSTORE_VALUE_NBITS;
- tid_i |= pg_rightmost_one_pos64(val);
+ compressed_tid = key << LOWER_OFFSET_NBITS;
+ compressed_tid |= pg_rightmost_one_pos64(off_bitmap);
- off = tid_i & ((UINT64CONST(1) << iter->ts->control->offset_nbits) - 1);
+ off = compressed_tid & ((UINT64CONST(1) << iter->ts->control->max_off_nbits) - 1);
- Assert(result->num_offsets < iter->ts->control->max_offset);
- result->offsets[result->num_offsets++] = off;
+ Assert(output->num_offsets < iter->ts->control->max_off);
+ output->offsets[output->num_offsets++] = off;
/* unset the rightmost bit */
- val &= ~pg_rightmost_one64(val);
+ off_bitmap &= ~pg_rightmost_one64(off_bitmap);
}
- result->blkno = key_get_blkno(iter->ts, key);
+ output->blkno = key_get_blkno(iter->ts, key);
}
/* Get block number from the given key */
static inline BlockNumber
-key_get_blkno(TidStore *ts, uint64 key)
+key_get_blkno(TidStore *ts, tidkey key)
{
- return (BlockNumber) (key >> ts->control->offset_key_nbits);
+ return (BlockNumber) (key >> ts->control->upper_off_nbits);
}
-/* Encode a tid to key and offset */
-static inline uint64
-tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit)
+/* Encode a tid to key and partial offset */
+static inline tidkey
+encode_tid(TidStore *ts, ItemPointer tid, offsetbm *off_bit)
{
- uint32 offset = ItemPointerGetOffsetNumber(tid);
+ OffsetNumber offset = ItemPointerGetOffsetNumber(tid);
BlockNumber block = ItemPointerGetBlockNumber(tid);
- return encode_key_off(ts, block, offset, off_bit);
+ return encode_blk_off(ts, block, offset, off_bit);
}
/* encode a block and offset to a key and partial offset */
-static inline uint64
-encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit)
+static inline tidkey
+encode_blk_off(TidStore *ts, BlockNumber block, OffsetNumber offset,
+ offsetbm *off_bit)
{
- uint64 key;
- uint64 tid_i;
+ tidkey key;
+ uint64 compressed_tid;
uint32 off_lower;
- off_lower = offset & TIDSTORE_OFFSET_MASK;
- Assert(off_lower < (sizeof(uint64) * BITS_PER_BYTE));
+ off_lower = offset & LOWER_OFFSET_MASK;
+ Assert(off_lower < (sizeof(offsetbm) * BITS_PER_BYTE));
*off_bit = UINT64CONST(1) << off_lower;
- tid_i = offset | ((uint64) block << ts->control->offset_nbits);
- key = tid_i >> TIDSTORE_VALUE_NBITS;
+ compressed_tid = offset | ((uint64) block << ts->control->max_off_nbits);
+ key = compressed_tid >> LOWER_OFFSET_NBITS;
return key;
}
diff --git a/src/include/access/tidstore.h b/src/include/access/tidstore.h
index a35a52124a..66f0fdd482 100644
--- a/src/include/access/tidstore.h
+++ b/src/include/access/tidstore.h
@@ -17,33 +17,34 @@
#include "storage/itemptr.h"
#include "utils/dsa.h"
-typedef dsa_pointer tidstore_handle;
+typedef dsa_pointer TidStoreHandle;
typedef struct TidStore TidStore;
typedef struct TidStoreIter TidStoreIter;
+/* Result struct for TidStoreIterateNext */
typedef struct TidStoreIterResult
{
BlockNumber blkno;
- OffsetNumber *offsets;
int num_offsets;
+ OffsetNumber offsets[FLEXIBLE_ARRAY_MEMBER];
} TidStoreIterResult;
-extern TidStore *tidstore_create(size_t max_bytes, int max_offset, dsa_area *dsa);
-extern TidStore *tidstore_attach(dsa_area *dsa, dsa_pointer handle);
-extern void tidstore_detach(TidStore *ts);
-extern void tidstore_destroy(TidStore *ts);
-extern void tidstore_reset(TidStore *ts);
-extern void tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
- int num_offsets);
-extern bool tidstore_lookup_tid(TidStore *ts, ItemPointer tid);
-extern TidStoreIter * tidstore_begin_iterate(TidStore *ts);
-extern TidStoreIterResult *tidstore_iterate_next(TidStoreIter *iter);
-extern void tidstore_end_iterate(TidStoreIter *iter);
-extern int64 tidstore_num_tids(TidStore *ts);
-extern bool tidstore_is_full(TidStore *ts);
-extern size_t tidstore_max_memory(TidStore *ts);
-extern size_t tidstore_memory_usage(TidStore *ts);
-extern tidstore_handle tidstore_get_handle(TidStore *ts);
+extern TidStore *TidStoreCreate(size_t max_bytes, int max_off, dsa_area *dsa);
+extern TidStore *TidStoreAttach(dsa_area *dsa, dsa_pointer handle);
+extern void TidStoreDetach(TidStore *ts);
+extern void TidStoreDestroy(TidStore *ts);
+extern void TidStoreReset(TidStore *ts);
+extern void TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets);
+extern bool TidStoreIsMember(TidStore *ts, ItemPointer tid);
+extern TidStoreIter * TidStoreBeginIterate(TidStore *ts);
+extern TidStoreIterResult *TidStoreIterateNext(TidStoreIter *iter);
+extern void TidStoreEndIterate(TidStoreIter *iter);
+extern int64 TidStoreNumTids(TidStore *ts);
+extern bool TidStoreIsFull(TidStore *ts);
+extern size_t TidStoreMaxMemory(TidStore *ts);
+extern size_t TidStoreMemoryUsage(TidStore *ts);
+extern TidStoreHandle TidStoreGetHandle(TidStore *ts);
#endif /* TIDSTORE_H */
diff --git a/src/test/modules/test_tidstore/test_tidstore.c b/src/test/modules/test_tidstore/test_tidstore.c
index 9a1217f833..8659e6780e 100644
--- a/src/test/modules/test_tidstore/test_tidstore.c
+++ b/src/test/modules/test_tidstore/test_tidstore.c
@@ -37,10 +37,10 @@ check_tid(TidStore *ts, BlockNumber blkno, OffsetNumber off, bool expect)
ItemPointerSet(&tid, blkno, off);
- found = tidstore_lookup_tid(ts, &tid);
+ found = TidStoreIsMember(ts, &tid);
if (found != expect)
- elog(ERROR, "lookup TID (%u, %u) returned %d, expected %d",
+ elog(ERROR, "TidStoreIsMember for TID (%u, %u) returned %d, expected %d",
blkno, off, found, expect);
}
@@ -69,9 +69,9 @@ test_basic(int max_offset)
LWLockRegisterTranche(tranche_id, "test_tidstore");
dsa = dsa_create(tranche_id);
- ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, dsa);
+ ts = TidStoreCreate(TEST_TIDSTORE_MAX_BYTES, max_offset, dsa);
#else
- ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
+ ts = TidStoreCreate(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
#endif
/* prepare the offset array */
@@ -83,7 +83,7 @@ test_basic(int max_offset)
/* add tids */
for (int i = 0; i < TEST_TIDSTORE_NUM_BLOCKS; i++)
- tidstore_add_tids(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
+ TidStoreSetBlockOffsets(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
/* lookup test */
for (OffsetNumber off = FirstOffsetNumber ; off < max_offset; off++)
@@ -105,30 +105,30 @@ test_basic(int max_offset)
}
/* test the number of tids */
- if (tidstore_num_tids(ts) != (TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS))
- elog(ERROR, "tidstore_num_tids returned " UINT64_FORMAT ", expected %d",
- tidstore_num_tids(ts),
+ if (TidStoreNumTids(ts) != (TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS))
+ elog(ERROR, "TidStoreNumTids returned " UINT64_FORMAT ", expected %d",
+ TidStoreNumTids(ts),
TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS);
/* iteration test */
- iter = tidstore_begin_iterate(ts);
+ iter = TidStoreBeginIterate(ts);
blk_idx = 0;
- while ((iter_result = tidstore_iterate_next(iter)) != NULL)
+ while ((iter_result = TidStoreIterateNext(iter)) != NULL)
{
/* check the returned block number */
if (blks_sorted[blk_idx] != iter_result->blkno)
- elog(ERROR, "tidstore_iterate_next returned block number %u, expected %u",
+ elog(ERROR, "TidStoreIterateNext returned block number %u, expected %u",
iter_result->blkno, blks_sorted[blk_idx]);
/* check the returned offset numbers */
if (TEST_TIDSTORE_NUM_OFFSETS != iter_result->num_offsets)
- elog(ERROR, "tidstore_iterate_next returned %u offsets, expected %u",
+ elog(ERROR, "TidStoreIterateNext %u offsets, expected %u",
iter_result->num_offsets, TEST_TIDSTORE_NUM_OFFSETS);
for (int i = 0; i < iter_result->num_offsets; i++)
{
if (offs[i] != iter_result->offsets[i])
- elog(ERROR, "tidstore_iterate_next returned offset number %u on block %u, expected %u",
+ elog(ERROR, "TidStoreIterateNext offset number %u on block %u, expected %u",
iter_result->offsets[i], iter_result->blkno, offs[i]);
}
@@ -136,15 +136,15 @@ test_basic(int max_offset)
}
if (blk_idx != TEST_TIDSTORE_NUM_BLOCKS)
- elog(ERROR, "tidstore_iterate_next returned %d blocks, expected %d",
+ elog(ERROR, "TidStoreIterateNext returned %d blocks, expected %d",
blk_idx, TEST_TIDSTORE_NUM_BLOCKS);
/* remove all tids */
- tidstore_reset(ts);
+ TidStoreReset(ts);
/* test the number of tids */
- if (tidstore_num_tids(ts) != 0)
- elog(ERROR, "tidstore_num_tids on empty store returned non-zero");
+ if (TidStoreNumTids(ts) != 0)
+ elog(ERROR, "TidStoreNumTids on empty store returned non-zero");
/* lookup test for empty store */
for (OffsetNumber off = FirstOffsetNumber ; off < MaxHeapTuplesPerPage;
@@ -156,7 +156,7 @@ test_basic(int max_offset)
check_tid(ts, MaxBlockNumber, off, false);
}
- tidstore_destroy(ts);
+ TidStoreDestroy(ts);
#ifdef TEST_SHARED_TIDSTORE
dsa_detach(dsa);
@@ -177,36 +177,37 @@ test_empty(void)
LWLockRegisterTranche(tranche_id, "test_tidstore");
dsa = dsa_create(tranche_id);
- ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, dsa);
+ ts = TidStoreCreate(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, dsa);
#else
- ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
+ ts = TidStoreCreate(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
#endif
elog(NOTICE, "testing empty tidstore");
ItemPointerSet(&tid, 0, FirstOffsetNumber);
- if (tidstore_lookup_tid(ts, &tid))
- elog(ERROR, "tidstore_lookup_tid for (0,1) on empty store returned true");
+ if (TidStoreIsMember(ts, &tid))
+ elog(ERROR, "TidStoreIsMember for TID (%u,%u) on empty store returned true",
+ 0, FirstOffsetNumber);
ItemPointerSet(&tid, MaxBlockNumber, MaxOffsetNumber);
- if (tidstore_lookup_tid(ts, &tid))
- elog(ERROR, "tidstore_lookup_tid for (%u,%u) on empty store returned true",
+ if (TidStoreIsMember(ts, &tid))
+ elog(ERROR, "TidStoreIsMember for TID (%u,%u) on empty store returned true",
MaxBlockNumber, MaxOffsetNumber);
- if (tidstore_num_tids(ts) != 0)
- elog(ERROR, "tidstore_num_entries on empty store returned non-zero");
+ if (TidStoreNumTids(ts) != 0)
+ elog(ERROR, "TidStoreNumTids on empty store returned non-zero");
- if (tidstore_is_full(ts))
- elog(ERROR, "tidstore_is_full on empty store returned true");
+ if (TidStoreIsFull(ts))
+ elog(ERROR, "TidStoreIsFull on empty store returned true");
- iter = tidstore_begin_iterate(ts);
+ iter = TidStoreBeginIterate(ts);
- if (tidstore_iterate_next(iter) != NULL)
- elog(ERROR, "tidstore_iterate_next on empty store returned TIDs");
+ if (TidStoreIterateNext(iter) != NULL)
+ elog(ERROR, "TidStoreIterateNext on empty store returned TIDs");
- tidstore_end_iterate(iter);
+ TidStoreEndIterate(iter);
- tidstore_destroy(ts);
+ TidStoreDestroy(ts);
#ifdef TEST_SHARED_TIDSTORE
dsa_detach(dsa);
@@ -221,6 +222,7 @@ test_tidstore(PG_FUNCTION_ARGS)
elog(NOTICE, "testing basic operations");
test_basic(MaxHeapTuplesPerPage);
test_basic(10);
+ test_basic(MaxHeapTuplesPerPage * 2);
PG_RETURN_VOID();
}
--
2.31.1
[application/octet-stream] v29-0005-Tool-for-measuring-radix-tree-and-tidstore-perfo.patch (24.7K, ../../CAD21AoAFiw2DS07nPhiS14eVXpnF-bBmjvab4y+Ct8igU2z_8A@mail.gmail.com/7-v29-0005-Tool-for-measuring-radix-tree-and-tidstore-perfo.patch)
download | inline diff:
From 848d68ee7c484a7041c6d0d703304cadfdfc36a2 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 16 Sep 2022 11:57:03 +0900
Subject: [PATCH v29 05/10] Tool for measuring radix tree and tidstore
performance
Includes Meson support, but commented out to avoid warnings
XXX: Not for commit
---
contrib/bench_radix_tree/Makefile | 21 +
.../bench_radix_tree--1.0.sql | 88 +++
contrib/bench_radix_tree/bench_radix_tree.c | 747 ++++++++++++++++++
.../bench_radix_tree/bench_radix_tree.control | 6 +
contrib/bench_radix_tree/expected/bench.out | 13 +
contrib/bench_radix_tree/meson.build | 33 +
contrib/bench_radix_tree/sql/bench.sql | 16 +
contrib/meson.build | 1 +
8 files changed, 925 insertions(+)
create mode 100644 contrib/bench_radix_tree/Makefile
create mode 100644 contrib/bench_radix_tree/bench_radix_tree--1.0.sql
create mode 100644 contrib/bench_radix_tree/bench_radix_tree.c
create mode 100644 contrib/bench_radix_tree/bench_radix_tree.control
create mode 100644 contrib/bench_radix_tree/expected/bench.out
create mode 100644 contrib/bench_radix_tree/meson.build
create mode 100644 contrib/bench_radix_tree/sql/bench.sql
diff --git a/contrib/bench_radix_tree/Makefile b/contrib/bench_radix_tree/Makefile
new file mode 100644
index 0000000000..952bb0ceae
--- /dev/null
+++ b/contrib/bench_radix_tree/Makefile
@@ -0,0 +1,21 @@
+# contrib/bench_radix_tree/Makefile
+
+MODULE_big = bench_radix_tree
+OBJS = \
+ bench_radix_tree.o
+
+EXTENSION = bench_radix_tree
+DATA = bench_radix_tree--1.0.sql
+
+REGRESS = bench_fixed_height
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/bench_radix_tree
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
new file mode 100644
index 0000000000..ad66265e23
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
@@ -0,0 +1,88 @@
+/* contrib/bench_radix_tree/bench_radix_tree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION bench_radix_tree" to load this file. \quit
+
+create function bench_shuffle_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_seq_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_load_random_int(
+cnt int8,
+OUT mem_allocated int8,
+OUT load_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_search_random_nodes(
+cnt int8,
+filter_str text DEFAULT NULL,
+OUT mem_allocated int8,
+OUT load_ms int8,
+OUT search_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C VOLATILE PARALLEL UNSAFE;
+
+create function bench_fixed_height_search(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT rt_search_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_node128_load(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_sparseload_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_tidstore_load(
+minblk int4,
+maxblk int4,
+OUT mem_allocated int8,
+OUT load_ms int8,
+OUT iter_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
new file mode 100644
index 0000000000..6e5149e2c4
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -0,0 +1,747 @@
+/*-------------------------------------------------------------------------
+ *
+ * bench_radix_tree.c
+ *
+ * Copyright (c) 2016-2022, PostgreSQL Global Development Group
+ *
+ * contrib/bench_radix_tree/bench_radix_tree.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/tidstore.h"
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "funcapi.h"
+#include "lib/radixtree.h"
+#include <math.h>
+#include "miscadmin.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/timestamp.h"
+
+PG_MODULE_MAGIC;
+
+/* run benchmark also for binary-search case? */
+/* #define MEASURE_BINARY_SEARCH 1 */
+
+#define TIDS_PER_BLOCK_FOR_LOAD 30
+#define TIDS_PER_BLOCK_FOR_LOOKUP 50
+
+//#define RT_DEBUG
+#define RT_PREFIX rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE uint64
+// WIP: compiles with warnings because rt_attach is defined but not used
+// #define RT_SHMEM
+#include "lib/radixtree.h"
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+ return tree->ctl->num_keys;
+}
+
+
+PG_FUNCTION_INFO_V1(bench_seq_search);
+PG_FUNCTION_INFO_V1(bench_shuffle_search);
+PG_FUNCTION_INFO_V1(bench_load_random_int);
+PG_FUNCTION_INFO_V1(bench_fixed_height_search);
+PG_FUNCTION_INFO_V1(bench_search_random_nodes);
+PG_FUNCTION_INFO_V1(bench_node128_load);
+PG_FUNCTION_INFO_V1(bench_tidstore_load);
+
+static uint64
+tid_to_key_off(ItemPointer tid, uint32 *off)
+{
+ uint64 upper;
+ uint32 shift = pg_ceil_log2_32(MaxHeapTuplesPerPage);
+ int64 tid_i;
+
+ Assert(ItemPointerGetOffsetNumber(tid) < MaxHeapTuplesPerPage);
+
+ tid_i = ItemPointerGetOffsetNumber(tid);
+ tid_i |= (uint64) ItemPointerGetBlockNumber(tid) << shift;
+
+ /* log(sizeof(uint64) * BITS_PER_BYTE, 2) = log(64, 2) = 6 */
+ *off = tid_i & ((1 << 6) - 1);
+ upper = tid_i >> 6;
+ Assert(*off < (sizeof(uint64) * BITS_PER_BYTE));
+
+ Assert(*off < 64);
+
+ return upper;
+}
+
+static int
+shuffle_randrange(pg_prng_state *state, int lower, int upper)
+{
+ return (int) floor(pg_prng_double(state) * ((upper - lower) + 0.999999)) + lower;
+}
+
+/* Naive Fisher-Yates implementation*/
+static void
+shuffle_itemptrs(ItemPointer itemptr, uint64 nitems)
+{
+ /* reproducability */
+ pg_prng_state state;
+
+ pg_prng_seed(&state, 0);
+
+ for (int i = 0; i < nitems - 1; i++)
+ {
+ int j = shuffle_randrange(&state, i, nitems - 1);
+ ItemPointerData t = itemptr[j];
+
+ itemptr[j] = itemptr[i];
+ itemptr[i] = t;
+ }
+}
+
+static ItemPointer
+generate_tids(BlockNumber minblk, BlockNumber maxblk, int ntids_per_blk, uint64 *ntids_p,
+ bool random_block)
+{
+ ItemPointer tids;
+ uint64 maxitems;
+ uint64 ntids = 0;
+ pg_prng_state state;
+
+ maxitems = (maxblk - minblk + 1) * ntids_per_blk;
+ tids = MemoryContextAllocHuge(TopTransactionContext,
+ sizeof(ItemPointerData) * maxitems);
+
+ if (random_block)
+ pg_prng_seed(&state, 0x9E3779B185EBCA87);
+
+ for (BlockNumber blk = minblk; blk < maxblk; blk++)
+ {
+ if (random_block && !pg_prng_bool(&state))
+ continue;
+
+ for (OffsetNumber off = FirstOffsetNumber;
+ off <= ntids_per_blk; off++)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ ItemPointerSetBlockNumber(&(tids[ntids]), blk);
+ ItemPointerSetOffsetNumber(&(tids[ntids]), off);
+
+ ntids++;
+ }
+ }
+
+ *ntids_p = ntids;
+ return tids;
+}
+
+#ifdef MEASURE_BINARY_SEARCH
+static int
+vac_cmp_itemptr(const void *left, const void *right)
+{
+ BlockNumber lblk,
+ rblk;
+ OffsetNumber loff,
+ roff;
+
+ lblk = ItemPointerGetBlockNumber((ItemPointer) left);
+ rblk = ItemPointerGetBlockNumber((ItemPointer) right);
+
+ if (lblk < rblk)
+ return -1;
+ if (lblk > rblk)
+ return 1;
+
+ loff = ItemPointerGetOffsetNumber((ItemPointer) left);
+ roff = ItemPointerGetOffsetNumber((ItemPointer) right);
+
+ if (loff < roff)
+ return -1;
+ if (loff > roff)
+ return 1;
+
+ return 0;
+}
+#endif
+
+Datum
+bench_tidstore_load(PG_FUNCTION_ARGS)
+{
+ BlockNumber minblk = PG_GETARG_INT32(0);
+ BlockNumber maxblk = PG_GETARG_INT32(1);
+ TidStore *ts;
+ TidStoreIter *iter;
+ TidStoreIterResult *result;
+ OffsetNumber *offs;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_ms;
+ int64 iter_ms;
+ TupleDesc tupdesc;
+ Datum values[3];
+ bool nulls[3] = {false};
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ offs = palloc(sizeof(OffsetNumber) * TIDS_PER_BLOCK_FOR_LOAD);
+ for (int i = 0; i < TIDS_PER_BLOCK_FOR_LOAD; i++)
+ offs[i] = i + 1; /* FirstOffsetNumber is 1 */
+
+ ts = tidstore_create(1 * 1024L * 1024L * 1024L, MaxHeapTuplesPerPage, NULL);
+
+ /* load tids */
+ start_time = GetCurrentTimestamp();
+ for (BlockNumber blkno = minblk; blkno < maxblk; blkno++)
+ tidstore_add_tids(ts, blkno, offs, TIDS_PER_BLOCK_FOR_LOAD);
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_ms = secs * 1000 + usecs / 1000;
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* iterate through tids */
+ iter = tidstore_begin_iterate(ts);
+ start_time = GetCurrentTimestamp();
+ while ((result = tidstore_iterate_next(iter)) != NULL)
+ ;
+ tidstore_end_iterate(iter);
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ iter_ms = secs * 1000 + usecs / 1000;
+
+ values[0] = Int64GetDatum(tidstore_memory_usage(ts));
+ values[1] = Int64GetDatum(load_ms);
+ values[2] = Int64GetDatum(iter_ms);
+
+ tidstore_destroy(ts);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+static Datum
+bench_search(FunctionCallInfo fcinfo, bool shuffle)
+{
+ BlockNumber minblk = PG_GETARG_INT32(0);
+ BlockNumber maxblk = PG_GETARG_INT32(1);
+ bool random_block = PG_GETARG_BOOL(2);
+ rt_radix_tree *rt = NULL;
+ uint64 ntids;
+ uint64 key;
+ uint64 last_key = PG_UINT64_MAX;
+ uint64 val = 0;
+ ItemPointer tids;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_load_ms,
+ rt_search_ms;
+ Datum values[7];
+ bool nulls[7];
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ tids = generate_tids(minblk, maxblk, TIDS_PER_BLOCK_FOR_LOAD, &ntids, random_block);
+
+ /* measure the load time of the radix tree */
+ rt = rt_create(CurrentMemoryContext);
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ uint32 off;
+
+ CHECK_FOR_INTERRUPTS();
+
+ key = tid_to_key_off(tid, &off);
+
+ if (last_key != PG_UINT64_MAX && last_key != key)
+ {
+ rt_set(rt, last_key, &val);
+ val = 0;
+ }
+
+ last_key = key;
+ val |= (uint64) 1 << off;
+ }
+ if (last_key != PG_UINT64_MAX)
+ rt_set(rt, last_key, &val);
+
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_load_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+ if (shuffle)
+ shuffle_itemptrs(tids, ntids);
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* meaure the serach time of the radix tree */
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ uint32 off;
+ volatile bool ret; /* prevent calling rt_search from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ key = tid_to_key_off(tid, &off);
+
+ ret = rt_search(rt, key, &val);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_search_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int64GetDatum(rt_num_entries(rt));
+ values[1] = Int64GetDatum(rt_memory_usage(rt));
+ values[2] = Int64GetDatum(sizeof(ItemPointerData) * ntids);
+ values[3] = Int64GetDatum(rt_load_ms);
+ nulls[4] = true; /* ar_load_ms */
+ values[5] = Int64GetDatum(rt_search_ms);
+ nulls[6] = true; /* ar_search_ms */
+
+#ifdef MEASURE_BINARY_SEARCH
+ {
+ ItemPointer itemptrs = NULL;
+
+ int64 ar_load_ms,
+ ar_search_ms;
+
+ /* measure the load time of the array */
+ itemptrs = MemoryContextAllocHuge(CurrentMemoryContext,
+ sizeof(ItemPointerData) * ntids);
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointerSetBlockNumber(&(itemptrs[i]),
+ ItemPointerGetBlockNumber(&(tids[i])));
+ ItemPointerSetOffsetNumber(&(itemptrs[i]),
+ ItemPointerGetOffsetNumber(&(tids[i])));
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ ar_load_ms = secs * 1000 + usecs / 1000;
+
+ /* next, measure the serach time of the array */
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ volatile bool ret; /* prevent calling bsearch from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ ret = bsearch((void *) tid,
+ (void *) itemptrs,
+ ntids,
+ sizeof(ItemPointerData),
+ vac_cmp_itemptr);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ ar_search_ms = secs * 1000 + usecs / 1000;
+
+ /* set the result */
+ nulls[4] = false;
+ values[4] = Int64GetDatum(ar_load_ms);
+ nulls[6] = false;
+ values[6] = Int64GetDatum(ar_search_ms);
+ }
+#endif
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_seq_search(PG_FUNCTION_ARGS)
+{
+ return bench_search(fcinfo, false);
+}
+
+Datum
+bench_shuffle_search(PG_FUNCTION_ARGS)
+{
+ return bench_search(fcinfo, true);
+}
+
+Datum
+bench_load_random_int(PG_FUNCTION_ARGS)
+{
+ uint64 cnt = (uint64) PG_GETARG_INT64(0);
+ rt_radix_tree *rt;
+ pg_prng_state state;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_time_ms;
+ Datum values[2];
+ bool nulls[2];
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ pg_prng_seed(&state, 0);
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 key = pg_prng_uint64(&state);
+
+ rt_set(rt, key, &key);
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_time_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int64GetDatum(rt_memory_usage(rt));
+ values[1] = Int64GetDatum(load_time_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+/* copy of splitmix64() */
+static uint64
+hash64(uint64 x)
+{
+ x ^= x >> 30;
+ x *= UINT64CONST(0xbf58476d1ce4e5b9);
+ x ^= x >> 27;
+ x *= UINT64CONST(0x94d049bb133111eb);
+ x ^= x >> 31;
+ return x;
+}
+
+/* attempts to have a relatively even population of node kinds */
+Datum
+bench_search_random_nodes(PG_FUNCTION_ARGS)
+{
+ uint64 cnt = (uint64) PG_GETARG_INT64(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_time_ms;
+ int64 search_time_ms;
+ Datum values[3] = {0};
+ bool nulls[3] = {0};
+ /* from trial and error */
+ uint64 filter = (((uint64) 0x7F<<32) | (0x07<<24) | (0xFF<<16) | 0xFF);
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (!PG_ARGISNULL(1))
+ {
+ char *filter_str = text_to_cstring(PG_GETARG_TEXT_P(1));
+
+ if (sscanf(filter_str, "0x%lX", &filter) == 0)
+ elog(ERROR, "invalid filter string %s", filter_str);
+ }
+ elog(NOTICE, "bench with filter 0x%lX", filter);
+
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 hash = hash64(i);
+ uint64 key = hash & filter;
+
+ rt_set(rt, key, &key);
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_time_ms = secs * 1000 + usecs / 1000;
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 hash = hash64(i);
+ uint64 key = hash & filter;
+ uint64 val;
+ volatile bool ret; /* prevent calling rt_search from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ ret = rt_search(rt, key, &val);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ search_time_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ values[0] = Int64GetDatum(rt_memory_usage(rt));
+ values[1] = Int64GetDatum(load_time_ms);
+ values[2] = Int64GetDatum(search_time_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_fixed_height_search(PG_FUNCTION_ARGS)
+{
+ int fanout = PG_GETARG_INT32(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_load_ms,
+ rt_search_ms;
+ Datum values[5];
+ bool nulls[5];
+
+ /* test boundary between vector and iteration */
+ const int n_keys = 5 * 16 * 16 * 16 * 16;
+ uint64 r,
+ h,
+ i,
+ j,
+ k;
+ uint64 key_id;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+
+ key_id = 0;
+
+ /*
+ * lower nodes have limited fanout, the top is only limited by
+ * bits-per-byte
+ */
+ for (r = 1;; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ for (i = 1; i <= fanout; i++)
+ {
+ for (j = 1; j <= fanout; j++)
+ {
+ for (k = 1; k <= fanout; k++)
+ {
+ uint64 key;
+
+ key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+ CHECK_FOR_INTERRUPTS();
+
+ key_id++;
+ if (key_id > n_keys)
+ goto finish_set;
+
+ rt_set(rt, key, &key_id);
+ }
+ }
+ }
+ }
+ }
+finish_set:
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_load_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ /* meaure the search time of the radix tree */
+ start_time = GetCurrentTimestamp();
+
+ key_id = 0;
+ for (r = 1;; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ for (i = 1; i <= fanout; i++)
+ {
+ for (j = 1; j <= fanout; j++)
+ {
+ for (k = 1; k <= fanout; k++)
+ {
+ uint64 key,
+ val;
+
+ key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+ CHECK_FOR_INTERRUPTS();
+
+ key_id++;
+ if (key_id > n_keys)
+ goto finish_search;
+
+ rt_search(rt, key, &val);
+ }
+ }
+ }
+ }
+ }
+finish_search:
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_search_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int32GetDatum(fanout);
+ values[1] = Int64GetDatum(rt_num_entries(rt));
+ values[2] = Int64GetDatum(rt_memory_usage(rt));
+ values[3] = Int64GetDatum(rt_load_ms);
+ values[4] = Int64GetDatum(rt_search_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_node128_load(PG_FUNCTION_ARGS)
+{
+ int fanout = PG_GETARG_INT32(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_sparseload_ms;
+ Datum values[5];
+ bool nulls[5];
+
+ uint64 r,
+ h;
+ uint64 key_id;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ rt = rt_create(CurrentMemoryContext);
+
+ key_id = 0;
+
+ for (r = 1; r <= fanout; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (h);
+
+ key_id++;
+ rt_set(rt, key, &key_id);
+ }
+ }
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ /* measure sparse deletion and re-loading */
+ start_time = GetCurrentTimestamp();
+
+ for (int t = 0; t<10000; t++)
+ {
+ /* delete one key in each leaf */
+ for (r = 1; r <= fanout; r++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (fanout);
+
+ rt_delete(rt, key);
+ }
+
+ /* add them all back */
+ key_id = 0;
+ for (r = 1; r <= fanout; r++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (fanout);
+
+ key_id++;
+ rt_set(rt, key, &key_id);
+ }
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_sparseload_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int32GetDatum(fanout);
+ values[1] = Int64GetDatum(rt_num_entries(rt));
+ values[2] = Int64GetDatum(rt_memory_usage(rt));
+ values[3] = Int64GetDatum(rt_sparseload_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+/* to silence warnings about unused iter functions */
+static void pg_attribute_unused()
+stub_iter()
+{
+ rt_radix_tree *rt;
+ rt_iter *iter;
+ uint64 key = 1;
+ uint64 value = 1;
+
+ rt = rt_create(CurrentMemoryContext);
+
+ iter = rt_begin_iterate(rt);
+ rt_iterate_next(iter, &key, &value);
+ rt_end_iterate(iter);
+}
\ No newline at end of file
diff --git a/contrib/bench_radix_tree/bench_radix_tree.control b/contrib/bench_radix_tree/bench_radix_tree.control
new file mode 100644
index 0000000000..1d988e6c9a
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.control
@@ -0,0 +1,6 @@
+# bench_radix_tree extension
+comment = 'benchmark suits for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/bench_radix_tree'
+relocatable = true
+trusted = true
diff --git a/contrib/bench_radix_tree/expected/bench.out b/contrib/bench_radix_tree/expected/bench.out
new file mode 100644
index 0000000000..60c303892e
--- /dev/null
+++ b/contrib/bench_radix_tree/expected/bench.out
@@ -0,0 +1,13 @@
+create extension bench_radix_tree;
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/bench_radix_tree/meson.build b/contrib/bench_radix_tree/meson.build
new file mode 100644
index 0000000000..332c1ae7df
--- /dev/null
+++ b/contrib/bench_radix_tree/meson.build
@@ -0,0 +1,33 @@
+bench_radix_tree_sources = files(
+ 'bench_radix_tree.c',
+)
+
+if host_system == 'windows'
+ bench_radix_tree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'bench_radix_tree',
+ '--FILEDESC', 'bench_radix_tree - performance test code for radix tree',])
+endif
+
+bench_radix_tree = shared_module('bench_radix_tree',
+ bench_radix_tree_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += bench_radix_tree
+
+install_data(
+ 'bench_radix_tree.control',
+ 'bench_radix_tree--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'bench_radix_tree',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'bench_radix_tree',
+ ],
+ },
+}
diff --git a/contrib/bench_radix_tree/sql/bench.sql b/contrib/bench_radix_tree/sql/bench.sql
new file mode 100644
index 0000000000..a46018c9d4
--- /dev/null
+++ b/contrib/bench_radix_tree/sql/bench.sql
@@ -0,0 +1,16 @@
+create extension bench_radix_tree;
+
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/meson.build b/contrib/meson.build
index bd4a57c43c..421d469f8c 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,6 +12,7 @@ subdir('amcheck')
subdir('auth_delay')
subdir('auto_explain')
subdir('basic_archive')
+subdir('bench_radix_tree')
subdir('bloom')
subdir('basebackup_to_shell')
subdir('bool_plperl')
--
2.31.1
[application/octet-stream] v29-0002-Move-some-bitmap-logic-out-of-bitmapset.c.patch (6.1K, ../../CAD21AoAFiw2DS07nPhiS14eVXpnF-bBmjvab4y+Ct8igU2z_8A@mail.gmail.com/8-v29-0002-Move-some-bitmap-logic-out-of-bitmapset.c.patch)
download | inline diff:
From 39f0e713854942fbad3678bce9138adea546f1be Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 6 Dec 2022 13:39:41 +0700
Subject: [PATCH v29 02/10] Move some bitmap logic out of bitmapset.c
Add pg_rightmost_one32/64 functions. This functionality was previously
private to bitmapset.c as the RIGHTMOST_ONE macro. It has practical
use in other contexts, so move to pg_bitutils.h.
Also move the logic for selecting appropriate pg_bitutils functions
based on word size to bitmapset.h for wider visibility and add
appropriate selection for bmw_rightmost_one().
Since the previous macro relied on casting to signedbitmapword,
and the new functions do not, remove that typedef.
Design input and review by Tom Lane
Discussion: https://www.postgresql.org/message-id/CAFBsxsFW2JjTo58jtDB%2B3sZhxMx3t-3evew8%3DAcr%2BGGhC%2BkFaA%40mail.gmail.com
---
src/backend/nodes/bitmapset.c | 36 ++------------------------------
src/include/nodes/bitmapset.h | 16 ++++++++++++--
src/include/port/pg_bitutils.h | 31 +++++++++++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 -
4 files changed, 47 insertions(+), 37 deletions(-)
diff --git a/src/backend/nodes/bitmapset.c b/src/backend/nodes/bitmapset.c
index 98660524ad..fcd8e2ccbc 100644
--- a/src/backend/nodes/bitmapset.c
+++ b/src/backend/nodes/bitmapset.c
@@ -32,39 +32,7 @@
#define BITMAPSET_SIZE(nwords) \
(offsetof(Bitmapset, words) + (nwords) * sizeof(bitmapword))
-/*----------
- * This is a well-known cute trick for isolating the rightmost one-bit
- * in a word. It assumes two's complement arithmetic. Consider any
- * nonzero value, and focus attention on the rightmost one. The value is
- * then something like
- * xxxxxx10000
- * where x's are unspecified bits. The two's complement negative is formed
- * by inverting all the bits and adding one. Inversion gives
- * yyyyyy01111
- * where each y is the inverse of the corresponding x. Incrementing gives
- * yyyyyy10000
- * and then ANDing with the original value gives
- * 00000010000
- * This works for all cases except original value = zero, where of course
- * we get zero.
- *----------
- */
-#define RIGHTMOST_ONE(x) ((signedbitmapword) (x) & -((signedbitmapword) (x)))
-
-#define HAS_MULTIPLE_ONES(x) ((bitmapword) RIGHTMOST_ONE(x) != (x))
-
-/* Select appropriate bit-twiddling functions for bitmap word size */
-#if BITS_PER_BITMAPWORD == 32
-#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w)
-#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w)
-#define bmw_popcount(w) pg_popcount32(w)
-#elif BITS_PER_BITMAPWORD == 64
-#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w)
-#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w)
-#define bmw_popcount(w) pg_popcount64(w)
-#else
-#error "invalid BITS_PER_BITMAPWORD"
-#endif
+#define HAS_MULTIPLE_ONES(x) (bmw_rightmost_one(x) != (x))
/*
@@ -1013,7 +981,7 @@ bms_first_member(Bitmapset *a)
{
int result;
- w = RIGHTMOST_ONE(w);
+ w = bmw_rightmost_one(w);
a->words[wordnum] &= ~w;
result = wordnum * BITS_PER_BITMAPWORD;
diff --git a/src/include/nodes/bitmapset.h b/src/include/nodes/bitmapset.h
index 3d2225e1ae..5f9a511b4a 100644
--- a/src/include/nodes/bitmapset.h
+++ b/src/include/nodes/bitmapset.h
@@ -38,13 +38,11 @@ struct List;
#define BITS_PER_BITMAPWORD 64
typedef uint64 bitmapword; /* must be an unsigned type */
-typedef int64 signedbitmapword; /* must be the matching signed type */
#else
#define BITS_PER_BITMAPWORD 32
typedef uint32 bitmapword; /* must be an unsigned type */
-typedef int32 signedbitmapword; /* must be the matching signed type */
#endif
@@ -75,6 +73,20 @@ typedef enum
BMS_MULTIPLE /* >1 member */
} BMS_Membership;
+/* Select appropriate bit-twiddling functions for bitmap word size */
+#if BITS_PER_BITMAPWORD == 32
+#define bmw_rightmost_one(w) pg_rightmost_one32(w)
+#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w)
+#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w)
+#define bmw_popcount(w) pg_popcount32(w)
+#elif BITS_PER_BITMAPWORD == 64
+#define bmw_rightmost_one(w) pg_rightmost_one64(w)
+#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w)
+#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w)
+#define bmw_popcount(w) pg_popcount64(w)
+#else
+#error "invalid BITS_PER_BITMAPWORD"
+#endif
/*
* function prototypes in nodes/bitmapset.c
diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h
index a49a9c03d9..7235ad25ee 100644
--- a/src/include/port/pg_bitutils.h
+++ b/src/include/port/pg_bitutils.h
@@ -17,6 +17,37 @@ extern PGDLLIMPORT const uint8 pg_leftmost_one_pos[256];
extern PGDLLIMPORT const uint8 pg_rightmost_one_pos[256];
extern PGDLLIMPORT const uint8 pg_number_of_ones[256];
+/*----------
+ * This is a well-known cute trick for isolating the rightmost one-bit
+ * in a word. It assumes two's complement arithmetic. Consider any
+ * nonzero value, and focus attention on the rightmost one. The value is
+ * then something like
+ * xxxxxx10000
+ * where x's are unspecified bits. The two's complement negative is formed
+ * by inverting all the bits and adding one. Inversion gives
+ * yyyyyy01111
+ * where each y is the inverse of the corresponding x. Incrementing gives
+ * yyyyyy10000
+ * and then ANDing with the original value gives
+ * 00000010000
+ * This works for all cases except original value = zero, where of course
+ * we get zero.
+ *----------
+ */
+static inline uint32
+pg_rightmost_one32(uint32 word)
+{
+ int32 result = (int32) word & -((int32) word);
+ return (uint32) result;
+}
+
+static inline uint64
+pg_rightmost_one64(uint64 word)
+{
+ int64 result = (int64) word & -((int64) word);
+ return (uint64) result;
+}
+
/*
* pg_leftmost_one_pos32
* Returns the position of the most significant set bit in "word",
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 45fc5759ce..f95d3dfd69 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3670,7 +3670,6 @@ shmem_request_hook_type
shmem_startup_hook_type
sig_atomic_t
sigjmp_buf
-signedbitmapword
sigset_t
size_t
slist_head
--
2.31.1
[application/octet-stream] v29-0001-Introduce-helper-SIMD-functions-for-small-byte-a.patch (2.9K, ../../CAD21AoAFiw2DS07nPhiS14eVXpnF-bBmjvab4y+Ct8igU2z_8A@mail.gmail.com/9-v29-0001-Introduce-helper-SIMD-functions-for-small-byte-a.patch)
download | inline diff:
From 58f8e2b82eb196d463114d8ec3dad343b2b027e0 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 24 Oct 2022 14:07:09 +0900
Subject: [PATCH v29 01/10] Introduce helper SIMD functions for small byte
arrays
vector8_min - helper for emulating ">=" semantics
vector8_highbit_mask - used to turn the result of a vector
comparison into a bitmask
Masahiko Sawada
Reviewed by Nathan Bossart, additional adjustments by me
Discussion: https://www.postgresql.org/message-id/CAD21AoDap240WDDdUDE0JMpCmuMMnGajrKrkCRxM7zn9Xk3JRA%40mail.gmail.com
---
src/include/port/simd.h | 47 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)
diff --git a/src/include/port/simd.h b/src/include/port/simd.h
index 1fa6c3bc6c..dfae14e463 100644
--- a/src/include/port/simd.h
+++ b/src/include/port/simd.h
@@ -79,6 +79,7 @@ static inline bool vector8_has_le(const Vector8 v, const uint8 c);
static inline bool vector8_is_highbit_set(const Vector8 v);
#ifndef USE_NO_SIMD
static inline bool vector32_is_highbit_set(const Vector32 v);
+static inline uint32 vector8_highbit_mask(const Vector8 v);
#endif
/* arithmetic operations */
@@ -96,6 +97,7 @@ static inline Vector8 vector8_ssub(const Vector8 v1, const Vector8 v2);
*/
#ifndef USE_NO_SIMD
static inline Vector8 vector8_eq(const Vector8 v1, const Vector8 v2);
+static inline Vector8 vector8_min(const Vector8 v1, const Vector8 v2);
static inline Vector32 vector32_eq(const Vector32 v1, const Vector32 v2);
#endif
@@ -299,6 +301,36 @@ vector32_is_highbit_set(const Vector32 v)
}
#endif /* ! USE_NO_SIMD */
+/*
+ * Return a bitmask formed from the high-bit of each element.
+ */
+#ifndef USE_NO_SIMD
+static inline uint32
+vector8_highbit_mask(const Vector8 v)
+{
+#ifdef USE_SSE2
+ return (uint32) _mm_movemask_epi8(v);
+#elif defined(USE_NEON)
+ /*
+ * Note: There is a faster way to do this, but it returns a uint64 and
+ * and if the caller wanted to extract the bit position using CTZ,
+ * it would have to divide that result by 4.
+ */
+ static const uint8 mask[16] = {
+ 1 << 0, 1 << 1, 1 << 2, 1 << 3,
+ 1 << 4, 1 << 5, 1 << 6, 1 << 7,
+ 1 << 0, 1 << 1, 1 << 2, 1 << 3,
+ 1 << 4, 1 << 5, 1 << 6, 1 << 7,
+ };
+
+ uint8x16_t masked = vandq_u8(vld1q_u8(mask), (uint8x16_t) vshrq_n_s8(v, 7));
+ uint8x16_t maskedhi = vextq_u8(masked, masked, 8);
+
+ return (uint32) vaddvq_u16((uint16x8_t) vzip1q_u8(masked, maskedhi));
+#endif
+}
+#endif /* ! USE_NO_SIMD */
+
/*
* Return the bitwise OR of the inputs
*/
@@ -372,4 +404,19 @@ vector32_eq(const Vector32 v1, const Vector32 v2)
}
#endif /* ! USE_NO_SIMD */
+/*
+ * Given two vectors, return a vector with the minimum element of each.
+ */
+#ifndef USE_NO_SIMD
+static inline Vector8
+vector8_min(const Vector8 v1, const Vector8 v2)
+{
+#ifdef USE_SSE2
+ return _mm_min_epu8(v1, v2);
+#elif defined(USE_NEON)
+ return vminq_u8(v1, v2);
+#endif
+}
+#endif /* ! USE_NO_SIMD */
+
#endif /* SIMD_H */
--
2.31.1
[application/octet-stream] v29-0003-Add-radixtree-template.patch (117.0K, ../../CAD21AoAFiw2DS07nPhiS14eVXpnF-bBmjvab4y+Ct8igU2z_8A@mail.gmail.com/10-v29-0003-Add-radixtree-template.patch)
download | inline diff:
From ab33774676db3e419dd56b2001f0cbf2bc291d3d Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 14 Sep 2022 12:38:51 +0000
Subject: [PATCH v29 03/10] Add radixtree template
WIP: commit message based on template comments
---
src/backend/utils/mmgr/dsa.c | 12 +
src/include/lib/radixtree.h | 2516 +++++++++++++++++
src/include/lib/radixtree_delete_impl.h | 122 +
src/include/lib/radixtree_insert_impl.h | 328 +++
src/include/lib/radixtree_iter_impl.h | 153 +
src/include/lib/radixtree_search_impl.h | 138 +
src/include/utils/dsa.h | 1 +
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_radixtree/.gitignore | 4 +
src/test/modules/test_radixtree/Makefile | 23 +
src/test/modules/test_radixtree/README | 7 +
.../expected/test_radixtree.out | 36 +
src/test/modules/test_radixtree/meson.build | 35 +
.../test_radixtree/sql/test_radixtree.sql | 7 +
.../test_radixtree/test_radixtree--1.0.sql | 8 +
.../modules/test_radixtree/test_radixtree.c | 681 +++++
.../test_radixtree/test_radixtree.control | 4 +
src/tools/pginclude/cpluspluscheck | 6 +
src/tools/pginclude/headerscheck | 6 +
20 files changed, 4089 insertions(+)
create mode 100644 src/include/lib/radixtree.h
create mode 100644 src/include/lib/radixtree_delete_impl.h
create mode 100644 src/include/lib/radixtree_insert_impl.h
create mode 100644 src/include/lib/radixtree_iter_impl.h
create mode 100644 src/include/lib/radixtree_search_impl.h
create mode 100644 src/test/modules/test_radixtree/.gitignore
create mode 100644 src/test/modules/test_radixtree/Makefile
create mode 100644 src/test/modules/test_radixtree/README
create mode 100644 src/test/modules/test_radixtree/expected/test_radixtree.out
create mode 100644 src/test/modules/test_radixtree/meson.build
create mode 100644 src/test/modules/test_radixtree/sql/test_radixtree.sql
create mode 100644 src/test/modules/test_radixtree/test_radixtree--1.0.sql
create mode 100644 src/test/modules/test_radixtree/test_radixtree.c
create mode 100644 src/test/modules/test_radixtree/test_radixtree.control
diff --git a/src/backend/utils/mmgr/dsa.c b/src/backend/utils/mmgr/dsa.c
index f5a62061a3..80555aefff 100644
--- a/src/backend/utils/mmgr/dsa.c
+++ b/src/backend/utils/mmgr/dsa.c
@@ -1024,6 +1024,18 @@ dsa_set_size_limit(dsa_area *area, size_t limit)
LWLockRelease(DSA_AREA_LOCK(area));
}
+size_t
+dsa_get_total_size(dsa_area *area)
+{
+ size_t size;
+
+ LWLockAcquire(DSA_AREA_LOCK(area), LW_SHARED);
+ size = area->control->total_segment_size;
+ LWLockRelease(DSA_AREA_LOCK(area));
+
+ return size;
+}
+
/*
* Aggressively free all spare memory in the hope of returning DSM segments to
* the operating system.
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
new file mode 100644
index 0000000000..e546bd705c
--- /dev/null
+++ b/src/include/lib/radixtree.h
@@ -0,0 +1,2516 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree.h
+ * Template for adaptive radix tree.
+ *
+ * This module employs the idea from the paper "The Adaptive Radix Tree: ARTful
+ * Indexing for Main-Memory Databases" by Viktor Leis, Alfons Kemper, and Thomas
+ * Neumann, 2013. The radix tree uses adaptive node sizes, a small number of node
+ * types, each with a different numbers of elements. Depending on the number of
+ * children, the appropriate node type is used.
+ *
+ * WIP: notes about traditional radix tree trading off span vs height...
+ *
+ * There are two kinds of nodes, inner nodes and leaves. Inner nodes
+ * map partial keys to child pointers.
+ *
+ * The ART paper mentions three ways to implement leaves:
+ *
+ * "- Single-value leaves: The values are stored using an addi-
+ * tional leaf node type which stores one value.
+ * - Multi-value leaves: The values are stored in one of four
+ * different leaf node types, which mirror the structure of
+ * inner nodes, but contain values instead of pointers.
+ * - Combined pointer/value slots: If values fit into point-
+ * ers, no separate node types are necessary. Instead, each
+ * pointer storage location in an inner node can either
+ * store a pointer or a value."
+ *
+ * We chose "multi-value leaves" to avoid the additional pointer traversal
+ * required by "single-value leaves"
+ *
+ * For simplicity, the key is assumed to be 64-bit unsigned integer. The
+ * tree doesn't need to contain paths where the highest bytes of all keys
+ * are zero. That way, the tree's height adapts to the distribution of keys.
+ *
+ * TODO: In the future it might be worthwhile to offer configurability of
+ * leaf implementation for different use cases. Single-values leaves would
+ * give more flexibility in key type, including variable-length keys.
+ *
+ * There are some optimizations not yet implemented, particularly path
+ * compression and lazy path expansion.
+ *
+ * To handle concurrency, we use a single reader-writer lock for the radix
+ * tree. The radix tree is exclusively locked during write operations such
+ * as RT_SET() and RT_DELETE(), and shared locked during read operations
+ * such as RT_SEARCH(). An iteration also holds the shared lock on the radix
+ * tree until it is completed.
+ *
+ * TODO: The current locking mechanism is not optimized for high concurrency
+ * with mixed read-write workloads. In the future it might be worthwhile
+ * to replace it with the Optimistic Lock Coupling or ROWEX mentioned in
+ * the paper "The ART of Practical Synchronization" by the same authors as
+ * the ART paper, 2016.
+ *
+ * WIP: the radix tree nodes don't shrink.
+ *
+ * To generate a radix tree and associated functions for a use case several
+ * macros have to be #define'ed before this file is included. Including
+ * the file #undef's all those, so a new radix tree can be generated
+ * afterwards.
+ * The relevant parameters are:
+ * - RT_PREFIX - prefix for all symbol names generated. A prefix of 'foo'
+ * will result in radix tree type 'foo_radix_tree' and functions like
+ * 'foo_create'/'foo_free' and so forth.
+ * - RT_DECLARE - if defined function prototypes and type declarations are
+ * generated
+ * - RT_DEFINE - if defined function definitions are generated
+ * - RT_SCOPE - in which scope (e.g. extern, static inline) do function
+ * declarations reside
+ * - RT_VALUE_TYPE - the type of the value.
+ *
+ * Optional parameters:
+ * - RT_SHMEM - if defined, the radix tree is created in the DSA area
+ * so that multiple processes can access it simultaneously.
+ * - RT_DEBUG - if defined add stats tracking and debugging functions
+ *
+ * Interface
+ * ---------
+ *
+ * RT_CREATE - Create a new, empty radix tree
+ * RT_FREE - Free the radix tree
+ * RT_SEARCH - Search a key-value pair
+ * RT_SET - Set a key-value pair
+ * RT_BEGIN_ITERATE - Begin iterating through all key-value pairs
+ * RT_ITERATE_NEXT - Return next key-value pair, if any
+ * RT_END_ITER - End iteration
+ * RT_MEMORY_USAGE - Get the memory usage
+ *
+ * Interface for Shared Memory
+ * ---------
+ *
+ * RT_ATTACH - Attach to the radix tree
+ * RT_DETACH - Detach from the radix tree
+ * RT_GET_HANDLE - Return the handle of the radix tree
+ *
+ * Optional Interface
+ * ---------
+ *
+ * RT_DELETE - Delete a key-value pair. Declared/define if RT_USE_DELETE is defined
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/include/lib/radixtree.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/stringinfo.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "port/pg_bitutils.h"
+#include "port/simd.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/* helpers */
+#define RT_MAKE_PREFIX(a) CppConcat(a,_)
+#define RT_MAKE_NAME(name) RT_MAKE_NAME_(RT_MAKE_PREFIX(RT_PREFIX),name)
+#define RT_MAKE_NAME_(a,b) CppConcat(a,b)
+
+/* function declarations */
+#define RT_CREATE RT_MAKE_NAME(create)
+#define RT_FREE RT_MAKE_NAME(free)
+#define RT_SEARCH RT_MAKE_NAME(search)
+#ifdef RT_SHMEM
+#define RT_ATTACH RT_MAKE_NAME(attach)
+#define RT_DETACH RT_MAKE_NAME(detach)
+#define RT_GET_HANDLE RT_MAKE_NAME(get_handle)
+#endif
+#define RT_SET RT_MAKE_NAME(set)
+#define RT_BEGIN_ITERATE RT_MAKE_NAME(begin_iterate)
+#define RT_ITERATE_NEXT RT_MAKE_NAME(iterate_next)
+#define RT_END_ITERATE RT_MAKE_NAME(end_iterate)
+#ifdef RT_USE_DELETE
+#define RT_DELETE RT_MAKE_NAME(delete)
+#endif
+#define RT_MEMORY_USAGE RT_MAKE_NAME(memory_usage)
+#ifdef RT_DEBUG
+#define RT_DUMP RT_MAKE_NAME(dump)
+#define RT_DUMP_NODE RT_MAKE_NAME(dump_node)
+#define RT_DUMP_SEARCH RT_MAKE_NAME(dump_search)
+#define RT_STATS RT_MAKE_NAME(stats)
+#endif
+
+/* internal helper functions (no externally visible prototypes) */
+#define RT_NEW_ROOT RT_MAKE_NAME(new_root)
+#define RT_ALLOC_NODE RT_MAKE_NAME(alloc_node)
+#define RT_INIT_NODE RT_MAKE_NAME(init_node)
+#define RT_FREE_NODE RT_MAKE_NAME(free_node)
+#define RT_FREE_RECURSE RT_MAKE_NAME(free_recurse)
+#define RT_EXTEND RT_MAKE_NAME(extend)
+#define RT_SET_EXTEND RT_MAKE_NAME(set_extend)
+#define RT_SWITCH_NODE_KIND RT_MAKE_NAME(grow_node_kind)
+#define RT_COPY_NODE RT_MAKE_NAME(copy_node)
+#define RT_REPLACE_NODE RT_MAKE_NAME(replace_node)
+#define RT_PTR_GET_LOCAL RT_MAKE_NAME(ptr_get_local)
+#define RT_PTR_ALLOC_IS_VALID RT_MAKE_NAME(ptr_stored_is_valid)
+#define RT_NODE_3_SEARCH_EQ RT_MAKE_NAME(node_3_search_eq)
+#define RT_NODE_32_SEARCH_EQ RT_MAKE_NAME(node_32_search_eq)
+#define RT_NODE_3_GET_INSERTPOS RT_MAKE_NAME(node_3_get_insertpos)
+#define RT_NODE_32_GET_INSERTPOS RT_MAKE_NAME(node_32_get_insertpos)
+#define RT_CHUNK_CHILDREN_ARRAY_SHIFT RT_MAKE_NAME(chunk_children_array_shift)
+#define RT_CHUNK_VALUES_ARRAY_SHIFT RT_MAKE_NAME(chunk_values_array_shift)
+#define RT_CHUNK_CHILDREN_ARRAY_DELETE RT_MAKE_NAME(chunk_children_array_delete)
+#define RT_CHUNK_VALUES_ARRAY_DELETE RT_MAKE_NAME(chunk_values_array_delete)
+#define RT_CHUNK_CHILDREN_ARRAY_COPY RT_MAKE_NAME(chunk_children_array_copy)
+#define RT_CHUNK_VALUES_ARRAY_COPY RT_MAKE_NAME(chunk_values_array_copy)
+#define RT_NODE_125_IS_CHUNK_USED RT_MAKE_NAME(node_125_is_chunk_used)
+#define RT_NODE_INNER_125_GET_CHILD RT_MAKE_NAME(node_inner_125_get_child)
+#define RT_NODE_LEAF_125_GET_VALUE RT_MAKE_NAME(node_leaf_125_get_value)
+#define RT_NODE_INNER_256_IS_CHUNK_USED RT_MAKE_NAME(node_inner_256_is_chunk_used)
+#define RT_NODE_LEAF_256_IS_CHUNK_USED RT_MAKE_NAME(node_leaf_256_is_chunk_used)
+#define RT_NODE_INNER_256_GET_CHILD RT_MAKE_NAME(node_inner_256_get_child)
+#define RT_NODE_LEAF_256_GET_VALUE RT_MAKE_NAME(node_leaf_256_get_value)
+#define RT_NODE_INNER_256_SET RT_MAKE_NAME(node_inner_256_set)
+#define RT_NODE_LEAF_256_SET RT_MAKE_NAME(node_leaf_256_set)
+#define RT_NODE_INNER_256_DELETE RT_MAKE_NAME(node_inner_256_delete)
+#define RT_NODE_LEAF_256_DELETE RT_MAKE_NAME(node_leaf_256_delete)
+#define RT_KEY_GET_SHIFT RT_MAKE_NAME(key_get_shift)
+#define RT_SHIFT_GET_MAX_VAL RT_MAKE_NAME(shift_get_max_val)
+#define RT_NODE_SEARCH_INNER RT_MAKE_NAME(node_search_inner)
+#define RT_NODE_SEARCH_LEAF RT_MAKE_NAME(node_search_leaf)
+#define RT_NODE_UPDATE_INNER RT_MAKE_NAME(node_update_inner)
+#define RT_NODE_DELETE_INNER RT_MAKE_NAME(node_delete_inner)
+#define RT_NODE_DELETE_LEAF RT_MAKE_NAME(node_delete_leaf)
+#define RT_NODE_INSERT_INNER RT_MAKE_NAME(node_insert_inner)
+#define RT_NODE_INSERT_LEAF RT_MAKE_NAME(node_insert_leaf)
+#define RT_NODE_INNER_ITERATE_NEXT RT_MAKE_NAME(node_inner_iterate_next)
+#define RT_NODE_LEAF_ITERATE_NEXT RT_MAKE_NAME(node_leaf_iterate_next)
+#define RT_UPDATE_ITER_STACK RT_MAKE_NAME(update_iter_stack)
+#define RT_ITER_UPDATE_KEY RT_MAKE_NAME(iter_update_key)
+#define RT_VERIFY_NODE RT_MAKE_NAME(verify_node)
+
+/* type declarations */
+#define RT_RADIX_TREE RT_MAKE_NAME(radix_tree)
+#define RT_RADIX_TREE_CONTROL RT_MAKE_NAME(radix_tree_control)
+#define RT_ITER RT_MAKE_NAME(iter)
+#ifdef RT_SHMEM
+#define RT_HANDLE RT_MAKE_NAME(handle)
+#endif
+#define RT_NODE RT_MAKE_NAME(node)
+#define RT_NODE_ITER RT_MAKE_NAME(node_iter)
+#define RT_NODE_BASE_3 RT_MAKE_NAME(node_base_3)
+#define RT_NODE_BASE_32 RT_MAKE_NAME(node_base_32)
+#define RT_NODE_BASE_125 RT_MAKE_NAME(node_base_125)
+#define RT_NODE_BASE_256 RT_MAKE_NAME(node_base_256)
+#define RT_NODE_INNER_3 RT_MAKE_NAME(node_inner_3)
+#define RT_NODE_INNER_32 RT_MAKE_NAME(node_inner_32)
+#define RT_NODE_INNER_125 RT_MAKE_NAME(node_inner_125)
+#define RT_NODE_INNER_256 RT_MAKE_NAME(node_inner_256)
+#define RT_NODE_LEAF_3 RT_MAKE_NAME(node_leaf_3)
+#define RT_NODE_LEAF_32 RT_MAKE_NAME(node_leaf_32)
+#define RT_NODE_LEAF_125 RT_MAKE_NAME(node_leaf_125)
+#define RT_NODE_LEAF_256 RT_MAKE_NAME(node_leaf_256)
+#define RT_SIZE_CLASS RT_MAKE_NAME(size_class)
+#define RT_SIZE_CLASS_ELEM RT_MAKE_NAME(size_class_elem)
+#define RT_SIZE_CLASS_INFO RT_MAKE_NAME(size_class_info)
+#define RT_CLASS_3 RT_MAKE_NAME(class_3)
+#define RT_CLASS_32_MIN RT_MAKE_NAME(class_32_min)
+#define RT_CLASS_32_MAX RT_MAKE_NAME(class_32_max)
+#define RT_CLASS_125 RT_MAKE_NAME(class_125)
+#define RT_CLASS_256 RT_MAKE_NAME(class_256)
+
+/* generate forward declarations necessary to use the radix tree */
+#ifdef RT_DECLARE
+
+typedef struct RT_RADIX_TREE RT_RADIX_TREE;
+typedef struct RT_ITER RT_ITER;
+
+#ifdef RT_SHMEM
+typedef dsa_pointer RT_HANDLE;
+#endif
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id);
+RT_SCOPE RT_RADIX_TREE * RT_ATTACH(dsa_area *dsa, dsa_pointer dp);
+RT_SCOPE void RT_DETACH(RT_RADIX_TREE *tree);
+RT_SCOPE RT_HANDLE RT_GET_HANDLE(RT_RADIX_TREE *tree);
+#else
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx);
+#endif
+RT_SCOPE void RT_FREE(RT_RADIX_TREE *tree);
+
+RT_SCOPE bool RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p);
+RT_SCOPE bool RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p);
+#ifdef RT_USE_DELETE
+RT_SCOPE bool RT_DELETE(RT_RADIX_TREE *tree, uint64 key);
+#endif
+
+RT_SCOPE RT_ITER * RT_BEGIN_ITERATE(RT_RADIX_TREE *tree);
+RT_SCOPE bool RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p);
+RT_SCOPE void RT_END_ITERATE(RT_ITER *iter);
+
+RT_SCOPE uint64 RT_MEMORY_USAGE(RT_RADIX_TREE *tree);
+
+#ifdef RT_DEBUG
+RT_SCOPE void RT_DUMP(RT_RADIX_TREE *tree);
+RT_SCOPE void RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key);
+RT_SCOPE void RT_STATS(RT_RADIX_TREE *tree);
+#endif
+
+#endif /* RT_DECLARE */
+
+
+/* generate implementation of the radix tree */
+#ifdef RT_DEFINE
+
+/* The number of bits encoded in one tree level */
+#define RT_NODE_SPAN BITS_PER_BYTE
+
+/* The number of maximum slots in the node */
+#define RT_NODE_MAX_SLOTS (1 << RT_NODE_SPAN)
+
+/* Mask for extracting a chunk from the key */
+#define RT_CHUNK_MASK ((1 << RT_NODE_SPAN) - 1)
+
+/* Maximum shift the radix tree uses */
+#define RT_MAX_SHIFT RT_KEY_GET_SHIFT(UINT64_MAX)
+
+/* Tree level the radix tree uses */
+#define RT_MAX_LEVEL ((sizeof(uint64) * BITS_PER_BYTE) / RT_NODE_SPAN)
+
+/*
+ * Number of bits necessary for isset array in the slot-index node.
+ * Since bitmapword can be 64 bits, the only values that make sense
+ * here are 64 and 128.
+ */
+#define RT_SLOT_IDX_LIMIT (RT_NODE_MAX_SLOTS / 2)
+
+/* Invalid index used in node-125 */
+#define RT_INVALID_SLOT_IDX 0xFF
+
+/* Get a chunk from the key */
+#define RT_GET_KEY_CHUNK(key, shift) ((uint8) (((key) >> (shift)) & RT_CHUNK_MASK))
+
+/* For accessing bitmaps */
+#define RT_BM_IDX(x) ((x) / BITS_PER_BITMAPWORD)
+#define RT_BM_BIT(x) ((x) % BITS_PER_BITMAPWORD)
+
+/*
+ * Node kinds
+ *
+ * The different node kinds are what make the tree "adaptive".
+ *
+ * Each node kind is associated with a different datatype and different
+ * search/set/delete/iterate algorithms adapted for its size. The largest
+ * kind, node256 is basically the same as a traditional radix tree,
+ * and would be most wasteful of memory when sparsely populated. The
+ * smaller nodes expend some additional CPU time to enable a smaller
+ * memory footprint.
+ *
+ * XXX There are 4 node kinds, and this should never be increased,
+ * for several reasons:
+ * 1. With 5 or more kinds, gcc tends to use a jump table for switch
+ * statements.
+ * 2. The 4 kinds can be represented with 2 bits, so we have the option
+ * in the future to tag the node pointer with the kind, even on
+ * platforms with 32-bit pointers. This might speed up node traversal
+ * in trees with highly random node kinds.
+ * 3. We can have multiple size classes per node kind.
+ */
+#define RT_NODE_KIND_3 0x00
+#define RT_NODE_KIND_32 0x01
+#define RT_NODE_KIND_125 0x02
+#define RT_NODE_KIND_256 0x03
+#define RT_NODE_KIND_COUNT 4
+
+/*
+ * Calculate the slab blocksize so that we can allocate at least 32 chunks
+ * from the block.
+ */
+#define RT_SLAB_BLOCK_SIZE(size) \
+ Max((SLAB_DEFAULT_BLOCK_SIZE / (size)) * (size), (size) * 32)
+
+/* Common type for all nodes types */
+typedef struct RT_NODE
+{
+ /*
+ * Number of children. We use uint16 to be able to indicate 256 children
+ * at the fanout of 8.
+ */
+ uint16 count;
+
+ /*
+ * Max capacity for the current size class. Storing this in the
+ * node enables multiple size classes per node kind.
+ * Technically, kinds with a single size class don't need this, so we could
+ * keep this in the individual base types, but the code is simpler this way.
+ * Note: node256 is unique in that it cannot possibly have more than a
+ * single size class, so for that kind we store zero, and uint8 is
+ * sufficient for other kinds.
+ */
+ uint8 fanout;
+
+ /*
+ * Shift indicates which part of the key space is represented by this
+ * node. That is, the key is shifted by 'shift' and the lowest
+ * RT_NODE_SPAN bits are then represented in chunk.
+ */
+ uint8 shift;
+
+ /* Node kind, one per search/set algorithm */
+ uint8 kind;
+} RT_NODE;
+
+
+#define RT_PTR_LOCAL RT_NODE *
+
+#ifdef RT_SHMEM
+#define RT_PTR_ALLOC dsa_pointer
+#else
+#define RT_PTR_ALLOC RT_PTR_LOCAL
+#endif
+
+
+#ifdef RT_SHMEM
+#define RT_INVALID_PTR_ALLOC InvalidDsaPointer
+#else
+#define RT_INVALID_PTR_ALLOC NULL
+#endif
+
+#ifdef RT_SHMEM
+#define RT_LOCK_EXCLUSIVE(tree) LWLockAcquire(&tree->ctl->lock, LW_EXCLUSIVE)
+#define RT_LOCK_SHARED(tree) LWLockAcquire(&tree->ctl->lock, LW_SHARED)
+#define RT_UNLOCK(tree) LWLockRelease(&tree->ctl->lock);
+#else
+#define RT_LOCK_EXCLUSIVE(tree) ((void) 0)
+#define RT_LOCK_SHARED(tree) ((void) 0)
+#define RT_UNLOCK(tree) ((void) 0)
+#endif
+
+/*
+ * Inner nodes and leaf nodes have analogous structure. To distinguish
+ * them at runtime, we take advantage of the fact that the key chunk
+ * is accessed by shifting: Inner tree nodes (shift > 0), store the
+ * pointer to its child node in the slot. In leaf nodes (shift == 0),
+ * the slot contains the value corresponding to the key.
+ */
+#define RT_NODE_IS_LEAF(n) (((RT_PTR_LOCAL) (n))->shift == 0)
+
+#define RT_NODE_MUST_GROW(node) \
+ ((node)->base.n.count == (node)->base.n.fanout)
+
+/*
+ * Base type of each node kinds for leaf and inner nodes.
+ * The base types must be a be able to accommodate the largest size
+ * class for variable-sized node kinds.
+ */
+typedef struct RT_NODE_BASE_3
+{
+ RT_NODE n;
+
+ /* 3 children, for key chunks */
+ uint8 chunks[3];
+} RT_NODE_BASE_3;
+
+typedef struct RT_NODE_BASE_32
+{
+ RT_NODE n;
+
+ /* 32 children, for key chunks */
+ uint8 chunks[32];
+} RT_NODE_BASE_32;
+
+/*
+ * node-125 uses slot_idx array, an array of RT_NODE_MAX_SLOTS length
+ * to store indexes into a second array that contains the values (or
+ * child pointers).
+ */
+typedef struct RT_NODE_BASE_125
+{
+ RT_NODE n;
+
+ /* The index of slots for each fanout */
+ uint8 slot_idxs[RT_NODE_MAX_SLOTS];
+
+ /* bitmap to track which slots are in use */
+ bitmapword isset[RT_BM_IDX(RT_SLOT_IDX_LIMIT)];
+} RT_NODE_BASE_125;
+
+typedef struct RT_NODE_BASE_256
+{
+ RT_NODE n;
+} RT_NODE_BASE_256;
+
+/*
+ * Inner and leaf nodes.
+ *
+ * These are separate because the value type might be different than
+ * something fitting into a pointer-width type.
+ */
+typedef struct RT_NODE_INNER_3
+{
+ RT_NODE_BASE_3 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_3;
+
+typedef struct RT_NODE_LEAF_3
+{
+ RT_NODE_BASE_3 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_3;
+
+typedef struct RT_NODE_INNER_32
+{
+ RT_NODE_BASE_32 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_32;
+
+typedef struct RT_NODE_LEAF_32
+{
+ RT_NODE_BASE_32 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_32;
+
+typedef struct RT_NODE_INNER_125
+{
+ RT_NODE_BASE_125 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_125;
+
+typedef struct RT_NODE_LEAF_125
+{
+ RT_NODE_BASE_125 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_125;
+
+/*
+ * node-256 is the largest node type. This node has an array
+ * for directly storing values (or child pointers in inner nodes).
+ * Unlike other node kinds, it's array size is by definition
+ * fixed.
+ */
+typedef struct RT_NODE_INNER_256
+{
+ RT_NODE_BASE_256 base;
+
+ /* Slots for 256 children */
+ RT_PTR_ALLOC children[RT_NODE_MAX_SLOTS];
+} RT_NODE_INNER_256;
+
+typedef struct RT_NODE_LEAF_256
+{
+ RT_NODE_BASE_256 base;
+
+ /*
+ * Unlike with inner256, zero is a valid value here, so we use a
+ * bitmap to track which slots are in use.
+ */
+ bitmapword isset[RT_BM_IDX(RT_NODE_MAX_SLOTS)];
+
+ /* Slots for 256 values */
+ RT_VALUE_TYPE values[RT_NODE_MAX_SLOTS];
+} RT_NODE_LEAF_256;
+
+/*
+ * Node size classes
+ *
+ * Nodes of different kinds necessarily belong to different size classes.
+ * The main innovation in our implementation compared to the ART paper
+ * is decoupling the notion of size class from kind.
+ *
+ * The size classes within a given node kind have the same underlying
+ * type, but a variable number of children/values. This is possible
+ * because the base type contains small fixed data structures that
+ * work the same way regardless of how full the node is. We store the
+ * node's allocated capacity in the "fanout" member of RT_NODE, to allow
+ * runtime introspection.
+ *
+ * Growing from one node kind to another requires special code for each
+ * case, but growing from one size class to another within the same kind
+ * is basically just allocate + memcpy.
+ *
+ * The size classes have been chosen so that inner nodes on platforms
+ * with 64-bit pointers (and leaf nodes when using a 64-bit key) are
+ * equal to or slightly smaller than some DSA size class.
+ */
+typedef enum RT_SIZE_CLASS
+{
+ RT_CLASS_3 = 0,
+ RT_CLASS_32_MIN,
+ RT_CLASS_32_MAX,
+ RT_CLASS_125,
+ RT_CLASS_256
+} RT_SIZE_CLASS;
+
+/* Information for each size class */
+typedef struct RT_SIZE_CLASS_ELEM
+{
+ const char *name;
+ int fanout;
+
+ /* slab chunk size */
+ Size inner_size;
+ Size leaf_size;
+} RT_SIZE_CLASS_ELEM;
+
+static const RT_SIZE_CLASS_ELEM RT_SIZE_CLASS_INFO[] = {
+ [RT_CLASS_3] = {
+ .name = "radix tree node 3",
+ .fanout = 3,
+ .inner_size = sizeof(RT_NODE_INNER_3) + 3 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_3) + 3 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_32_MIN] = {
+ .name = "radix tree node 15",
+ .fanout = 15,
+ .inner_size = sizeof(RT_NODE_INNER_32) + 15 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_32) + 15 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_32_MAX] = {
+ .name = "radix tree node 32",
+ .fanout = 32,
+ .inner_size = sizeof(RT_NODE_INNER_32) + 32 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_32) + 32 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_125] = {
+ .name = "radix tree node 125",
+ .fanout = 125,
+ .inner_size = sizeof(RT_NODE_INNER_125) + 125 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_125) + 125 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_256] = {
+ .name = "radix tree node 256",
+ .fanout = 256,
+ .inner_size = sizeof(RT_NODE_INNER_256),
+ .leaf_size = sizeof(RT_NODE_LEAF_256),
+ },
+};
+
+#define RT_SIZE_CLASS_COUNT lengthof(RT_SIZE_CLASS_INFO)
+
+#ifdef RT_SHMEM
+/* A magic value used to identify our radix tree */
+#define RT_RADIX_TREE_MAGIC 0x54A48167
+#endif
+
+/* Contains the actual tree and ancillary info */
+// WIP: this name is a bit strange
+typedef struct RT_RADIX_TREE_CONTROL
+{
+#ifdef RT_SHMEM
+ RT_HANDLE handle;
+ uint32 magic;
+ LWLock lock;
+#endif
+
+ RT_PTR_ALLOC root;
+ uint64 max_val;
+ uint64 num_keys;
+
+ /* statistics */
+#ifdef RT_DEBUG
+ int32 cnt[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE_CONTROL;
+
+/* Entry point for allocating and accessing the tree */
+typedef struct RT_RADIX_TREE
+{
+ MemoryContext context;
+
+ /* pointing to either local memory or DSA */
+ RT_RADIX_TREE_CONTROL *ctl;
+
+#ifdef RT_SHMEM
+ dsa_area *dsa;
+#else
+ MemoryContextData *inner_slabs[RT_SIZE_CLASS_COUNT];
+ MemoryContextData *leaf_slabs[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE;
+
+/*
+ * Iteration support.
+ *
+ * Iterating the radix tree returns each pair of key and value in the ascending
+ * order of the key. To support this, the we iterate nodes of each level.
+ *
+ * RT_NODE_ITER struct is used to track the iteration within a node.
+ *
+ * RT_ITER is the struct for iteration of the radix tree, and uses RT_NODE_ITER
+ * in order to track the iteration of each level. During iteration, we also
+ * construct the key whenever updating the node iteration information, e.g., when
+ * advancing the current index within the node or when moving to the next node
+ * at the same level.
+ *
+ * XXX: Currently we allow only one process to do iteration. Therefore, rt_node_iter
+ * has the local pointers to nodes, rather than RT_PTR_ALLOC.
+ * We need either a safeguard to disallow other processes to begin the iteration
+ * while one process is doing or to allow multiple processes to do the iteration.
+ */
+typedef struct RT_NODE_ITER
+{
+ RT_PTR_LOCAL node; /* current node being iterated */
+ int current_idx; /* current position. -1 for initial value */
+} RT_NODE_ITER;
+
+typedef struct RT_ITER
+{
+ RT_RADIX_TREE *tree;
+
+ /* Track the iteration on nodes of each level */
+ RT_NODE_ITER stack[RT_MAX_LEVEL];
+ int stack_len;
+
+ /* The key is constructed during iteration */
+ uint64 key;
+} RT_ITER;
+
+
+static void RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_PTR_ALLOC child);
+static bool RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_VALUE_TYPE *value_p);
+
+/* verification (available only with assertion) */
+static void RT_VERIFY_NODE(RT_PTR_LOCAL node);
+
+/* Get the local address of an allocated node */
+static inline RT_PTR_LOCAL
+RT_PTR_GET_LOCAL(RT_RADIX_TREE *tree, RT_PTR_ALLOC node)
+{
+#ifdef RT_SHMEM
+ return dsa_get_address(tree->dsa, (dsa_pointer) node);
+#else
+ return node;
+#endif
+}
+
+static inline bool
+RT_PTR_ALLOC_IS_VALID(RT_PTR_ALLOC ptr)
+{
+#ifdef RT_SHMEM
+ return DsaPointerIsValid(ptr);
+#else
+ return PointerIsValid(ptr);
+#endif
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_3_SEARCH_EQ(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+ int idx = -1;
+
+ for (int i = 0; i < node->n.count; i++)
+ {
+ if (node->chunks[i] == chunk)
+ {
+ idx = i;
+ break;
+ }
+ }
+
+ return idx;
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_3_GET_INSERTPOS(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+ int idx;
+
+ for (idx = 0; idx < node->n.count; idx++)
+ {
+ if (node->chunks[idx] >= chunk)
+ break;
+ }
+
+ return idx;
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_32_SEARCH_EQ(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+ int count = node->n.count;
+#ifndef USE_NO_SIMD
+ Vector8 spread_chunk;
+ Vector8 haystack1;
+ Vector8 haystack2;
+ Vector8 cmp1;
+ Vector8 cmp2;
+ uint32 bitfield;
+ int index_simd = -1;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+ int index = -1;
+
+ for (int i = 0; i < count; i++)
+ {
+ if (node->chunks[i] == chunk)
+ {
+ index = i;
+ break;
+ }
+ }
+#endif
+
+#ifndef USE_NO_SIMD
+ /* replicate the search key */
+ spread_chunk = vector8_broadcast(chunk);
+
+ /* compare to all 32 keys stored in the node */
+ vector8_load(&haystack1, &node->chunks[0]);
+ vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+ cmp1 = vector8_eq(spread_chunk, haystack1);
+ cmp2 = vector8_eq(spread_chunk, haystack2);
+
+ /* convert comparison to a bitfield */
+ bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+
+ /* mask off invalid entries */
+ bitfield &= ((UINT64CONST(1) << count) - 1);
+
+ /* convert bitfield to index by counting trailing zeros */
+ if (bitfield)
+ index_simd = pg_rightmost_one_pos32(bitfield);
+
+ Assert(index_simd == index);
+ return index_simd;
+#else
+ return index;
+#endif
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_32_GET_INSERTPOS(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+ int count = node->n.count;
+#ifndef USE_NO_SIMD
+ Vector8 spread_chunk;
+ Vector8 haystack1;
+ Vector8 haystack2;
+ Vector8 cmp1;
+ Vector8 cmp2;
+ Vector8 min1;
+ Vector8 min2;
+ uint32 bitfield;
+ int index_simd;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+ int index;
+
+ for (index = 0; index < count; index++)
+ {
+ /*
+ * This is coded with '>=' to match what we can do with SIMD,
+ * with an assert to keep us honest.
+ */
+ if (node->chunks[index] >= chunk)
+ {
+ Assert(node->chunks[index] != chunk);
+ break;
+ }
+ }
+#endif
+
+#ifndef USE_NO_SIMD
+ /*
+ * This is a bit more complicated than RT_NODE_32_SEARCH_EQ(), because
+ * no unsigned uint8 comparison instruction exists, at least for SSE2. So
+ * we need to play some trickery using vector8_min() to effectively get
+ * >=. There'll never be any equal elements in current uses, but that's
+ * what we get here...
+ */
+ spread_chunk = vector8_broadcast(chunk);
+ vector8_load(&haystack1, &node->chunks[0]);
+ vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+ min1 = vector8_min(spread_chunk, haystack1);
+ min2 = vector8_min(spread_chunk, haystack2);
+ cmp1 = vector8_eq(spread_chunk, min1);
+ cmp2 = vector8_eq(spread_chunk, min2);
+ bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+ bitfield &= ((UINT64CONST(1) << count) - 1);
+
+ if (bitfield)
+ index_simd = pg_rightmost_one_pos32(bitfield);
+ else
+ index_simd = count;
+
+ Assert(index_simd == index);
+ return index_simd;
+#else
+ return index;
+#endif
+}
+
+
+/*
+ * Functions to manipulate both chunks array and children/values array.
+ * These are used for node-3 and node-32.
+ */
+
+/* Shift the elements right at 'idx' by one */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_SHIFT(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+ memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+ memmove(&(children[idx + 1]), &(children[idx]), sizeof(RT_PTR_ALLOC) * (count - idx));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_SHIFT(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+ memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+ memmove(&(values[idx + 1]), &(values[idx]), sizeof(RT_VALUE_TYPE) * (count - idx));
+}
+
+/* Delete the element at 'idx' */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_DELETE(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+ memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+ memmove(&(children[idx]), &(children[idx + 1]), sizeof(RT_PTR_ALLOC) * (count - idx - 1));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_DELETE(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+ memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+ memmove(&(values[idx]), &(values[idx + 1]), sizeof(RT_VALUE_TYPE) * (count - idx - 1));
+}
+
+/* Copy both chunks and children/values arrays */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_COPY(uint8 *src_chunks, RT_PTR_ALLOC *src_children,
+ uint8 *dst_chunks, RT_PTR_ALLOC *dst_children)
+{
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const Size chunk_size = sizeof(uint8) * fanout;
+ const Size children_size = sizeof(RT_PTR_ALLOC) * fanout;
+
+ memcpy(dst_chunks, src_chunks, chunk_size);
+ memcpy(dst_children, src_children, children_size);
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_COPY(uint8 *src_chunks, RT_VALUE_TYPE *src_values,
+ uint8 *dst_chunks, RT_VALUE_TYPE *dst_values)
+{
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const Size chunk_size = sizeof(uint8) * fanout;
+ const Size values_size = sizeof(RT_VALUE_TYPE) * fanout;
+
+ memcpy(dst_chunks, src_chunks, chunk_size);
+ memcpy(dst_values, src_values, values_size);
+}
+
+/* Functions to manipulate inner and leaf node-125 */
+
+/* Does the given chunk in the node has the value? */
+static inline bool
+RT_NODE_125_IS_CHUNK_USED(RT_NODE_BASE_125 *node, uint8 chunk)
+{
+ return node->slot_idxs[chunk] != RT_INVALID_SLOT_IDX;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_125_GET_CHILD(RT_NODE_INNER_125 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ return node->children[node->base.slot_idxs[chunk]];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_125_GET_VALUE(RT_NODE_LEAF_125 *node, uint8 chunk)
+{
+ Assert(RT_NODE_IS_LEAF(node));
+ Assert(((RT_NODE_BASE_125 *) node)->slot_idxs[chunk] != RT_INVALID_SLOT_IDX);
+ return node->values[node->base.slot_idxs[chunk]];
+}
+
+/* Functions to manipulate inner and leaf node-256 */
+
+/* Return true if the slot corresponding to the given chunk is in use */
+static inline bool
+RT_NODE_INNER_256_IS_CHUNK_USED(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ return node->children[chunk] != RT_INVALID_PTR_ALLOC;
+}
+
+static inline bool
+RT_NODE_LEAF_256_IS_CHUNK_USED(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ return (node->isset[idx] & ((bitmapword) 1 << bitnum)) != 0;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_256_GET_CHILD(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ Assert(RT_NODE_INNER_256_IS_CHUNK_USED(node, chunk));
+ return node->children[chunk];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_256_GET_VALUE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ Assert(RT_NODE_IS_LEAF(node));
+ Assert(RT_NODE_LEAF_256_IS_CHUNK_USED(node, chunk));
+ return node->values[chunk];
+}
+
+/* Set the child in the node-256 */
+static inline void
+RT_NODE_INNER_256_SET(RT_NODE_INNER_256 *node, uint8 chunk, RT_PTR_ALLOC child)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ node->children[chunk] = child;
+}
+
+/* Set the value in the node-256 */
+static inline void
+RT_NODE_LEAF_256_SET(RT_NODE_LEAF_256 *node, uint8 chunk, RT_VALUE_TYPE value)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ node->isset[idx] |= ((bitmapword) 1 << bitnum);
+ node->values[chunk] = value;
+}
+
+/* Set the slot at the given chunk position */
+static inline void
+RT_NODE_INNER_256_DELETE(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ node->children[chunk] = RT_INVALID_PTR_ALLOC;
+}
+
+static inline void
+RT_NODE_LEAF_256_DELETE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ node->isset[idx] &= ~((bitmapword) 1 << bitnum);
+}
+
+/*
+ * Return the largest shift that will allowing storing the given key.
+ */
+static inline int
+RT_KEY_GET_SHIFT(uint64 key)
+{
+ if (key == 0)
+ return 0;
+ else
+ return (pg_leftmost_one_pos64(key) / RT_NODE_SPAN) * RT_NODE_SPAN;
+}
+
+/*
+ * Return the max value that can be stored in the tree with the given shift.
+ */
+static uint64
+RT_SHIFT_GET_MAX_VAL(int shift)
+{
+ if (shift == RT_MAX_SHIFT)
+ return UINT64_MAX;
+
+ return (UINT64CONST(1) << (shift + RT_NODE_SPAN)) - 1;
+}
+
+/*
+ * Allocate a new node with the given node kind.
+ */
+static RT_PTR_ALLOC
+RT_ALLOC_NODE(RT_RADIX_TREE *tree, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+ RT_PTR_ALLOC allocnode;
+ size_t allocsize;
+
+ if (is_leaf)
+ allocsize = RT_SIZE_CLASS_INFO[size_class].leaf_size;
+ else
+ allocsize = RT_SIZE_CLASS_INFO[size_class].inner_size;
+
+#ifdef RT_SHMEM
+ allocnode = dsa_allocate(tree->dsa, allocsize);
+#else
+ if (is_leaf)
+ allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->leaf_slabs[size_class],
+ allocsize);
+ else
+ allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->inner_slabs[size_class],
+ allocsize);
+#endif
+
+#ifdef RT_DEBUG
+ /* update the statistics */
+ tree->ctl->cnt[size_class]++;
+#endif
+
+ return allocnode;
+}
+
+/* Initialize the node contents */
+static inline void
+RT_INIT_NODE(RT_PTR_LOCAL node, uint8 kind, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+ if (is_leaf)
+ MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].leaf_size);
+ else
+ MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].inner_size);
+
+ node->kind = kind;
+
+ if (kind == RT_NODE_KIND_256)
+ /* See comment for the RT_NODE type */
+ Assert(node->fanout == 0);
+ else
+ node->fanout = RT_SIZE_CLASS_INFO[size_class].fanout;
+
+ /* Initialize slot_idxs to invalid values */
+ if (kind == RT_NODE_KIND_125)
+ {
+ RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+
+ memset(n125->slot_idxs, RT_INVALID_SLOT_IDX, sizeof(n125->slot_idxs));
+ }
+}
+
+/*
+ * Create a new node as the root. Subordinate nodes will be created during
+ * the insertion.
+ */
+static pg_noinline void
+RT_NEW_ROOT(RT_RADIX_TREE *tree, uint64 key)
+{
+ int shift = RT_KEY_GET_SHIFT(key);
+ bool is_leaf = shift == 0;
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(newnode, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ newnode->shift = shift;
+ tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(shift);
+ tree->ctl->root = allocnode;
+}
+
+static inline void
+RT_COPY_NODE(RT_PTR_LOCAL newnode, RT_PTR_LOCAL oldnode)
+{
+ newnode->shift = oldnode->shift;
+ newnode->count = oldnode->count;
+}
+
+/*
+ * Given a new allocated node and an old node, initalize the new
+ * node with the necessary fields and return its local pointer.
+ */
+static inline RT_PTR_LOCAL
+RT_SWITCH_NODE_KIND(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, RT_PTR_LOCAL node,
+ uint8 new_kind, uint8 new_class, bool is_leaf)
+{
+ RT_PTR_LOCAL newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(newnode, new_kind, new_class, is_leaf);
+ RT_COPY_NODE(newnode, node);
+
+ return newnode;
+}
+
+/* Free the given node */
+static void
+RT_FREE_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode)
+{
+ /* If we're deleting the root node, make the tree empty */
+ if (tree->ctl->root == allocnode)
+ {
+ tree->ctl->root = RT_INVALID_PTR_ALLOC;
+ tree->ctl->max_val = 0;
+ }
+
+#ifdef RT_DEBUG
+ {
+ int i;
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+ /* update the statistics */
+ for (i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ if (node->fanout == RT_SIZE_CLASS_INFO[i].fanout)
+ break;
+ }
+
+ /* fanout of node256 is intentionally 0 */
+ if (i == RT_SIZE_CLASS_COUNT)
+ i = RT_CLASS_256;
+
+ tree->ctl->cnt[i]--;
+ Assert(tree->ctl->cnt[i] >= 0);
+ }
+#endif
+
+#ifdef RT_SHMEM
+ dsa_free(tree->dsa, allocnode);
+#else
+ pfree(allocnode);
+#endif
+}
+
+/* Update the parent's pointer when growing a node */
+static inline void
+RT_NODE_UPDATE_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC new_child)
+{
+#define RT_ACTION_UPDATE
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+#undef RT_ACTION_UPDATE
+}
+
+/*
+ * Replace old_child with new_child, and free the old one.
+ */
+static inline void
+RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
+ RT_PTR_ALLOC stored_old_child, RT_PTR_LOCAL old_child,
+ RT_PTR_ALLOC new_child, uint64 key)
+{
+#ifdef USE_ASSERT_CHECKING
+ RT_PTR_LOCAL new = RT_PTR_GET_LOCAL(tree, new_child);
+
+ Assert(old_child->shift == new->shift);
+ Assert(old_child->count == new->count);
+#endif
+
+ if (parent == old_child)
+ {
+ /* Replace the root node with the new larger node */
+ tree->ctl->root = new_child;
+ }
+ else
+ RT_NODE_UPDATE_INNER(parent, key, new_child);
+
+ RT_FREE_NODE(tree, stored_old_child);
+}
+
+/*
+ * The radix tree doesn't have sufficient height. Extend the radix tree so
+ * it can store the key.
+ */
+static pg_noinline void
+RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
+{
+ int target_shift;
+ RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+ int shift = root->shift + RT_NODE_SPAN;
+
+ target_shift = RT_KEY_GET_SHIFT(key);
+
+ /* Grow tree from 'shift' to 'target_shift' */
+ while (shift <= target_shift)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL node;
+ RT_NODE_INNER_3 *n3;
+
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, true);
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(node, RT_NODE_KIND_3, RT_CLASS_3, true);
+ node->shift = shift;
+ node->count = 1;
+
+ n3 = (RT_NODE_INNER_3 *) node;
+ n3->base.chunks[0] = 0;
+ n3->children[0] = tree->ctl->root;
+
+ /* Update the root */
+ tree->ctl->root = allocnode;
+
+ shift += RT_NODE_SPAN;
+ }
+
+ tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(target_shift);
+}
+
+/*
+ * The radix tree doesn't have inner and leaf nodes for given key-value pair.
+ * Insert inner and leaf nodes from 'node' to bottom.
+ */
+static pg_noinline void
+RT_SET_EXTEND(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
+ RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node)
+{
+ int shift = node->shift;
+
+ Assert(RT_PTR_GET_LOCAL(tree, stored_node) == node);
+
+ while (shift >= RT_NODE_SPAN)
+ {
+ RT_PTR_ALLOC allocchild;
+ RT_PTR_LOCAL newchild;
+ int newshift = shift - RT_NODE_SPAN;
+ bool is_leaf = newshift == 0;
+
+ allocchild = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ newchild = RT_PTR_GET_LOCAL(tree, allocchild);
+ RT_INIT_NODE(newchild, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ newchild->shift = newshift;
+ RT_NODE_INSERT_INNER(tree, parent, stored_node, node, key, allocchild);
+
+ parent = node;
+ node = newchild;
+ stored_node = allocchild;
+ shift -= RT_NODE_SPAN;
+ }
+
+ RT_NODE_INSERT_LEAF(tree, parent, stored_node, node, key, value_p);
+ tree->ctl->num_keys++;
+}
+
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the child
+ * pointer is set to child_p.
+ */
+static inline bool
+RT_NODE_SEARCH_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC *child_p)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the pointer
+ * to the value is set to value_p.
+ */
+static inline bool
+RT_NODE_SEARCH_LEAF(RT_PTR_LOCAL node, uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_INNER(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_LEAF(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+#endif
+
+/*
+ * Insert "child" into "node".
+ *
+ * "parent" is the parent of "node", so the grandparent of the child.
+ * If the node we're inserting into needs to grow, we update the parent's
+ * child pointer with the pointer to the new larger node.
+ */
+static void
+RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_PTR_ALLOC child)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/* Like RT_NODE_INSERT_INNER, but for leaf nodes */
+static bool
+RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Create the radix tree in the given memory context and return it.
+ */
+RT_SCOPE RT_RADIX_TREE *
+#ifdef RT_SHMEM
+RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id)
+#else
+RT_CREATE(MemoryContext ctx)
+#endif
+{
+ RT_RADIX_TREE *tree;
+ MemoryContext old_ctx;
+#ifdef RT_SHMEM
+ dsa_pointer dp;
+#endif
+
+ old_ctx = MemoryContextSwitchTo(ctx);
+
+ tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+ tree->context = ctx;
+
+#ifdef RT_SHMEM
+ tree->dsa = dsa;
+ dp = dsa_allocate0(dsa, sizeof(RT_RADIX_TREE_CONTROL));
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, dp);
+ tree->ctl->handle = dp;
+ tree->ctl->magic = RT_RADIX_TREE_MAGIC;
+ LWLockInitialize(&tree->ctl->lock, tranche_id);
+#else
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) palloc0(sizeof(RT_RADIX_TREE_CONTROL));
+
+ /* Create a slab context for each size class */
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ RT_SIZE_CLASS_ELEM size_class = RT_SIZE_CLASS_INFO[i];
+ size_t inner_blocksize = RT_SLAB_BLOCK_SIZE(size_class.inner_size);
+ size_t leaf_blocksize = RT_SLAB_BLOCK_SIZE(size_class.leaf_size);
+
+ tree->inner_slabs[i] = SlabContextCreate(ctx,
+ size_class.name,
+ inner_blocksize,
+ size_class.inner_size);
+ tree->leaf_slabs[i] = SlabContextCreate(ctx,
+ size_class.name,
+ leaf_blocksize,
+ size_class.leaf_size);
+ }
+#endif
+
+ tree->ctl->root = RT_INVALID_PTR_ALLOC;
+
+ MemoryContextSwitchTo(old_ctx);
+
+ return tree;
+}
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE *
+RT_ATTACH(dsa_area *dsa, RT_HANDLE handle)
+{
+ RT_RADIX_TREE *tree;
+ dsa_pointer control;
+
+ tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+
+ /* Find the control object in shard memory */
+ control = handle;
+
+ tree->dsa = dsa;
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, control);
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+ return tree;
+}
+
+RT_SCOPE void
+RT_DETACH(RT_RADIX_TREE *tree)
+{
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ pfree(tree);
+}
+
+RT_SCOPE RT_HANDLE
+RT_GET_HANDLE(RT_RADIX_TREE *tree)
+{
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ return tree->ctl->handle;
+}
+
+/*
+ * Recursively free all nodes allocated to the DSA area.
+ */
+static void
+RT_FREE_RECURSE(RT_RADIX_TREE *tree, RT_PTR_ALLOC ptr)
+{
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, ptr);
+
+ check_stack_depth();
+ CHECK_FOR_INTERRUPTS();
+
+ /* The leaf node doesn't have child pointers */
+ if (RT_NODE_IS_LEAF(node))
+ {
+ dsa_free(tree->dsa, ptr);
+ return;
+ }
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+ for (int i = 0; i < n3->base.n.count; i++)
+ RT_FREE_RECURSE(tree, n3->children[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+ for (int i = 0; i < n32->base.n.count; i++)
+ RT_FREE_RECURSE(tree, n32->children[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) node;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+ continue;
+
+ RT_FREE_RECURSE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+ }
+
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ RT_FREE_RECURSE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+ }
+
+ break;
+ }
+ }
+
+ /* Free the inner node */
+ dsa_free(tree->dsa, ptr);
+}
+#endif
+
+/*
+ * Free the given radix tree.
+ */
+RT_SCOPE void
+RT_FREE(RT_RADIX_TREE *tree)
+{
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+ /* Free all memory used for radix tree nodes */
+ if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ RT_FREE_RECURSE(tree, tree->ctl->root);
+
+ /*
+ * Vandalize the control block to help catch programming error where
+ * other backends access the memory formerly occupied by this radix tree.
+ */
+ tree->ctl->magic = 0;
+ dsa_free(tree->dsa, tree->ctl->handle);
+#else
+ pfree(tree->ctl);
+
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ MemoryContextDelete(tree->inner_slabs[i]);
+ MemoryContextDelete(tree->leaf_slabs[i]);
+ }
+#endif
+
+ pfree(tree);
+}
+
+/*
+ * Set key to value. If the entry already exists, we update its value to 'value'
+ * and return true. Returns false if entry doesn't yet exist.
+ */
+RT_SCOPE bool
+RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+ int shift;
+ bool updated;
+ RT_PTR_LOCAL parent;
+ RT_PTR_ALLOC stored_child;
+ RT_PTR_LOCAL child;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ RT_LOCK_EXCLUSIVE(tree);
+
+ /* Empty tree, create the root */
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ RT_NEW_ROOT(tree, key);
+
+ /* Extend the tree if necessary */
+ if (key > tree->ctl->max_val)
+ RT_EXTEND(tree, key);
+
+ stored_child = tree->ctl->root;
+ parent = RT_PTR_GET_LOCAL(tree, stored_child);
+ shift = parent->shift;
+
+ /* Descend the tree until we reach a leaf node */
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC new_child = RT_INVALID_PTR_ALLOC;
+
+ child = RT_PTR_GET_LOCAL(tree, stored_child);
+
+ if (RT_NODE_IS_LEAF(child))
+ break;
+
+ if (!RT_NODE_SEARCH_INNER(child, key, &new_child))
+ {
+ RT_SET_EXTEND(tree, key, value_p, parent, stored_child, child);
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ parent = child;
+ stored_child = new_child;
+ shift -= RT_NODE_SPAN;
+ }
+
+ updated = RT_NODE_INSERT_LEAF(tree, parent, stored_child, child, key, value_p);
+
+ /* Update the statistics */
+ if (!updated)
+ tree->ctl->num_keys++;
+
+ RT_UNLOCK(tree);
+ return updated;
+}
+
+/*
+ * Search the given key in the radix tree. Return true if there is the key,
+ * otherwise return false. On success, we set the value to *value_p so it must
+ * not be NULL.
+ */
+RT_SCOPE bool
+RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+ RT_PTR_LOCAL node;
+ int shift;
+ bool found;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+ Assert(value_p != NULL);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ node = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+ shift = node->shift;
+
+ /* Descend the tree until a leaf node */
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+ if (RT_NODE_IS_LEAF(node))
+ break;
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ node = RT_PTR_GET_LOCAL(tree, child);
+ shift -= RT_NODE_SPAN;
+ }
+
+ found = RT_NODE_SEARCH_LEAF(node, key, value_p);
+
+ RT_UNLOCK(tree);
+ return found;
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Delete the given key from the radix tree. Return true if the key is found (and
+ * deleted), otherwise do nothing and return false.
+ */
+RT_SCOPE bool
+RT_DELETE(RT_RADIX_TREE *tree, uint64 key)
+{
+ RT_PTR_LOCAL node;
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_ALLOC stack[RT_MAX_LEVEL] = {0};
+ int shift;
+ int level;
+ bool deleted;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ RT_LOCK_EXCLUSIVE(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ /*
+ * Descend the tree to search the key while building a stack of nodes we
+ * visited.
+ */
+ allocnode = tree->ctl->root;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift = node->shift;
+ level = -1;
+ while (shift > 0)
+ {
+ RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+ /* Push the current node to the stack */
+ stack[++level] = allocnode;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ allocnode = child;
+ shift -= RT_NODE_SPAN;
+ }
+
+ /* Delete the key from the leaf node if exists */
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ deleted = RT_NODE_DELETE_LEAF(node, key);
+
+ if (!deleted)
+ {
+ /* no key is found in the leaf node */
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ /* Found the key to delete. Update the statistics */
+ tree->ctl->num_keys--;
+
+ /*
+ * Return if the leaf node still has keys and we don't need to delete the
+ * node.
+ */
+ if (node->count > 0)
+ {
+ RT_UNLOCK(tree);
+ return true;
+ }
+
+ /* Free the empty leaf node */
+ RT_FREE_NODE(tree, allocnode);
+
+ /* Delete the key in inner nodes recursively */
+ while (level >= 0)
+ {
+ allocnode = stack[level--];
+
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ deleted = RT_NODE_DELETE_INNER(node, key);
+ Assert(deleted);
+
+ /* If the node didn't become empty, we stop deleting the key */
+ if (node->count > 0)
+ break;
+
+ /* The node became empty */
+ RT_FREE_NODE(tree, allocnode);
+ }
+
+ RT_UNLOCK(tree);
+ return true;
+}
+#endif
+
+static inline void
+RT_ITER_UPDATE_KEY(RT_ITER *iter, uint8 chunk, uint8 shift)
+{
+ iter->key &= ~(((uint64) RT_CHUNK_MASK) << shift);
+ iter->key |= (((uint64) chunk) << shift);
+}
+
+/*
+ * Advance the slot in the inner node. Return the child if exists, otherwise
+ * null.
+ */
+static inline RT_PTR_LOCAL
+RT_NODE_INNER_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Advance the slot in the leaf node. On success, return true and the value
+ * is set to value_p, otherwise return false.
+ */
+static inline bool
+RT_NODE_LEAF_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter,
+ RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Update each node_iter for inner nodes in the iterator node stack.
+ */
+static void
+RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
+{
+ int level = from;
+ RT_PTR_LOCAL node = from_node;
+
+ for (;;)
+ {
+ RT_NODE_ITER *node_iter = &(iter->stack[level--]);
+
+ node_iter->node = node;
+ node_iter->current_idx = -1;
+
+ /* We don't advance the leaf node iterator here */
+ if (RT_NODE_IS_LEAF(node))
+ return;
+
+ /* Advance to the next slot in the inner node */
+ node = RT_NODE_INNER_ITERATE_NEXT(iter, node_iter);
+
+ /* We must find the first children in the node */
+ Assert(node);
+ }
+}
+
+/*
+ * Create and return the iterator for the given radix tree.
+ *
+ * The radix tree is locked in shared mode during the iteration, so
+ * RT_END_ITERATE needs to be called when finished to release the lock.
+ */
+RT_SCOPE RT_ITER *
+RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
+{
+ MemoryContext old_ctx;
+ RT_ITER *iter;
+ RT_PTR_LOCAL root;
+ int top_level;
+
+ old_ctx = MemoryContextSwitchTo(tree->context);
+
+ iter = (RT_ITER *) palloc0(sizeof(RT_ITER));
+ iter->tree = tree;
+
+ RT_LOCK_SHARED(tree);
+
+ /* empty tree */
+ if (!iter->tree->ctl->root)
+ return iter;
+
+ root = RT_PTR_GET_LOCAL(tree, iter->tree->ctl->root);
+ top_level = root->shift / RT_NODE_SPAN;
+ iter->stack_len = top_level;
+
+ /*
+ * Descend to the left most leaf node from the root. The key is being
+ * constructed while descending to the leaf.
+ */
+ RT_UPDATE_ITER_STACK(iter, root, top_level);
+
+ MemoryContextSwitchTo(old_ctx);
+
+ return iter;
+}
+
+/*
+ * Return true with setting key_p and value_p if there is next key. Otherwise
+ * return false.
+ */
+RT_SCOPE bool
+RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p)
+{
+ /* Empty tree */
+ if (!iter->tree->ctl->root)
+ return false;
+
+ for (;;)
+ {
+ RT_PTR_LOCAL child = NULL;
+ RT_VALUE_TYPE value;
+ int level;
+ bool found;
+
+ /* Advance the leaf node iterator to get next key-value pair */
+ found = RT_NODE_LEAF_ITERATE_NEXT(iter, &(iter->stack[0]), &value);
+
+ if (found)
+ {
+ *key_p = iter->key;
+ *value_p = value;
+ return true;
+ }
+
+ /*
+ * We've visited all values in the leaf node, so advance inner node
+ * iterators from the level=1 until we find the next child node.
+ */
+ for (level = 1; level <= iter->stack_len; level++)
+ {
+ child = RT_NODE_INNER_ITERATE_NEXT(iter, &(iter->stack[level]));
+
+ if (child)
+ break;
+ }
+
+ /* the iteration finished */
+ if (!child)
+ return false;
+
+ /*
+ * Set the node to the node iterator and update the iterator stack
+ * from this node.
+ */
+ RT_UPDATE_ITER_STACK(iter, child, level - 1);
+
+ /* Node iterators are updated, so try again from the leaf */
+ }
+
+ return false;
+}
+
+/*
+ * Terminate the iteration and release the lock.
+ *
+ * This function needs to be called after finishing or when exiting an
+ * iteration.
+ */
+RT_SCOPE void
+RT_END_ITERATE(RT_ITER *iter)
+{
+#ifdef RT_SHMEM
+ Assert(LWLockHeldByMe(&iter->tree->ctl->lock));
+#endif
+
+ RT_UNLOCK(iter->tree);
+ pfree(iter);
+}
+
+/*
+ * Return the statistics of the amount of memory used by the radix tree.
+ */
+RT_SCOPE uint64
+RT_MEMORY_USAGE(RT_RADIX_TREE *tree)
+{
+ Size total = 0;
+
+ RT_LOCK_SHARED(tree);
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ total = dsa_get_total_size(tree->dsa);
+#else
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ total += MemoryContextMemAllocated(tree->inner_slabs[i], true);
+ total += MemoryContextMemAllocated(tree->leaf_slabs[i], true);
+ }
+#endif
+
+ RT_UNLOCK(tree);
+ return total;
+}
+
+/*
+ * Verify the radix tree node.
+ */
+static void
+RT_VERIFY_NODE(RT_PTR_LOCAL node)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(node->count >= 0);
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE_BASE_3 *n3 = (RT_NODE_BASE_3 *) node;
+
+ for (int i = 1; i < n3->n.count; i++)
+ Assert(n3->chunks[i - 1] < n3->chunks[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE_BASE_32 *n32 = (RT_NODE_BASE_32 *) node;
+
+ for (int i = 1; i < n32->n.count; i++)
+ Assert(n32->chunks[i - 1] < n32->chunks[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+ int cnt = 0;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ uint8 slot = n125->slot_idxs[i];
+ int idx = RT_BM_IDX(slot);
+ int bitnum = RT_BM_BIT(slot);
+
+ if (!RT_NODE_125_IS_CHUNK_USED(n125, i))
+ continue;
+
+ /* Check if the corresponding slot is used */
+ Assert(slot < node->fanout);
+ Assert((n125->isset[idx] & ((bitmapword) 1 << bitnum)) != 0);
+
+ cnt++;
+ }
+
+ Assert(n125->n.count == cnt);
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+ int cnt = 0;
+
+ for (int i = 0; i < RT_BM_IDX(RT_NODE_MAX_SLOTS); i++)
+ cnt += bmw_popcount(n256->isset[i]);
+
+ /* Check if the number of used chunk matches */
+ Assert(n256->base.n.count == cnt);
+
+ break;
+ }
+ }
+ }
+#endif
+}
+
+/***************** DEBUG FUNCTIONS *****************/
+#ifdef RT_DEBUG
+
+#define RT_UINT64_FORMAT_HEX "%" INT64_MODIFIER "X"
+
+RT_SCOPE void
+RT_STATS(RT_RADIX_TREE *tree)
+{
+ RT_LOCK_SHARED(tree);
+
+ fprintf(stderr, "max_val = " UINT64_FORMAT "\n", tree->ctl->max_val);
+ fprintf(stderr, "num_keys = " UINT64_FORMAT "\n", tree->ctl->num_keys);
+
+#ifdef RT_SHMEM
+ fprintf(stderr, "handle = " UINT64_FORMAT "\n", tree->ctl->handle);
+#endif
+
+ if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+
+ fprintf(stderr, "height = %d, n3 = %u, n15 = %u, n32 = %u, n125 = %u, n256 = %u\n",
+ root->shift / RT_NODE_SPAN,
+ tree->ctl->cnt[RT_CLASS_3],
+ tree->ctl->cnt[RT_CLASS_32_MIN],
+ tree->ctl->cnt[RT_CLASS_32_MAX],
+ tree->ctl->cnt[RT_CLASS_125],
+ tree->ctl->cnt[RT_CLASS_256]);
+ }
+
+ RT_UNLOCK(tree);
+}
+
+static void
+RT_DUMP_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, int level,
+ bool recurse, StringInfo buf)
+{
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+ StringInfoData spaces;
+
+ initStringInfo(&spaces);
+ appendStringInfoSpaces(&spaces, (level * 4) + 1);
+
+ appendStringInfo(buf, "%s%s[%s] kind %d, fanout %d, count %u, shift %u:\n",
+ spaces.data,
+ level == 0 ? "" : "-> ",
+ RT_NODE_IS_LEAF(node) ? "LEAF" : "INNR",
+ (node->kind == RT_NODE_KIND_3) ? 3 :
+ (node->kind == RT_NODE_KIND_32) ? 32 :
+ (node->kind == RT_NODE_KIND_125) ? 125 : 256,
+ node->fanout == 0 ? 256 : node->fanout,
+ node->count, node->shift);
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ for (int i = 0; i < node->count; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_3 *n3 = (RT_NODE_LEAF_3 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+ spaces.data, i, n3->base.chunks[i]);
+ }
+ else
+ {
+ RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X",
+ spaces.data, i, n3->base.chunks[i]);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, n3->children[i], level + 1,
+ recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ for (int i = 0; i < node->count; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_32 *n32 = (RT_NODE_LEAF_32 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+ spaces.data, i, n32->base.chunks[i]);
+ }
+ else
+ {
+ RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X",
+ spaces.data, i, n32->base.chunks[i]);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, n32->children[i], level + 1,
+ recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_BASE_125 *b125 = (RT_NODE_BASE_125 *) node;
+ char *sep = "";
+
+ appendStringInfo(buf, "%sslot_idxs: ", spaces.data);
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+ continue;
+
+ appendStringInfo(buf, "%s[%d]=%d ",
+ sep, i, b125->slot_idxs[i]);
+ sep = ",";
+ }
+
+ appendStringInfo(buf, "\n%sisset-bitmap: ", spaces.data);
+ for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+ appendStringInfo(buf, "%X ", ((uint8 *) b125->isset)[i]);
+ appendStringInfo(buf, "\n");
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+ continue;
+
+ if (RT_NODE_IS_LEAF(node))
+ appendStringInfo(buf, "%schunk 0x%X\n",
+ spaces.data, i);
+ else
+ {
+ RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) b125;
+
+ appendStringInfo(buf, "%schunk 0x%X",
+ spaces.data, i);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i),
+ level + 1, recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+ appendStringInfo(buf, "%sisset-bitmap: ", spaces.data);
+ for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+ appendStringInfo(buf, "%X ", ((uint8 *) n256->isset)[i]);
+ appendStringInfo(buf, "\n");
+ }
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ appendStringInfo(buf, "%schunk 0x%X\n",
+ spaces.data, i);
+ }
+ else
+ {
+ RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ appendStringInfo(buf, "%schunk 0x%X",
+ spaces.data, i);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i),
+ level + 1, recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ }
+}
+
+RT_SCOPE void
+RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key)
+{
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL node;
+ StringInfoData buf;
+ int shift;
+ int level = 0;
+
+ RT_STATS(tree);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "empty tree\n");
+ return;
+ }
+
+ if (key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "key " UINT64_FORMAT "(0x" RT_UINT64_FORMAT_HEX ") is larger than max val\n",
+ key, key);
+ return;
+ }
+
+ initStringInfo(&buf);
+ allocnode = tree->ctl->root;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift = node->shift;
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC child;
+
+ RT_DUMP_NODE(tree, allocnode, level, false, &buf);
+
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_VALUE_TYPE dummy;
+
+ /* We reached at a leaf node, find the corresponding slot */
+ RT_NODE_SEARCH_LEAF(node, key, &dummy);
+
+ break;
+ }
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ break;
+
+ allocnode = child;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift -= RT_NODE_SPAN;
+ level++;
+ }
+ RT_UNLOCK(tree);
+
+ fprintf(stderr, "%s", buf.data);
+}
+
+RT_SCOPE void
+RT_DUMP(RT_RADIX_TREE *tree)
+{
+ StringInfoData buf;
+
+ RT_STATS(tree);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "empty tree\n");
+ return;
+ }
+
+ initStringInfo(&buf);
+
+ RT_DUMP_NODE(tree, tree->ctl->root, 0, true, &buf);
+ RT_UNLOCK(tree);
+
+ fprintf(stderr, "%s",buf.data);
+}
+#endif
+
+#endif /* RT_DEFINE */
+
+
+/* undefine external parameters, so next radix tree can be defined */
+#undef RT_PREFIX
+#undef RT_SCOPE
+#undef RT_DECLARE
+#undef RT_DEFINE
+#undef RT_VALUE_TYPE
+
+/* locally declared macros */
+#undef RT_MAKE_PREFIX
+#undef RT_MAKE_NAME
+#undef RT_MAKE_NAME_
+#undef RT_NODE_SPAN
+#undef RT_NODE_MAX_SLOTS
+#undef RT_CHUNK_MASK
+#undef RT_MAX_SHIFT
+#undef RT_MAX_LEVEL
+#undef RT_GET_KEY_CHUNK
+#undef RT_BM_IDX
+#undef RT_BM_BIT
+#undef RT_LOCK_EXCLUSIVE
+#undef RT_LOCK_SHARED
+#undef RT_UNLOCK
+#undef RT_NODE_IS_LEAF
+#undef RT_NODE_MUST_GROW
+#undef RT_NODE_KIND_COUNT
+#undef RT_SIZE_CLASS_COUNT
+#undef RT_SLOT_IDX_LIMIT
+#undef RT_INVALID_SLOT_IDX
+#undef RT_SLAB_BLOCK_SIZE
+#undef RT_RADIX_TREE_MAGIC
+#undef RT_UINT64_FORMAT_HEX
+
+/* type declarations */
+#undef RT_RADIX_TREE
+#undef RT_RADIX_TREE_CONTROL
+#undef RT_PTR_LOCAL
+#undef RT_PTR_ALLOC
+#undef RT_INVALID_PTR_ALLOC
+#undef RT_HANDLE
+#undef RT_ITER
+#undef RT_NODE
+#undef RT_NODE_ITER
+#undef RT_NODE_KIND_3
+#undef RT_NODE_KIND_32
+#undef RT_NODE_KIND_125
+#undef RT_NODE_KIND_256
+#undef RT_NODE_BASE_3
+#undef RT_NODE_BASE_32
+#undef RT_NODE_BASE_125
+#undef RT_NODE_BASE_256
+#undef RT_NODE_INNER_3
+#undef RT_NODE_INNER_32
+#undef RT_NODE_INNER_125
+#undef RT_NODE_INNER_256
+#undef RT_NODE_LEAF_3
+#undef RT_NODE_LEAF_32
+#undef RT_NODE_LEAF_125
+#undef RT_NODE_LEAF_256
+#undef RT_SIZE_CLASS
+#undef RT_SIZE_CLASS_ELEM
+#undef RT_SIZE_CLASS_INFO
+#undef RT_CLASS_3
+#undef RT_CLASS_32_MIN
+#undef RT_CLASS_32_MAX
+#undef RT_CLASS_125
+#undef RT_CLASS_256
+
+/* function declarations */
+#undef RT_CREATE
+#undef RT_FREE
+#undef RT_ATTACH
+#undef RT_DETACH
+#undef RT_GET_HANDLE
+#undef RT_SEARCH
+#undef RT_SET
+#undef RT_BEGIN_ITERATE
+#undef RT_ITERATE_NEXT
+#undef RT_END_ITERATE
+#undef RT_USE_DELETE
+#undef RT_DELETE
+#undef RT_MEMORY_USAGE
+#undef RT_DUMP
+#undef RT_DUMP_NODE
+#undef RT_DUMP_SEARCH
+#undef RT_STATS
+
+/* internal helper functions */
+#undef RT_NEW_ROOT
+#undef RT_ALLOC_NODE
+#undef RT_INIT_NODE
+#undef RT_FREE_NODE
+#undef RT_FREE_RECURSE
+#undef RT_EXTEND
+#undef RT_SET_EXTEND
+#undef RT_SWITCH_NODE_KIND
+#undef RT_COPY_NODE
+#undef RT_REPLACE_NODE
+#undef RT_PTR_GET_LOCAL
+#undef RT_PTR_ALLOC_IS_VALID
+#undef RT_NODE_3_SEARCH_EQ
+#undef RT_NODE_32_SEARCH_EQ
+#undef RT_NODE_3_GET_INSERTPOS
+#undef RT_NODE_32_GET_INSERTPOS
+#undef RT_CHUNK_CHILDREN_ARRAY_SHIFT
+#undef RT_CHUNK_VALUES_ARRAY_SHIFT
+#undef RT_CHUNK_CHILDREN_ARRAY_DELETE
+#undef RT_CHUNK_VALUES_ARRAY_DELETE
+#undef RT_CHUNK_CHILDREN_ARRAY_COPY
+#undef RT_CHUNK_VALUES_ARRAY_COPY
+#undef RT_NODE_125_IS_CHUNK_USED
+#undef RT_NODE_INNER_125_GET_CHILD
+#undef RT_NODE_LEAF_125_GET_VALUE
+#undef RT_NODE_INNER_256_IS_CHUNK_USED
+#undef RT_NODE_LEAF_256_IS_CHUNK_USED
+#undef RT_NODE_INNER_256_GET_CHILD
+#undef RT_NODE_LEAF_256_GET_VALUE
+#undef RT_NODE_INNER_256_SET
+#undef RT_NODE_LEAF_256_SET
+#undef RT_NODE_INNER_256_DELETE
+#undef RT_NODE_LEAF_256_DELETE
+#undef RT_KEY_GET_SHIFT
+#undef RT_SHIFT_GET_MAX_VAL
+#undef RT_NODE_SEARCH_INNER
+#undef RT_NODE_SEARCH_LEAF
+#undef RT_NODE_UPDATE_INNER
+#undef RT_NODE_DELETE_INNER
+#undef RT_NODE_DELETE_LEAF
+#undef RT_NODE_INSERT_INNER
+#undef RT_NODE_INSERT_LEAF
+#undef RT_NODE_INNER_ITERATE_NEXT
+#undef RT_NODE_LEAF_ITERATE_NEXT
+#undef RT_UPDATE_ITER_STACK
+#undef RT_ITER_UPDATE_KEY
+#undef RT_VERIFY_NODE
+
+#undef RT_DEBUG
diff --git a/src/include/lib/radixtree_delete_impl.h b/src/include/lib/radixtree_delete_impl.h
new file mode 100644
index 0000000000..5f6dda1f12
--- /dev/null
+++ b/src/include/lib/radixtree_delete_impl.h
@@ -0,0 +1,122 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_delete_impl.h
+ * Common implementation for deletion in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ * TODO: Shrink nodes when deletion would allow them to fit in a smaller
+ * size class.
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_delete_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_DELETE(n3->base.chunks, n3->values,
+ n3->base.n.count, idx);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_DELETE(n3->base.chunks, n3->children,
+ n3->base.n.count, idx);
+#endif
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_DELETE(n32->base.chunks, n32->values,
+ n32->base.n.count, idx);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_DELETE(n32->base.chunks, n32->children,
+ n32->base.n.count, idx);
+#endif
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+ int idx;
+ int bitnum;
+
+ if (slotpos == RT_INVALID_SLOT_IDX)
+ return false;
+
+ idx = RT_BM_IDX(slotpos);
+ bitnum = RT_BM_BIT(slotpos);
+ n125->base.isset[idx] &= ~((bitmapword) 1 << bitnum);
+ n125->base.slot_idxs[chunk] = RT_INVALID_SLOT_IDX;
+
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_DELETE(n256, chunk);
+#else
+ RT_NODE_INNER_256_DELETE(n256, chunk);
+#endif
+ break;
+ }
+ }
+
+ /* update statistics */
+ node->count--;
+
+ return true;
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
new file mode 100644
index 0000000000..d56e58dcac
--- /dev/null
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -0,0 +1,328 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_insert_impl.h
+ * Common implementation for insertion in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_insert_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ const bool is_leaf = true;
+ bool chunk_exists = false;
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+ const bool is_leaf = false;
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ int idx = RT_NODE_3_SEARCH_EQ(&n3->base, chunk);
+
+ if (idx != -1)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n3->values[idx] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n3)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE32_TYPE *new32;
+ const uint8 new_kind = RT_NODE_KIND_32;
+ const RT_SIZE_CLASS new_class = RT_CLASS_32_MIN;
+
+ /* grow node from 3 to 32 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_COPY(n3->base.chunks, n3->values,
+ new32->base.chunks, new32->values);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_COPY(n3->base.chunks, n3->children,
+ new32->base.chunks, new32->children);
+#endif
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
+ int count = n3->base.n.count;
+
+ /* shift chunks and children */
+ if (insertpos < count)
+ {
+ Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_SHIFT(n3->base.chunks, n3->values,
+ count, insertpos);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_SHIFT(n3->base.chunks, n3->children,
+ count, insertpos);
+#endif
+ }
+
+ n3->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+ n3->values[insertpos] = *value_p;
+#else
+ n3->children[insertpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_32:
+ {
+ const RT_SIZE_CLASS_ELEM class32_max = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX];
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ int idx = RT_NODE_32_SEARCH_EQ(&n32->base, chunk);
+
+ if (idx != -1)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n32->values[idx] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n32)) &&
+ n32->base.n.fanout < class32_max.fanout)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ const RT_SIZE_CLASS_ELEM class32_min = RT_SIZE_CLASS_INFO[RT_CLASS_32_MIN];
+ const RT_SIZE_CLASS new_class = RT_CLASS_32_MAX;
+
+ Assert(n32->base.n.fanout == class32_min.fanout);
+
+ /* grow to the next size class of this kind */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ n32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ memcpy(newnode, node, class32_min.leaf_size);
+#else
+ memcpy(newnode, node, class32_min.inner_size);
+#endif
+ newnode->fanout = class32_max.fanout;
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+
+ if (unlikely(RT_NODE_MUST_GROW(n32)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE125_TYPE *new125;
+ const uint8 new_kind = RT_NODE_KIND_125;
+ const RT_SIZE_CLASS new_class = RT_CLASS_125;
+
+ Assert(n32->base.n.fanout == class32_max.fanout);
+
+ /* grow node from 32 to 125 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new125 = (RT_NODE125_TYPE *) newnode;
+
+ for (int i = 0; i < class32_max.fanout; i++)
+ {
+ new125->base.slot_idxs[n32->base.chunks[i]] = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ new125->values[i] = n32->values[i];
+#else
+ new125->children[i] = n32->children[i];
+#endif
+ }
+
+ /*
+ * Since we just copied a dense array, we can set the bits
+ * using a single store, provided the length of that array
+ * is at most the number of bits in a bitmapword.
+ */
+ Assert(class32_max.fanout <= sizeof(bitmapword) * BITS_PER_BYTE);
+ new125->base.isset[0] = (bitmapword) (((uint64) 1 << class32_max.fanout) - 1);
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
+ int count = n32->base.n.count;
+
+ if (insertpos < count)
+ {
+ Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_SHIFT(n32->base.chunks, n32->values,
+ count, insertpos);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_SHIFT(n32->base.chunks, n32->children,
+ count, insertpos);
+#endif
+ }
+
+ n32->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+ n32->values[insertpos] = *value_p;
+#else
+ n32->children[insertpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos;
+ int cnt = 0;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ slotpos = n125->base.slot_idxs[chunk];
+ if (slotpos != RT_INVALID_SLOT_IDX)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n125->values[slotpos] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n125)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE256_TYPE *new256;
+ const uint8 new_kind = RT_NODE_KIND_256;
+ const RT_SIZE_CLASS new_class = RT_CLASS_256;
+
+ /* grow node from 125 to 256 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new256 = (RT_NODE256_TYPE *) newnode;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS && cnt < n125->base.n.count; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+ continue;
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_SET(new256, i, RT_NODE_LEAF_125_GET_VALUE(n125, i));
+#else
+ RT_NODE_INNER_256_SET(new256, i, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+ cnt++;
+ }
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int idx;
+ bitmapword inverse;
+
+ /* get the first word with at least one bit not set */
+ for (idx = 0; idx < RT_BM_IDX(RT_SLOT_IDX_LIMIT); idx++)
+ {
+ if (n125->base.isset[idx] < ~((bitmapword) 0))
+ break;
+ }
+
+ /* To get the first unset bit in X, get the first set bit in ~X */
+ inverse = ~(n125->base.isset[idx]);
+ slotpos = idx * BITS_PER_BITMAPWORD;
+ slotpos += bmw_rightmost_one_pos(inverse);
+ Assert(slotpos < node->fanout);
+
+ /* mark the slot used */
+ n125->base.isset[idx] |= bmw_rightmost_one(inverse);
+ n125->base.slot_idxs[chunk] = slotpos;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ n125->values[slotpos] = *value_p;
+#else
+ n125->children[slotpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ chunk_exists = RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk);
+ Assert(chunk_exists || node->count < RT_NODE_MAX_SLOTS);
+ RT_NODE_LEAF_256_SET(n256, chunk, *value_p);
+#else
+ Assert(node->count < RT_NODE_MAX_SLOTS);
+ RT_NODE_INNER_256_SET(n256, chunk, child);
+#endif
+ break;
+ }
+ }
+
+ /* Update statistics */
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!chunk_exists)
+ node->count++;
+#else
+ node->count++;
+#endif
+
+ /*
+ * Done. Finally, verify the chunk and value is inserted or replaced
+ * properly in the node.
+ */
+ RT_VERIFY_NODE(node);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ return chunk_exists;
+#else
+ return;
+#endif
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_iter_impl.h b/src/include/lib/radixtree_iter_impl.h
new file mode 100644
index 0000000000..98c78eb237
--- /dev/null
+++ b/src/include/lib/radixtree_iter_impl.h
@@ -0,0 +1,153 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_iter_impl.h
+ * Common implementation for iteration in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_iter_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ bool found = false;
+ uint8 key_chunk;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_VALUE_TYPE value;
+
+ Assert(RT_NODE_IS_LEAF(node_iter->node));
+#else
+ RT_PTR_LOCAL child = NULL;
+
+ Assert(!RT_NODE_IS_LEAF(node_iter->node));
+#endif
+
+#ifdef RT_SHMEM
+ Assert(iter->tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ switch (node_iter->node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node_iter->node;
+
+ node_iter->current_idx++;
+ if (node_iter->current_idx >= n3->base.n.count)
+ break;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = n3->values[node_iter->current_idx];
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, n3->children[node_iter->current_idx]);
+#endif
+ key_chunk = n3->base.chunks[node_iter->current_idx];
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node_iter->node;
+
+ node_iter->current_idx++;
+ if (node_iter->current_idx >= n32->base.n.count)
+ break;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ value = n32->values[node_iter->current_idx];
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, n32->children[node_iter->current_idx]);
+#endif
+ key_chunk = n32->base.chunks[node_iter->current_idx];
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node_iter->node;
+ int i;
+
+ for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (RT_NODE_125_IS_CHUNK_USED((RT_NODE_BASE_125 *) n125, i))
+ break;
+ }
+
+ if (i >= RT_NODE_MAX_SLOTS)
+ break;
+
+ node_iter->current_idx = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = RT_NODE_LEAF_125_GET_VALUE(n125, i);
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+ key_chunk = i;
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node_iter->node;
+ int i;
+
+ for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ {
+#ifdef RT_NODE_LEVEL_LEAF
+ if (RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+#else
+ if (RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+#endif
+ break;
+ }
+
+ if (i >= RT_NODE_MAX_SLOTS)
+ break;
+
+ node_iter->current_idx = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = RT_NODE_LEAF_256_GET_VALUE(n256, i);
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+#endif
+ key_chunk = i;
+ found = true;
+ break;
+ }
+ }
+
+ if (found)
+ {
+ RT_ITER_UPDATE_KEY(iter, key_chunk, node_iter->node->shift);
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = value;
+#endif
+ }
+
+#ifdef RT_NODE_LEVEL_LEAF
+ return found;
+#else
+ return child;
+#endif
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_search_impl.h b/src/include/lib/radixtree_search_impl.h
new file mode 100644
index 0000000000..a8925c75d0
--- /dev/null
+++ b/src/include/lib/radixtree_search_impl.h
@@ -0,0 +1,138 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_search_impl.h
+ * Common implementation for search in leaf and inner nodes, plus
+ * update for inner nodes only.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_search_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(value_p != NULL);
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+#ifndef RT_ACTION_UPDATE
+ Assert(child_p != NULL);
+#endif
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+#ifdef RT_ACTION_UPDATE
+ Assert(idx >= 0);
+ n3->children[idx] = new_child;
+#else
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n3->values[idx];
+#else
+ *child_p = n3->children[idx];
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+#ifdef RT_ACTION_UPDATE
+ Assert(idx >= 0);
+ n32->children[idx] = new_child;
+#else
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n32->values[idx];
+#else
+ *child_p = n32->children[idx];
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+
+#ifdef RT_ACTION_UPDATE
+ Assert(slotpos != RT_INVALID_SLOT_IDX);
+ n125->children[slotpos] = new_child;
+#else
+ if (slotpos == RT_INVALID_SLOT_IDX)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_125_GET_VALUE(n125, chunk);
+#else
+ *child_p = RT_NODE_INNER_125_GET_CHILD(n125, chunk);
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_ACTION_UPDATE
+ RT_NODE_INNER_256_SET(n256, chunk, new_child);
+#else
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_256_GET_VALUE(n256, chunk);
+#else
+ *child_p = RT_NODE_INNER_256_GET_CHILD(n256, chunk);
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ }
+
+#ifdef RT_ACTION_UPDATE
+ return;
+#else
+ return true;
+#endif /* RT_ACTION_UPDATE */
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/utils/dsa.h b/src/include/utils/dsa.h
index 3ce4ee300a..2af215484f 100644
--- a/src/include/utils/dsa.h
+++ b/src/include/utils/dsa.h
@@ -121,6 +121,7 @@ extern dsa_handle dsa_get_handle(dsa_area *area);
extern dsa_pointer dsa_allocate_extended(dsa_area *area, size_t size, int flags);
extern void dsa_free(dsa_area *area, dsa_pointer dp);
extern void *dsa_get_address(dsa_area *area, dsa_pointer dp);
+extern size_t dsa_get_total_size(dsa_area *area);
extern void dsa_trim(dsa_area *area);
extern void dsa_dump(dsa_area *area);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index c629cbe383..9659eb85d7 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -28,6 +28,7 @@ SUBDIRS = \
test_pg_db_role_setting \
test_pg_dump \
test_predtest \
+ test_radixtree \
test_rbtree \
test_regex \
test_rls_hooks \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 1baa6b558d..232cbdac80 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -24,6 +24,7 @@ subdir('test_parser')
subdir('test_pg_db_role_setting')
subdir('test_pg_dump')
subdir('test_predtest')
+subdir('test_radixtree')
subdir('test_rbtree')
subdir('test_regex')
subdir('test_rls_hooks')
diff --git a/src/test/modules/test_radixtree/.gitignore b/src/test/modules/test_radixtree/.gitignore
new file mode 100644
index 0000000000..5dcb3ff972
--- /dev/null
+++ b/src/test/modules/test_radixtree/.gitignore
@@ -0,0 +1,4 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
diff --git a/src/test/modules/test_radixtree/Makefile b/src/test/modules/test_radixtree/Makefile
new file mode 100644
index 0000000000..da06b93da3
--- /dev/null
+++ b/src/test/modules/test_radixtree/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_radixtree/Makefile
+
+MODULE_big = test_radixtree
+OBJS = \
+ $(WIN32RES) \
+ test_radixtree.o
+PGFILEDESC = "test_radixtree - test code for src/backend/lib/radixtree.c"
+
+EXTENSION = test_radixtree
+DATA = test_radixtree--1.0.sql
+
+REGRESS = test_radixtree
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_radixtree
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_radixtree/README b/src/test/modules/test_radixtree/README
new file mode 100644
index 0000000000..a8b271869a
--- /dev/null
+++ b/src/test/modules/test_radixtree/README
@@ -0,0 +1,7 @@
+test_integerset contains unit tests for testing the integer set implementation
+in src/backend/lib/integerset.c.
+
+The tests verify the correctness of the implementation, but they can also be
+used as a micro-benchmark. If you set the 'intset_test_stats' flag in
+test_integerset.c, the tests will print extra information about execution time
+and memory usage.
diff --git a/src/test/modules/test_radixtree/expected/test_radixtree.out b/src/test/modules/test_radixtree/expected/test_radixtree.out
new file mode 100644
index 0000000000..ce645cb8b5
--- /dev/null
+++ b/src/test/modules/test_radixtree/expected/test_radixtree.out
@@ -0,0 +1,36 @@
+CREATE EXTENSION test_radixtree;
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
+NOTICE: testing basic operations with leaf node 4
+NOTICE: testing basic operations with inner node 4
+NOTICE: testing basic operations with leaf node 32
+NOTICE: testing basic operations with inner node 32
+NOTICE: testing basic operations with leaf node 125
+NOTICE: testing basic operations with inner node 125
+NOTICE: testing basic operations with leaf node 256
+NOTICE: testing basic operations with inner node 256
+NOTICE: testing radix tree node types with shift "0"
+NOTICE: testing radix tree node types with shift "8"
+NOTICE: testing radix tree node types with shift "16"
+NOTICE: testing radix tree node types with shift "24"
+NOTICE: testing radix tree node types with shift "32"
+NOTICE: testing radix tree node types with shift "40"
+NOTICE: testing radix tree node types with shift "48"
+NOTICE: testing radix tree node types with shift "56"
+NOTICE: testing radix tree with pattern "all ones"
+NOTICE: testing radix tree with pattern "alternating bits"
+NOTICE: testing radix tree with pattern "clusters of ten"
+NOTICE: testing radix tree with pattern "clusters of hundred"
+NOTICE: testing radix tree with pattern "one-every-64k"
+NOTICE: testing radix tree with pattern "sparse"
+NOTICE: testing radix tree with pattern "single values, distance > 2^32"
+NOTICE: testing radix tree with pattern "clusters, distance > 2^32"
+NOTICE: testing radix tree with pattern "clusters, distance > 2^60"
+ test_radixtree
+----------------
+
+(1 row)
+
diff --git a/src/test/modules/test_radixtree/meson.build b/src/test/modules/test_radixtree/meson.build
new file mode 100644
index 0000000000..6add06bbdb
--- /dev/null
+++ b/src/test/modules/test_radixtree/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_radixtree_sources = files(
+ 'test_radixtree.c',
+)
+
+if host_system == 'windows'
+ test_radixtree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_radixtree',
+ '--FILEDESC', 'test_radixtree - test code for src/include//lib/radixtree.h',])
+endif
+
+test_radixtree = shared_module('test_radixtree',
+ test_radixtree_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += test_radixtree
+
+install_data(
+ 'test_radixtree.control',
+ 'test_radixtree--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'test_radixtree',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'test_radixtree',
+ ],
+ },
+}
diff --git a/src/test/modules/test_radixtree/sql/test_radixtree.sql b/src/test/modules/test_radixtree/sql/test_radixtree.sql
new file mode 100644
index 0000000000..41ece5e9f5
--- /dev/null
+++ b/src/test/modules/test_radixtree/sql/test_radixtree.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_radixtree;
+
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
diff --git a/src/test/modules/test_radixtree/test_radixtree--1.0.sql b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
new file mode 100644
index 0000000000..074a5a7ea7
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_radixtree/test_radixtree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_radixtree" to load this file. \quit
+
+CREATE FUNCTION test_radixtree()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
new file mode 100644
index 0000000000..afe53382f3
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -0,0 +1,681 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_radixtree.c
+ * Test radixtree set data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_radixtree/test_radixtree.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "storage/lwlock.h"
+#include "utils/memutils.h"
+#include "utils/timestamp.h"
+
+#define UINT64_HEX_FORMAT "%" INT64_MODIFIER "X"
+
+/*
+ * The tests pass with uint32, but build with warnings because the string
+ * format expects uint64.
+ */
+typedef uint64 TestValueType;
+
+/*
+ * If you enable this, the "pattern" tests will print information about
+ * how long populating, probing, and iterating the test set takes, and
+ * how much memory the test set consumed. That can be used as
+ * micro-benchmark of various operations and input patterns (you might
+ * want to increase the number of values used in each of the test, if
+ * you do that, to reduce noise).
+ *
+ * The information is printed to the server's stderr, mostly because
+ * that's where MemoryContextStats() output goes.
+ */
+static const bool rt_test_stats = false;
+
+static int rt_node_kind_fanouts[] = {
+ 0,
+ 4, /* RT_NODE_KIND_4 */
+ 32, /* RT_NODE_KIND_32 */
+ 125, /* RT_NODE_KIND_125 */
+ 256 /* RT_NODE_KIND_256 */
+};
+/*
+ * A struct to define a pattern of integers, for use with the test_pattern()
+ * function.
+ */
+typedef struct
+{
+ char *test_name; /* short name of the test, for humans */
+ char *pattern_str; /* a bit pattern */
+ uint64 spacing; /* pattern repeats at this interval */
+ uint64 num_values; /* number of integers to set in total */
+} test_spec;
+
+/* Test patterns borrowed from test_integerset.c */
+static const test_spec test_specs[] = {
+ {
+ "all ones", "1111111111",
+ 10, 1000000
+ },
+ {
+ "alternating bits", "0101010101",
+ 10, 1000000
+ },
+ {
+ "clusters of ten", "1111111111",
+ 10000, 1000000
+ },
+ {
+ "clusters of hundred",
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111",
+ 10000, 1000000
+ },
+ {
+ "one-every-64k", "1",
+ 65536, 1000000
+ },
+ {
+ "sparse", "100000000000000000000000000000001",
+ 10000000, 1000000
+ },
+ {
+ "single values, distance > 2^32", "1",
+ UINT64CONST(10000000000), 100000
+ },
+ {
+ "clusters, distance > 2^32", "10101010",
+ UINT64CONST(10000000000), 1000000
+ },
+ {
+ "clusters, distance > 2^60", "10101010",
+ UINT64CONST(2000000000000000000),
+ 23 /* can't be much higher than this, or we
+ * overflow uint64 */
+ }
+};
+
+/* define the radix tree implementation to test */
+#define RT_PREFIX rt
+#define RT_SCOPE
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE TestValueType
+/* #define RT_SHMEM */
+#include "lib/radixtree.h"
+
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+ return tree->ctl->num_keys;
+}
+
+PG_MODULE_MAGIC;
+
+PG_FUNCTION_INFO_V1(test_radixtree);
+
+static void
+test_empty(void)
+{
+ rt_radix_tree *radixtree;
+ rt_iter *iter;
+ TestValueType dummy;
+ uint64 key;
+ TestValueType val;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
+ dsa = dsa_create(tranche_id);
+
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ if (rt_search(radixtree, 0, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_search(radixtree, 1, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_search(radixtree, PG_UINT64_MAX, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_delete(radixtree, 0))
+ elog(ERROR, "rt_delete on empty tree returned true");
+
+ if (rt_num_entries(radixtree) != 0)
+ elog(ERROR, "rt_num_entries on empty tree return non-zero");
+
+ iter = rt_begin_iterate(radixtree);
+
+ if (rt_iterate_next(iter, &key, &val))
+ elog(ERROR, "rt_itereate_next on empty tree returned true");
+
+ rt_end_iterate(iter);
+
+ rt_free(radixtree);
+
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+static void
+test_basic(int children, bool test_inner)
+{
+ rt_radix_tree *radixtree;
+ uint64 *keys;
+ int shift = test_inner ? 8 : 0;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing basic operations with %s node %d",
+ test_inner ? "inner" : "leaf", children);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ /* prepare keys in order like 1, 32, 2, 31, 2, ... */
+ keys = palloc(sizeof(uint64) * children);
+ for (int i = 0; i < children; i++)
+ {
+ if (i % 2 == 0)
+ keys[i] = (uint64) ((i / 2) + 1) << shift;
+ else
+ keys[i] = (uint64) (children - (i / 2)) << shift;
+ }
+
+ /* insert keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (rt_set(radixtree, keys[i], (TestValueType*) &keys[i]))
+ elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+ }
+
+ /* look up keys */
+ for (int i = 0; i < children; i++)
+ {
+ TestValueType value;
+
+ if (!rt_search(radixtree, keys[i], &value))
+ elog(ERROR, "could not find key 0x" UINT64_HEX_FORMAT, keys[i]);
+ if (value != (TestValueType) keys[i])
+ elog(ERROR, "rt_search returned 0x" UINT64_HEX_FORMAT ", expected " UINT64_HEX_FORMAT,
+ value, (TestValueType) keys[i]);
+ }
+
+ /* update keys */
+ for (int i = 0; i < children; i++)
+ {
+ TestValueType update = keys[i] + 1;
+ if (!rt_set(radixtree, keys[i], (TestValueType*) &update))
+ elog(ERROR, "could not update key 0x" UINT64_HEX_FORMAT, keys[i]);
+ }
+
+ /* repeat deleting and inserting keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (!rt_delete(radixtree, keys[i]))
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, keys[i]);
+ if (rt_set(radixtree, keys[i], (TestValueType*) &keys[i]))
+ elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+ }
+
+ pfree(keys);
+ rt_free(radixtree);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Check if keys from start to end with the shift exist in the tree.
+ */
+static void
+check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end,
+ int incr)
+{
+ for (int i = start; i < end; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ TestValueType val;
+
+ if (!rt_search(radixtree, key, &val))
+ elog(ERROR, "key 0x" UINT64_HEX_FORMAT " is not found on node-%d",
+ key, end);
+ if (val != (TestValueType) key)
+ elog(ERROR, "rt_search with key 0x" UINT64_HEX_FORMAT " returns 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+ key, val, key);
+ }
+}
+
+static void
+test_node_types_insert(rt_radix_tree *radixtree, uint8 shift, bool insert_asc)
+{
+ uint64 num_entries;
+ int ninserted = 0;
+ int start = insert_asc ? 0 : 256;
+ int incr = insert_asc ? 1 : -1;
+ int end = insert_asc ? 256 : 0;
+ int node_kind_idx = 1;
+
+ for (int i = start; i != end; i += incr)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_set(radixtree, key, (TestValueType*) &key);
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " is found", key);
+
+ /*
+ * After filling all slots in each node type, check if the values
+ * are stored properly.
+ */
+ if (ninserted == rt_node_kind_fanouts[node_kind_idx] - 1)
+ {
+ int check_start = insert_asc
+ ? rt_node_kind_fanouts[node_kind_idx - 1]
+ : rt_node_kind_fanouts[node_kind_idx];
+ int check_end = insert_asc
+ ? rt_node_kind_fanouts[node_kind_idx]
+ : rt_node_kind_fanouts[node_kind_idx - 1];
+
+ check_search_on_node(radixtree, shift, check_start, check_end, incr);
+ node_kind_idx++;
+ }
+
+ ninserted++;
+ }
+
+ num_entries = rt_num_entries(radixtree);
+
+ if (num_entries != 256)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+static void
+test_node_types_delete(rt_radix_tree *radixtree, uint8 shift)
+{
+ uint64 num_entries;
+
+ for (int i = 0; i < 256; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_delete(radixtree, key);
+
+ if (!found)
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, key);
+ }
+
+ num_entries = rt_num_entries(radixtree);
+
+ /* The tree must be empty */
+ if (num_entries != 0)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+/*
+ * Test for inserting and deleting key-value pairs to each node type at the given shift
+ * level.
+ */
+static void
+test_node_types(uint8 shift)
+{
+ rt_radix_tree *radixtree;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing radix tree node types with shift \"%d\"", shift);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ /*
+ * Insert and search entries for every node type at the 'shift' level,
+ * then delete all entries to make it empty, and insert and search entries
+ * again.
+ */
+ test_node_types_insert(radixtree, shift, true);
+ test_node_types_delete(radixtree, shift);
+ test_node_types_insert(radixtree, shift, false);
+
+ rt_free(radixtree);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Test with a repeating pattern, defined by the 'spec'.
+ */
+static void
+test_pattern(const test_spec * spec)
+{
+ rt_radix_tree *radixtree;
+ rt_iter *iter;
+ MemoryContext radixtree_ctx;
+ TimestampTz starttime;
+ TimestampTz endtime;
+ uint64 n;
+ uint64 last_int;
+ uint64 ndeleted;
+ uint64 nbefore;
+ uint64 nafter;
+ int patternlen;
+ uint64 *pattern_values;
+ uint64 pattern_num_values;
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing radix tree with pattern \"%s\"", spec->test_name);
+ if (rt_test_stats)
+ fprintf(stderr, "-----\ntesting radix tree with pattern \"%s\"\n", spec->test_name);
+
+ /* Pre-process the pattern, creating an array of integers from it. */
+ patternlen = strlen(spec->pattern_str);
+ pattern_values = palloc(patternlen * sizeof(uint64));
+ pattern_num_values = 0;
+ for (int i = 0; i < patternlen; i++)
+ {
+ if (spec->pattern_str[i] == '1')
+ pattern_values[pattern_num_values++] = i;
+ }
+
+ /*
+ * Allocate the radix tree.
+ *
+ * Allocate it in a separate memory context, so that we can print its
+ * memory usage easily.
+ */
+ radixtree_ctx = AllocSetContextCreate(CurrentMemoryContext,
+ "radixtree test",
+ ALLOCSET_SMALL_SIZES);
+ MemoryContextSetIdentifier(radixtree_ctx, spec->test_name);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(radixtree_ctx, dsa, tranche_id);
+#else
+ radixtree = rt_create(radixtree_ctx);
+#endif
+
+
+ /*
+ * Add values to the set.
+ */
+ starttime = GetCurrentTimestamp();
+
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ uint64 x = 0;
+
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ bool found;
+
+ x = last_int + pattern_values[i];
+
+ found = rt_set(radixtree, x, (TestValueType*) &x);
+
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " found", x);
+
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+
+ endtime = GetCurrentTimestamp();
+
+ if (rt_test_stats)
+ fprintf(stderr, "added " UINT64_FORMAT " values in %d ms\n",
+ spec->num_values, (int) (endtime - starttime) / 1000);
+
+ /*
+ * Print stats on the amount of memory used.
+ *
+ * We print the usage reported by rt_memory_usage(), as well as the stats
+ * from the memory context. They should be in the same ballpark, but it's
+ * hard to automate testing that, so if you're making changes to the
+ * implementation, just observe that manually.
+ */
+ if (rt_test_stats)
+ {
+ uint64 mem_usage;
+
+ /*
+ * Also print memory usage as reported by rt_memory_usage(). It
+ * should be in the same ballpark as the usage reported by
+ * MemoryContextStats().
+ */
+ mem_usage = rt_memory_usage(radixtree);
+ fprintf(stderr, "rt_memory_usage() reported " UINT64_FORMAT " (%0.2f bytes / integer)\n",
+ mem_usage, (double) mem_usage / spec->num_values);
+
+ MemoryContextStats(radixtree_ctx);
+ }
+
+ /* Check that rt_num_entries works */
+ n = rt_num_entries(radixtree);
+ if (n != spec->num_values)
+ elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT, n, spec->num_values);
+
+ /*
+ * Test random-access probes with rt_search()
+ */
+ starttime = GetCurrentTimestamp();
+
+ for (n = 0; n < 100000; n++)
+ {
+ bool found;
+ bool expected;
+ uint64 x;
+ TestValueType v;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit values that are actually in the set.
+ */
+ x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+ /* Do we expect this value to be present in the set? */
+ if (x >= last_int)
+ expected = false;
+ else
+ {
+ uint64 idx = x % spec->spacing;
+
+ if (idx >= patternlen)
+ expected = false;
+ else if (spec->pattern_str[idx] == '1')
+ expected = true;
+ else
+ expected = false;
+ }
+
+ /* Is it present according to rt_search() ? */
+ found = rt_search(radixtree, x, &v);
+
+ if (found != expected)
+ elog(ERROR, "mismatch at 0x" UINT64_HEX_FORMAT ": %d vs %d", x, found, expected);
+ if (found && (v != (TestValueType) x))
+ elog(ERROR, "found 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+ v, x);
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "probed " UINT64_FORMAT " values in %d ms\n",
+ n, (int) (endtime - starttime) / 1000);
+
+ /*
+ * Test iterator
+ */
+ starttime = GetCurrentTimestamp();
+
+ iter = rt_begin_iterate(radixtree);
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ uint64 expected = last_int + pattern_values[i];
+ uint64 x;
+ TestValueType val;
+
+ if (!rt_iterate_next(iter, &x, &val))
+ break;
+
+ if (x != expected)
+ elog(ERROR,
+ "iterate returned wrong key; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d",
+ x, expected, i);
+ if (val != (TestValueType) expected)
+ elog(ERROR,
+ "iterate returned wrong value; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d", x, expected, i);
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "iterated " UINT64_FORMAT " values in %d ms\n",
+ n, (int) (endtime - starttime) / 1000);
+
+ rt_end_iterate(iter);
+
+ if (n < spec->num_values)
+ elog(ERROR, "iterator stopped short after " UINT64_FORMAT " entries, expected " UINT64_FORMAT, n, spec->num_values);
+ if (n > spec->num_values)
+ elog(ERROR, "iterator returned " UINT64_FORMAT " entries, " UINT64_FORMAT " was expected", n, spec->num_values);
+
+ /*
+ * Test random-access probes with rt_delete()
+ */
+ starttime = GetCurrentTimestamp();
+
+ nbefore = rt_num_entries(radixtree);
+ ndeleted = 0;
+ for (n = 0; n < 1; n++)
+ {
+ bool found;
+ uint64 x;
+ TestValueType v;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit values that are actually in the set.
+ */
+ x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+ /* Is it present according to rt_search() ? */
+ found = rt_search(radixtree, x, &v);
+
+ if (!found)
+ continue;
+
+ /* If the key is found, delete it and check again */
+ if (!rt_delete(radixtree, x))
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, x);
+ if (rt_search(radixtree, x, &v))
+ elog(ERROR, "found deleted key 0x" UINT64_HEX_FORMAT, x);
+ if (rt_delete(radixtree, x))
+ elog(ERROR, "deleted already-deleted key 0x" UINT64_HEX_FORMAT, x);
+
+ ndeleted++;
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "deleted " UINT64_FORMAT " values in %d ms\n",
+ ndeleted, (int) (endtime - starttime) / 1000);
+
+ nafter = rt_num_entries(radixtree);
+
+ /* Check that rt_num_entries works */
+ if ((nbefore - ndeleted) != nafter)
+ elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT "after " UINT64_FORMAT " deletion",
+ nafter, (nbefore - ndeleted), ndeleted);
+
+ rt_free(radixtree);
+ MemoryContextDelete(radixtree_ctx);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+Datum
+test_radixtree(PG_FUNCTION_ARGS)
+{
+ test_empty();
+
+ for (int i = 1; i < lengthof(rt_node_kind_fanouts); i++)
+ {
+ test_basic(rt_node_kind_fanouts[i], false);
+ test_basic(rt_node_kind_fanouts[i], true);
+ }
+
+ for (int shift = 0; shift <= (64 - 8); shift += 8)
+ test_node_types(shift);
+
+ /* Test different test patterns, with lots of entries */
+ for (int i = 0; i < lengthof(test_specs); i++)
+ test_pattern(&test_specs[i]);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_radixtree/test_radixtree.control b/src/test/modules/test_radixtree/test_radixtree.control
new file mode 100644
index 0000000000..e53f2a3e0c
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.control
@@ -0,0 +1,4 @@
+comment = 'Test code for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/test_radixtree'
+relocatable = true
diff --git a/src/tools/pginclude/cpluspluscheck b/src/tools/pginclude/cpluspluscheck
index e52fe9f509..09fa6e7432 100755
--- a/src/tools/pginclude/cpluspluscheck
+++ b/src/tools/pginclude/cpluspluscheck
@@ -101,6 +101,12 @@ do
test "$f" = src/include/nodes/nodetags.h && continue
test "$f" = src/backend/nodes/nodetags.h && continue
+ # radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+ test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+ test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+ test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+ test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
# These files are not meant to be included standalone, because
# they contain lists that might have multiple use-cases.
test "$f" = src/include/access/rmgrlist.h && continue
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index abbba7aa63..d4d2f1da03 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -96,6 +96,12 @@ do
test "$f" = src/include/nodes/nodetags.h && continue
test "$f" = src/backend/nodes/nodetags.h && continue
+ # radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+ test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+ test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+ test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+ test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
# These files are not meant to be included standalone, because
# they contain lists that might have multiple use-cases.
test "$f" = src/include/access/rmgrlist.h && continue
--
2.31.1
[application/octet-stream] v29-0004-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch (36.0K, ../../CAD21AoAFiw2DS07nPhiS14eVXpnF-bBmjvab4y+Ct8igU2z_8A@mail.gmail.com/11-v29-0004-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch)
download | inline diff:
From 7a4bf52d585e41926b6a85cb7ae64be177cc0d04 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 23 Dec 2022 23:50:39 +0900
Subject: [PATCH v29 04/10] Add TIDStore, to store sets of TIDs
(ItemPointerData) efficiently.
The TIDStore is designed to store large sets of TIDs efficiently, and
is backed by the radix tree. A TID is encoded into 64-bit key and
value and inserted to the radix tree.
The TIDStore is not used for anything yet, aside from the test code,
but the follow up patch integrates the TIDStore with lazy vacuum,
reducing lazy vacuum memory usage and lifting the 1GB limit on its
size, by storing the list of dead TIDs more efficiently.
This includes a unit test module, in src/test/modules/test_tidstore.
---
doc/src/sgml/monitoring.sgml | 4 +
src/backend/access/common/Makefile | 1 +
src/backend/access/common/meson.build | 1 +
src/backend/access/common/tidstore.c | 681 ++++++++++++++++++
src/backend/storage/lmgr/lwlock.c | 2 +
src/include/access/tidstore.h | 49 ++
src/include/storage/lwlock.h | 1 +
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_tidstore/Makefile | 23 +
.../test_tidstore/expected/test_tidstore.out | 13 +
src/test/modules/test_tidstore/meson.build | 35 +
.../test_tidstore/sql/test_tidstore.sql | 7 +
.../test_tidstore/test_tidstore--1.0.sql | 8 +
.../modules/test_tidstore/test_tidstore.c | 226 ++++++
.../test_tidstore/test_tidstore.control | 4 +
16 files changed, 1057 insertions(+)
create mode 100644 src/backend/access/common/tidstore.c
create mode 100644 src/include/access/tidstore.h
create mode 100644 src/test/modules/test_tidstore/Makefile
create mode 100644 src/test/modules/test_tidstore/expected/test_tidstore.out
create mode 100644 src/test/modules/test_tidstore/meson.build
create mode 100644 src/test/modules/test_tidstore/sql/test_tidstore.sql
create mode 100644 src/test/modules/test_tidstore/test_tidstore--1.0.sql
create mode 100644 src/test/modules/test_tidstore/test_tidstore.c
create mode 100644 src/test/modules/test_tidstore/test_tidstore.control
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index dca50707ad..e28206e056 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2198,6 +2198,10 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<entry>Waiting to access a shared TID bitmap during a parallel bitmap
index scan.</entry>
</row>
+ <row>
+ <entry><literal>SharedTidStore</literal></entry>
+ <entry>Waiting to access a shared TID store.</entry>
+ </row>
<row>
<entry><literal>SharedTupleStore</literal></entry>
<entry>Waiting to access a shared tuple store during parallel
diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile
index b9aff0ccfd..67b8cc6108 100644
--- a/src/backend/access/common/Makefile
+++ b/src/backend/access/common/Makefile
@@ -27,6 +27,7 @@ OBJS = \
syncscan.o \
toast_compression.o \
toast_internals.o \
+ tidstore.o \
tupconvert.o \
tupdesc.o
diff --git a/src/backend/access/common/meson.build b/src/backend/access/common/meson.build
index f5ac17b498..fce19c09ce 100644
--- a/src/backend/access/common/meson.build
+++ b/src/backend/access/common/meson.build
@@ -15,6 +15,7 @@ backend_sources += files(
'syncscan.c',
'toast_compression.c',
'toast_internals.c',
+ 'tidstore.c',
'tupconvert.c',
'tupdesc.c',
)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
new file mode 100644
index 0000000000..8c05e60d92
--- /dev/null
+++ b/src/backend/access/common/tidstore.c
@@ -0,0 +1,681 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.c
+ * Tid (ItemPointerData) storage implementation.
+ *
+ * This module provides a in-memory data structure to store Tids (ItemPointer).
+ * Internally, a tid is encoded as a pair of 64-bit key and 64-bit value, and
+ * stored in the radix tree.
+ *
+ * A TidStore can be shared among parallel worker processes by passing DSA area
+ * to tidstore_create(). Other backends can attach to the shared TidStore by
+ * tidstore_attach().
+ *
+ * Regarding the concurrency, it basically relies on the concurrency support in
+ * the radix tree, but we acquires the lock on a TidStore in some cases, for
+ * example, when to reset the store and when to access the number tids in the
+ * store (num_tids).
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/access/common/tidstore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/tidstore.h"
+#include "miscadmin.h"
+#include "port/pg_bitutils.h"
+#include "storage/lwlock.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/*
+ * For encoding purposes, tids are represented as a pair of 64-bit key and
+ * 64-bit value. First, we construct 64-bit unsigned integer by combining
+ * the block number and the offset number. The number of bits used for the
+ * offset number is specified by max_offsets in tidstore_create(). We are
+ * frugal with the bits, because smaller keys could help keeping the radix
+ * tree shallow.
+ *
+ * For example, a tid of heap with 8kB blocks uses the lowest 9 bits for
+ * the offset number and uses the next 32 bits for the block number. That
+ * is, only 41 bits are used:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ *
+ * X = bits used for offset number
+ * Y = bits used for block number
+ * u = unused bit
+ * (high on the left, low on the right)
+ *
+ * 9 bits are enough for the offset number, because MaxHeapTuplesPerPage < 2^9
+ * on 8kB blocks.
+ *
+ * The 64-bit value is the bitmap representation of the lowest 6 bits
+ * (TIDSTORE_VALUE_NBITS) of the integer, and the rest 35 bits are used
+ * as the key:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ * |----| value
+ * |---------------------------------------------| key
+ *
+ * The maximum height of the radix tree is 5 in this case.
+ */
+#define TIDSTORE_VALUE_NBITS 6 /* log(64, 2) */
+#define TIDSTORE_OFFSET_MASK ((1 << TIDSTORE_VALUE_NBITS) - 1)
+
+/* A magic value used to identify our TidStores. */
+#define TIDSTORE_MAGIC 0x826f6a10
+
+#define RT_PREFIX local_rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE uint64
+#include "lib/radixtree.h"
+
+#define RT_PREFIX shared_rt
+#define RT_SHMEM
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE uint64
+#include "lib/radixtree.h"
+
+/* The control object for a TidStore */
+typedef struct TidStoreControl
+{
+ /* the number of tids in the store */
+ int64 num_tids;
+
+ /* These values are never changed after creation */
+ size_t max_bytes; /* the maximum bytes a TidStore can use */
+ int max_offset; /* the maximum offset number */
+ int offset_nbits; /* the number of bits required for an offset
+ * number */
+ int offset_key_nbits; /* the number of bits of an offset number
+ * used in a key */
+
+ /* The below fields are used only in shared case */
+
+ uint32 magic;
+ LWLock lock;
+
+ /* handles for TidStore and radix tree */
+ tidstore_handle handle;
+ shared_rt_handle tree_handle;
+} TidStoreControl;
+
+/* Per-backend state for a TidStore */
+struct TidStore
+{
+ /*
+ * Control object. This is allocated in DSA area 'area' in the shared
+ * case, otherwise in backend-local memory.
+ */
+ TidStoreControl *control;
+
+ /* Storage for Tids. Use either one depending on TidStoreIsShared() */
+ union
+ {
+ local_rt_radix_tree *local;
+ shared_rt_radix_tree *shared;
+ } tree;
+
+ /* DSA area for TidStore if used */
+ dsa_area *area;
+};
+#define TidStoreIsShared(ts) ((ts)->area != NULL)
+
+/* Iterator for TidStore */
+typedef struct TidStoreIter
+{
+ TidStore *ts;
+
+ /* iterator of radix tree. Use either one depending on TidStoreIsShared() */
+ union
+ {
+ shared_rt_iter *shared;
+ local_rt_iter *local;
+ } tree_iter;
+
+ /* we returned all tids? */
+ bool finished;
+
+ /* save for the next iteration */
+ uint64 next_key;
+ uint64 next_val;
+
+ /* output for the caller */
+ TidStoreIterResult result;
+} TidStoreIter;
+
+static void tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val);
+static inline BlockNumber key_get_blkno(TidStore *ts, uint64 key);
+static inline uint64 encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit);
+static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit);
+
+/*
+ * Create a TidStore. The returned object is allocated in backend-local memory.
+ * The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
+ */
+TidStore *
+tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
+{
+ TidStore *ts;
+
+ ts = palloc0(sizeof(TidStore));
+
+ /*
+ * Create the radix tree for the main storage.
+ *
+ * Memory consumption depends on the number of stored tids, but also on the
+ * distribution of them, how the radix tree stores, and the memory management
+ * that backed the radix tree. The maximum bytes that a TidStore can
+ * use is specified by the max_bytes in tidstore_create(). We want the total
+ * amount of memory consumption by a TidStore not to exceed the max_bytes.
+ *
+ * In local TidStore cases, the radix tree uses slab allocators for each kind
+ * of node class. The most memory consuming case while adding Tids associated
+ * with one page (i.e. during tidstore_add_tids()) is that we allocate a new
+ * slab block for a new radix tree node, which is approximately 70kB. Therefore,
+ * we deduct 70kB from the max_bytes.
+ *
+ * In shared cases, DSA allocates the memory segments big enough to follow
+ * a geometric series that approximately doubles the total DSA size (see
+ * make_new_segment() in dsa.c). We simulated the how DSA increases segment
+ * size and the simulation revealed, the 75% threshold for the maximum bytes
+ * perfectly works in case where the max_bytes is a power-of-2, and the 60%
+ * threshold works for other cases.
+ */
+ if (area != NULL)
+ {
+ dsa_pointer dp;
+ float ratio = ((max_bytes & (max_bytes - 1)) == 0) ? 0.75 : 0.6;
+
+ ts->tree.shared = shared_rt_create(CurrentMemoryContext, area,
+ LWTRANCHE_SHARED_TIDSTORE);
+
+ dp = dsa_allocate0(area, sizeof(TidStoreControl));
+ ts->control = (TidStoreControl *) dsa_get_address(area, dp);
+ ts->control->max_bytes = (uint64) (max_bytes * ratio);
+ ts->area = area;
+
+ ts->control->magic = TIDSTORE_MAGIC;
+ LWLockInitialize(&ts->control->lock, LWTRANCHE_SHARED_TIDSTORE);
+ ts->control->handle = dp;
+ ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+ }
+ else
+ {
+ ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+ ts->control = (TidStoreControl *) palloc0(sizeof(TidStoreControl));
+ ts->control->max_bytes = max_bytes - (70 * 1024);
+ }
+
+ ts->control->max_offset = max_offset;
+ ts->control->offset_nbits = pg_ceil_log2_32(max_offset);
+
+ if (ts->control->offset_nbits < TIDSTORE_VALUE_NBITS)
+ ts->control->offset_nbits = TIDSTORE_VALUE_NBITS;
+
+ ts->control->offset_key_nbits =
+ ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
+
+ return ts;
+}
+
+/*
+ * Attach to the shared TidStore using a handle. The returned object is
+ * allocated in backend-local memory using the CurrentMemoryContext.
+ */
+TidStore *
+tidstore_attach(dsa_area *area, tidstore_handle handle)
+{
+ TidStore *ts;
+ dsa_pointer control;
+
+ Assert(area != NULL);
+ Assert(DsaPointerIsValid(handle));
+
+ /* create per-backend state */
+ ts = palloc0(sizeof(TidStore));
+
+ /* Find the control object in shared memory */
+ control = handle;
+
+ /* Set up the TidStore */
+ ts->control = (TidStoreControl *) dsa_get_address(area, control);
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ ts->tree.shared = shared_rt_attach(area, ts->control->tree_handle);
+ ts->area = area;
+
+ return ts;
+}
+
+/*
+ * Detach from a TidStore. This detaches from radix tree and frees the
+ * backend-local resources. The radix tree will continue to exist until
+ * it is either explicitly destroyed, or the area that backs it is returned
+ * to the operating system.
+ */
+void
+tidstore_detach(TidStore *ts)
+{
+ Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+ shared_rt_detach(ts->tree.shared);
+ pfree(ts);
+}
+
+/*
+ * Destroy a TidStore, returning all memory.
+ *
+ * TODO: The caller must be certain that no other backend will attempt to
+ * access the TidStore before calling this function. Other backend must
+ * explicitly call tidstore_detach to free up backend-local memory associated
+ * with the TidStore. The backend that calls tidstore_destroy must not call
+ * tidstore_detach.
+ */
+void
+tidstore_destroy(TidStore *ts)
+{
+ if (TidStoreIsShared(ts))
+ {
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ /*
+ * Vandalize the control block to help catch programming error where
+ * other backends access the memory formerly occupied by this radix
+ * tree.
+ */
+ ts->control->magic = 0;
+ dsa_free(ts->area, ts->control->handle);
+ shared_rt_free(ts->tree.shared);
+ }
+ else
+ {
+ pfree(ts->control);
+ local_rt_free(ts->tree.local);
+ }
+
+ pfree(ts);
+}
+
+/*
+ * Forget all collected Tids. It's similar to tidstore_destroy but we don't free
+ * entire TidStore but recreate only the radix tree storage.
+ */
+void
+tidstore_reset(TidStore *ts)
+{
+ if (TidStoreIsShared(ts))
+ {
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+ /*
+ * Free the radix tree and return allocated DSA segments to
+ * the operating system.
+ */
+ shared_rt_free(ts->tree.shared);
+ dsa_trim(ts->area);
+
+ /* Recreate the radix tree */
+ ts->tree.shared = shared_rt_create(CurrentMemoryContext, ts->area,
+ LWTRANCHE_SHARED_TIDSTORE);
+
+ /* update the radix tree handle as we recreated it */
+ ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
+
+ LWLockRelease(&ts->control->lock);
+ }
+ else
+ {
+ local_rt_free(ts->tree.local);
+ ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
+ }
+}
+
+/* Add Tids on a block to TidStore */
+void
+tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets)
+{
+ uint64 *values;
+ uint64 key;
+ uint64 prev_key;
+ uint64 off_bitmap = 0;
+ int idx;
+ const uint64 key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
+ const int nkeys = UINT64CONST(1) << ts->control->offset_key_nbits;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ values = palloc(sizeof(uint64) * nkeys);
+ key = prev_key = key_base;
+
+ for (int i = 0; i < num_offsets; i++)
+ {
+ uint64 off_bit;
+
+ /* encode the tid to a key and partial offset */
+ key = encode_key_off(ts, blkno, offsets[i], &off_bit);
+
+ /* make sure we scanned the line pointer array in order */
+ Assert(key >= prev_key);
+
+ if (key > prev_key)
+ {
+ idx = prev_key - key_base;
+ Assert(idx >= 0 && idx < nkeys);
+
+ /* write out offset bitmap for this key */
+ values[idx] = off_bitmap;
+
+ /* zero out any gaps up to the current key */
+ for (int empty_idx = idx + 1; empty_idx < key - key_base; empty_idx++)
+ values[empty_idx] = 0;
+
+ /* reset for current key -- the current offset will be handled below */
+ off_bitmap = 0;
+ prev_key = key;
+ }
+
+ off_bitmap |= off_bit;
+ }
+
+ /* save the final index for later */
+ idx = key - key_base;
+ /* write out last offset bitmap */
+ values[idx] = off_bitmap;
+
+ if (TidStoreIsShared(ts))
+ LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+ /* insert the calculated key-values to the tree */
+ for (int i = 0; i <= idx; i++)
+ {
+ if (values[i])
+ {
+ key = key_base + i;
+
+ if (TidStoreIsShared(ts))
+ shared_rt_set(ts->tree.shared, key, &values[i]);
+ else
+ local_rt_set(ts->tree.local, key, &values[i]);
+ }
+ }
+
+ /* update statistics */
+ ts->control->num_tids += num_offsets;
+
+ if (TidStoreIsShared(ts))
+ LWLockRelease(&ts->control->lock);
+
+ pfree(values);
+}
+
+/* Return true if the given tid is present in the TidStore */
+bool
+tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
+{
+ uint64 key;
+ uint64 val = 0;
+ uint64 off_bit;
+ bool found;
+
+ key = tid_to_key_off(ts, tid, &off_bit);
+
+ if (TidStoreIsShared(ts))
+ found = shared_rt_search(ts->tree.shared, key, &val);
+ else
+ found = local_rt_search(ts->tree.local, key, &val);
+
+ if (!found)
+ return false;
+
+ return (val & off_bit) != 0;
+}
+
+/*
+ * Prepare to iterate through a TidStore. Since the radix tree is locked during the
+ * iteration, so tidstore_end_iterate() needs to called when finished.
+ *
+ * Concurrent updates during the iteration will be blocked when inserting a
+ * key-value to the radix tree.
+ */
+TidStoreIter *
+tidstore_begin_iterate(TidStore *ts)
+{
+ TidStoreIter *iter;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ iter = palloc0(sizeof(TidStoreIter));
+ iter->ts = ts;
+
+ iter->result.blkno = InvalidBlockNumber;
+ iter->result.offsets = palloc(sizeof(OffsetNumber) * ts->control->max_offset);
+
+ if (TidStoreIsShared(ts))
+ iter->tree_iter.shared = shared_rt_begin_iterate(ts->tree.shared);
+ else
+ iter->tree_iter.local = local_rt_begin_iterate(ts->tree.local);
+
+ /* If the TidStore is empty, there is no business */
+ if (tidstore_num_tids(ts) == 0)
+ iter->finished = true;
+
+ return iter;
+}
+
+static inline bool
+tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
+{
+ if (TidStoreIsShared(iter->ts))
+ return shared_rt_iterate_next(iter->tree_iter.shared, key, val);
+
+ return local_rt_iterate_next(iter->tree_iter.local, key, val);
+}
+
+/*
+ * Scan the TidStore and return a pointer to TidStoreIterResult that has tids
+ * in one block. We return the block numbers in ascending order and the offset
+ * numbers in each result is also sorted in ascending order.
+ */
+TidStoreIterResult *
+tidstore_iterate_next(TidStoreIter *iter)
+{
+ uint64 key;
+ uint64 val;
+ TidStoreIterResult *result = &(iter->result);
+
+ if (iter->finished)
+ return NULL;
+
+ if (BlockNumberIsValid(result->blkno))
+ {
+ /* Process the previously collected key-value */
+ result->num_offsets = 0;
+ tidstore_iter_extract_tids(iter, iter->next_key, iter->next_val);
+ }
+
+ while (tidstore_iter_kv(iter, &key, &val))
+ {
+ BlockNumber blkno;
+
+ blkno = key_get_blkno(iter->ts, key);
+
+ if (BlockNumberIsValid(result->blkno) && result->blkno != blkno)
+ {
+ /*
+ * We got a key-value pair for a different block. So return the
+ * collected tids, and remember the key-value for the next iteration.
+ */
+ iter->next_key = key;
+ iter->next_val = val;
+ return result;
+ }
+
+ /* Collect tids extracted from the key-value pair */
+ tidstore_iter_extract_tids(iter, key, val);
+ }
+
+ iter->finished = true;
+ return result;
+}
+
+/*
+ * Finish an iteration over TidStore. This needs to be called after finishing
+ * or when existing an iteration.
+ */
+void
+tidstore_end_iterate(TidStoreIter *iter)
+{
+ if (TidStoreIsShared(iter->ts))
+ shared_rt_end_iterate(iter->tree_iter.shared);
+ else
+ local_rt_end_iterate(iter->tree_iter.local);
+
+ pfree(iter->result.offsets);
+ pfree(iter);
+}
+
+/* Return the number of tids we collected so far */
+int64
+tidstore_num_tids(TidStore *ts)
+{
+ uint64 num_tids;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ if (!TidStoreIsShared(ts))
+ return ts->control->num_tids;
+
+ LWLockAcquire(&ts->control->lock, LW_SHARED);
+ num_tids = ts->control->num_tids;
+ LWLockRelease(&ts->control->lock);
+
+ return num_tids;
+}
+
+/* Return true if the current memory usage of TidStore exceeds the limit */
+bool
+tidstore_is_full(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ return (tidstore_memory_usage(ts) > ts->control->max_bytes);
+}
+
+/* Return the maximum memory TidStore can use */
+size_t
+tidstore_max_memory(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ return ts->control->max_bytes;
+}
+
+/* Return the memory usage of TidStore */
+size_t
+tidstore_memory_usage(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ /*
+ * In the shared case, TidStoreControl and radix_tree are backed by the
+ * same DSA area and rt_memory_usage() returns the value including both.
+ * So we don't need to add the size of TidStoreControl separately.
+ */
+ if (TidStoreIsShared(ts))
+ return sizeof(TidStore) + shared_rt_memory_usage(ts->tree.shared);
+
+ return sizeof(TidStore) + sizeof(TidStore) + local_rt_memory_usage(ts->tree.local);
+}
+
+/*
+ * Get a handle that can be used by other processes to attach to this TidStore
+ */
+tidstore_handle
+tidstore_get_handle(TidStore *ts)
+{
+ Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+ return ts->control->handle;
+}
+
+/* Extract tids from the given key-value pair */
+static void
+tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
+{
+ TidStoreIterResult *result = (&iter->result);
+
+ while (val)
+ {
+ uint64 tid_i;
+ OffsetNumber off;
+
+ tid_i = key << TIDSTORE_VALUE_NBITS;
+ tid_i |= pg_rightmost_one_pos64(val);
+
+ off = tid_i & ((UINT64CONST(1) << iter->ts->control->offset_nbits) - 1);
+
+ Assert(result->num_offsets < iter->ts->control->max_offset);
+ result->offsets[result->num_offsets++] = off;
+
+ /* unset the rightmost bit */
+ val &= ~pg_rightmost_one64(val);
+ }
+
+ result->blkno = key_get_blkno(iter->ts, key);
+}
+
+/* Get block number from the given key */
+static inline BlockNumber
+key_get_blkno(TidStore *ts, uint64 key)
+{
+ return (BlockNumber) (key >> ts->control->offset_key_nbits);
+}
+
+/* Encode a tid to key and offset */
+static inline uint64
+tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit)
+{
+ uint32 offset = ItemPointerGetOffsetNumber(tid);
+ BlockNumber block = ItemPointerGetBlockNumber(tid);
+
+ return encode_key_off(ts, block, offset, off_bit);
+}
+
+/* encode a block and offset to a key and partial offset */
+static inline uint64
+encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit)
+{
+ uint64 key;
+ uint64 tid_i;
+ uint32 off_lower;
+
+ off_lower = offset & TIDSTORE_OFFSET_MASK;
+ Assert(off_lower < (sizeof(uint64) * BITS_PER_BYTE));
+
+ *off_bit = UINT64CONST(1) << off_lower;
+ tid_i = offset | ((uint64) block << ts->control->offset_nbits);
+ key = tid_i >> TIDSTORE_VALUE_NBITS;
+
+ return key;
+}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index d2ec396045..55b3a04097 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -176,6 +176,8 @@ static const char *const BuiltinTrancheNames[] = {
"SharedTupleStore",
/* LWTRANCHE_SHARED_TIDBITMAP: */
"SharedTidBitmap",
+ /* LWTRANCHE_SHARED_TIDSTORE: */
+ "SharedTidStore",
/* LWTRANCHE_PARALLEL_APPEND: */
"ParallelAppend",
/* LWTRANCHE_PER_XACT_PREDICATE_LIST: */
diff --git a/src/include/access/tidstore.h b/src/include/access/tidstore.h
new file mode 100644
index 0000000000..a35a52124a
--- /dev/null
+++ b/src/include/access/tidstore.h
@@ -0,0 +1,49 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.h
+ * Tid storage.
+ *
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/tidstore.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef TIDSTORE_H
+#define TIDSTORE_H
+
+#include "storage/itemptr.h"
+#include "utils/dsa.h"
+
+typedef dsa_pointer tidstore_handle;
+
+typedef struct TidStore TidStore;
+typedef struct TidStoreIter TidStoreIter;
+
+typedef struct TidStoreIterResult
+{
+ BlockNumber blkno;
+ OffsetNumber *offsets;
+ int num_offsets;
+} TidStoreIterResult;
+
+extern TidStore *tidstore_create(size_t max_bytes, int max_offset, dsa_area *dsa);
+extern TidStore *tidstore_attach(dsa_area *dsa, dsa_pointer handle);
+extern void tidstore_detach(TidStore *ts);
+extern void tidstore_destroy(TidStore *ts);
+extern void tidstore_reset(TidStore *ts);
+extern void tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets);
+extern bool tidstore_lookup_tid(TidStore *ts, ItemPointer tid);
+extern TidStoreIter * tidstore_begin_iterate(TidStore *ts);
+extern TidStoreIterResult *tidstore_iterate_next(TidStoreIter *iter);
+extern void tidstore_end_iterate(TidStoreIter *iter);
+extern int64 tidstore_num_tids(TidStore *ts);
+extern bool tidstore_is_full(TidStore *ts);
+extern size_t tidstore_max_memory(TidStore *ts);
+extern size_t tidstore_memory_usage(TidStore *ts);
+extern tidstore_handle tidstore_get_handle(TidStore *ts);
+
+#endif /* TIDSTORE_H */
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index d2c7afb8f4..07002fdfbe 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -199,6 +199,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_PER_SESSION_RECORD_TYPMOD,
LWTRANCHE_SHARED_TUPLESTORE,
LWTRANCHE_SHARED_TIDBITMAP,
+ LWTRANCHE_SHARED_TIDSTORE,
LWTRANCHE_PARALLEL_APPEND,
LWTRANCHE_PER_XACT_PREDICATE_LIST,
LWTRANCHE_PGSTATS_DSA,
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 9659eb85d7..bddc16ada7 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
test_rls_hooks \
test_shm_mq \
test_slru \
+ test_tidstore \
unsafe_tests \
worker_spi
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 232cbdac80..c0d5645ad8 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -30,5 +30,6 @@ subdir('test_regex')
subdir('test_rls_hooks')
subdir('test_shm_mq')
subdir('test_slru')
+subdir('test_tidstore')
subdir('unsafe_tests')
subdir('worker_spi')
diff --git a/src/test/modules/test_tidstore/Makefile b/src/test/modules/test_tidstore/Makefile
new file mode 100644
index 0000000000..dab107d70c
--- /dev/null
+++ b/src/test/modules/test_tidstore/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_tidstore/Makefile
+
+MODULE_big = test_tidstore
+OBJS = \
+ $(WIN32RES) \
+ test_tidstore.o
+PGFILEDESC = "test_tidstore - test code for src/backend/access/common/tidstore.c"
+
+EXTENSION = test_tidstore
+DATA = test_tidstore--1.0.sql
+
+REGRESS = test_tidstore
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_tidstore
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_tidstore/expected/test_tidstore.out b/src/test/modules/test_tidstore/expected/test_tidstore.out
new file mode 100644
index 0000000000..7ff2f9af87
--- /dev/null
+++ b/src/test/modules/test_tidstore/expected/test_tidstore.out
@@ -0,0 +1,13 @@
+CREATE EXTENSION test_tidstore;
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
+NOTICE: testing empty tidstore
+NOTICE: testing basic operations
+ test_tidstore
+---------------
+
+(1 row)
+
diff --git a/src/test/modules/test_tidstore/meson.build b/src/test/modules/test_tidstore/meson.build
new file mode 100644
index 0000000000..31f2da7b61
--- /dev/null
+++ b/src/test/modules/test_tidstore/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_tidstore_sources = files(
+ 'test_tidstore.c',
+)
+
+if host_system == 'windows'
+ test_tidstore_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_tidstore',
+ '--FILEDESC', 'test_tidstore - test code for src/backend/access/common/tidstore.c',])
+endif
+
+test_tidstore = shared_module('test_tidstore',
+ test_tidstore_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += test_tidstore
+
+install_data(
+ 'test_tidstore.control',
+ 'test_tidstore--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'test_tidstore',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'test_tidstore',
+ ],
+ },
+}
diff --git a/src/test/modules/test_tidstore/sql/test_tidstore.sql b/src/test/modules/test_tidstore/sql/test_tidstore.sql
new file mode 100644
index 0000000000..03aea31815
--- /dev/null
+++ b/src/test/modules/test_tidstore/sql/test_tidstore.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_tidstore;
+
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
diff --git a/src/test/modules/test_tidstore/test_tidstore--1.0.sql b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
new file mode 100644
index 0000000000..47e9149900
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_tidstore/test_tidstore--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_tidstore" to load this file. \quit
+
+CREATE FUNCTION test_tidstore()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_tidstore/test_tidstore.c b/src/test/modules/test_tidstore/test_tidstore.c
new file mode 100644
index 0000000000..9a1217f833
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.c
@@ -0,0 +1,226 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_tidstore.c
+ * Test TidStore data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_tidstore/test_tidstore.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/tidstore.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "storage/lwlock.h"
+#include "utils/memutils.h"
+
+PG_MODULE_MAGIC;
+
+/* #define TEST_SHARED_TIDSTORE 1 */
+
+#define TEST_TIDSTORE_MAX_BYTES (2 * 1024 * 1024L) /* 2MB */
+
+PG_FUNCTION_INFO_V1(test_tidstore);
+
+static void
+check_tid(TidStore *ts, BlockNumber blkno, OffsetNumber off, bool expect)
+{
+ ItemPointerData tid;
+ bool found;
+
+ ItemPointerSet(&tid, blkno, off);
+
+ found = tidstore_lookup_tid(ts, &tid);
+
+ if (found != expect)
+ elog(ERROR, "lookup TID (%u, %u) returned %d, expected %d",
+ blkno, off, found, expect);
+}
+
+static void
+test_basic(int max_offset)
+{
+#define TEST_TIDSTORE_NUM_BLOCKS 5
+#define TEST_TIDSTORE_NUM_OFFSETS 5
+
+ TidStore *ts;
+ TidStoreIter *iter;
+ TidStoreIterResult *iter_result;
+ BlockNumber blks[TEST_TIDSTORE_NUM_BLOCKS] = {
+ 0, MaxBlockNumber, MaxBlockNumber - 1, 1, MaxBlockNumber / 2,
+ };
+ BlockNumber blks_sorted[TEST_TIDSTORE_NUM_BLOCKS] = {
+ 0, 1, MaxBlockNumber / 2, MaxBlockNumber - 1, MaxBlockNumber
+ };
+ OffsetNumber offs[TEST_TIDSTORE_NUM_OFFSETS];
+ int blk_idx;
+
+#ifdef TEST_SHARED_TIDSTORE
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_tidstore");
+ dsa = dsa_create(tranche_id);
+
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, dsa);
+#else
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
+#endif
+
+ /* prepare the offset array */
+ offs[0] = FirstOffsetNumber;
+ offs[1] = FirstOffsetNumber + 1;
+ offs[2] = max_offset / 2;
+ offs[3] = max_offset - 1;
+ offs[4] = max_offset;
+
+ /* add tids */
+ for (int i = 0; i < TEST_TIDSTORE_NUM_BLOCKS; i++)
+ tidstore_add_tids(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
+
+ /* lookup test */
+ for (OffsetNumber off = FirstOffsetNumber ; off < max_offset; off++)
+ {
+ bool expect = false;
+ for (int i = 0; i < TEST_TIDSTORE_NUM_OFFSETS; i++)
+ {
+ if (offs[i] == off)
+ {
+ expect = true;
+ break;
+ }
+ }
+
+ check_tid(ts, 0, off, expect);
+ check_tid(ts, 2, off, false);
+ check_tid(ts, MaxBlockNumber - 2, off, false);
+ check_tid(ts, MaxBlockNumber, off, expect);
+ }
+
+ /* test the number of tids */
+ if (tidstore_num_tids(ts) != (TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS))
+ elog(ERROR, "tidstore_num_tids returned " UINT64_FORMAT ", expected %d",
+ tidstore_num_tids(ts),
+ TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS);
+
+ /* iteration test */
+ iter = tidstore_begin_iterate(ts);
+ blk_idx = 0;
+ while ((iter_result = tidstore_iterate_next(iter)) != NULL)
+ {
+ /* check the returned block number */
+ if (blks_sorted[blk_idx] != iter_result->blkno)
+ elog(ERROR, "tidstore_iterate_next returned block number %u, expected %u",
+ iter_result->blkno, blks_sorted[blk_idx]);
+
+ /* check the returned offset numbers */
+ if (TEST_TIDSTORE_NUM_OFFSETS != iter_result->num_offsets)
+ elog(ERROR, "tidstore_iterate_next returned %u offsets, expected %u",
+ iter_result->num_offsets, TEST_TIDSTORE_NUM_OFFSETS);
+
+ for (int i = 0; i < iter_result->num_offsets; i++)
+ {
+ if (offs[i] != iter_result->offsets[i])
+ elog(ERROR, "tidstore_iterate_next returned offset number %u on block %u, expected %u",
+ iter_result->offsets[i], iter_result->blkno, offs[i]);
+ }
+
+ blk_idx++;
+ }
+
+ if (blk_idx != TEST_TIDSTORE_NUM_BLOCKS)
+ elog(ERROR, "tidstore_iterate_next returned %d blocks, expected %d",
+ blk_idx, TEST_TIDSTORE_NUM_BLOCKS);
+
+ /* remove all tids */
+ tidstore_reset(ts);
+
+ /* test the number of tids */
+ if (tidstore_num_tids(ts) != 0)
+ elog(ERROR, "tidstore_num_tids on empty store returned non-zero");
+
+ /* lookup test for empty store */
+ for (OffsetNumber off = FirstOffsetNumber ; off < MaxHeapTuplesPerPage;
+ off++)
+ {
+ check_tid(ts, 0, off, false);
+ check_tid(ts, 2, off, false);
+ check_tid(ts, MaxBlockNumber - 2, off, false);
+ check_tid(ts, MaxBlockNumber, off, false);
+ }
+
+ tidstore_destroy(ts);
+
+#ifdef TEST_SHARED_TIDSTORE
+ dsa_detach(dsa);
+#endif
+}
+
+static void
+test_empty(void)
+{
+ TidStore *ts;
+ TidStoreIter *iter;
+ ItemPointerData tid;
+
+#ifdef TEST_SHARED_TIDSTORE
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_tidstore");
+ dsa = dsa_create(tranche_id);
+
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, dsa);
+#else
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
+#endif
+
+ elog(NOTICE, "testing empty tidstore");
+
+ ItemPointerSet(&tid, 0, FirstOffsetNumber);
+ if (tidstore_lookup_tid(ts, &tid))
+ elog(ERROR, "tidstore_lookup_tid for (0,1) on empty store returned true");
+
+ ItemPointerSet(&tid, MaxBlockNumber, MaxOffsetNumber);
+ if (tidstore_lookup_tid(ts, &tid))
+ elog(ERROR, "tidstore_lookup_tid for (%u,%u) on empty store returned true",
+ MaxBlockNumber, MaxOffsetNumber);
+
+ if (tidstore_num_tids(ts) != 0)
+ elog(ERROR, "tidstore_num_entries on empty store returned non-zero");
+
+ if (tidstore_is_full(ts))
+ elog(ERROR, "tidstore_is_full on empty store returned true");
+
+ iter = tidstore_begin_iterate(ts);
+
+ if (tidstore_iterate_next(iter) != NULL)
+ elog(ERROR, "tidstore_iterate_next on empty store returned TIDs");
+
+ tidstore_end_iterate(iter);
+
+ tidstore_destroy(ts);
+
+#ifdef TEST_SHARED_TIDSTORE
+ dsa_detach(dsa);
+#endif
+}
+
+Datum
+test_tidstore(PG_FUNCTION_ARGS)
+{
+ test_empty();
+
+ elog(NOTICE, "testing basic operations");
+ test_basic(MaxHeapTuplesPerPage);
+ test_basic(10);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_tidstore/test_tidstore.control b/src/test/modules/test_tidstore/test_tidstore.control
new file mode 100644
index 0000000000..9b6bd4638f
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.control
@@ -0,0 +1,4 @@
+comment = 'Test code for tidstore'
+default_version = '1.0'
+module_pathname = '$libdir/test_tidstore'
+relocatable = true
--
2.31.1
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-02-22 06:15 ` Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-02-22 06:15 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Mon, Feb 20, 2023 at 2:56 PM Masahiko Sawada <[email protected]> wrote:
>
> On Thu, Feb 16, 2023 at 6:23 PM John Naylor
> <[email protected]> wrote:
> >
> > On Thu, Feb 16, 2023 at 10:24 AM Masahiko Sawada <[email protected]> wrote:
> > >
> > > On Tue, Feb 14, 2023 at 8:24 PM John Naylor
> > > <[email protected]> wrote:
> >
> > > > > I can think that something like traversing a HOT chain could visit
> > > > > offsets out of order. But fortunately we prune such collected TIDs
> > > > > before heap vacuum in heap case.
> > > >
> > > > Further, currently we *already* assume we populate the tid array in order (for binary search), so we can just continue assuming that (with an assert added since it's more public in this form). I'm not sure why such basic common sense evaded me a few versions ago...
> > >
> > > Right. TidStore is implemented not only for heap, so loading
> > > out-of-order TIDs might be important in the future.
> >
> > That's what I was probably thinking about some weeks ago, but I'm having a hard time imagining how it would come up, even for something like the conveyor-belt concept.
> >
> > > We have the following WIP comment in test_radixtree:
> > >
> > > // WIP: compiles with warnings because rt_attach is defined but not used
> > > // #define RT_SHMEM
> > >
> > > How about unsetting RT_SCOPE to suppress warnings for unused rt_attach
> > > and friends?
> >
> > Sounds good to me, and the other fixes make sense as well.
>
> Thanks, I merged them.
>
> >
> > > FYI I've briefly tested the TidStore with blocksize = 32kb, and it
> > > seems to work fine.
> >
> > That was on my list, so great! How about the other end -- nominally we allow 512b. (In practice it won't matter, but this would make sure I didn't mess anything up when forcing all MaxTuplesPerPage to encode.)
>
> According to the doc, the minimum block size is 1kB. It seems to work
> fine with 1kB blocks.
>
> >
> > > You removed the vacuum integration patch from v27, is there any reason for that?
> >
> > Just an oversight.
> >
> > Now for some general comments on the tid store...
> >
> > + * TODO: The caller must be certain that no other backend will attempt to
> > + * access the TidStore before calling this function. Other backend must
> > + * explicitly call tidstore_detach to free up backend-local memory associated
> > + * with the TidStore. The backend that calls tidstore_destroy must not call
> > + * tidstore_detach.
> > + */
> > +void
> > +tidstore_destroy(TidStore *ts)
> >
> > Do we need to do anything for this todo?
>
> Since it's practically no problem, I think we can live with it for
> now. dshash also has the same todo.
>
> >
> > It might help readability to have a concept of "off_upper/off_lower", just so we can describe things more clearly. The key is block + off_upper, and the value is a bitmap of all the off_lower bits. I hinted at that in my addition of encode_key_off(). Along those lines, maybe s/TIDSTORE_OFFSET_MASK/TIDSTORE_OFFSET_LOWER_MASK/. Actually, I'm not even sure the TIDSTORE_ prefix is valuable for these local macros.
> >
> > The word "value" as a variable name is pretty generic in this context, and it might be better to call it the off_lower_bitmap, at least in some places. The "key" doesn't have a good short term for naming, but in comments we should make sure we're clear it's "block# + off_upper".
> >
> > I'm not a fan of the name "tid_i", even as a temp variable -- maybe "compressed_tid"?
> >
> > maybe s/tid_to_key_off/encode_tid/ and s/encode_key_off/encode_block_offset/
> >
> > It might be worth using typedefs for key and value type. Actually, since key type is fixed for the foreseeable future, maybe the radix tree template should define a key typedef?
> >
> > The term "result" is probably fine within the tidstore, but as a public name used by vacuum, it's not very descriptive. I don't have a good idea, though.
> >
> > Some files in backend/access use CamelCase for public functions, although it's not consistent. I think doing that for tidstore would help readability, since they would stand out from rt_* functions and vacuum functions. It's a matter of taste, though.
> >
> > I don't understand the control flow in tidstore_iterate_next(), or when BlockNumberIsValid() is true. If this is the best way to code this, it needs more commentary.
>
> The attached 0008 patch addressed all above comments on tidstore.
>
> > Some comments on vacuum:
> >
> > I think we'd better get some real-world testing of this, fairly soon.
> >
> > I had an idea: If it's not too much effort, it might be worth splitting it into two parts: one that just adds the store (not caring about its memory limits or progress reporting etc). During index scan, check both the new store and the array and log a warning (we don't want to exit or crash, better to try to investigate while live if possible) if the result doesn't match. Then perhaps set up an instance and let something like TPC-C run for a few days. The second patch would just restore the rest of the current patch. That would help reassure us it's working as designed.
>
> Yeah, I did a similar thing in an earlier version of tidstore patch.
> Since we're trying to introduce two new components: radix tree and
> tidstore, I sometimes find it hard to investigate failures happening
> during lazy (parallel) vacuum due to a bug either in tidstore or radix
> tree. If there is a bug in lazy vacuum, we cannot even do initdb. So
> it might be a good idea to do such checks in USE_ASSERT_CHECKING (or
> with another macro say DEBUG_TIDSTORE) builds. For example, TidStore
> stores tids to both the radix tree and array, and checks if the
> results match when lookup or iteration. It will use more memory but it
> would not be a big problem in USE_ASSERT_CHECKING builds. It would
> also be great if we can enable such checks on some bf animals.
I've tried this idea. Enabling this check on all debug builds (i.e.,
with USE_ASSERT_CHECKING macro) seems not a good idea so I use a
special macro for that, TIDSTORE_DEBUG. I think we can define this
macro on some bf animals (or possibly a new bf animal).
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
From 107aa2af2966c10ce750e6b410ae570462423aab Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 22 Feb 2023 14:43:15 +0900
Subject: [PATCH v29 11/11] Debug TIDStore.
---
src/backend/access/common/tidstore.c | 242 ++++++++++++++++++++++++++-
1 file changed, 238 insertions(+), 4 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 9360520482..438bf0c800 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -28,12 +28,20 @@
#include "postgres.h"
#include "access/tidstore.h"
+#include "catalog/index.h"
#include "miscadmin.h"
#include "port/pg_bitutils.h"
#include "storage/lwlock.h"
#include "utils/dsa.h"
#include "utils/memutils.h"
+#define TIDSTORE_DEBUG
+
+/* Enable TidStore debugging only when USE_ASSERT_CHECKING */
+#if defined(TIDSTORE_DEBUG) && !defined(USE_ASSERT_CHECKING)
+#undef TIDSTORE_DEBUG
+#endif
+
/*
* For encoding purposes, a tid is represented as a pair of 64-bit key and
* 64-bit value.
@@ -115,6 +123,12 @@ typedef struct TidStoreControl
/* handles for TidStore and radix tree */
TidStoreHandle handle;
shared_rt_handle tree_handle;
+
+#ifdef TIDSTORE_DEBUG
+ dsm_handle tids_handle;
+ int64 max_tids;
+ bool tids_unordered;
+#endif
} TidStoreControl;
/* Per-backend state for a TidStore */
@@ -135,6 +149,11 @@ struct TidStore
/* DSA area for TidStore if used */
dsa_area *area;
+
+#ifdef TIDSTORE_DEBUG
+ dsm_segment *tids_seg;
+ ItemPointerData *tids;
+#endif
};
#define TidStoreIsShared(ts) ((ts)->area != NULL)
@@ -157,6 +176,11 @@ typedef struct TidStoreIter
tidkey next_tidkey;
offsetbm next_off_bitmap;
+#ifdef TIDSTORE_DEBUG
+ /* iterator index for the ts->tids array */
+ int64 tids_idx;
+#endif
+
/*
* output for the caller. Must be last because variable-size.
*/
@@ -169,6 +193,15 @@ static inline tidkey encode_blk_off(TidStore *ts, BlockNumber block,
OffsetNumber offset, offsetbm *off_bit);
static inline tidkey encode_tid(TidStore *ts, ItemPointer tid, offsetbm *off_bit);
+/* debug functions available only when TIDSTORE_DEBUG */
+#ifdef TIDSTORE_DEBUG
+static void ts_debug_set_block_offsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets);
+static void ts_debug_iter_check_tids(TidStoreIter *iter);
+static bool ts_debug_is_member(TidStore *ts, ItemPointer tid);
+static int itemptr_cmp(const void *left, const void *right);
+#endif
+
/*
* Create a TidStore. The returned object is allocated in backend-local memory.
* The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
@@ -237,6 +270,26 @@ TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
ts->control->upper_off_nbits =
ts->control->max_off_nbits - LOWER_OFFSET_NBITS;
+#ifdef TIDSTORE_DEBUG
+ {
+ int64 max_tids = max_bytes / sizeof(ItemPointerData);
+
+ /* Allocate the array of tids too */
+ if (TidStoreIsShared(ts))
+ {
+ ts->tids_seg = dsm_create(sizeof(ItemPointerData) * max_tids, 0);
+ ts->tids = dsm_segment_address(ts->tids_seg);
+ ts->control->tids_handle = dsm_segment_handle(ts->tids_seg);
+ ts->control->max_tids = max_tids;
+ }
+ else
+ {
+ ts->tids = palloc(sizeof(ItemPointerData) * max_tids);
+ ts->control->max_tids = max_tids;
+ }
+ }
+#endif
+
return ts;
}
@@ -266,6 +319,11 @@ TidStoreAttach(dsa_area *area, TidStoreHandle handle)
ts->tree.shared = shared_rt_attach(area, ts->control->tree_handle);
ts->area = area;
+#ifdef TIDSTORE_DEBUG
+ ts->tids_seg = dsm_attach(ts->control->tids_handle);
+ ts->tids = (ItemPointer) dsm_segment_address(ts->tids_seg);
+#endif
+
return ts;
}
@@ -280,6 +338,11 @@ TidStoreDetach(TidStore *ts)
{
Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+#ifdef TIDSTORE_DEBUG
+ if (TidStoreIsShared(ts))
+ dsm_detach(ts->tids_seg);
+#endif
+
shared_rt_detach(ts->tree.shared);
pfree(ts);
}
@@ -315,6 +378,13 @@ TidStoreDestroy(TidStore *ts)
local_rt_free(ts->tree.local);
}
+#ifdef TIDSTORE_DEBUG
+ if (TidStoreIsShared(ts))
+ dsm_detach(ts->tids_seg);
+ else
+ pfree(ts->tids);
+#endif
+
pfree(ts);
}
@@ -434,6 +504,11 @@ TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
}
}
+#ifdef TIDSTORE_DEBUG
+ /* Insert tids into the tid array too */
+ ts_debug_set_block_offsets(ts, blkno, offsets, num_offsets);
+#endif
+
/* update statistics */
ts->control->num_tids += num_offsets;
@@ -451,6 +526,11 @@ TidStoreIsMember(TidStore *ts, ItemPointer tid)
offsetbm off_bitmap = 0;
offsetbm off_bit;
bool found;
+ bool ret;
+
+#ifdef TIDSTORE_DEBUG
+ bool ret_debug = ts_debug_is_member(ts, tid);
+#endif
key = encode_tid(ts, tid, &off_bit);
@@ -460,9 +540,20 @@ TidStoreIsMember(TidStore *ts, ItemPointer tid)
found = local_rt_search(ts->tree.local, key, &off_bitmap);
if (!found)
+ {
+#ifdef TIDSTORE_DEBUG
+ Assert(!ret_debug);
+#endif
return false;
+ }
+
+ ret = (off_bitmap & off_bit) != 0;
- return (off_bitmap & off_bit) != 0;
+#ifdef TIDSTORE_DEBUG
+ Assert(ret == ret_debug);
+#endif
+
+ return ret;
}
/*
@@ -494,6 +585,10 @@ TidStoreBeginIterate(TidStore *ts)
if (TidStoreNumTids(ts) == 0)
iter->finished = true;
+#ifdef TIDSTORE_DEBUG
+ iter->tids_idx = 0;
+#endif
+
return iter;
}
@@ -515,6 +610,7 @@ TidStoreIterResult *
TidStoreIterateNext(TidStoreIter *iter)
{
tidkey key;
+ bool iter_found;
offsetbm off_bitmap = 0;
TidStoreIterResult *output = &(iter->output);
@@ -532,7 +628,7 @@ TidStoreIterateNext(TidStoreIter *iter)
if (iter->next_off_bitmap > 0)
iter_decode_key_off(iter, iter->next_tidkey, iter->next_off_bitmap);
- while (tidstore_iter(iter, &key, &off_bitmap))
+ while ((iter_found = tidstore_iter(iter, &key, &off_bitmap)))
{
BlockNumber blkno = key_get_blkno(iter->ts, key);
@@ -545,14 +641,20 @@ TidStoreIterateNext(TidStoreIter *iter)
*/
iter->next_tidkey = key;
iter->next_off_bitmap = off_bitmap;
- return output;
+ break;
}
/* Collect tids decoded from the key and offset bitmap */
iter_decode_key_off(iter, key, off_bitmap);
}
- iter->finished = true;
+ if (!iter_found)
+ iter->finished = true;
+
+#ifdef TIDSTORE_DEBUG
+ ts_debug_iter_check_tids(iter);
+#endif
+
return output;
}
@@ -699,3 +801,135 @@ encode_blk_off(TidStore *ts, BlockNumber block, OffsetNumber offset,
return key;
}
+
+#ifdef TIDSTORE_DEBUG
+/* Comparator routines for ItemPointer */
+static int
+itemptr_cmp(const void *left, const void *right)
+{
+ BlockNumber lblk,
+ rblk;
+ OffsetNumber loff,
+ roff;
+
+ lblk = ItemPointerGetBlockNumber((ItemPointer) left);
+ rblk = ItemPointerGetBlockNumber((ItemPointer) right);
+
+ if (lblk < rblk)
+ return -1;
+ if (lblk > rblk)
+ return 1;
+
+ loff = ItemPointerGetOffsetNumber((ItemPointer) left);
+ roff = ItemPointerGetOffsetNumber((ItemPointer) right);
+
+ if (loff < roff)
+ return -1;
+ if (loff > roff)
+ return 1;
+
+ return 0;
+}
+
+/* Insert tids to the tid array for debugging */
+static void
+ts_debug_set_block_offsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets)
+{
+ if (ts->control->num_tids > 0 &&
+ blkno < ItemPointerGetBlockNumber(&(ts->tids[ts->control->num_tids - 1])))
+ {
+ /* The array will be sorted at ts_debug_is_member() */
+ ts->control->tids_unordered = true;
+ }
+
+ for (int i = 0; i < num_offsets; i++)
+ {
+ ItemPointer tid;
+ int idx = ts->control->num_tids + i;
+
+ /* Enlarge the tid array if necessary */
+ if (idx >= ts->control->max_tids)
+ {
+ ts->control->max_tids *= 2;
+
+ if (TidStoreIsShared(ts))
+ {
+ dsm_segment *new_seg =
+ dsm_create(sizeof(ItemPointerData) * ts->control->max_tids, 0);
+ ItemPointer new_tids = dsm_segment_address(new_seg);
+
+ /* copy tids from old to new array */
+ memcpy(new_tids, ts->tids,
+ sizeof(ItemPointerData) * (ts->control->max_tids / 2));
+
+ dsm_detach(ts->tids_seg);
+ ts->tids = new_tids;
+ }
+ else
+ ts->tids = repalloc(ts->tids,
+ sizeof(ItemPointerData) * ts->control->max_tids);
+ }
+
+ tid = &(ts->tids[idx]);
+ ItemPointerSetBlockNumber(tid, blkno);
+ ItemPointerSetOffsetNumber(tid, offsets[i]);
+ }
+}
+
+/* Return true if the given tid is present in the tid array */
+static bool
+ts_debug_is_member(TidStore *ts, ItemPointer tid)
+{
+ int64 litem,
+ ritem,
+ item;
+ ItemPointer res;
+
+ if (ts->control->num_tids == 0)
+ return false;
+
+ /* Make sure the tid array is sorted */
+ if (ts->control->tids_unordered)
+ {
+ qsort(ts->tids, ts->control->num_tids, sizeof(ItemPointerData), itemptr_cmp);
+ ts->control->tids_unordered = false;
+ }
+
+ litem = itemptr_encode(&ts->tids[0]);
+ ritem = itemptr_encode(&ts->tids[ts->control->num_tids - 1]);
+ item = itemptr_encode(tid);
+
+ /*
+ * Doing a simple bound check before bsearch() is useful to avoid the
+ * extra cost of bsearch(), especially if dead items on the heap are
+ * concentrated in a certain range. Since this function is called for
+ * every index tuple, it pays to be really fast.
+ */
+ if (item < litem || item > ritem)
+ return false;
+
+ res = bsearch(tid, ts->tids, ts->control->num_tids, sizeof(ItemPointerData),
+ itemptr_cmp);
+
+ return (res != NULL);
+}
+
+/* Verify if the iterator output matches the tids in the array for debugging */
+static void
+ts_debug_iter_check_tids(TidStoreIter *iter)
+{
+ BlockNumber blkno = iter->output.blkno;
+
+ for (int i = 0; i < iter->output.num_offsets; i++)
+ {
+ ItemPointer tid = &(iter->ts->tids[iter->tids_idx + i]);
+
+ Assert((iter->tids_idx + i) < iter->ts->control->max_tids);
+ Assert(ItemPointerGetBlockNumber(tid) == blkno);
+ Assert(ItemPointerGetOffsetNumber(tid) == iter->output.offsets[i]);
+ }
+
+ iter->tids_idx += iter->output.num_offsets;
+}
+#endif
--
2.31.1
Attachments:
[text/plain] v29-0011-Debug-TIDStore.patch.txt (9.7K, ../../CAD21AoDCuTO6_wKF9mPm2hb9Fo-dc9kwbec6iXpwptsYaoFdyA@mail.gmail.com/2-v29-0011-Debug-TIDStore.patch.txt)
download | inline diff:
From 107aa2af2966c10ce750e6b410ae570462423aab Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 22 Feb 2023 14:43:15 +0900
Subject: [PATCH v29 11/11] Debug TIDStore.
---
src/backend/access/common/tidstore.c | 242 ++++++++++++++++++++++++++-
1 file changed, 238 insertions(+), 4 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 9360520482..438bf0c800 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -28,12 +28,20 @@
#include "postgres.h"
#include "access/tidstore.h"
+#include "catalog/index.h"
#include "miscadmin.h"
#include "port/pg_bitutils.h"
#include "storage/lwlock.h"
#include "utils/dsa.h"
#include "utils/memutils.h"
+#define TIDSTORE_DEBUG
+
+/* Enable TidStore debugging only when USE_ASSERT_CHECKING */
+#if defined(TIDSTORE_DEBUG) && !defined(USE_ASSERT_CHECKING)
+#undef TIDSTORE_DEBUG
+#endif
+
/*
* For encoding purposes, a tid is represented as a pair of 64-bit key and
* 64-bit value.
@@ -115,6 +123,12 @@ typedef struct TidStoreControl
/* handles for TidStore and radix tree */
TidStoreHandle handle;
shared_rt_handle tree_handle;
+
+#ifdef TIDSTORE_DEBUG
+ dsm_handle tids_handle;
+ int64 max_tids;
+ bool tids_unordered;
+#endif
} TidStoreControl;
/* Per-backend state for a TidStore */
@@ -135,6 +149,11 @@ struct TidStore
/* DSA area for TidStore if used */
dsa_area *area;
+
+#ifdef TIDSTORE_DEBUG
+ dsm_segment *tids_seg;
+ ItemPointerData *tids;
+#endif
};
#define TidStoreIsShared(ts) ((ts)->area != NULL)
@@ -157,6 +176,11 @@ typedef struct TidStoreIter
tidkey next_tidkey;
offsetbm next_off_bitmap;
+#ifdef TIDSTORE_DEBUG
+ /* iterator index for the ts->tids array */
+ int64 tids_idx;
+#endif
+
/*
* output for the caller. Must be last because variable-size.
*/
@@ -169,6 +193,15 @@ static inline tidkey encode_blk_off(TidStore *ts, BlockNumber block,
OffsetNumber offset, offsetbm *off_bit);
static inline tidkey encode_tid(TidStore *ts, ItemPointer tid, offsetbm *off_bit);
+/* debug functions available only when TIDSTORE_DEBUG */
+#ifdef TIDSTORE_DEBUG
+static void ts_debug_set_block_offsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets);
+static void ts_debug_iter_check_tids(TidStoreIter *iter);
+static bool ts_debug_is_member(TidStore *ts, ItemPointer tid);
+static int itemptr_cmp(const void *left, const void *right);
+#endif
+
/*
* Create a TidStore. The returned object is allocated in backend-local memory.
* The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
@@ -237,6 +270,26 @@ TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
ts->control->upper_off_nbits =
ts->control->max_off_nbits - LOWER_OFFSET_NBITS;
+#ifdef TIDSTORE_DEBUG
+ {
+ int64 max_tids = max_bytes / sizeof(ItemPointerData);
+
+ /* Allocate the array of tids too */
+ if (TidStoreIsShared(ts))
+ {
+ ts->tids_seg = dsm_create(sizeof(ItemPointerData) * max_tids, 0);
+ ts->tids = dsm_segment_address(ts->tids_seg);
+ ts->control->tids_handle = dsm_segment_handle(ts->tids_seg);
+ ts->control->max_tids = max_tids;
+ }
+ else
+ {
+ ts->tids = palloc(sizeof(ItemPointerData) * max_tids);
+ ts->control->max_tids = max_tids;
+ }
+ }
+#endif
+
return ts;
}
@@ -266,6 +319,11 @@ TidStoreAttach(dsa_area *area, TidStoreHandle handle)
ts->tree.shared = shared_rt_attach(area, ts->control->tree_handle);
ts->area = area;
+#ifdef TIDSTORE_DEBUG
+ ts->tids_seg = dsm_attach(ts->control->tids_handle);
+ ts->tids = (ItemPointer) dsm_segment_address(ts->tids_seg);
+#endif
+
return ts;
}
@@ -280,6 +338,11 @@ TidStoreDetach(TidStore *ts)
{
Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+#ifdef TIDSTORE_DEBUG
+ if (TidStoreIsShared(ts))
+ dsm_detach(ts->tids_seg);
+#endif
+
shared_rt_detach(ts->tree.shared);
pfree(ts);
}
@@ -315,6 +378,13 @@ TidStoreDestroy(TidStore *ts)
local_rt_free(ts->tree.local);
}
+#ifdef TIDSTORE_DEBUG
+ if (TidStoreIsShared(ts))
+ dsm_detach(ts->tids_seg);
+ else
+ pfree(ts->tids);
+#endif
+
pfree(ts);
}
@@ -434,6 +504,11 @@ TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
}
}
+#ifdef TIDSTORE_DEBUG
+ /* Insert tids into the tid array too */
+ ts_debug_set_block_offsets(ts, blkno, offsets, num_offsets);
+#endif
+
/* update statistics */
ts->control->num_tids += num_offsets;
@@ -451,6 +526,11 @@ TidStoreIsMember(TidStore *ts, ItemPointer tid)
offsetbm off_bitmap = 0;
offsetbm off_bit;
bool found;
+ bool ret;
+
+#ifdef TIDSTORE_DEBUG
+ bool ret_debug = ts_debug_is_member(ts, tid);
+#endif
key = encode_tid(ts, tid, &off_bit);
@@ -460,9 +540,20 @@ TidStoreIsMember(TidStore *ts, ItemPointer tid)
found = local_rt_search(ts->tree.local, key, &off_bitmap);
if (!found)
+ {
+#ifdef TIDSTORE_DEBUG
+ Assert(!ret_debug);
+#endif
return false;
+ }
+
+ ret = (off_bitmap & off_bit) != 0;
- return (off_bitmap & off_bit) != 0;
+#ifdef TIDSTORE_DEBUG
+ Assert(ret == ret_debug);
+#endif
+
+ return ret;
}
/*
@@ -494,6 +585,10 @@ TidStoreBeginIterate(TidStore *ts)
if (TidStoreNumTids(ts) == 0)
iter->finished = true;
+#ifdef TIDSTORE_DEBUG
+ iter->tids_idx = 0;
+#endif
+
return iter;
}
@@ -515,6 +610,7 @@ TidStoreIterResult *
TidStoreIterateNext(TidStoreIter *iter)
{
tidkey key;
+ bool iter_found;
offsetbm off_bitmap = 0;
TidStoreIterResult *output = &(iter->output);
@@ -532,7 +628,7 @@ TidStoreIterateNext(TidStoreIter *iter)
if (iter->next_off_bitmap > 0)
iter_decode_key_off(iter, iter->next_tidkey, iter->next_off_bitmap);
- while (tidstore_iter(iter, &key, &off_bitmap))
+ while ((iter_found = tidstore_iter(iter, &key, &off_bitmap)))
{
BlockNumber blkno = key_get_blkno(iter->ts, key);
@@ -545,14 +641,20 @@ TidStoreIterateNext(TidStoreIter *iter)
*/
iter->next_tidkey = key;
iter->next_off_bitmap = off_bitmap;
- return output;
+ break;
}
/* Collect tids decoded from the key and offset bitmap */
iter_decode_key_off(iter, key, off_bitmap);
}
- iter->finished = true;
+ if (!iter_found)
+ iter->finished = true;
+
+#ifdef TIDSTORE_DEBUG
+ ts_debug_iter_check_tids(iter);
+#endif
+
return output;
}
@@ -699,3 +801,135 @@ encode_blk_off(TidStore *ts, BlockNumber block, OffsetNumber offset,
return key;
}
+
+#ifdef TIDSTORE_DEBUG
+/* Comparator routines for ItemPointer */
+static int
+itemptr_cmp(const void *left, const void *right)
+{
+ BlockNumber lblk,
+ rblk;
+ OffsetNumber loff,
+ roff;
+
+ lblk = ItemPointerGetBlockNumber((ItemPointer) left);
+ rblk = ItemPointerGetBlockNumber((ItemPointer) right);
+
+ if (lblk < rblk)
+ return -1;
+ if (lblk > rblk)
+ return 1;
+
+ loff = ItemPointerGetOffsetNumber((ItemPointer) left);
+ roff = ItemPointerGetOffsetNumber((ItemPointer) right);
+
+ if (loff < roff)
+ return -1;
+ if (loff > roff)
+ return 1;
+
+ return 0;
+}
+
+/* Insert tids to the tid array for debugging */
+static void
+ts_debug_set_block_offsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets)
+{
+ if (ts->control->num_tids > 0 &&
+ blkno < ItemPointerGetBlockNumber(&(ts->tids[ts->control->num_tids - 1])))
+ {
+ /* The array will be sorted at ts_debug_is_member() */
+ ts->control->tids_unordered = true;
+ }
+
+ for (int i = 0; i < num_offsets; i++)
+ {
+ ItemPointer tid;
+ int idx = ts->control->num_tids + i;
+
+ /* Enlarge the tid array if necessary */
+ if (idx >= ts->control->max_tids)
+ {
+ ts->control->max_tids *= 2;
+
+ if (TidStoreIsShared(ts))
+ {
+ dsm_segment *new_seg =
+ dsm_create(sizeof(ItemPointerData) * ts->control->max_tids, 0);
+ ItemPointer new_tids = dsm_segment_address(new_seg);
+
+ /* copy tids from old to new array */
+ memcpy(new_tids, ts->tids,
+ sizeof(ItemPointerData) * (ts->control->max_tids / 2));
+
+ dsm_detach(ts->tids_seg);
+ ts->tids = new_tids;
+ }
+ else
+ ts->tids = repalloc(ts->tids,
+ sizeof(ItemPointerData) * ts->control->max_tids);
+ }
+
+ tid = &(ts->tids[idx]);
+ ItemPointerSetBlockNumber(tid, blkno);
+ ItemPointerSetOffsetNumber(tid, offsets[i]);
+ }
+}
+
+/* Return true if the given tid is present in the tid array */
+static bool
+ts_debug_is_member(TidStore *ts, ItemPointer tid)
+{
+ int64 litem,
+ ritem,
+ item;
+ ItemPointer res;
+
+ if (ts->control->num_tids == 0)
+ return false;
+
+ /* Make sure the tid array is sorted */
+ if (ts->control->tids_unordered)
+ {
+ qsort(ts->tids, ts->control->num_tids, sizeof(ItemPointerData), itemptr_cmp);
+ ts->control->tids_unordered = false;
+ }
+
+ litem = itemptr_encode(&ts->tids[0]);
+ ritem = itemptr_encode(&ts->tids[ts->control->num_tids - 1]);
+ item = itemptr_encode(tid);
+
+ /*
+ * Doing a simple bound check before bsearch() is useful to avoid the
+ * extra cost of bsearch(), especially if dead items on the heap are
+ * concentrated in a certain range. Since this function is called for
+ * every index tuple, it pays to be really fast.
+ */
+ if (item < litem || item > ritem)
+ return false;
+
+ res = bsearch(tid, ts->tids, ts->control->num_tids, sizeof(ItemPointerData),
+ itemptr_cmp);
+
+ return (res != NULL);
+}
+
+/* Verify if the iterator output matches the tids in the array for debugging */
+static void
+ts_debug_iter_check_tids(TidStoreIter *iter)
+{
+ BlockNumber blkno = iter->output.blkno;
+
+ for (int i = 0; i < iter->output.num_offsets; i++)
+ {
+ ItemPointer tid = &(iter->ts->tids[iter->tids_idx + i]);
+
+ Assert((iter->tids_idx + i) < iter->ts->control->max_tids);
+ Assert(ItemPointerGetBlockNumber(tid) == blkno);
+ Assert(ItemPointerGetOffsetNumber(tid) == iter->output.offsets[i]);
+ }
+
+ iter->tids_idx += iter->output.num_offsets;
+}
+#endif
--
2.31.1
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-02-22 07:35 ` John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: John Naylor @ 2023-02-22 07:35 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Wed, Feb 22, 2023 at 1:16 PM Masahiko Sawada <[email protected]>
wrote:
>
> On Mon, Feb 20, 2023 at 2:56 PM Masahiko Sawada <[email protected]>
wrote:
> >
> > Yeah, I did a similar thing in an earlier version of tidstore patch.
Okay, if you had checks against the old array lookup in development, that
gives us better confidence.
> > Since we're trying to introduce two new components: radix tree and
> > tidstore, I sometimes find it hard to investigate failures happening
> > during lazy (parallel) vacuum due to a bug either in tidstore or radix
> > tree. If there is a bug in lazy vacuum, we cannot even do initdb. So
> > it might be a good idea to do such checks in USE_ASSERT_CHECKING (or
> > with another macro say DEBUG_TIDSTORE) builds. For example, TidStore
> > stores tids to both the radix tree and array, and checks if the
> > results match when lookup or iteration. It will use more memory but it
> > would not be a big problem in USE_ASSERT_CHECKING builds. It would
> > also be great if we can enable such checks on some bf animals.
>
> I've tried this idea. Enabling this check on all debug builds (i.e.,
> with USE_ASSERT_CHECKING macro) seems not a good idea so I use a
> special macro for that, TIDSTORE_DEBUG. I think we can define this
> macro on some bf animals (or possibly a new bf animal).
I don't think any vacuum calls in regression tests would stress any of
this code very much, so it's not worth carrying the old way forward. I was
thinking of only doing this as a short-time sanity check for testing a
real-world workload.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-02-22 08:29 ` Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-02-22 08:29 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Wed, Feb 22, 2023 at 4:35 PM John Naylor
<[email protected]> wrote:
>
>
> On Wed, Feb 22, 2023 at 1:16 PM Masahiko Sawada <[email protected]> wrote:
> >
> > On Mon, Feb 20, 2023 at 2:56 PM Masahiko Sawada <[email protected]> wrote:
> > >
> > > Yeah, I did a similar thing in an earlier version of tidstore patch.
>
> Okay, if you had checks against the old array lookup in development, that gives us better confidence.
>
> > > Since we're trying to introduce two new components: radix tree and
> > > tidstore, I sometimes find it hard to investigate failures happening
> > > during lazy (parallel) vacuum due to a bug either in tidstore or radix
> > > tree. If there is a bug in lazy vacuum, we cannot even do initdb. So
> > > it might be a good idea to do such checks in USE_ASSERT_CHECKING (or
> > > with another macro say DEBUG_TIDSTORE) builds. For example, TidStore
> > > stores tids to both the radix tree and array, and checks if the
> > > results match when lookup or iteration. It will use more memory but it
> > > would not be a big problem in USE_ASSERT_CHECKING builds. It would
> > > also be great if we can enable such checks on some bf animals.
> >
> > I've tried this idea. Enabling this check on all debug builds (i.e.,
> > with USE_ASSERT_CHECKING macro) seems not a good idea so I use a
> > special macro for that, TIDSTORE_DEBUG. I think we can define this
> > macro on some bf animals (or possibly a new bf animal).
>
> I don't think any vacuum calls in regression tests would stress any of this code very much, so it's not worth carrying the old way forward. I was thinking of only doing this as a short-time sanity check for testing a real-world workload.
I guess that It would also be helpful at least until the GA release.
People will be able to test them easily on their workloads or their
custom test scenarios.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-02-22 09:55 ` John Naylor <[email protected]>
2023-02-23 09:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
0 siblings, 2 replies; 78+ messages in thread
From: John Naylor @ 2023-02-22 09:55 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Wed, Feb 22, 2023 at 3:29 PM Masahiko Sawada <[email protected]>
wrote:
>
> On Wed, Feb 22, 2023 at 4:35 PM John Naylor
> <[email protected]> wrote:
> >
> > I don't think any vacuum calls in regression tests would stress any of
this code very much, so it's not worth carrying the old way forward. I was
thinking of only doing this as a short-time sanity check for testing a
real-world workload.
>
> I guess that It would also be helpful at least until the GA release.
> People will be able to test them easily on their workloads or their
> custom test scenarios.
That doesn't seem useful to me. If we've done enough testing to reassure us
the new way always gives the same answer, the old way is not needed at
commit time. If there is any doubt it will always give the same answer,
then the whole patchset won't be committed.
TPC-C was just an example. It should have testing comparing the old and new
methods. If you have already done that to some degree, that might be
enough. After performance tests, I'll also try some vacuums that use the
comparison patch.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-02-23 09:40 ` John Naylor <[email protected]>
2023-02-24 08:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
1 sibling, 1 reply; 78+ messages in thread
From: John Naylor @ 2023-02-23 09:40 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
I ran a couple "in situ" tests on server hardware using UUID columns, since
they are common in the real world and have bad correlation to heap
order, so are a challenge for index vacuum.
=== test 1, delete everything from a small table, with very small
maintenance_work_mem:
alter system set shared_buffers ='4GB';
alter system set max_wal_size ='10GB';
alter system set checkpoint_timeout ='30 min';
alter system set autovacuum =off;
-- unrealistically low
alter system set maintenance_work_mem = '32MB';
create table if not exists test (x uuid);
truncate table test;
insert into test (x) select gen_random_uuid() from
generate_series(1,50*1000*1000);
create index on test (x);
delete from test;
vacuum (verbose, truncate off) test;
--
master:
INFO: finished vacuuming "john.naylor.public.test": index scans: 9
system usage: CPU: user: 70.04 s, system: 19.85 s, elapsed: 802.06 s
v29 patch:
INFO: finished vacuuming "john.naylor.public.test": index scans: 1
system usage: CPU: user: 9.80 s, system: 2.62 s, elapsed: 36.68 s
This is a bit artificial, but it's easy to construct cases where the array
leads to multiple index scans but the new tid store can fit everythin
without breaking a sweat. I didn't save the progress reporting, but v29 was
using about 11MB for tid storage.
=== test 2: try to stress tid lookup with production maintenance_work_mem:
1. use unlogged table to reduce noise
2. vacuum freeze first to reduce heap scan time
3. delete some records at the beginning and end of heap to defeat binary
search's pre-check
alter system set shared_buffers ='4GB';
alter system set max_wal_size ='10GB';
alter system set checkpoint_timeout ='30 min';
alter system set autovacuum =off;
alter system set maintenance_work_mem = '1GB';
create unlogged table if not exists test (x uuid);
truncate table test;
insert into test (x) select gen_random_uuid() from
generate_series(1,1000*1000*1000);
vacuum_freeze test;
select pg_size_pretty(pg_table_size('test'));
pg_size_pretty
----------------
41 GB
create index on test (x);
select pg_size_pretty(pg_total_relation_size('test'));
pg_size_pretty
----------------
71 GB
select max(ctid) from test;
max
--------------
(5405405,75)
delete from test where ctid < '(100000,0)'::tid;
delete from test where ctid > '(5300000,0)'::tid;
vacuum (verbose, truncate off) test;
both:
INFO: vacuuming "john.naylor.public.test"
INFO: finished vacuuming "john.naylor.public.test": index scans: 1
index scan needed: 205406 pages from table (3.80% of total) had 38000000
dead item identifiers removed
--
master:
system usage: CPU: user: 134.32 s, system: 19.24 s, elapsed: 286.14 s
v29 patch:
system usage: CPU: user: 97.71 s, system: 45.78 s, elapsed: 573.94 s
The entire vacuum took 25% less wall clock time. Reminder that this is
without wal logging, and also unscientific because only one run.
--
I took 10 seconds of perf data while index vacuuming was going on (showing
calls > 2%):
master:
40.59% postgres postgres [.] vac_cmp_itemptr
24.97% postgres libc-2.17.so [.] bsearch
6.67% postgres postgres [.] btvacuumpage
4.61% postgres [kernel.kallsyms] [k] copy_user_enhanced_fast_string
3.48% postgres postgres [.] PageIndexMultiDelete
2.67% postgres postgres [.] vac_tid_reaped
2.03% postgres postgres [.] compactify_tuples
2.01% postgres libc-2.17.so [.] __memcpy_ssse3_back
v29 patch:
29.22% postgres postgres [.] TidStoreIsMember
9.30% postgres postgres [.] btvacuumpage
7.76% postgres postgres [.] PageIndexMultiDelete
6.31% postgres [kernel.kallsyms] [k] copy_user_enhanced_fast_string
5.60% postgres postgres [.] compactify_tuples
4.26% postgres libc-2.17.so [.] __memcpy_ssse3_back
4.12% postgres postgres [.] hash_search_with_hash_value
--
master:
psql -c "select phase, heap_blks_total, heap_blks_scanned, max_dead_tuples,
num_dead_tuples from pg_stat_progress_vacuum"
phase | heap_blks_total | heap_blks_scanned | max_dead_tuples
| num_dead_tuples
-------------------+-----------------+-------------------+-----------------+-----------------
vacuuming indexes | 5405406 | 5405406 | 178956969
| 38000000
v29 patch:
psql -c "select phase, heap_blks_total, heap_blks_scanned,
max_dead_tuple_bytes, dead_tuple_bytes from pg_stat_progress_vacuum"
phase | heap_blks_total | heap_blks_scanned |
max_dead_tuple_bytes | dead_tuple_bytes
-------------------+-----------------+-------------------+----------------------+------------------
vacuuming indexes | 5405406 | 5405406 |
1073670144 | 8678064
Here, the old array pessimistically needs 1GB allocated (as for any table >
~5GB), but only fills 228MB for tid lookup. The patch reports 8.7MB. Tables
that only fit, say, 30-50 tuples per page will have less extreme
differences in memory use. Same for the case where only a couple dead items
occur per page, with many uninteresting pages in between. Even so, the
allocation will be much more accurately sized in the patch, especially in
non-parallel vacuum.
There are other cases that could be tested (I mentioned some above), but
this is enough to show the improvements possible.
I still need to do some cosmetic follow-up to v29 as well as a status
report, and I will try to get back to that soon.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-23 09:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-02-24 08:40 ` Masahiko Sawada <[email protected]>
2023-02-27 17:07 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-02-24 08:40 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Thu, Feb 23, 2023 at 6:41 PM John Naylor
<[email protected]> wrote:
>
> I ran a couple "in situ" tests on server hardware using UUID columns, since they are common in the real world and have bad correlation to heap order, so are a challenge for index vacuum.
Thank you for the test!
>
> === test 1, delete everything from a small table, with very small maintenance_work_mem:
>
> alter system set shared_buffers ='4GB';
> alter system set max_wal_size ='10GB';
> alter system set checkpoint_timeout ='30 min';
> alter system set autovacuum =off;
>
> -- unrealistically low
> alter system set maintenance_work_mem = '32MB';
>
> create table if not exists test (x uuid);
> truncate table test;
> insert into test (x) select gen_random_uuid() from generate_series(1,50*1000*1000);
> create index on test (x);
>
> delete from test;
> vacuum (verbose, truncate off) test;
> --
>
> master:
> INFO: finished vacuuming "john.naylor.public.test": index scans: 9
> system usage: CPU: user: 70.04 s, system: 19.85 s, elapsed: 802.06 s
>
> v29 patch:
> INFO: finished vacuuming "john.naylor.public.test": index scans: 1
> system usage: CPU: user: 9.80 s, system: 2.62 s, elapsed: 36.68 s
>
> This is a bit artificial, but it's easy to construct cases where the array leads to multiple index scans but the new tid store can fit everythin without breaking a sweat. I didn't save the progress reporting, but v29 was using about 11MB for tid storage.
Cool.
>
>
> === test 2: try to stress tid lookup with production maintenance_work_mem:
> 1. use unlogged table to reduce noise
> 2. vacuum freeze first to reduce heap scan time
> 3. delete some records at the beginning and end of heap to defeat binary search's pre-check
>
> alter system set shared_buffers ='4GB';
> alter system set max_wal_size ='10GB';
> alter system set checkpoint_timeout ='30 min';
> alter system set autovacuum =off;
>
> alter system set maintenance_work_mem = '1GB';
>
> create unlogged table if not exists test (x uuid);
> truncate table test;
> insert into test (x) select gen_random_uuid() from generate_series(1,1000*1000*1000);
> vacuum_freeze test;
>
> select pg_size_pretty(pg_table_size('test'));
> pg_size_pretty
> ----------------
> 41 GB
>
> create index on test (x);
>
> select pg_size_pretty(pg_total_relation_size('test'));
> pg_size_pretty
> ----------------
> 71 GB
>
> select max(ctid) from test;
> max
> --------------
> (5405405,75)
>
> delete from test where ctid < '(100000,0)'::tid;
> delete from test where ctid > '(5300000,0)'::tid;
>
> vacuum (verbose, truncate off) test;
>
> both:
> INFO: vacuuming "john.naylor.public.test"
> INFO: finished vacuuming "john.naylor.public.test": index scans: 1
> index scan needed: 205406 pages from table (3.80% of total) had 38000000 dead item identifiers removed
>
> --
> master:
> system usage: CPU: user: 134.32 s, system: 19.24 s, elapsed: 286.14 s
>
> v29 patch:
> system usage: CPU: user: 97.71 s, system: 45.78 s, elapsed: 573.94 s
In v29 vacuum took twice as long (286 s vs. 573 s)?
>
> The entire vacuum took 25% less wall clock time. Reminder that this is without wal logging, and also unscientific because only one run.
>
> --
> I took 10 seconds of perf data while index vacuuming was going on (showing calls > 2%):
>
> master:
> 40.59% postgres postgres [.] vac_cmp_itemptr
> 24.97% postgres libc-2.17.so [.] bsearch
> 6.67% postgres postgres [.] btvacuumpage
> 4.61% postgres [kernel.kallsyms] [k] copy_user_enhanced_fast_string
> 3.48% postgres postgres [.] PageIndexMultiDelete
> 2.67% postgres postgres [.] vac_tid_reaped
> 2.03% postgres postgres [.] compactify_tuples
> 2.01% postgres libc-2.17.so [.] __memcpy_ssse3_back
>
> v29 patch:
>
> 29.22% postgres postgres [.] TidStoreIsMember
> 9.30% postgres postgres [.] btvacuumpage
> 7.76% postgres postgres [.] PageIndexMultiDelete
> 6.31% postgres [kernel.kallsyms] [k] copy_user_enhanced_fast_string
> 5.60% postgres postgres [.] compactify_tuples
> 4.26% postgres libc-2.17.so [.] __memcpy_ssse3_back
> 4.12% postgres postgres [.] hash_search_with_hash_value
>
> --
> master:
> psql -c "select phase, heap_blks_total, heap_blks_scanned, max_dead_tuples, num_dead_tuples from pg_stat_progress_vacuum"
> phase | heap_blks_total | heap_blks_scanned | max_dead_tuples | num_dead_tuples
> -------------------+-----------------+-------------------+-----------------+-----------------
> vacuuming indexes | 5405406 | 5405406 | 178956969 | 38000000
>
> v29 patch:
> psql -c "select phase, heap_blks_total, heap_blks_scanned, max_dead_tuple_bytes, dead_tuple_bytes from pg_stat_progress_vacuum"
> phase | heap_blks_total | heap_blks_scanned | max_dead_tuple_bytes | dead_tuple_bytes
> -------------------+-----------------+-------------------+----------------------+------------------
> vacuuming indexes | 5405406 | 5405406 | 1073670144 | 8678064
>
> Here, the old array pessimistically needs 1GB allocated (as for any table > ~5GB), but only fills 228MB for tid lookup. The patch reports 8.7MB. Tables that only fit, say, 30-50 tuples per page will have less extreme differences in memory use. Same for the case where only a couple dead items occur per page, with many uninteresting pages in between. Even so, the allocation will be much more accurately sized in the patch, especially in non-parallel vacuum.
Agreed.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-23 09:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 08:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-02-27 17:07 ` John Naylor <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: John Naylor @ 2023-02-27 17:07 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Fri, Feb 24, 2023 at 3:41 PM Masahiko Sawada <[email protected]>
wrote:
>
> In v29 vacuum took twice as long (286 s vs. 573 s)?
Not sure what happened there, and clearly I was looking at the wrong number
:/
I scripted the test for reproducibility and ran it three times. Also
included some variations (attached):
UUID times look comparable here, so no speedup or regression:
master:
system usage: CPU: user: 216.05 s, system: 35.81 s, elapsed: 634.22 s
system usage: CPU: user: 173.71 s, system: 31.24 s, elapsed: 599.04 s
system usage: CPU: user: 171.16 s, system: 30.21 s, elapsed: 583.21 s
v29:
system usage: CPU: user: 93.47 s, system: 40.92 s, elapsed: 594.10 s
system usage: CPU: user: 99.58 s, system: 44.73 s, elapsed: 606.80 s
system usage: CPU: user: 96.29 s, system: 42.74 s, elapsed: 600.10 s
Then, I tried sequential integers, which is a much more favorable access
pattern in general, and the new tid storage shows substantial improvement:
master:
system usage: CPU: user: 100.39 s, system: 7.79 s, elapsed: 121.57 s
system usage: CPU: user: 104.90 s, system: 8.81 s, elapsed: 124.24 s
system usage: CPU: user: 95.04 s, system: 7.55 s, elapsed: 116.44 s
v29:
system usage: CPU: user: 24.57 s, system: 8.53 s, elapsed: 61.07 s
system usage: CPU: user: 23.18 s, system: 8.25 s, elapsed: 58.99 s
system usage: CPU: user: 23.20 s, system: 8.98 s, elapsed: 66.86 s
That's fast enough that I thought an improvement would show up even with
standard WAL logging (no separate attachment, since it's a trivial change).
Seems a bit faster:
master:
system usage: CPU: user: 152.27 s, system: 11.76 s, elapsed: 216.86 s
system usage: CPU: user: 137.25 s, system: 11.07 s, elapsed: 213.62 s
system usage: CPU: user: 149.48 s, system: 12.15 s, elapsed: 220.96 s
v29:
system usage: CPU: user: 40.88 s, system: 15.99 s, elapsed: 170.98 s
system usage: CPU: user: 41.33 s, system: 15.45 s, elapsed: 166.75 s
system usage: CPU: user: 41.51 s, system: 18.20 s, elapsed: 203.94 s
There is more we could test here, but I feel better about these numbers.
In the next few days, I'll resume style review and list the remaining
issues we need to address.
--
John Naylor
EDB: http://www.enterprisedb.com
Attachments:
[application/sql] vacuum-test-lookup-int.sql (1.2K, ../../CAFBsxsHUxmXYy0y4RrhMcNe-R11Bm099Xe-wUdb78pOu0+PT2Q@mail.gmail.com/3-vacuum-test-lookup-int.sql)
download
[application/sql] vacuum-test-lookup-uuid.sql (1.0K, ../../CAFBsxsHUxmXYy0y4RrhMcNe-R11Bm099Xe-wUdb78pOu0+PT2Q@mail.gmail.com/4-vacuum-test-lookup-uuid.sql)
download
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-02-24 05:50 ` Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
1 sibling, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-02-24 05:50 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Wed, Feb 22, 2023 at 6:55 PM John Naylor
<[email protected]> wrote:
>
>
> On Wed, Feb 22, 2023 at 3:29 PM Masahiko Sawada <[email protected]> wrote:
> >
> > On Wed, Feb 22, 2023 at 4:35 PM John Naylor
> > <[email protected]> wrote:
> > >
> > > I don't think any vacuum calls in regression tests would stress any of this code very much, so it's not worth carrying the old way forward. I was thinking of only doing this as a short-time sanity check for testing a real-world workload.
> >
> > I guess that It would also be helpful at least until the GA release.
> > People will be able to test them easily on their workloads or their
> > custom test scenarios.
>
> That doesn't seem useful to me. If we've done enough testing to reassure us the new way always gives the same answer, the old way is not needed at commit time. If there is any doubt it will always give the same answer, then the whole patchset won't be committed.
True. Even if we're done enough testing we cannot claim there is no
bug. My idea is to make the bug investigation easier but on
reflection, it seems not the best idea given this purpose. Instead, it
seems to be better to add more necessary assertions. What do you think
about the attached patch? Please note that it also includes the
changes for minimum memory requirement.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 9360520482..fc20e58a95 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -75,6 +75,14 @@ typedef uint64 offsetbm;
#define LOWER_OFFSET_NBITS 6 /* log(sizeof(offsetbm), 2) */
#define LOWER_OFFSET_MASK ((1 << LOWER_OFFSET_NBITS) - 1)
+/*
+ * The minimum amount of memory required by TidStore is 2MB, the current minimum
+ * valid value for the maintenance_work_mem GUC. This is required to allocate the
+ * DSA initial segment, 1MB, and some meta data. This number is applied also to
+ * the local TidStore cases for simplicity.
+ */
+#define TIDSTORE_MIN_MEMORY (2 * 1024 * 1024L) /* 2MB */
+
/* A magic value used to identify our TidStore. */
#define TIDSTORE_MAGIC 0x826f6a10
@@ -101,7 +109,7 @@ typedef struct TidStoreControl
/* These values are never changed after creation */
size_t max_bytes; /* the maximum bytes a TidStore can use */
- int max_off; /* the maximum offset number */
+ OffsetNumber max_off; /* the maximum offset number */
int max_off_nbits; /* the number of bits required for offset
* numbers */
int upper_off_nbits; /* the number of bits of offset numbers
@@ -174,10 +182,17 @@ static inline tidkey encode_tid(TidStore *ts, ItemPointer tid, offsetbm *off_bit
* The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
*/
TidStore *
-TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
+TidStoreCreate(size_t max_bytes, OffsetNumber max_off, dsa_area *area)
{
TidStore *ts;
+ Assert(max_off <= MaxOffsetNumber);
+
+ /* Sanity check for the max_bytes */
+ if (max_bytes < TIDSTORE_MIN_MEMORY)
+ elog(ERROR, "memory for TidStore must be at least %ld, but %zu provided",
+ TIDSTORE_MIN_MEMORY, max_bytes);
+
ts = palloc0(sizeof(TidStore));
/*
@@ -192,8 +207,8 @@ TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
* In local TidStore cases, the radix tree uses slab allocators for each kind
* of node class. The most memory consuming case while adding Tids associated
* with one page (i.e. during TidStoreSetBlockOffsets()) is that we allocate a new
- * slab block for a new radix tree node, which is approximately 70kB. Therefore,
- * we deduct 70kB from the max_bytes.
+ * slab block for a new radix tree node, which is approximately 70kB at most.
+ * Therefore, we deduct 70kB from the max_bytes.
*
* In shared cases, DSA allocates the memory segments big enough to follow
* a geometric series that approximately doubles the total DSA size (see
@@ -378,6 +393,7 @@ TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
const int nkeys = UINT64CONST(1) << ts->control->upper_off_nbits;
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+ Assert(BlockNumberIsValid(blkno));
bitmaps = palloc(sizeof(offsetbm) * nkeys);
key = prev_key = key_base;
@@ -386,6 +402,8 @@ TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
{
offsetbm off_bit;
+ Assert(offsets[i] <= ts->control->max_off);
+
/* encode the tid to a key and partial offset */
key = encode_blk_off(ts, blkno, offsets[i], &off_bit);
@@ -452,6 +470,8 @@ TidStoreIsMember(TidStore *ts, ItemPointer tid)
offsetbm off_bit;
bool found;
+ Assert(ItemPointerIsValid(tid));
+
key = encode_tid(ts, tid, &off_bit);
if (TidStoreIsShared(ts))
@@ -535,6 +555,7 @@ TidStoreIterateNext(TidStoreIter *iter)
while (tidstore_iter(iter, &key, &off_bitmap))
{
BlockNumber blkno = key_get_blkno(iter->ts, key);
+ Assert(BlockNumberIsValid(blkno));
if (BlockNumberIsValid(output->blkno) && output->blkno != blkno)
{
@@ -586,6 +607,7 @@ TidStoreNumTids(TidStore *ts)
num_tids = ts->control->num_tids;
LWLockRelease(&ts->control->lock);
+ Assert(num_tids >= 0);
return num_tids;
}
diff --git a/src/include/access/tidstore.h b/src/include/access/tidstore.h
index 66f0fdd482..d1cc93cbb6 100644
--- a/src/include/access/tidstore.h
+++ b/src/include/access/tidstore.h
@@ -30,7 +30,7 @@ typedef struct TidStoreIterResult
OffsetNumber offsets[FLEXIBLE_ARRAY_MEMBER];
} TidStoreIterResult;
-extern TidStore *TidStoreCreate(size_t max_bytes, int max_off, dsa_area *dsa);
+extern TidStore *TidStoreCreate(size_t max_bytes, OffsetNumber max_off, dsa_area *dsa);
extern TidStore *TidStoreAttach(dsa_area *dsa, dsa_pointer handle);
extern void TidStoreDetach(TidStore *ts);
extern void TidStoreDestroy(TidStore *ts);
Attachments:
[text/plain] add_assertions.patch.txt (4.5K, ../../CAD21AoC=sazrXJUwH61CMz_+GjM__5BhQzMjN4-=gtOXWvkbAg@mail.gmail.com/2-add_assertions.patch.txt)
download | inline diff:
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 9360520482..fc20e58a95 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -75,6 +75,14 @@ typedef uint64 offsetbm;
#define LOWER_OFFSET_NBITS 6 /* log(sizeof(offsetbm), 2) */
#define LOWER_OFFSET_MASK ((1 << LOWER_OFFSET_NBITS) - 1)
+/*
+ * The minimum amount of memory required by TidStore is 2MB, the current minimum
+ * valid value for the maintenance_work_mem GUC. This is required to allocate the
+ * DSA initial segment, 1MB, and some meta data. This number is applied also to
+ * the local TidStore cases for simplicity.
+ */
+#define TIDSTORE_MIN_MEMORY (2 * 1024 * 1024L) /* 2MB */
+
/* A magic value used to identify our TidStore. */
#define TIDSTORE_MAGIC 0x826f6a10
@@ -101,7 +109,7 @@ typedef struct TidStoreControl
/* These values are never changed after creation */
size_t max_bytes; /* the maximum bytes a TidStore can use */
- int max_off; /* the maximum offset number */
+ OffsetNumber max_off; /* the maximum offset number */
int max_off_nbits; /* the number of bits required for offset
* numbers */
int upper_off_nbits; /* the number of bits of offset numbers
@@ -174,10 +182,17 @@ static inline tidkey encode_tid(TidStore *ts, ItemPointer tid, offsetbm *off_bit
* The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
*/
TidStore *
-TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
+TidStoreCreate(size_t max_bytes, OffsetNumber max_off, dsa_area *area)
{
TidStore *ts;
+ Assert(max_off <= MaxOffsetNumber);
+
+ /* Sanity check for the max_bytes */
+ if (max_bytes < TIDSTORE_MIN_MEMORY)
+ elog(ERROR, "memory for TidStore must be at least %ld, but %zu provided",
+ TIDSTORE_MIN_MEMORY, max_bytes);
+
ts = palloc0(sizeof(TidStore));
/*
@@ -192,8 +207,8 @@ TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
* In local TidStore cases, the radix tree uses slab allocators for each kind
* of node class. The most memory consuming case while adding Tids associated
* with one page (i.e. during TidStoreSetBlockOffsets()) is that we allocate a new
- * slab block for a new radix tree node, which is approximately 70kB. Therefore,
- * we deduct 70kB from the max_bytes.
+ * slab block for a new radix tree node, which is approximately 70kB at most.
+ * Therefore, we deduct 70kB from the max_bytes.
*
* In shared cases, DSA allocates the memory segments big enough to follow
* a geometric series that approximately doubles the total DSA size (see
@@ -378,6 +393,7 @@ TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
const int nkeys = UINT64CONST(1) << ts->control->upper_off_nbits;
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+ Assert(BlockNumberIsValid(blkno));
bitmaps = palloc(sizeof(offsetbm) * nkeys);
key = prev_key = key_base;
@@ -386,6 +402,8 @@ TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
{
offsetbm off_bit;
+ Assert(offsets[i] <= ts->control->max_off);
+
/* encode the tid to a key and partial offset */
key = encode_blk_off(ts, blkno, offsets[i], &off_bit);
@@ -452,6 +470,8 @@ TidStoreIsMember(TidStore *ts, ItemPointer tid)
offsetbm off_bit;
bool found;
+ Assert(ItemPointerIsValid(tid));
+
key = encode_tid(ts, tid, &off_bit);
if (TidStoreIsShared(ts))
@@ -535,6 +555,7 @@ TidStoreIterateNext(TidStoreIter *iter)
while (tidstore_iter(iter, &key, &off_bitmap))
{
BlockNumber blkno = key_get_blkno(iter->ts, key);
+ Assert(BlockNumberIsValid(blkno));
if (BlockNumberIsValid(output->blkno) && output->blkno != blkno)
{
@@ -586,6 +607,7 @@ TidStoreNumTids(TidStore *ts)
num_tids = ts->control->num_tids;
LWLockRelease(&ts->control->lock);
+ Assert(num_tids >= 0);
return num_tids;
}
diff --git a/src/include/access/tidstore.h b/src/include/access/tidstore.h
index 66f0fdd482..d1cc93cbb6 100644
--- a/src/include/access/tidstore.h
+++ b/src/include/access/tidstore.h
@@ -30,7 +30,7 @@ typedef struct TidStoreIterResult
OffsetNumber offsets[FLEXIBLE_ARRAY_MEMBER];
} TidStoreIterResult;
-extern TidStore *TidStoreCreate(size_t max_bytes, int max_off, dsa_area *dsa);
+extern TidStore *TidStoreCreate(size_t max_bytes, OffsetNumber max_off, dsa_area *dsa);
extern TidStore *TidStoreAttach(dsa_area *dsa, dsa_pointer handle);
extern void TidStoreDetach(TidStore *ts);
extern void TidStoreDestroy(TidStore *ts);
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-02-28 06:42 ` John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: John Naylor @ 2023-02-28 06:42 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Fri, Feb 24, 2023 at 12:50 PM Masahiko Sawada <[email protected]>
wrote:
>
> On Wed, Feb 22, 2023 at 6:55 PM John Naylor
> <[email protected]> wrote:
> >
> > That doesn't seem useful to me. If we've done enough testing to
reassure us the new way always gives the same answer, the old way is not
needed at commit time. If there is any doubt it will always give the same
answer, then the whole patchset won't be committed.
> My idea is to make the bug investigation easier but on
> reflection, it seems not the best idea given this purpose.
My concern with TIDSTORE_DEBUG is that it adds new code that mimics the old
tid array. As I've said, that doesn't seem like a good thing to carry
forward forevermore, in any form. Plus, comparing new code with new code is
not the same thing as comparing existing code with new code. That was my
idea upthread.
Maybe the effort my idea requires is too much vs. the likelihood of finding
a problem. In any case, it's clear that if I want that level of paranoia,
I'm going to have to do it myself.
> What do you think
> about the attached patch? Please note that it also includes the
> changes for minimum memory requirement.
Most of the asserts look logical, or at least harmless.
- int max_off; /* the maximum offset number */
+ OffsetNumber max_off; /* the maximum offset number */
I agree with using the specific type for offsets here, but I'm not sure why
this change belongs in this patch. If we decided against the new asserts,
this would be easy to lose.
This change, however, defies common sense:
+/*
+ * The minimum amount of memory required by TidStore is 2MB, the current
minimum
+ * valid value for the maintenance_work_mem GUC. This is required to
allocate the
+ * DSA initial segment, 1MB, and some meta data. This number is applied
also to
+ * the local TidStore cases for simplicity.
+ */
+#define TIDSTORE_MIN_MEMORY (2 * 1024 * 1024L) /* 2MB */
+ /* Sanity check for the max_bytes */
+ if (max_bytes < TIDSTORE_MIN_MEMORY)
+ elog(ERROR, "memory for TidStore must be at least %ld, but %zu provided",
+ TIDSTORE_MIN_MEMORY, max_bytes);
Aside from the fact that this elog's something that would never get past
development, the #define just adds a hard-coded copy of something that is
already hard-coded somewhere else, whose size depends on an implementation
detail in a third place.
This also assumes that all users of tid store are limited by
maintenance_work_mem. Andres thought of an example of some day unifying
with tidbitmap.c, and maybe other applications will be limited by work_mem.
But now that I'm looking at the guc tables, I am reminded that work_mem's
minimum is 64kB, so this highlights a design problem: There is obviously no
requirement that the minimum work_mem has to be >= a single DSA segment,
even though operations like parallel hash and parallel bitmap heap scan are
limited by work_mem. It would be nice to find out what happens with these
parallel features when work_mem is tiny (maybe parallelism is not even
considered?).
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-02-28 13:20 ` Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-02-28 13:20 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Tue, Feb 28, 2023 at 3:42 PM John Naylor
<[email protected]> wrote:
>
>
> On Fri, Feb 24, 2023 at 12:50 PM Masahiko Sawada <[email protected]> wrote:
> >
> > On Wed, Feb 22, 2023 at 6:55 PM John Naylor
> > <[email protected]> wrote:
> > >
> > > That doesn't seem useful to me. If we've done enough testing to reassure us the new way always gives the same answer, the old way is not needed at commit time. If there is any doubt it will always give the same answer, then the whole patchset won't be committed.
>
> > My idea is to make the bug investigation easier but on
> > reflection, it seems not the best idea given this purpose.
>
> My concern with TIDSTORE_DEBUG is that it adds new code that mimics the old tid array. As I've said, that doesn't seem like a good thing to carry forward forevermore, in any form. Plus, comparing new code with new code is not the same thing as comparing existing code with new code. That was my idea upthread.
>
> Maybe the effort my idea requires is too much vs. the likelihood of finding a problem. In any case, it's clear that if I want that level of paranoia, I'm going to have to do it myself.
>
> > What do you think
> > about the attached patch? Please note that it also includes the
> > changes for minimum memory requirement.
>
> Most of the asserts look logical, or at least harmless.
>
> - int max_off; /* the maximum offset number */
> + OffsetNumber max_off; /* the maximum offset number */
>
> I agree with using the specific type for offsets here, but I'm not sure why this change belongs in this patch. If we decided against the new asserts, this would be easy to lose.
Right. I'll separate this change as a separate patch.
>
> This change, however, defies common sense:
>
> +/*
> + * The minimum amount of memory required by TidStore is 2MB, the current minimum
> + * valid value for the maintenance_work_mem GUC. This is required to allocate the
> + * DSA initial segment, 1MB, and some meta data. This number is applied also to
> + * the local TidStore cases for simplicity.
> + */
> +#define TIDSTORE_MIN_MEMORY (2 * 1024 * 1024L) /* 2MB */
>
> + /* Sanity check for the max_bytes */
> + if (max_bytes < TIDSTORE_MIN_MEMORY)
> + elog(ERROR, "memory for TidStore must be at least %ld, but %zu provided",
> + TIDSTORE_MIN_MEMORY, max_bytes);
>
> Aside from the fact that this elog's something that would never get past development, the #define just adds a hard-coded copy of something that is already hard-coded somewhere else, whose size depends on an implementation detail in a third place.
>
> This also assumes that all users of tid store are limited by maintenance_work_mem. Andres thought of an example of some day unifying with tidbitmap.c, and maybe other applications will be limited by work_mem.
>
> But now that I'm looking at the guc tables, I am reminded that work_mem's minimum is 64kB, so this highlights a design problem: There is obviously no requirement that the minimum work_mem has to be >= a single DSA segment, even though operations like parallel hash and parallel bitmap heap scan are limited by work_mem.
Right.
> It would be nice to find out what happens with these parallel features when work_mem is tiny (maybe parallelism is not even considered?).
IIUC both don't care about the allocated DSA segment size. Parallel
hash accounts actual tuple (+ header) size as used memory but doesn't
consider how much DSA segment is allocated behind. Both parallel hash
and parallel bitmap scan can work even with work_mem = 64kB, but when
checking the total DSA segment size allocated during these operations,
it was 1MB.
I realized that there is a similar memory limit design issue also on
the non-shared tidstore cases. We deduct 70kB from max_bytes but it
won't work fine with work_mem = 64kB. Probably we need to reconsider
it. FYI 70kB comes from the maximum slab block size for node256.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-02-28 15:09 ` Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-02-28 15:09 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Tue, Feb 28, 2023 at 10:20 PM Masahiko Sawada <[email protected]> wrote:
>
> On Tue, Feb 28, 2023 at 3:42 PM John Naylor
> <[email protected]> wrote:
> >
> >
> > On Fri, Feb 24, 2023 at 12:50 PM Masahiko Sawada <[email protected]> wrote:
> > >
> > > On Wed, Feb 22, 2023 at 6:55 PM John Naylor
> > > <[email protected]> wrote:
> > > >
> > > > That doesn't seem useful to me. If we've done enough testing to reassure us the new way always gives the same answer, the old way is not needed at commit time. If there is any doubt it will always give the same answer, then the whole patchset won't be committed.
> >
> > > My idea is to make the bug investigation easier but on
> > > reflection, it seems not the best idea given this purpose.
> >
> > My concern with TIDSTORE_DEBUG is that it adds new code that mimics the old tid array. As I've said, that doesn't seem like a good thing to carry forward forevermore, in any form. Plus, comparing new code with new code is not the same thing as comparing existing code with new code. That was my idea upthread.
> >
> > Maybe the effort my idea requires is too much vs. the likelihood of finding a problem. In any case, it's clear that if I want that level of paranoia, I'm going to have to do it myself.
> >
> > > What do you think
> > > about the attached patch? Please note that it also includes the
> > > changes for minimum memory requirement.
> >
> > Most of the asserts look logical, or at least harmless.
> >
> > - int max_off; /* the maximum offset number */
> > + OffsetNumber max_off; /* the maximum offset number */
> >
> > I agree with using the specific type for offsets here, but I'm not sure why this change belongs in this patch. If we decided against the new asserts, this would be easy to lose.
>
> Right. I'll separate this change as a separate patch.
>
> >
> > This change, however, defies common sense:
> >
> > +/*
> > + * The minimum amount of memory required by TidStore is 2MB, the current minimum
> > + * valid value for the maintenance_work_mem GUC. This is required to allocate the
> > + * DSA initial segment, 1MB, and some meta data. This number is applied also to
> > + * the local TidStore cases for simplicity.
> > + */
> > +#define TIDSTORE_MIN_MEMORY (2 * 1024 * 1024L) /* 2MB */
> >
> > + /* Sanity check for the max_bytes */
> > + if (max_bytes < TIDSTORE_MIN_MEMORY)
> > + elog(ERROR, "memory for TidStore must be at least %ld, but %zu provided",
> > + TIDSTORE_MIN_MEMORY, max_bytes);
> >
> > Aside from the fact that this elog's something that would never get past development, the #define just adds a hard-coded copy of something that is already hard-coded somewhere else, whose size depends on an implementation detail in a third place.
> >
> > This also assumes that all users of tid store are limited by maintenance_work_mem. Andres thought of an example of some day unifying with tidbitmap.c, and maybe other applications will be limited by work_mem.
> >
> > But now that I'm looking at the guc tables, I am reminded that work_mem's minimum is 64kB, so this highlights a design problem: There is obviously no requirement that the minimum work_mem has to be >= a single DSA segment, even though operations like parallel hash and parallel bitmap heap scan are limited by work_mem.
>
> Right.
>
> > It would be nice to find out what happens with these parallel features when work_mem is tiny (maybe parallelism is not even considered?).
>
> IIUC both don't care about the allocated DSA segment size. Parallel
> hash accounts actual tuple (+ header) size as used memory but doesn't
> consider how much DSA segment is allocated behind. Both parallel hash
> and parallel bitmap scan can work even with work_mem = 64kB, but when
> checking the total DSA segment size allocated during these operations,
> it was 1MB.
>
> I realized that there is a similar memory limit design issue also on
> the non-shared tidstore cases. We deduct 70kB from max_bytes but it
> won't work fine with work_mem = 64kB. Probably we need to reconsider
> it. FYI 70kB comes from the maximum slab block size for node256.
Currently, we calculate the slab block size enough to allocate 32
chunks from there. For node256, the leaf node is 2,088 bytes and the
slab block size is 66,816 bytes. One idea to fix this issue to
decrease it. For example, with 16 chunks the slab block size is 33,408
bytes and with 8 chunks it's 16,704 bytes. I ran a brief benchmark
test with 70kB block size and 16kB block size:
* 70kB slab blocks:
select * from bench_search_random_nodes(20 * 1000 * 1000, '0xFFFFFF');
height = 2, n3 = 0, n15 = 0, n32 = 0, n125 = 0, n256 = 65793
mem_allocated | load_ms | search_ms
---------------+---------+-----------
143085184 | 1216 | 750
(1 row)
* 16kB slab blocks:
select * from bench_search_random_nodes(20 * 1000 * 1000, '0xFFFFFF');
height = 2, n3 = 0, n15 = 0, n32 = 0, n125 = 0, n256 = 65793
mem_allocated | load_ms | search_ms
---------------+---------+-----------
157601248 | 1220 | 786
(1 row)
There is a performance difference a bit but a smaller slab block size
seems to be acceptable if there is no other better way.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-03-01 06:37 ` John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: John Naylor @ 2023-03-01 06:37 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Tue, Feb 28, 2023 at 10:09 PM Masahiko Sawada <[email protected]>
wrote:
>
> On Tue, Feb 28, 2023 at 10:20 PM Masahiko Sawada <[email protected]>
wrote:
> >
> > On Tue, Feb 28, 2023 at 3:42 PM John Naylor
> > <[email protected]> wrote:
> > >
> > >
> > > On Fri, Feb 24, 2023 at 12:50 PM Masahiko Sawada <
[email protected]> wrote:
> > > >
> > > > On Wed, Feb 22, 2023 at 6:55 PM John Naylor
> > > > <[email protected]> wrote:
> > > > >
> > > > > That doesn't seem useful to me. If we've done enough testing to
reassure us the new way always gives the same answer, the old way is not
needed at commit time. If there is any doubt it will always give the same
answer, then the whole patchset won't be committed.
> > >
> > > > My idea is to make the bug investigation easier but on
> > > > reflection, it seems not the best idea given this purpose.
> > >
> > > My concern with TIDSTORE_DEBUG is that it adds new code that mimics
the old tid array. As I've said, that doesn't seem like a good thing to
carry forward forevermore, in any form. Plus, comparing new code with new
code is not the same thing as comparing existing code with new code. That
was my idea upthread.
> > >
> > > Maybe the effort my idea requires is too much vs. the likelihood of
finding a problem. In any case, it's clear that if I want that level of
paranoia, I'm going to have to do it myself.
> > >
> > > > What do you think
> > > > about the attached patch? Please note that it also includes the
> > > > changes for minimum memory requirement.
> > >
> > > Most of the asserts look logical, or at least harmless.
> > >
> > > - int max_off; /* the maximum offset number */
> > > + OffsetNumber max_off; /* the maximum offset number */
> > >
> > > I agree with using the specific type for offsets here, but I'm not
sure why this change belongs in this patch. If we decided against the new
asserts, this would be easy to lose.
> >
> > Right. I'll separate this change as a separate patch.
> >
> > >
> > > This change, however, defies common sense:
> > >
> > > +/*
> > > + * The minimum amount of memory required by TidStore is 2MB, the
current minimum
> > > + * valid value for the maintenance_work_mem GUC. This is required to
allocate the
> > > + * DSA initial segment, 1MB, and some meta data. This number is
applied also to
> > > + * the local TidStore cases for simplicity.
> > > + */
> > > +#define TIDSTORE_MIN_MEMORY (2 * 1024 * 1024L) /* 2MB */
> > >
> > > + /* Sanity check for the max_bytes */
> > > + if (max_bytes < TIDSTORE_MIN_MEMORY)
> > > + elog(ERROR, "memory for TidStore must be at least %ld, but %zu
provided",
> > > + TIDSTORE_MIN_MEMORY, max_bytes);
> > >
> > > Aside from the fact that this elog's something that would never get
past development, the #define just adds a hard-coded copy of something that
is already hard-coded somewhere else, whose size depends on an
implementation detail in a third place.
> > >
> > > This also assumes that all users of tid store are limited by
maintenance_work_mem. Andres thought of an example of some day unifying
with tidbitmap.c, and maybe other applications will be limited by work_mem.
> > >
> > > But now that I'm looking at the guc tables, I am reminded that
work_mem's minimum is 64kB, so this highlights a design problem: There is
obviously no requirement that the minimum work_mem has to be >= a single
DSA segment, even though operations like parallel hash and parallel bitmap
heap scan are limited by work_mem.
> >
> > Right.
> >
> > > It would be nice to find out what happens with these parallel
features when work_mem is tiny (maybe parallelism is not even considered?).
> >
> > IIUC both don't care about the allocated DSA segment size. Parallel
> > hash accounts actual tuple (+ header) size as used memory but doesn't
> > consider how much DSA segment is allocated behind. Both parallel hash
> > and parallel bitmap scan can work even with work_mem = 64kB, but when
> > checking the total DSA segment size allocated during these operations,
> > it was 1MB.
> >
> > I realized that there is a similar memory limit design issue also on
> > the non-shared tidstore cases. We deduct 70kB from max_bytes but it
> > won't work fine with work_mem = 64kB. Probably we need to reconsider
> > it. FYI 70kB comes from the maximum slab block size for node256.
>
> Currently, we calculate the slab block size enough to allocate 32
> chunks from there. For node256, the leaf node is 2,088 bytes and the
> slab block size is 66,816 bytes. One idea to fix this issue to
> decrease it.
I think we're trying to solve the wrong problem here. I need to study this
more, but it seems that code that needs to stay within a memory limit only
needs to track what's been allocated in chunks within a block, since
writing there is what invokes a page fault. If we're not keeping track of
each and every chunk space, for speed, it doesn't follow that we need to
keep every block allocation within the configured limit. I'm guessing we
can just ask the context if the block space has gone *over* the limit, and
we can assume that the last allocation we perform will only fault one
additional page. We need to have a clear answer on this before doing
anything else.
If that's correct, and I'm not positive yet, we can get rid of all the
fragile assumptions about things the tid store has no business knowing
about, as well as the guc change. I'm not sure how this affects progress
reporting, because it would be nice if it didn't report dead_tuple_bytes
bigger than max_dead_tuple_bytes.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-03-01 11:58 ` Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-03-01 11:58 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Wed, Mar 1, 2023 at 3:37 PM John Naylor <[email protected]> wrote:
>
> On Tue, Feb 28, 2023 at 10:09 PM Masahiko Sawada <[email protected]> wrote:
> >
> > On Tue, Feb 28, 2023 at 10:20 PM Masahiko Sawada <[email protected]> wrote:
> > >
> > > On Tue, Feb 28, 2023 at 3:42 PM John Naylor
> > > <[email protected]> wrote:
> > > >
> > > >
> > > > On Fri, Feb 24, 2023 at 12:50 PM Masahiko Sawada <[email protected]> wrote:
> > > > >
> > > > > On Wed, Feb 22, 2023 at 6:55 PM John Naylor
> > > > > <[email protected]> wrote:
> > > > > >
> > > > > > That doesn't seem useful to me. If we've done enough testing to reassure us the new way always gives the same answer, the old way is not needed at commit time. If there is any doubt it will always give the same answer, then the whole patchset won't be committed.
> > > >
> > > > > My idea is to make the bug investigation easier but on
> > > > > reflection, it seems not the best idea given this purpose.
> > > >
> > > > My concern with TIDSTORE_DEBUG is that it adds new code that mimics the old tid array. As I've said, that doesn't seem like a good thing to carry forward forevermore, in any form. Plus, comparing new code with new code is not the same thing as comparing existing code with new code. That was my idea upthread.
> > > >
> > > > Maybe the effort my idea requires is too much vs. the likelihood of finding a problem. In any case, it's clear that if I want that level of paranoia, I'm going to have to do it myself.
> > > >
> > > > > What do you think
> > > > > about the attached patch? Please note that it also includes the
> > > > > changes for minimum memory requirement.
> > > >
> > > > Most of the asserts look logical, or at least harmless.
> > > >
> > > > - int max_off; /* the maximum offset number */
> > > > + OffsetNumber max_off; /* the maximum offset number */
> > > >
> > > > I agree with using the specific type for offsets here, but I'm not sure why this change belongs in this patch. If we decided against the new asserts, this would be easy to lose.
> > >
> > > Right. I'll separate this change as a separate patch.
> > >
> > > >
> > > > This change, however, defies common sense:
> > > >
> > > > +/*
> > > > + * The minimum amount of memory required by TidStore is 2MB, the current minimum
> > > > + * valid value for the maintenance_work_mem GUC. This is required to allocate the
> > > > + * DSA initial segment, 1MB, and some meta data. This number is applied also to
> > > > + * the local TidStore cases for simplicity.
> > > > + */
> > > > +#define TIDSTORE_MIN_MEMORY (2 * 1024 * 1024L) /* 2MB */
> > > >
> > > > + /* Sanity check for the max_bytes */
> > > > + if (max_bytes < TIDSTORE_MIN_MEMORY)
> > > > + elog(ERROR, "memory for TidStore must be at least %ld, but %zu provided",
> > > > + TIDSTORE_MIN_MEMORY, max_bytes);
> > > >
> > > > Aside from the fact that this elog's something that would never get past development, the #define just adds a hard-coded copy of something that is already hard-coded somewhere else, whose size depends on an implementation detail in a third place.
> > > >
> > > > This also assumes that all users of tid store are limited by maintenance_work_mem. Andres thought of an example of some day unifying with tidbitmap.c, and maybe other applications will be limited by work_mem.
> > > >
> > > > But now that I'm looking at the guc tables, I am reminded that work_mem's minimum is 64kB, so this highlights a design problem: There is obviously no requirement that the minimum work_mem has to be >= a single DSA segment, even though operations like parallel hash and parallel bitmap heap scan are limited by work_mem.
> > >
> > > Right.
> > >
> > > > It would be nice to find out what happens with these parallel features when work_mem is tiny (maybe parallelism is not even considered?).
> > >
> > > IIUC both don't care about the allocated DSA segment size. Parallel
> > > hash accounts actual tuple (+ header) size as used memory but doesn't
> > > consider how much DSA segment is allocated behind. Both parallel hash
> > > and parallel bitmap scan can work even with work_mem = 64kB, but when
> > > checking the total DSA segment size allocated during these operations,
> > > it was 1MB.
> > >
> > > I realized that there is a similar memory limit design issue also on
> > > the non-shared tidstore cases. We deduct 70kB from max_bytes but it
> > > won't work fine with work_mem = 64kB. Probably we need to reconsider
> > > it. FYI 70kB comes from the maximum slab block size for node256.
> >
> > Currently, we calculate the slab block size enough to allocate 32
> > chunks from there. For node256, the leaf node is 2,088 bytes and the
> > slab block size is 66,816 bytes. One idea to fix this issue to
> > decrease it.
>
> I think we're trying to solve the wrong problem here. I need to study this more, but it seems that code that needs to stay within a memory limit only needs to track what's been allocated in chunks within a block, since writing there is what invokes a page fault.
Right. I guess we've discussed what we use for calculating the *used*
memory amount but I don't remember.
I think I was confused by the fact that we use some different
approaches to calculate the amount of used memory. Parallel hash and
tidbitmap use the allocated chunk size whereas hash_agg_check_limits()
in nodeAgg.c uses MemoryContextMemAllocated(), which uses the
allocated block size.
> If we're not keeping track of each and every chunk space, for speed, it doesn't follow that we need to keep every block allocation within the configured limit. I'm guessing we can just ask the context if the block space has gone *over* the limit, and we can assume that the last allocation we perform will only fault one additional page. We need to have a clear answer on this before doing anything else.
>
> If that's correct, and I'm not positive yet, we can get rid of all the fragile assumptions about things the tid store has no business knowing about, as well as the guc change.
True.
> I'm not sure how this affects progress reporting, because it would be nice if it didn't report dead_tuple_bytes bigger than max_dead_tuple_bytes.
Yes, the progress reporting could be confusable. Particularly, in
shared tidstore cases, the dead_tuple_bytes could be much bigger than
max_dead_tuple_bytes. Probably what we need might be functions for
MemoryContext and dsa_area to get the amount of memory that has been
allocated, by not tracking every chunk space. For example, the
functions would be like what SlabStats() does; iterate over every
block and calculates the total/free memory usage.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-03-03 11:03 ` John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: John Naylor @ 2023-03-03 11:03 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Wed, Mar 1, 2023 at 6:59 PM Masahiko Sawada <[email protected]>
wrote:
>
> On Wed, Mar 1, 2023 at 3:37 PM John Naylor <[email protected]>
wrote:
> >
> > I think we're trying to solve the wrong problem here. I need to study
this more, but it seems that code that needs to stay within a memory limit
only needs to track what's been allocated in chunks within a block, since
writing there is what invokes a page fault.
>
> Right. I guess we've discussed what we use for calculating the *used*
> memory amount but I don't remember.
>
> I think I was confused by the fact that we use some different
> approaches to calculate the amount of used memory. Parallel hash and
> tidbitmap use the allocated chunk size whereas hash_agg_check_limits()
> in nodeAgg.c uses MemoryContextMemAllocated(), which uses the
> allocated block size.
That's good to know. The latter says:
* After adding a new group to the hash table, check whether we need to
enter
* spill mode. Allocations may happen without adding new groups (for
instance,
* if the transition state size grows), so this check is imperfect.
I'm willing to claim that vacuum can be imperfect also, given the tid
store's properties: 1) on average much more efficient in used space, and 2)
no longer bound by the 1GB limit.
> > I'm not sure how this affects progress reporting, because it would be
nice if it didn't report dead_tuple_bytes bigger than max_dead_tuple_bytes.
>
> Yes, the progress reporting could be confusable. Particularly, in
> shared tidstore cases, the dead_tuple_bytes could be much bigger than
> max_dead_tuple_bytes. Probably what we need might be functions for
> MemoryContext and dsa_area to get the amount of memory that has been
> allocated, by not tracking every chunk space. For example, the
> functions would be like what SlabStats() does; iterate over every
> block and calculates the total/free memory usage.
I'm not sure we need to invent new infrastructure for this. Looking at v29
in vacuumlazy.c, the order of operations for memory accounting is:
First, get the block-level space -- stop and vacuum indexes if we exceed
the limit:
/*
* Consider if we definitely have enough space to process TIDs on page
* already. If we are close to overrunning the available space for
* dead_items TIDs, pause and do a cycle of vacuuming before we tackle
* this page.
*/
if (TidStoreIsFull(vacrel->dead_items)) --> which is basically "if
(TidStoreMemoryUsage(ts) > ts->control->max_bytes)"
Then, after pruning the current page, store the tids and then get the
block-level space again:
else if (prunestate.num_offsets > 0)
{
/* Save details of the LP_DEAD items from the page in dead_items */
TidStoreSetBlockOffsets(...);
pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
TidStoreMemoryUsage(dead_items));
}
Since the block-level measurement is likely overestimating quite a bit, I
propose to simply reverse the order of the actions here, effectively
reporting progress for the *last page* and not the current one: First
update progress with the current memory usage, then add tids for this page.
If this allocated a new block, only a small bit of that will be written to.
If this block pushes it over the limit, we will detect that up at the top
of the loop. It's kind of like our earlier attempts at a "fudge factor",
but simpler and less brittle. And, as far as OS pages we have actually
written to, I think it'll effectively respect the memory limit, at least in
the local mem case. And the numbers will make sense.
Thoughts?
But now that I'm looking more closely at the details of memory accounting,
I don't like that TidStoreMemoryUsage() is called twice per page pruned
(see above). Maybe it wouldn't noticeably slow things down, but it's a bit
sloppy. It seems like we should call it once per loop and save the result
somewhere. If that's the right way to go, that possibly indicates that
TidStoreIsFull() is not a useful interface, at least in this form.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-03-06 06:27 ` Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-03-06 06:27 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Fri, Mar 3, 2023 at 8:04 PM John Naylor <[email protected]> wrote:
>
> On Wed, Mar 1, 2023 at 6:59 PM Masahiko Sawada <[email protected]> wrote:
> >
> > On Wed, Mar 1, 2023 at 3:37 PM John Naylor <[email protected]> wrote:
> > >
> > > I think we're trying to solve the wrong problem here. I need to study this more, but it seems that code that needs to stay within a memory limit only needs to track what's been allocated in chunks within a block, since writing there is what invokes a page fault.
> >
> > Right. I guess we've discussed what we use for calculating the *used*
> > memory amount but I don't remember.
> >
> > I think I was confused by the fact that we use some different
> > approaches to calculate the amount of used memory. Parallel hash and
> > tidbitmap use the allocated chunk size whereas hash_agg_check_limits()
> > in nodeAgg.c uses MemoryContextMemAllocated(), which uses the
> > allocated block size.
>
> That's good to know. The latter says:
>
> * After adding a new group to the hash table, check whether we need to enter
> * spill mode. Allocations may happen without adding new groups (for instance,
> * if the transition state size grows), so this check is imperfect.
>
> I'm willing to claim that vacuum can be imperfect also, given the tid store's properties: 1) on average much more efficient in used space, and 2) no longer bound by the 1GB limit.
>
> > > I'm not sure how this affects progress reporting, because it would be nice if it didn't report dead_tuple_bytes bigger than max_dead_tuple_bytes.
> >
> > Yes, the progress reporting could be confusable. Particularly, in
> > shared tidstore cases, the dead_tuple_bytes could be much bigger than
> > max_dead_tuple_bytes. Probably what we need might be functions for
> > MemoryContext and dsa_area to get the amount of memory that has been
> > allocated, by not tracking every chunk space. For example, the
> > functions would be like what SlabStats() does; iterate over every
> > block and calculates the total/free memory usage.
>
> I'm not sure we need to invent new infrastructure for this. Looking at v29 in vacuumlazy.c, the order of operations for memory accounting is:
>
> First, get the block-level space -- stop and vacuum indexes if we exceed the limit:
>
> /*
> * Consider if we definitely have enough space to process TIDs on page
> * already. If we are close to overrunning the available space for
> * dead_items TIDs, pause and do a cycle of vacuuming before we tackle
> * this page.
> */
> if (TidStoreIsFull(vacrel->dead_items)) --> which is basically "if (TidStoreMemoryUsage(ts) > ts->control->max_bytes)"
>
> Then, after pruning the current page, store the tids and then get the block-level space again:
>
> else if (prunestate.num_offsets > 0)
> {
> /* Save details of the LP_DEAD items from the page in dead_items */
> TidStoreSetBlockOffsets(...);
>
> pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
> TidStoreMemoryUsage(dead_items));
> }
>
> Since the block-level measurement is likely overestimating quite a bit, I propose to simply reverse the order of the actions here, effectively reporting progress for the *last page* and not the current one: First update progress with the current memory usage, then add tids for this page. If this allocated a new block, only a small bit of that will be written to. If this block pushes it over the limit, we will detect that up at the top of the loop. It's kind of like our earlier attempts at a "fudge factor", but simpler and less brittle. And, as far as OS pages we have actually written to, I think it'll effectively respect the memory limit, at least in the local mem case. And the numbers will make sense.
>
> Thoughts?
It looks to work but it still doesn't work in a case where a shared
tidstore is created with a 64kB memory limit, right?
TidStoreMemoryUsage() returns 1MB and TidStoreIsFull() returns true
from the beginning.
BTW I realized that since the caller can pass dsa_area to tidstore
(and radix tree), if other data are allocated in the same DSA are,
TidStoreMemoryUsage() (and RT_MEMORY_USAGE()) returns the memory usage
that includes not only itself but also other data. Probably it's
better to comment that the passed dsa_area should be dedicated to a
tidstore (or a radix tree).
>
> But now that I'm looking more closely at the details of memory accounting, I don't like that TidStoreMemoryUsage() is called twice per page pruned (see above). Maybe it wouldn't noticeably slow things down, but it's a bit sloppy. It seems like we should call it once per loop and save the result somewhere. If that's the right way to go, that possibly indicates that TidStoreIsFull() is not a useful interface, at least in this form.
Agreed.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-03-06 16:00 ` John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: John Naylor @ 2023-03-06 16:00 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Mon, Mar 6, 2023 at 1:28 PM Masahiko Sawada <[email protected]>
wrote:
> > Since the block-level measurement is likely overestimating quite a bit,
I propose to simply reverse the order of the actions here, effectively
reporting progress for the *last page* and not the current one: First
update progress with the current memory usage, then add tids for this page.
If this allocated a new block, only a small bit of that will be written to.
If this block pushes it over the limit, we will detect that up at the top
of the loop. It's kind of like our earlier attempts at a "fudge factor",
but simpler and less brittle. And, as far as OS pages we have actually
written to, I think it'll effectively respect the memory limit, at least in
the local mem case. And the numbers will make sense.
> >
> > Thoughts?
>
> It looks to work but it still doesn't work in a case where a shared
> tidstore is created with a 64kB memory limit, right?
> TidStoreMemoryUsage() returns 1MB and TidStoreIsFull() returns true
> from the beginning.
I have two ideas:
1. Make it optional to track chunk memory space by a template parameter. It
might be tiny compared to everything else that vacuum does. That would
allow other users to avoid that overhead.
2. When context block usage exceeds the limit (rare), make the additional
effort to get the precise usage -- I'm not sure such a top-down facility
exists, and I'm not feeling well enough today to study this further.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-03-07 01:24 ` Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-03-07 01:24 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Tue, Mar 7, 2023 at 1:01 AM John Naylor <[email protected]> wrote:
>
> On Mon, Mar 6, 2023 at 1:28 PM Masahiko Sawada <[email protected]> wrote:
>
> > > Since the block-level measurement is likely overestimating quite a bit, I propose to simply reverse the order of the actions here, effectively reporting progress for the *last page* and not the current one: First update progress with the current memory usage, then add tids for this page. If this allocated a new block, only a small bit of that will be written to. If this block pushes it over the limit, we will detect that up at the top of the loop. It's kind of like our earlier attempts at a "fudge factor", but simpler and less brittle. And, as far as OS pages we have actually written to, I think it'll effectively respect the memory limit, at least in the local mem case. And the numbers will make sense.
> > >
> > > Thoughts?
> >
> > It looks to work but it still doesn't work in a case where a shared
> > tidstore is created with a 64kB memory limit, right?
> > TidStoreMemoryUsage() returns 1MB and TidStoreIsFull() returns true
> > from the beginning.
>
> I have two ideas:
>
> 1. Make it optional to track chunk memory space by a template parameter. It might be tiny compared to everything else that vacuum does. That would allow other users to avoid that overhead.
> 2. When context block usage exceeds the limit (rare), make the additional effort to get the precise usage -- I'm not sure such a top-down facility exists, and I'm not feeling well enough today to study this further.
I prefer option (1) as it's straight forward. I mentioned a similar
idea before[1]. RT_MEMORY_USAGE() is defined only when the macro is
defined. It might be worth checking if there is visible overhead of
tracking chunk memory space. IIRC we've not evaluated it yet.
[1] https://www.postgresql.org/message-id/CAD21AoDK3gbX-jVxT6Pfso1Na0Krzr8Q15498Aj6tmXgzMFksA%40mail.gma...
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-03-08 04:40 ` John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: John Naylor @ 2023-03-08 04:40 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Tue, Mar 7, 2023 at 8:25 AM Masahiko Sawada <[email protected]>
wrote:
> > 1. Make it optional to track chunk memory space by a template
parameter. It might be tiny compared to everything else that vacuum does.
That would allow other users to avoid that overhead.
> > 2. When context block usage exceeds the limit (rare), make the
additional effort to get the precise usage -- I'm not sure such a top-down
facility exists, and I'm not feeling well enough today to study this
further.
>
> I prefer option (1) as it's straight forward. I mentioned a similar
> idea before[1]. RT_MEMORY_USAGE() is defined only when the macro is
> defined. It might be worth checking if there is visible overhead of
> tracking chunk memory space. IIRC we've not evaluated it yet.
Ok, let's try this -- I can test and profile later this week.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-03-09 06:51 ` Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-03-09 06:51 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Wed, Mar 8, 2023 at 1:40 PM John Naylor <[email protected]> wrote:
>
>
> On Tue, Mar 7, 2023 at 8:25 AM Masahiko Sawada <[email protected]> wrote:
>
> > > 1. Make it optional to track chunk memory space by a template parameter. It might be tiny compared to everything else that vacuum does. That would allow other users to avoid that overhead.
> > > 2. When context block usage exceeds the limit (rare), make the additional effort to get the precise usage -- I'm not sure such a top-down facility exists, and I'm not feeling well enough today to study this further.
> >
> > I prefer option (1) as it's straight forward. I mentioned a similar
> > idea before[1]. RT_MEMORY_USAGE() is defined only when the macro is
> > defined. It might be worth checking if there is visible overhead of
> > tracking chunk memory space. IIRC we've not evaluated it yet.
>
> Ok, let's try this -- I can test and profile later this week.
Thanks!
I've attached the new version patches. I merged improvements and fixes
I did in the v29 patch. 0007 through 0010 are updates from v29. The
main change made in v30 is to make the memory measurement and
RT_MEMORY_USAGE() optional, which is done in 0007 patch. The 0008 and
0009 patches are the updates for tidstore and the vacuum integration
patches. Here are results of quick tests (an average of 3 executions):
query: select * from bench_load_random_int(10 * 1000 * 1000)
* w/ RT_MEASURE_MEMORY_USAGE:
mem_allocated | load_ms
---------------+---------
1996512000 | 3305
(1 row)
* w/o RT_MEASURE_MEMORY_USAGE:
mem_allocated | load_ms
---------------+---------
0 | 3258
(1 row)
It seems to be within a noise level but I agree to make it optional.
Apart from the memory measurement stuff, I've done another todo item
on my list; adding min max classes for node3 and node125. I've done
that in 0010 patch, and here is a quick test result:
query: select * from bench_load_random_int(10 * 1000 * 1000)
* w/ 0000 patch
mem_allocated | load_ms
---------------+---------
1268630080 | 3275
(1 row)
* w/o 0000 patch
mem_allocated | load_ms
---------------+---------
1996512000 | 3214
(1 row)
That's a good improvement on the memory usage, without a noticeable
performance overhead. FYI CLASS_3_MIN has 1 fanout and is 24 bytes in
size, and CLASS_125_MIN has 61 fanouts and is 768 bytes in size.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] v30-0008-Remove-the-max-memory-deduction-from-TidStore.patch (3.9K, ../../CAD21AoC+kjqkSnLLcbqtu=Td5HSx2H+iAwNHCeZcWqjy1w7k5g@mail.gmail.com/2-v30-0008-Remove-the-max-memory-deduction-from-TidStore.patch)
download | inline diff:
From 5e3e7098eb12ec1d7ee546cc8f6e635638f131be Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 8 Mar 2023 15:08:58 +0900
Subject: [PATCH v30 08/11] Remove the max memory deduction from TidStore.
---
src/backend/access/common/tidstore.c | 43 +++++++---------------------
1 file changed, 10 insertions(+), 33 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 2d6f2b3ab9..54e2ef29db 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -82,6 +82,7 @@ typedef uint64 offsetbm;
#define RT_SCOPE static
#define RT_DECLARE
#define RT_DEFINE
+#define RT_MEASURE_MEMORY_USAGE
#define RT_VALUE_TYPE tidkey
#include "lib/radixtree.h"
@@ -90,6 +91,7 @@ typedef uint64 offsetbm;
#define RT_SCOPE static
#define RT_DECLARE
#define RT_DEFINE
+#define RT_MEASURE_MEMORY_USAGE
#define RT_VALUE_TYPE tidkey
#include "lib/radixtree.h"
@@ -182,39 +184,15 @@ TidStoreCreate(size_t max_bytes, OffsetNumber max_off, dsa_area *area)
ts = palloc0(sizeof(TidStore));
- /*
- * Create the radix tree for the main storage.
- *
- * Memory consumption depends on the number of stored tids, but also on the
- * distribution of them, how the radix tree stores, and the memory management
- * that backed the radix tree. The maximum bytes that a TidStore can
- * use is specified by the max_bytes in TidStoreCreate(). We want the total
- * amount of memory consumption by a TidStore not to exceed the max_bytes.
- *
- * In local TidStore cases, the radix tree uses slab allocators for each kind
- * of node class. The most memory consuming case while adding Tids associated
- * with one page (i.e. during TidStoreSetBlockOffsets()) is that we allocate a new
- * slab block for a new radix tree node, which is approximately 70kB. Therefore,
- * we deduct 70kB from the max_bytes.
- *
- * In shared cases, DSA allocates the memory segments big enough to follow
- * a geometric series that approximately doubles the total DSA size (see
- * make_new_segment() in dsa.c). We simulated the how DSA increases segment
- * size and the simulation revealed, the 75% threshold for the maximum bytes
- * perfectly works in case where the max_bytes is a power-of-2, and the 60%
- * threshold works for other cases.
- */
if (area != NULL)
{
dsa_pointer dp;
- float ratio = ((max_bytes & (max_bytes - 1)) == 0) ? 0.75 : 0.6;
ts->tree.shared = shared_rt_create(CurrentMemoryContext, area,
LWTRANCHE_SHARED_TIDSTORE);
dp = dsa_allocate0(area, sizeof(TidStoreControl));
ts->control = (TidStoreControl *) dsa_get_address(area, dp);
- ts->control->max_bytes = (size_t) (max_bytes * ratio);
ts->area = area;
ts->control->magic = TIDSTORE_MAGIC;
@@ -225,11 +203,15 @@ TidStoreCreate(size_t max_bytes, OffsetNumber max_off, dsa_area *area)
else
{
ts->tree.local = local_rt_create(CurrentMemoryContext);
-
ts->control = (TidStoreControl *) palloc0(sizeof(TidStoreControl));
- ts->control->max_bytes = max_bytes - (70 * 1024);
}
+ /*
+ * max_bytes is forced to be at least 64KB, the current minimum valid value
+ * for the work_mem GUC.
+ */
+ ts->control->max_bytes = Max(64 * 1024L, max_bytes);
+
ts->control->max_off = max_off;
ts->control->max_off_nbits = pg_ceil_log2_32(max_off);
@@ -333,14 +315,8 @@ TidStoreReset(TidStore *ts)
LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
- /*
- * Free the radix tree and return allocated DSA segments to
- * the operating system.
- */
- shared_rt_free(ts->tree.shared);
- dsa_trim(ts->area);
-
/* Recreate the radix tree */
+ shared_rt_free(ts->tree.shared);
ts->tree.shared = shared_rt_create(CurrentMemoryContext, ts->area,
LWTRANCHE_SHARED_TIDSTORE);
@@ -354,6 +330,7 @@ TidStoreReset(TidStore *ts)
}
else
{
+ /* Recreate the radix tree */
local_rt_free(ts->tree.local);
ts->tree.local = local_rt_create(CurrentMemoryContext);
--
2.31.1
[application/octet-stream] v30-0011-Revert-building-benchmark-module-for-CI.patch (694B, ../../CAD21AoC+kjqkSnLLcbqtu=Td5HSx2H+iAwNHCeZcWqjy1w7k5g@mail.gmail.com/3-v30-0011-Revert-building-benchmark-module-for-CI.patch)
download | inline diff:
From 7c16882823a3d5b65f32c0147ff9f59e77500390 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 14 Feb 2023 19:31:34 +0700
Subject: [PATCH v30 11/11] Revert building benchmark module for CI
---
contrib/meson.build | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/contrib/meson.build b/contrib/meson.build
index 421d469f8c..52253de793 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,7 +12,7 @@ subdir('amcheck')
subdir('auth_delay')
subdir('auto_explain')
subdir('basic_archive')
-subdir('bench_radix_tree')
+#subdir('bench_radix_tree')
subdir('bloom')
subdir('basebackup_to_shell')
subdir('bool_plperl')
--
2.31.1
[application/octet-stream] v30-0007-Radix-tree-optionally-tracks-memory-usage-when-R.patch (8.0K, ../../CAD21AoC+kjqkSnLLcbqtu=Td5HSx2H+iAwNHCeZcWqjy1w7k5g@mail.gmail.com/4-v30-0007-Radix-tree-optionally-tracks-memory-usage-when-R.patch)
download | inline diff:
From d271f527e12d91ea238f1bfef4e88220793fee76 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 8 Mar 2023 15:08:19 +0900
Subject: [PATCH v30 07/11] Radix tree optionally tracks memory usage, when
RT_MEASURE_MEMORY_USAGE.
---
contrib/bench_radix_tree/bench_radix_tree.c | 1 +
src/backend/utils/mmgr/dsa.c | 12 ---
src/include/lib/radixtree.h | 93 +++++++++++++++++--
src/include/utils/dsa.h | 1 -
.../modules/test_radixtree/test_radixtree.c | 1 +
5 files changed, 85 insertions(+), 23 deletions(-)
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
index 63e842395d..fc6e4cb699 100644
--- a/contrib/bench_radix_tree/bench_radix_tree.c
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -34,6 +34,7 @@ PG_MODULE_MAGIC;
#define RT_DECLARE
#define RT_DEFINE
#define RT_USE_DELETE
+#define RT_MEASURE_MEMORY_USAGE
#define RT_VALUE_TYPE uint64
// WIP: compiles with warnings because rt_attach is defined but not used
// #define RT_SHMEM
diff --git a/src/backend/utils/mmgr/dsa.c b/src/backend/utils/mmgr/dsa.c
index 80555aefff..f5a62061a3 100644
--- a/src/backend/utils/mmgr/dsa.c
+++ b/src/backend/utils/mmgr/dsa.c
@@ -1024,18 +1024,6 @@ dsa_set_size_limit(dsa_area *area, size_t limit)
LWLockRelease(DSA_AREA_LOCK(area));
}
-size_t
-dsa_get_total_size(dsa_area *area)
-{
- size_t size;
-
- LWLockAcquire(DSA_AREA_LOCK(area), LW_SHARED);
- size = area->control->total_segment_size;
- LWLockRelease(DSA_AREA_LOCK(area));
-
- return size;
-}
-
/*
* Aggressively free all spare memory in the hope of returning DSM segments to
* the operating system.
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index 2e3963c3d5..6d65544dd0 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -84,7 +84,6 @@
* RT_BEGIN_ITERATE - Begin iterating through all key-value pairs
* RT_ITERATE_NEXT - Return next key-value pair, if any
* RT_END_ITERATE - End iteration
- * RT_MEMORY_USAGE - Get the memory usage
*
* Interface for Shared Memory
* ---------
@@ -97,6 +96,8 @@
* ---------
*
* RT_DELETE - Delete a key-value pair. Declared/define if RT_USE_DELETE is defined
+ * RT_MEMORY_USAGE - Get the memory usage. Declared/define if
+ * RT_MEASURE_MEMORY_USAGE is defined.
*
*
* Copyright (c) 2023, PostgreSQL Global Development Group
@@ -138,7 +139,9 @@
#ifdef RT_USE_DELETE
#define RT_DELETE RT_MAKE_NAME(delete)
#endif
+#ifdef RT_MEASURE_MEMORY_USAGE
#define RT_MEMORY_USAGE RT_MAKE_NAME(memory_usage)
+#endif
#ifdef RT_DEBUG
#define RT_DUMP RT_MAKE_NAME(dump)
#define RT_DUMP_NODE RT_MAKE_NAME(dump_node)
@@ -150,6 +153,9 @@
#define RT_NEW_ROOT RT_MAKE_NAME(new_root)
#define RT_ALLOC_NODE RT_MAKE_NAME(alloc_node)
#define RT_INIT_NODE RT_MAKE_NAME(init_node)
+#ifdef RT_MEASURE_MEMORY_USAGE
+#define RT_FANOUT_GET_NODE_SIZE RT_MAKE_NAME(fanout_get_node_size)
+#endif
#define RT_FREE_NODE RT_MAKE_NAME(free_node)
#define RT_FREE_RECURSE RT_MAKE_NAME(free_recurse)
#define RT_EXTEND_UP RT_MAKE_NAME(extend_up)
@@ -255,7 +261,9 @@ RT_SCOPE RT_ITER * RT_BEGIN_ITERATE(RT_RADIX_TREE *tree);
RT_SCOPE bool RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p);
RT_SCOPE void RT_END_ITERATE(RT_ITER *iter);
+#ifdef RT_MEASURE_MEMORY_USAGE
RT_SCOPE uint64 RT_MEMORY_USAGE(RT_RADIX_TREE *tree);
+#endif
#ifdef RT_DEBUG
RT_SCOPE void RT_DUMP(RT_RADIX_TREE *tree);
@@ -624,6 +632,10 @@ typedef struct RT_RADIX_TREE_CONTROL
uint64 max_val;
uint64 num_keys;
+#ifdef RT_MEASURE_MEMORY_USAGE
+ int64 mem_used;
+#endif
+
/* statistics */
#ifdef RT_DEBUG
int32 cnt[RT_SIZE_CLASS_COUNT];
@@ -1089,6 +1101,11 @@ RT_ALLOC_NODE(RT_RADIX_TREE *tree, RT_SIZE_CLASS size_class, bool is_leaf)
allocsize);
#endif
+#ifdef RT_MEASURE_MEMORY_USAGE
+ /* update memory usage */
+ tree->ctl->mem_used += allocsize;
+#endif
+
#ifdef RT_DEBUG
/* update the statistics */
tree->ctl->cnt[size_class]++;
@@ -1165,6 +1182,54 @@ RT_SWITCH_NODE_KIND(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, RT_PTR_LOCAL no
return newnode;
}
+#ifdef RT_MEASURE_MEMORY_USAGE
+/* Return the node size of the given fanout of the size class */
+static inline Size
+RT_FANOUT_GET_NODE_SIZE(int fanout, bool is_leaf)
+{
+ const Size fanout_inner_node_size[] = {
+ [3] = RT_SIZE_CLASS_INFO[RT_CLASS_3].inner_size,
+ [15] = RT_SIZE_CLASS_INFO[RT_CLASS_32_MIN].inner_size,
+ [32] = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX].inner_size,
+ [125] = RT_SIZE_CLASS_INFO[RT_CLASS_125].inner_size,
+ [256] = RT_SIZE_CLASS_INFO[RT_CLASS_256].inner_size,
+ };
+ const Size fanout_leaf_node_size[] = {
+ [3] = RT_SIZE_CLASS_INFO[RT_CLASS_3].leaf_size,
+ [15] = RT_SIZE_CLASS_INFO[RT_CLASS_32_MIN].leaf_size,
+ [32] = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX].leaf_size,
+ [125] = RT_SIZE_CLASS_INFO[RT_CLASS_125].leaf_size,
+ [256] = RT_SIZE_CLASS_INFO[RT_CLASS_256].leaf_size,
+ };
+ Size node_size;
+
+ node_size = is_leaf ?
+ fanout_leaf_node_size[fanout] : fanout_inner_node_size[fanout];
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ Size assert_node_size = 0;
+
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ RT_SIZE_CLASS_ELEM size_class = RT_SIZE_CLASS_INFO[i];
+
+ if (size_class.fanout == fanout)
+ {
+ assert_node_size = is_leaf ?
+ size_class.leaf_size : size_class.inner_size;
+ break;
+ }
+ }
+
+ Assert(node_size == assert_node_size);
+ }
+#endif
+
+ return node_size;
+}
+#endif
+
/* Free the given node */
static void
RT_FREE_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode)
@@ -1197,11 +1262,22 @@ RT_FREE_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode)
}
#endif
+#ifdef RT_MEASURE_MEMORY_USAGE
+ /* update memory usage */
+ {
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+ tree->ctl->mem_used -= RT_FANOUT_GET_NODE_SIZE(node->fanout,
+ RT_NODE_IS_LEAF(node));
+ Assert(tree->ctl->mem_used >= 0);
+ }
+#endif
+
#ifdef RT_SHMEM
dsa_free(tree->dsa, allocnode);
#else
pfree(allocnode);
#endif
+
}
/* Update the parent's pointer when growing a node */
@@ -1989,27 +2065,23 @@ RT_END_ITERATE(RT_ITER *iter)
/*
* Return the statistics of the amount of memory used by the radix tree.
*/
+#ifdef RT_MEASURE_MEMORY_USAGE
RT_SCOPE uint64
RT_MEMORY_USAGE(RT_RADIX_TREE *tree)
{
Size total = 0;
- RT_LOCK_SHARED(tree);
-
#ifdef RT_SHMEM
Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
- total = dsa_get_total_size(tree->dsa);
-#else
- for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
- {
- total += MemoryContextMemAllocated(tree->inner_slabs[i], true);
- total += MemoryContextMemAllocated(tree->leaf_slabs[i], true);
- }
#endif
+ RT_LOCK_SHARED(tree);
+ total = tree->ctl->mem_used;
RT_UNLOCK(tree);
+
return total;
}
+#endif
/*
* Verify the radix tree node.
@@ -2476,6 +2548,7 @@ RT_DUMP(RT_RADIX_TREE *tree)
#undef RT_NEW_ROOT
#undef RT_ALLOC_NODE
#undef RT_INIT_NODE
+#undef RT_FANOUT_GET_NODE_SIZE
#undef RT_FREE_NODE
#undef RT_FREE_RECURSE
#undef RT_EXTEND_UP
diff --git a/src/include/utils/dsa.h b/src/include/utils/dsa.h
index 2af215484f..3ce4ee300a 100644
--- a/src/include/utils/dsa.h
+++ b/src/include/utils/dsa.h
@@ -121,7 +121,6 @@ extern dsa_handle dsa_get_handle(dsa_area *area);
extern dsa_pointer dsa_allocate_extended(dsa_area *area, size_t size, int flags);
extern void dsa_free(dsa_area *area, dsa_pointer dp);
extern void *dsa_get_address(dsa_area *area, dsa_pointer dp);
-extern size_t dsa_get_total_size(dsa_area *area);
extern void dsa_trim(dsa_area *area);
extern void dsa_dump(dsa_area *area);
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
index 5a169854d9..19d286d84b 100644
--- a/src/test/modules/test_radixtree/test_radixtree.c
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -114,6 +114,7 @@ static const test_spec test_specs[] = {
#define RT_DECLARE
#define RT_DEFINE
#define RT_USE_DELETE
+#define RT_MEASURE_MEMORY_USAGE
#define RT_VALUE_TYPE TestValueType
/* #define RT_SHMEM */
#include "lib/radixtree.h"
--
2.31.1
[application/octet-stream] v30-0009-Revert-the-update-for-the-minimum-value-of-maint.patch (1.4K, ../../CAD21AoC+kjqkSnLLcbqtu=Td5HSx2H+iAwNHCeZcWqjy1w7k5g@mail.gmail.com/5-v30-0009-Revert-the-update-for-the-minimum-value-of-maint.patch)
download | inline diff:
From f7013c9023ff3f9a6707276303443f0b4e00ccbf Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 8 Mar 2023 15:09:22 +0900
Subject: [PATCH v30 09/11] Revert the update for the minimum value of
maintenance_work_mem.
---
src/backend/postmaster/autovacuum.c | 6 +++---
src/backend/utils/misc/guc_tables.c | 2 +-
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index a371f6fbba..ff6149a179 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -3397,12 +3397,12 @@ check_autovacuum_work_mem(int *newval, void **extra, GucSource source)
return true;
/*
- * We clamp manually-set values to at least 2MB. Since
+ * We clamp manually-set values to at least 1MB. Since
* maintenance_work_mem is always set to at least this value, do the same
* here.
*/
- if (*newval < 2048)
- *newval = 2048;
+ if (*newval < 1024)
+ *newval = 1024;
return true;
}
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 8a64614cd1..1c0583fe26 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2313,7 +2313,7 @@ struct config_int ConfigureNamesInt[] =
GUC_UNIT_KB
},
&maintenance_work_mem,
- 65536, 2048, MAX_KILOBYTES,
+ 65536, 1024, MAX_KILOBYTES,
NULL, NULL, NULL
},
--
2.31.1
[application/octet-stream] v30-0010-Add-min-and-max-classes-for-node3-and-node125.patch (11.9K, ../../CAD21AoC+kjqkSnLLcbqtu=Td5HSx2H+iAwNHCeZcWqjy1w7k5g@mail.gmail.com/6-v30-0010-Add-min-and-max-classes-for-node3-and-node125.patch)
download | inline diff:
From ba41d3bfcf0d3016c61948ce6acc0d9582d8aad8 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Thu, 9 Mar 2023 11:42:17 +0900
Subject: [PATCH v30 10/11] Add min and max classes for node3 and node125.
---
src/include/lib/radixtree.h | 70 +++++++++++++------
src/include/lib/radixtree_insert_impl.h | 56 ++++++++++++++-
.../expected/test_radixtree.out | 4 ++
.../modules/test_radixtree/test_radixtree.c | 6 +-
4 files changed, 110 insertions(+), 26 deletions(-)
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index 6d65544dd0..b655f4a2a2 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -225,10 +225,12 @@
#define RT_SIZE_CLASS RT_MAKE_NAME(size_class)
#define RT_SIZE_CLASS_ELEM RT_MAKE_NAME(size_class_elem)
#define RT_SIZE_CLASS_INFO RT_MAKE_NAME(size_class_info)
-#define RT_CLASS_3 RT_MAKE_NAME(class_3)
+#define RT_CLASS_3_MIN RT_MAKE_NAME(class_3_min)
+#define RT_CLASS_3_MAX RT_MAKE_NAME(class_3_max)
#define RT_CLASS_32_MIN RT_MAKE_NAME(class_32_min)
#define RT_CLASS_32_MAX RT_MAKE_NAME(class_32_max)
-#define RT_CLASS_125 RT_MAKE_NAME(class_125)
+#define RT_CLASS_125_MIN RT_MAKE_NAME(class_125_min)
+#define RT_CLASS_125_MAX RT_MAKE_NAME(class_125_max)
#define RT_CLASS_256 RT_MAKE_NAME(class_256)
/* generate forward declarations necessary to use the radix tree */
@@ -561,10 +563,12 @@ typedef struct RT_NODE_LEAF_256
*/
typedef enum RT_SIZE_CLASS
{
- RT_CLASS_3 = 0,
+ RT_CLASS_3_MIN = 0,
+ RT_CLASS_3_MAX,
RT_CLASS_32_MIN,
RT_CLASS_32_MAX,
- RT_CLASS_125,
+ RT_CLASS_125_MIN,
+ RT_CLASS_125_MAX,
RT_CLASS_256
} RT_SIZE_CLASS;
@@ -580,7 +584,13 @@ typedef struct RT_SIZE_CLASS_ELEM
} RT_SIZE_CLASS_ELEM;
static const RT_SIZE_CLASS_ELEM RT_SIZE_CLASS_INFO[] = {
- [RT_CLASS_3] = {
+ [RT_CLASS_3_MIN] = {
+ .name = "radix tree node 1",
+ .fanout = 1,
+ .inner_size = sizeof(RT_NODE_INNER_3) + 1 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_3) + 1 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_3_MAX] = {
.name = "radix tree node 3",
.fanout = 3,
.inner_size = sizeof(RT_NODE_INNER_3) + 3 * sizeof(RT_PTR_ALLOC),
@@ -598,7 +608,13 @@ static const RT_SIZE_CLASS_ELEM RT_SIZE_CLASS_INFO[] = {
.inner_size = sizeof(RT_NODE_INNER_32) + 32 * sizeof(RT_PTR_ALLOC),
.leaf_size = sizeof(RT_NODE_LEAF_32) + 32 * sizeof(RT_VALUE_TYPE),
},
- [RT_CLASS_125] = {
+ [RT_CLASS_125_MIN] = {
+ .name = "radix tree node 125",
+ .fanout = 61,
+ .inner_size = sizeof(RT_NODE_INNER_125) + 61 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_125) + 61 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_125_MAX] = {
.name = "radix tree node 125",
.fanout = 125,
.inner_size = sizeof(RT_NODE_INNER_125) + 125 * sizeof(RT_PTR_ALLOC),
@@ -934,7 +950,7 @@ static inline void
RT_CHUNK_CHILDREN_ARRAY_COPY(uint8 *src_chunks, RT_PTR_ALLOC *src_children,
uint8 *dst_chunks, RT_PTR_ALLOC *dst_children)
{
- const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3_MAX].fanout;
const Size chunk_size = sizeof(uint8) * fanout;
const Size children_size = sizeof(RT_PTR_ALLOC) * fanout;
@@ -946,7 +962,7 @@ static inline void
RT_CHUNK_VALUES_ARRAY_COPY(uint8 *src_chunks, RT_VALUE_TYPE *src_values,
uint8 *dst_chunks, RT_VALUE_TYPE *dst_values)
{
- const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3_MAX].fanout;
const Size chunk_size = sizeof(uint8) * fanout;
const Size values_size = sizeof(RT_VALUE_TYPE) * fanout;
@@ -1152,9 +1168,9 @@ RT_NEW_ROOT(RT_RADIX_TREE *tree, uint64 key)
RT_PTR_ALLOC allocnode;
RT_PTR_LOCAL newnode;
- allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3_MIN, is_leaf);
newnode = RT_PTR_GET_LOCAL(tree, allocnode);
- RT_INIT_NODE(newnode, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ RT_INIT_NODE(newnode, RT_NODE_KIND_3, RT_CLASS_3_MIN, is_leaf);
newnode->shift = shift;
tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(shift);
tree->ctl->root = allocnode;
@@ -1188,17 +1204,21 @@ static inline Size
RT_FANOUT_GET_NODE_SIZE(int fanout, bool is_leaf)
{
const Size fanout_inner_node_size[] = {
- [3] = RT_SIZE_CLASS_INFO[RT_CLASS_3].inner_size,
+ [1] = RT_SIZE_CLASS_INFO[RT_CLASS_3_MIN].inner_size,
+ [3] = RT_SIZE_CLASS_INFO[RT_CLASS_3_MAX].inner_size,
[15] = RT_SIZE_CLASS_INFO[RT_CLASS_32_MIN].inner_size,
[32] = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX].inner_size,
- [125] = RT_SIZE_CLASS_INFO[RT_CLASS_125].inner_size,
+ [61] = RT_SIZE_CLASS_INFO[RT_CLASS_125_MIN].inner_size,
+ [125] = RT_SIZE_CLASS_INFO[RT_CLASS_125_MAX].inner_size,
[256] = RT_SIZE_CLASS_INFO[RT_CLASS_256].inner_size,
};
const Size fanout_leaf_node_size[] = {
- [3] = RT_SIZE_CLASS_INFO[RT_CLASS_3].leaf_size,
+ [1] = RT_SIZE_CLASS_INFO[RT_CLASS_3_MIN].leaf_size,
+ [3] = RT_SIZE_CLASS_INFO[RT_CLASS_3_MAX].leaf_size,
[15] = RT_SIZE_CLASS_INFO[RT_CLASS_32_MIN].leaf_size,
[32] = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX].leaf_size,
- [125] = RT_SIZE_CLASS_INFO[RT_CLASS_125].leaf_size,
+ [61] = RT_SIZE_CLASS_INFO[RT_CLASS_125_MIN].leaf_size,
+ [125] = RT_SIZE_CLASS_INFO[RT_CLASS_125_MAX].leaf_size,
[256] = RT_SIZE_CLASS_INFO[RT_CLASS_256].leaf_size,
};
Size node_size;
@@ -1337,9 +1357,9 @@ RT_EXTEND_UP(RT_RADIX_TREE *tree, uint64 key)
RT_PTR_LOCAL node;
RT_NODE_INNER_3 *n3;
- allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, true);
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3_MIN, true);
node = RT_PTR_GET_LOCAL(tree, allocnode);
- RT_INIT_NODE(node, RT_NODE_KIND_3, RT_CLASS_3, true);
+ RT_INIT_NODE(node, RT_NODE_KIND_3, RT_CLASS_3_MIN, true);
node->shift = shift;
node->count = 1;
@@ -1375,9 +1395,9 @@ RT_EXTEND_DOWN(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_L
int newshift = shift - RT_NODE_SPAN;
bool is_leaf = newshift == 0;
- allocchild = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ allocchild = RT_ALLOC_NODE(tree, RT_CLASS_3_MIN, is_leaf);
newchild = RT_PTR_GET_LOCAL(tree, allocchild);
- RT_INIT_NODE(newchild, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ RT_INIT_NODE(newchild, RT_NODE_KIND_3, RT_CLASS_3_MIN, is_leaf);
newchild->shift = newshift;
RT_NODE_INSERT_INNER(tree, parent, stored_node, node, key, allocchild);
@@ -2177,12 +2197,14 @@ RT_STATS(RT_RADIX_TREE *tree)
{
RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
- fprintf(stderr, "height = %d, n3 = %u, n15 = %u, n32 = %u, n125 = %u, n256 = %u\n",
+ fprintf(stderr, "height = %d, n1 = %u, n3 = %u, n15 = %u, n32 = %u, n61 = %u, n125 = %u, n256 = %u\n",
root->shift / RT_NODE_SPAN,
- tree->ctl->cnt[RT_CLASS_3],
+ tree->ctl->cnt[RT_CLASS_3_MIN],
+ tree->ctl->cnt[RT_CLASS_3_MAX],
tree->ctl->cnt[RT_CLASS_32_MIN],
tree->ctl->cnt[RT_CLASS_32_MAX],
- tree->ctl->cnt[RT_CLASS_125],
+ tree->ctl->cnt[RT_CLASS_125_MIN],
+ tree->ctl->cnt[RT_CLASS_125_MAX],
tree->ctl->cnt[RT_CLASS_256]);
}
@@ -2519,10 +2541,12 @@ RT_DUMP(RT_RADIX_TREE *tree)
#undef RT_SIZE_CLASS
#undef RT_SIZE_CLASS_ELEM
#undef RT_SIZE_CLASS_INFO
-#undef RT_CLASS_3
+#undef RT_CLASS_3_MIN
+#undef RT_CLASS_3_MAX
#undef RT_CLASS_32_MIN
#undef RT_CLASS_32_MAX
-#undef RT_CLASS_125
+#undef RT_CLASS_125_MIN
+#undef RT_CLASS_125_MAX
#undef RT_CLASS_256
/* function declarations */
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
index d56e58dcac..d10093dfba 100644
--- a/src/include/lib/radixtree_insert_impl.h
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -42,6 +42,7 @@
{
case RT_NODE_KIND_3:
{
+ const RT_SIZE_CLASS_ELEM class3_max = RT_SIZE_CLASS_INFO[RT_CLASS_3_MAX];
RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
#ifdef RT_NODE_LEVEL_LEAF
@@ -55,6 +56,32 @@
break;
}
#endif
+ if (unlikely(RT_NODE_MUST_GROW(n3)) &&
+ n3->base.n.fanout < class3_max.fanout)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ const RT_SIZE_CLASS_ELEM class3_min = RT_SIZE_CLASS_INFO[RT_CLASS_3_MIN];
+ const RT_SIZE_CLASS new_class = RT_CLASS_3_MAX;
+
+ Assert(n3->base.n.fanout == class3_min.fanout);
+
+ /* grow to the next size class of this kind */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ n3 = (RT_NODE3_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ memcpy(newnode, node, class3_min.leaf_size);
+#else
+ memcpy(newnode, node, class3_min.inner_size);
+#endif
+ newnode->fanout = class3_max.fanout;
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+
if (unlikely(RT_NODE_MUST_GROW(n3)))
{
RT_PTR_ALLOC allocnode;
@@ -154,7 +181,7 @@
RT_PTR_LOCAL newnode;
RT_NODE125_TYPE *new125;
const uint8 new_kind = RT_NODE_KIND_125;
- const RT_SIZE_CLASS new_class = RT_CLASS_125;
+ const RT_SIZE_CLASS new_class = RT_CLASS_125_MIN;
Assert(n32->base.n.fanout == class32_max.fanout);
@@ -213,6 +240,7 @@
/* FALLTHROUGH */
case RT_NODE_KIND_125:
{
+ const RT_SIZE_CLASS_ELEM class125_max = RT_SIZE_CLASS_INFO[RT_CLASS_125_MAX];
RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
int slotpos;
int cnt = 0;
@@ -227,6 +255,32 @@
break;
}
#endif
+ if (unlikely(RT_NODE_MUST_GROW(n125)) &&
+ n125->base.n.fanout < class125_max.fanout)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ const RT_SIZE_CLASS_ELEM class125_min = RT_SIZE_CLASS_INFO[RT_CLASS_125_MIN];
+ const RT_SIZE_CLASS new_class = RT_CLASS_125_MAX;
+
+ Assert(n125->base.n.fanout == class125_min.fanout);
+
+ /* grow to the next size class of this kind */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ n125 = (RT_NODE125_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ memcpy(newnode, node, class125_min.leaf_size);
+#else
+ memcpy(newnode, node, class125_min.inner_size);
+#endif
+ newnode->fanout = class125_max.fanout;
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+
if (unlikely(RT_NODE_MUST_GROW(n125)))
{
RT_PTR_ALLOC allocnode;
diff --git a/src/test/modules/test_radixtree/expected/test_radixtree.out b/src/test/modules/test_radixtree/expected/test_radixtree.out
index 7ad1ce3605..f2b1d7e4f8 100644
--- a/src/test/modules/test_radixtree/expected/test_radixtree.out
+++ b/src/test/modules/test_radixtree/expected/test_radixtree.out
@@ -4,12 +4,16 @@ CREATE EXTENSION test_radixtree;
-- an error if something fails.
--
SELECT test_radixtree();
+NOTICE: testing basic operations with leaf node 1
+NOTICE: testing basic operations with inner node 1
NOTICE: testing basic operations with leaf node 3
NOTICE: testing basic operations with inner node 3
NOTICE: testing basic operations with leaf node 15
NOTICE: testing basic operations with inner node 15
NOTICE: testing basic operations with leaf node 32
NOTICE: testing basic operations with inner node 32
+NOTICE: testing basic operations with leaf node 61
+NOTICE: testing basic operations with inner node 61
NOTICE: testing basic operations with leaf node 125
NOTICE: testing basic operations with inner node 125
NOTICE: testing basic operations with leaf node 256
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
index 19d286d84b..4f38b6e3de 100644
--- a/src/test/modules/test_radixtree/test_radixtree.c
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -47,10 +47,12 @@ static const bool rt_test_stats = false;
* XXX: should we expose and use RT_SIZE_CLASS and RT_SIZE_CLASS_INFO?
*/
static int rt_node_class_fanouts[] = {
- 3, /* RT_CLASS_3 */
+ 1, /* RT_CLASS_3_MIN */
+ 3, /* RT_CLASS_3_MAX */
15, /* RT_CLASS_32_MIN */
32, /* RT_CLASS_32_MAX */
- 125, /* RT_CLASS_125 */
+ 61, /* RT_CLASS_125_MIN */
+ 125, /* RT_CLASS_125_MAX */
256 /* RT_CLASS_256 */
};
/*
--
2.31.1
[application/octet-stream] v30-0006-Use-TIDStore-for-storing-dead-tuple-TID-during-l.patch (47.7K, ../../CAD21AoC+kjqkSnLLcbqtu=Td5HSx2H+iAwNHCeZcWqjy1w7k5g@mail.gmail.com/7-v30-0006-Use-TIDStore-for-storing-dead-tuple-TID-during-l.patch)
download | inline diff:
From b4e4ea5f22ee8898fa7ef58a21d0da1d4d661a0a Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Tue, 7 Feb 2023 17:19:29 +0700
Subject: [PATCH v30 06/11] Use TIDStore for storing dead tuple TID during lazy
vacuum
Previously, we used an array of ItemPointerData to store dead tuple
TIDs, which was not space efficient and slow to lookup. Also, we had
the 1GB limit on its size.
Now we use TIDStore to store dead tuple TIDs. Since the TIDStore,
backed by the radix tree, incrementaly allocates the memory, we get
rid of the 1GB limit.
Since we are no longer able to exactly estimate the maximum number of
TIDs can be stored the pg_stat_progress_vacuum shows the progress
information based on the amount of memory in bytes. The column names
are also changed to max_dead_tuple_bytes and dead_tuple_bytes.
In addition, since the TIDStore use the radix tree internally, the
minimum amount of memory required by TIDStore is 1MB, the inital DSA
segment size. Due to that, we increase the minimum value of
maintenance_work_mem (also autovacuum_work_mem) from 1MB to 2MB.
XXX: needs to bump catalog version
---
doc/src/sgml/monitoring.sgml | 8 +-
src/backend/access/heap/vacuumlazy.c | 279 ++++++++-------------
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/vacuum.c | 78 +-----
src/backend/commands/vacuumparallel.c | 66 +++--
src/backend/postmaster/autovacuum.c | 6 +-
src/backend/storage/lmgr/lwlock.c | 2 +
src/backend/utils/misc/guc_tables.c | 2 +-
src/include/commands/progress.h | 4 +-
src/include/commands/vacuum.h | 25 +-
src/include/storage/lwlock.h | 1 +
src/test/regress/expected/cluster.out | 2 +-
src/test/regress/expected/create_index.out | 2 +-
src/test/regress/expected/rules.out | 4 +-
src/test/regress/sql/cluster.sql | 2 +-
src/test/regress/sql/create_index.sql | 2 +-
16 files changed, 174 insertions(+), 311 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 97d588b1d8..61e163636a 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -7170,10 +7170,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>max_dead_tuples</structfield> <type>bigint</type>
+ <structfield>max_dead_tuple_bytes</structfield> <type>bigint</type>
</para>
<para>
- Number of dead tuples that we can store before needing to perform
+ Amount of dead tuple data that we can store before needing to perform
an index vacuum cycle, based on
<xref linkend="guc-maintenance-work-mem"/>.
</para></entry>
@@ -7181,10 +7181,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>num_dead_tuples</structfield> <type>bigint</type>
+ <structfield>dead_tuple_bytes</structfield> <type>bigint</type>
</para>
<para>
- Number of dead tuples collected since the last index vacuum cycle.
+ Amount of dead tuple data collected since the last index vacuum cycle.
</para></entry>
</row>
</tbody>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 8f14cf85f3..edb9079124 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3,18 +3,17 @@
* vacuumlazy.c
* Concurrent ("lazy") vacuuming.
*
- * The major space usage for vacuuming is storage for the array of dead TIDs
+ * The major space usage for vacuuming is TidStore, a storage for dead TIDs
* that are to be removed from indexes. We want to ensure we can vacuum even
* the very largest relations with finite memory space usage. To do that, we
- * set upper bounds on the number of TIDs we can keep track of at once.
+ * set upper bounds on the maximum memory that can be used for keeping track
+ * of dead TIDs at once.
*
* We are willing to use at most maintenance_work_mem (or perhaps
- * autovacuum_work_mem) memory space to keep track of dead TIDs. We initially
- * allocate an array of TIDs of that size, with an upper limit that depends on
- * table size (this limit ensures we don't allocate a huge area uselessly for
- * vacuuming small tables). If the array threatens to overflow, we must call
- * lazy_vacuum to vacuum indexes (and to vacuum the pages that we've pruned).
- * This frees up the memory space dedicated to storing dead TIDs.
+ * autovacuum_work_mem) memory space to keep track of dead TIDs. If the
+ * TidStore is full, we must call lazy_vacuum to vacuum indexes (and to vacuum
+ * the pages that we've pruned). This frees up the memory space dedicated to
+ * to store dead TIDs.
*
* In practice VACUUM will often complete its initial pass over the target
* heap relation without ever running out of space to store TIDs. This means
@@ -40,6 +39,7 @@
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
#include "access/multixact.h"
+#include "access/tidstore.h"
#include "access/transam.h"
#include "access/visibilitymap.h"
#include "access/xact.h"
@@ -188,7 +188,7 @@ typedef struct LVRelState
* lazy_vacuum_heap_rel, which marks the same LP_DEAD line pointers as
* LP_UNUSED during second heap pass.
*/
- VacDeadItems *dead_items; /* TIDs whose index tuples we'll delete */
+ TidStore *dead_items; /* TIDs whose index tuples we'll delete */
BlockNumber rel_pages; /* total number of pages */
BlockNumber scanned_pages; /* # pages examined (not skipped via VM) */
BlockNumber removed_pages; /* # pages removed by relation truncation */
@@ -220,11 +220,14 @@ typedef struct LVRelState
typedef struct LVPagePruneState
{
bool hastup; /* Page prevents rel truncation? */
- bool has_lpdead_items; /* includes existing LP_DEAD items */
+
+ /* collected offsets of LP_DEAD items including existing ones */
+ OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
+ int num_offsets;
/*
* State describes the proper VM bit states to set for the page following
- * pruning and freezing. all_visible implies !has_lpdead_items, but don't
+ * pruning and freezing. all_visible implies num_offsets == 0, but don't
* trust all_frozen result unless all_visible is also set to true.
*/
bool all_visible; /* Every item visible to all? */
@@ -259,8 +262,9 @@ static bool lazy_scan_noprune(LVRelState *vacrel, Buffer buf,
static void lazy_vacuum(LVRelState *vacrel);
static bool lazy_vacuum_all_indexes(LVRelState *vacrel);
static void lazy_vacuum_heap_rel(LVRelState *vacrel);
-static int lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
- Buffer buffer, int index, Buffer vmbuffer);
+static void lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
+ OffsetNumber *offsets, int num_offsets,
+ Buffer buffer, Buffer vmbuffer);
static bool lazy_check_wraparound_failsafe(LVRelState *vacrel);
static void lazy_cleanup_all_indexes(LVRelState *vacrel);
static IndexBulkDeleteResult *lazy_vacuum_one_index(Relation indrel,
@@ -487,11 +491,11 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
}
/*
- * Allocate dead_items array memory using dead_items_alloc. This handles
- * parallel VACUUM initialization as part of allocating shared memory
- * space used for dead_items. (But do a failsafe precheck first, to
- * ensure that parallel VACUUM won't be attempted at all when relfrozenxid
- * is already dangerously old.)
+ * Allocate dead_items memory using dead_items_alloc. This handles parallel
+ * VACUUM initialization as part of allocating shared memory space used for
+ * dead_items. (But do a failsafe precheck first, to ensure that parallel
+ * VACUUM won't be attempted at all when relfrozenxid is already dangerously
+ * old.)
*/
lazy_check_wraparound_failsafe(vacrel);
dead_items_alloc(vacrel, params->nworkers);
@@ -797,7 +801,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* have collected the TIDs whose index tuples need to be removed.
*
* Finally, invokes lazy_vacuum_heap_rel to vacuum heap pages, which
- * largely consists of marking LP_DEAD items (from collected TID array)
+ * largely consists of marking LP_DEAD items (from vacrel->dead_items)
* as LP_UNUSED. This has to happen in a second, final pass over the
* heap, to preserve a basic invariant that all index AMs rely on: no
* extant index tuple can ever be allowed to contain a TID that points to
@@ -825,21 +829,21 @@ lazy_scan_heap(LVRelState *vacrel)
blkno,
next_unskippable_block,
next_fsm_block_to_vacuum = 0;
- VacDeadItems *dead_items = vacrel->dead_items;
+ TidStore *dead_items = vacrel->dead_items;
Buffer vmbuffer = InvalidBuffer;
bool next_unskippable_allvis,
skipping_current_range;
const int initprog_index[] = {
PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_TOTAL_HEAP_BLKS,
- PROGRESS_VACUUM_MAX_DEAD_TUPLES
+ PROGRESS_VACUUM_MAX_DEAD_TUPLE_BYTES
};
int64 initprog_val[3];
/* Report that we're scanning the heap, advertising total # of blocks */
initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
initprog_val[1] = rel_pages;
- initprog_val[2] = dead_items->max_items;
+ initprog_val[2] = TidStoreMaxMemory(vacrel->dead_items);
pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
/* Set up an initial range of skippable blocks using the visibility map */
@@ -906,8 +910,7 @@ lazy_scan_heap(LVRelState *vacrel)
* dead_items TIDs, pause and do a cycle of vacuuming before we tackle
* this page.
*/
- Assert(dead_items->max_items >= MaxHeapTuplesPerPage);
- if (dead_items->max_items - dead_items->num_items < MaxHeapTuplesPerPage)
+ if (TidStoreIsFull(vacrel->dead_items))
{
/*
* Before beginning index vacuuming, we release any pin we may
@@ -969,7 +972,7 @@ lazy_scan_heap(LVRelState *vacrel)
continue;
}
- /* Collect LP_DEAD items in dead_items array, count tuples */
+ /* Collect LP_DEAD items in dead_items, count tuples */
if (lazy_scan_noprune(vacrel, buf, blkno, page, &hastup,
&recordfreespace))
{
@@ -1011,14 +1014,14 @@ lazy_scan_heap(LVRelState *vacrel)
* Prune, freeze, and count tuples.
*
* Accumulates details of remaining LP_DEAD line pointers on page in
- * dead_items array. This includes LP_DEAD line pointers that we
- * pruned ourselves, as well as existing LP_DEAD line pointers that
- * were pruned some time earlier. Also considers freezing XIDs in the
- * tuple headers of remaining items with storage.
+ * dead_items. This includes LP_DEAD line pointers that we pruned
+ * ourselves, as well as existing LP_DEAD line pointers that were pruned
+ * some time earlier. Also considers freezing XIDs in the tuple headers
+ * of remaining items with storage.
*/
lazy_scan_prune(vacrel, buf, blkno, page, &prunestate);
- Assert(!prunestate.all_visible || !prunestate.has_lpdead_items);
+ Assert(!prunestate.all_visible || (prunestate.num_offsets == 0));
/* Remember the location of the last page with nonremovable tuples */
if (prunestate.hastup)
@@ -1034,14 +1037,12 @@ lazy_scan_heap(LVRelState *vacrel)
* performed here can be thought of as the one-pass equivalent of
* a call to lazy_vacuum().
*/
- if (prunestate.has_lpdead_items)
+ if (prunestate.num_offsets > 0)
{
Size freespace;
- lazy_vacuum_heap_page(vacrel, blkno, buf, 0, vmbuffer);
-
- /* Forget the LP_DEAD items that we just vacuumed */
- dead_items->num_items = 0;
+ lazy_vacuum_heap_page(vacrel, blkno, prunestate.deadoffsets,
+ prunestate.num_offsets, buf, vmbuffer);
/*
* Periodically perform FSM vacuuming to make newly-freed
@@ -1078,7 +1079,16 @@ lazy_scan_heap(LVRelState *vacrel)
* with prunestate-driven visibility map and FSM steps (just like
* the two-pass strategy).
*/
- Assert(dead_items->num_items == 0);
+ Assert(TidStoreNumTids(dead_items) == 0);
+ }
+ else if (prunestate.num_offsets > 0)
+ {
+ /* Save details of the LP_DEAD items from the page in dead_items */
+ TidStoreSetBlockOffsets(dead_items, blkno, prunestate.deadoffsets,
+ prunestate.num_offsets);
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
+ TidStoreMemoryUsage(dead_items));
}
/*
@@ -1145,7 +1155,7 @@ lazy_scan_heap(LVRelState *vacrel)
* There should never be LP_DEAD items on a page with PD_ALL_VISIBLE
* set, however.
*/
- else if (prunestate.has_lpdead_items && PageIsAllVisible(page))
+ else if ((prunestate.num_offsets > 0) && PageIsAllVisible(page))
{
elog(WARNING, "page containing LP_DEAD items is marked as all-visible in relation \"%s\" page %u",
vacrel->relname, blkno);
@@ -1193,7 +1203,7 @@ lazy_scan_heap(LVRelState *vacrel)
* Final steps for block: drop cleanup lock, record free space in the
* FSM
*/
- if (prunestate.has_lpdead_items && vacrel->do_index_vacuuming)
+ if ((prunestate.num_offsets > 0) && vacrel->do_index_vacuuming)
{
/*
* Wait until lazy_vacuum_heap_rel() to save free space. This
@@ -1249,7 +1259,7 @@ lazy_scan_heap(LVRelState *vacrel)
* Do index vacuuming (call each index's ambulkdelete routine), then do
* related heap vacuuming
*/
- if (dead_items->num_items > 0)
+ if (TidStoreNumTids(dead_items) > 0)
lazy_vacuum(vacrel);
/*
@@ -1524,9 +1534,9 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* The approach we take now is to restart pruning when the race condition is
* detected. This allows heap_page_prune() to prune the tuples inserted by
* the now-aborted transaction. This is a little crude, but it guarantees
- * that any items that make it into the dead_items array are simple LP_DEAD
- * line pointers, and that every remaining item with tuple storage is
- * considered as a candidate for freezing.
+ * that any items that make it into the dead_items are simple LP_DEAD line
+ * pointers, and that every remaining item with tuple storage is considered
+ * as a candidate for freezing.
*/
static void
lazy_scan_prune(LVRelState *vacrel,
@@ -1543,13 +1553,11 @@ lazy_scan_prune(LVRelState *vacrel,
HTSV_Result res;
int tuples_deleted,
tuples_frozen,
- lpdead_items,
live_tuples,
recently_dead_tuples;
int nnewlpdead;
HeapPageFreeze pagefrz;
int64 fpi_before = pgWalUsage.wal_fpi;
- OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
@@ -1571,7 +1579,6 @@ retry:
pagefrz.NoFreezePageRelminMxid = vacrel->NewRelminMxid;
tuples_deleted = 0;
tuples_frozen = 0;
- lpdead_items = 0;
live_tuples = 0;
recently_dead_tuples = 0;
@@ -1580,9 +1587,9 @@ retry:
*
* We count tuples removed by the pruning step as tuples_deleted. Its
* final value can be thought of as the number of tuples that have been
- * deleted from the table. It should not be confused with lpdead_items;
- * lpdead_items's final value can be thought of as the number of tuples
- * that were deleted from indexes.
+ * deleted from the table. It should not be confused with
+ * prunestate->deadoffsets; prunestate->deadoffsets's final value can
+ * be thought of as the number of tuples that were deleted from indexes.
*/
tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest,
InvalidTransactionId, 0, &nnewlpdead,
@@ -1593,7 +1600,7 @@ retry:
* requiring freezing among remaining tuples with storage
*/
prunestate->hastup = false;
- prunestate->has_lpdead_items = false;
+ prunestate->num_offsets = 0;
prunestate->all_visible = true;
prunestate->all_frozen = true;
prunestate->visibility_cutoff_xid = InvalidTransactionId;
@@ -1638,7 +1645,7 @@ retry:
* (This is another case where it's useful to anticipate that any
* LP_DEAD items will become LP_UNUSED during the ongoing VACUUM.)
*/
- deadoffsets[lpdead_items++] = offnum;
+ prunestate->deadoffsets[prunestate->num_offsets++] = offnum;
continue;
}
@@ -1875,7 +1882,7 @@ retry:
*/
#ifdef USE_ASSERT_CHECKING
/* Note that all_frozen value does not matter when !all_visible */
- if (prunestate->all_visible && lpdead_items == 0)
+ if (prunestate->all_visible && prunestate->num_offsets == 0)
{
TransactionId cutoff;
bool all_frozen;
@@ -1888,28 +1895,9 @@ retry:
}
#endif
- /*
- * Now save details of the LP_DEAD items from the page in vacrel
- */
- if (lpdead_items > 0)
+ if (prunestate->num_offsets > 0)
{
- VacDeadItems *dead_items = vacrel->dead_items;
- ItemPointerData tmp;
-
vacrel->lpdead_item_pages++;
- prunestate->has_lpdead_items = true;
-
- ItemPointerSetBlockNumber(&tmp, blkno);
-
- for (int i = 0; i < lpdead_items; i++)
- {
- ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
- dead_items->items[dead_items->num_items++] = tmp;
- }
-
- Assert(dead_items->num_items <= dead_items->max_items);
- pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
- dead_items->num_items);
/*
* It was convenient to ignore LP_DEAD items in all_visible earlier on
@@ -1928,7 +1916,7 @@ retry:
/* Finally, add page-local counts to whole-VACUUM counts */
vacrel->tuples_deleted += tuples_deleted;
vacrel->tuples_frozen += tuples_frozen;
- vacrel->lpdead_items += lpdead_items;
+ vacrel->lpdead_items += prunestate->num_offsets;
vacrel->live_tuples += live_tuples;
vacrel->recently_dead_tuples += recently_dead_tuples;
}
@@ -1940,7 +1928,7 @@ retry:
* lazy_scan_prune, which requires a full cleanup lock. While pruning isn't
* performed here, it's quite possible that an earlier opportunistic pruning
* operation left LP_DEAD items behind. We'll at least collect any such items
- * in the dead_items array for removal from indexes.
+ * in the dead_items for removal from indexes.
*
* For aggressive VACUUM callers, we may return false to indicate that a full
* cleanup lock is required for processing by lazy_scan_prune. This is only
@@ -2099,7 +2087,7 @@ lazy_scan_noprune(LVRelState *vacrel,
vacrel->NewRelfrozenXid = NoFreezePageRelfrozenXid;
vacrel->NewRelminMxid = NoFreezePageRelminMxid;
- /* Save any LP_DEAD items found on the page in dead_items array */
+ /* Save any LP_DEAD items found on the page in dead_items */
if (vacrel->nindexes == 0)
{
/* Using one-pass strategy (since table has no indexes) */
@@ -2129,8 +2117,7 @@ lazy_scan_noprune(LVRelState *vacrel,
}
else
{
- VacDeadItems *dead_items = vacrel->dead_items;
- ItemPointerData tmp;
+ TidStore *dead_items = vacrel->dead_items;
/*
* Page has LP_DEAD items, and so any references/TIDs that remain in
@@ -2139,17 +2126,10 @@ lazy_scan_noprune(LVRelState *vacrel,
*/
vacrel->lpdead_item_pages++;
- ItemPointerSetBlockNumber(&tmp, blkno);
+ TidStoreSetBlockOffsets(dead_items, blkno, deadoffsets, lpdead_items);
- for (int i = 0; i < lpdead_items; i++)
- {
- ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
- dead_items->items[dead_items->num_items++] = tmp;
- }
-
- Assert(dead_items->num_items <= dead_items->max_items);
- pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
- dead_items->num_items);
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
+ TidStoreMemoryUsage(dead_items));
vacrel->lpdead_items += lpdead_items;
@@ -2198,7 +2178,7 @@ lazy_vacuum(LVRelState *vacrel)
if (!vacrel->do_index_vacuuming)
{
Assert(!vacrel->do_index_cleanup);
- vacrel->dead_items->num_items = 0;
+ TidStoreReset(vacrel->dead_items);
return;
}
@@ -2227,7 +2207,7 @@ lazy_vacuum(LVRelState *vacrel)
BlockNumber threshold;
Assert(vacrel->num_index_scans == 0);
- Assert(vacrel->lpdead_items == vacrel->dead_items->num_items);
+ Assert(vacrel->lpdead_items == TidStoreNumTids(vacrel->dead_items));
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2254,8 +2234,8 @@ lazy_vacuum(LVRelState *vacrel)
* cases then this may need to be reconsidered.
*/
threshold = (double) vacrel->rel_pages * BYPASS_THRESHOLD_PAGES;
- bypass = (vacrel->lpdead_item_pages < threshold &&
- vacrel->lpdead_items < MAXDEADITEMS(32L * 1024L * 1024L));
+ bypass = (vacrel->lpdead_item_pages < threshold) &&
+ TidStoreMemoryUsage(vacrel->dead_items) < (32L * 1024L * 1024L);
}
if (bypass)
@@ -2300,7 +2280,7 @@ lazy_vacuum(LVRelState *vacrel)
* Forget the LP_DEAD items that we just vacuumed (or just decided to not
* vacuum)
*/
- vacrel->dead_items->num_items = 0;
+ TidStoreReset(vacrel->dead_items);
}
/*
@@ -2373,7 +2353,7 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
* place).
*/
Assert(vacrel->num_index_scans > 0 ||
- vacrel->dead_items->num_items == vacrel->lpdead_items);
+ TidStoreNumTids(vacrel->dead_items) == vacrel->lpdead_items);
Assert(allindexes || vacrel->failsafe_active);
/*
@@ -2392,9 +2372,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
/*
* lazy_vacuum_heap_rel() -- second pass over the heap for two pass strategy
*
- * This routine marks LP_DEAD items in vacrel->dead_items array as LP_UNUSED.
- * Pages that never had lazy_scan_prune record LP_DEAD items are not visited
- * at all.
+ * This routine marks LP_DEAD items in vacrel->dead_items as LP_UNUSED. Pages
+ * that never had lazy_scan_prune record LP_DEAD items are not visited at all.
*
* We may also be able to truncate the line pointer array of the heap pages we
* visit. If there is a contiguous group of LP_UNUSED items at the end of the
@@ -2410,10 +2389,11 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
static void
lazy_vacuum_heap_rel(LVRelState *vacrel)
{
- int index = 0;
BlockNumber vacuumed_pages = 0;
Buffer vmbuffer = InvalidBuffer;
LVSavedErrInfo saved_err_info;
+ TidStoreIter *iter;
+ TidStoreIterResult *iter_result;
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2428,7 +2408,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
VACUUM_ERRCB_PHASE_VACUUM_HEAP,
InvalidBlockNumber, InvalidOffsetNumber);
- while (index < vacrel->dead_items->num_items)
+ iter = TidStoreBeginIterate(vacrel->dead_items);
+ while ((iter_result = TidStoreIterateNext(iter)) != NULL)
{
BlockNumber blkno;
Buffer buf;
@@ -2437,7 +2418,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
vacuum_delay_point();
- blkno = ItemPointerGetBlockNumber(&vacrel->dead_items->items[index]);
+ blkno = iter_result->blkno;
vacrel->blkno = blkno;
/*
@@ -2451,7 +2432,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
buf = ReadBufferExtended(vacrel->rel, MAIN_FORKNUM, blkno, RBM_NORMAL,
vacrel->bstrategy);
LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
- index = lazy_vacuum_heap_page(vacrel, blkno, buf, index, vmbuffer);
+ lazy_vacuum_heap_page(vacrel, blkno, iter_result->offsets,
+ iter_result->num_offsets, buf, vmbuffer);
/* Now that we've vacuumed the page, record its available space */
page = BufferGetPage(buf);
@@ -2461,6 +2443,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
vacuumed_pages++;
}
+ TidStoreEndIterate(iter);
vacrel->blkno = InvalidBlockNumber;
if (BufferIsValid(vmbuffer))
@@ -2470,36 +2453,31 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
* We set all LP_DEAD items from the first heap pass to LP_UNUSED during
* the second heap pass. No more, no less.
*/
- Assert(index > 0);
Assert(vacrel->num_index_scans > 1 ||
- (index == vacrel->lpdead_items &&
+ (TidStoreNumTids(vacrel->dead_items) == vacrel->lpdead_items &&
vacuumed_pages == vacrel->lpdead_item_pages));
ereport(DEBUG2,
- (errmsg("table \"%s\": removed %lld dead item identifiers in %u pages",
- vacrel->relname, (long long) index, vacuumed_pages)));
+ (errmsg("table \"%s\": removed " INT64_FORMAT "dead item identifiers in %u pages",
+ vacrel->relname, TidStoreNumTids(vacrel->dead_items),
+ vacuumed_pages)));
/* Revert to the previous phase information for error traceback */
restore_vacuum_error_info(vacrel, &saved_err_info);
}
/*
- * lazy_vacuum_heap_page() -- free page's LP_DEAD items listed in the
- * vacrel->dead_items array.
+ * lazy_vacuum_heap_page() -- free page's LP_DEAD items.
*
* Caller must have an exclusive buffer lock on the buffer (though a full
* cleanup lock is also acceptable). vmbuffer must be valid and already have
* a pin on blkno's visibility map page.
- *
- * index is an offset into the vacrel->dead_items array for the first listed
- * LP_DEAD item on the page. The return value is the first index immediately
- * after all LP_DEAD items for the same page in the array.
*/
-static int
-lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
- int index, Buffer vmbuffer)
+static void
+lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
+ OffsetNumber *deadoffsets, int num_offsets, Buffer buffer,
+ Buffer vmbuffer)
{
- VacDeadItems *dead_items = vacrel->dead_items;
Page page = BufferGetPage(buffer);
OffsetNumber unused[MaxHeapTuplesPerPage];
int nunused = 0;
@@ -2518,16 +2496,11 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
START_CRIT_SECTION();
- for (; index < dead_items->num_items; index++)
+ for (int i = 0; i < num_offsets; i++)
{
- BlockNumber tblk;
- OffsetNumber toff;
ItemId itemid;
+ OffsetNumber toff = deadoffsets[i];
- tblk = ItemPointerGetBlockNumber(&dead_items->items[index]);
- if (tblk != blkno)
- break; /* past end of tuples for this block */
- toff = ItemPointerGetOffsetNumber(&dead_items->items[index]);
itemid = PageGetItemId(page, toff);
Assert(ItemIdIsDead(itemid) && !ItemIdHasStorage(itemid));
@@ -2597,7 +2570,6 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
/* Revert to the previous phase information for error traceback */
restore_vacuum_error_info(vacrel, &saved_err_info);
- return index;
}
/*
@@ -2687,8 +2659,8 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
* lazy_vacuum_one_index() -- vacuum index relation.
*
* Delete all the index tuples containing a TID collected in
- * vacrel->dead_items array. Also update running statistics.
- * Exact details depend on index AM's ambulkdelete routine.
+ * vacrel->dead_items. Also update running statistics. Exact
+ * details depend on index AM's ambulkdelete routine.
*
* reltuples is the number of heap tuples to be passed to the
* bulkdelete callback. It's always assumed to be estimated.
@@ -3094,48 +3066,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
}
/*
- * Returns the number of dead TIDs that VACUUM should allocate space to
- * store, given a heap rel of size vacrel->rel_pages, and given current
- * maintenance_work_mem setting (or current autovacuum_work_mem setting,
- * when applicable).
- *
- * See the comments at the head of this file for rationale.
- */
-static int
-dead_items_max_items(LVRelState *vacrel)
-{
- int64 max_items;
- int vac_work_mem = IsAutoVacuumWorkerProcess() &&
- autovacuum_work_mem != -1 ?
- autovacuum_work_mem : maintenance_work_mem;
-
- if (vacrel->nindexes > 0)
- {
- BlockNumber rel_pages = vacrel->rel_pages;
-
- max_items = MAXDEADITEMS(vac_work_mem * 1024L);
- max_items = Min(max_items, INT_MAX);
- max_items = Min(max_items, MAXDEADITEMS(MaxAllocSize));
-
- /* curious coding here to ensure the multiplication can't overflow */
- if ((BlockNumber) (max_items / MaxHeapTuplesPerPage) > rel_pages)
- max_items = rel_pages * MaxHeapTuplesPerPage;
-
- /* stay sane if small maintenance_work_mem */
- max_items = Max(max_items, MaxHeapTuplesPerPage);
- }
- else
- {
- /* One-pass case only stores a single heap page's TIDs at a time */
- max_items = MaxHeapTuplesPerPage;
- }
-
- return (int) max_items;
-}
-
-/*
- * Allocate dead_items (either using palloc, or in dynamic shared memory).
- * Sets dead_items in vacrel for caller.
+ * Allocate a (local or shared) TidStore for storing dead TIDs. Sets dead_items
+ * in vacrel for caller.
*
* Also handles parallel initialization as part of allocating dead_items in
* DSM when required.
@@ -3143,11 +3075,9 @@ dead_items_max_items(LVRelState *vacrel)
static void
dead_items_alloc(LVRelState *vacrel, int nworkers)
{
- VacDeadItems *dead_items;
- int max_items;
-
- max_items = dead_items_max_items(vacrel);
- Assert(max_items >= MaxHeapTuplesPerPage);
+ int vac_work_mem = IsAutoVacuumWorkerProcess() &&
+ autovacuum_work_mem != -1 ?
+ autovacuum_work_mem * 1024L : maintenance_work_mem * 1024L;
/*
* Initialize state for a parallel vacuum. As of now, only one worker can
@@ -3174,7 +3104,7 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
else
vacrel->pvs = parallel_vacuum_init(vacrel->rel, vacrel->indrels,
vacrel->nindexes, nworkers,
- max_items,
+ vac_work_mem, MaxHeapTuplesPerPage,
vacrel->verbose ? INFO : DEBUG2,
vacrel->bstrategy);
@@ -3187,11 +3117,8 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
}
/* Serial VACUUM case */
- dead_items = (VacDeadItems *) palloc(vac_max_items_to_alloc_size(max_items));
- dead_items->max_items = max_items;
- dead_items->num_items = 0;
-
- vacrel->dead_items = dead_items;
+ vacrel->dead_items = TidStoreCreate(vac_work_mem, MaxHeapTuplesPerPage,
+ NULL);
}
/*
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 34ca0e739f..149d41b41c 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1180,7 +1180,7 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuple_bytes, S.param7 AS dead_tuple_bytes
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index aa79d9de4d..5fb30d7e62 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -97,7 +97,6 @@ static bool vacuum_rel(Oid relid, RangeVar *relation, VacuumParams *params,
static double compute_parallel_delay(void);
static VacOptValue get_vacoptval_from_boolean(DefElem *def);
static bool vac_tid_reaped(ItemPointer itemptr, void *state);
-static int vac_cmp_itemptr(const void *left, const void *right);
/*
* Primary entry point for manual VACUUM and ANALYZE commands
@@ -2303,16 +2302,16 @@ get_vacoptval_from_boolean(DefElem *def)
*/
IndexBulkDeleteResult *
vac_bulkdel_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat,
- VacDeadItems *dead_items)
+ TidStore *dead_items)
{
/* Do bulk deletion */
istat = index_bulk_delete(ivinfo, istat, vac_tid_reaped,
(void *) dead_items);
ereport(ivinfo->message_level,
- (errmsg("scanned index \"%s\" to remove %d row versions",
+ (errmsg("scanned index \"%s\" to remove " UINT64_FORMAT " row versions",
RelationGetRelationName(ivinfo->index),
- dead_items->num_items)));
+ TidStoreNumTids(dead_items))));
return istat;
}
@@ -2343,82 +2342,15 @@ vac_cleanup_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat)
return istat;
}
-/*
- * Returns the total required space for VACUUM's dead_items array given a
- * max_items value.
- */
-Size
-vac_max_items_to_alloc_size(int max_items)
-{
- Assert(max_items <= MAXDEADITEMS(MaxAllocSize));
-
- return offsetof(VacDeadItems, items) + sizeof(ItemPointerData) * max_items;
-}
-
/*
* vac_tid_reaped() -- is a particular tid deletable?
*
* This has the right signature to be an IndexBulkDeleteCallback.
- *
- * Assumes dead_items array is sorted (in ascending TID order).
*/
static bool
vac_tid_reaped(ItemPointer itemptr, void *state)
{
- VacDeadItems *dead_items = (VacDeadItems *) state;
- int64 litem,
- ritem,
- item;
- ItemPointer res;
-
- litem = itemptr_encode(&dead_items->items[0]);
- ritem = itemptr_encode(&dead_items->items[dead_items->num_items - 1]);
- item = itemptr_encode(itemptr);
-
- /*
- * Doing a simple bound check before bsearch() is useful to avoid the
- * extra cost of bsearch(), especially if dead items on the heap are
- * concentrated in a certain range. Since this function is called for
- * every index tuple, it pays to be really fast.
- */
- if (item < litem || item > ritem)
- return false;
-
- res = (ItemPointer) bsearch(itemptr,
- dead_items->items,
- dead_items->num_items,
- sizeof(ItemPointerData),
- vac_cmp_itemptr);
-
- return (res != NULL);
-}
-
-/*
- * Comparator routines for use with qsort() and bsearch().
- */
-static int
-vac_cmp_itemptr(const void *left, const void *right)
-{
- BlockNumber lblk,
- rblk;
- OffsetNumber loff,
- roff;
-
- lblk = ItemPointerGetBlockNumber((ItemPointer) left);
- rblk = ItemPointerGetBlockNumber((ItemPointer) right);
-
- if (lblk < rblk)
- return -1;
- if (lblk > rblk)
- return 1;
-
- loff = ItemPointerGetOffsetNumber((ItemPointer) left);
- roff = ItemPointerGetOffsetNumber((ItemPointer) right);
-
- if (loff < roff)
- return -1;
- if (loff > roff)
- return 1;
+ TidStore *dead_items = (TidStore *) state;
- return 0;
+ return TidStoreIsMember(dead_items, itemptr);
}
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index bcd40c80a1..9225daf3ab 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -9,10 +9,10 @@
* In a parallel vacuum, we perform both index bulk deletion and index cleanup
* with parallel worker processes. Individual indexes are processed by one
* vacuum process. ParalleVacuumState contains shared information as well as
- * the memory space for storing dead items allocated in the DSM segment. We
+ * the memory space for storing dead items allocated in the DSA area. We
* launch parallel worker processes at the start of parallel index
* bulk-deletion and index cleanup and once all indexes are processed, the
- * parallel worker processes exit. Each time we process indexes in parallel,
+ * parallel worker processes exit. Each time we process indexes in parallel,
* the parallel context is re-initialized so that the same DSM can be used for
* multiple passes of index bulk-deletion and index cleanup.
*
@@ -103,6 +103,9 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /* Handle of the shared TidStore */
+ TidStoreHandle dead_items_handle;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -166,7 +169,8 @@ struct ParallelVacuumState
PVIndStats *indstats;
/* Shared dead items space among parallel vacuum workers */
- VacDeadItems *dead_items;
+ TidStore *dead_items;
+ dsa_area *dead_items_area;
/* Points to buffer usage area in DSM */
BufferUsage *buffer_usage;
@@ -222,20 +226,23 @@ static void parallel_vacuum_error_callback(void *arg);
*/
ParallelVacuumState *
parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
- int nrequested_workers, int max_items,
- int elevel, BufferAccessStrategy bstrategy)
+ int nrequested_workers, int vac_work_mem,
+ int max_offset, int elevel,
+ BufferAccessStrategy bstrategy)
{
ParallelVacuumState *pvs;
ParallelContext *pcxt;
PVShared *shared;
- VacDeadItems *dead_items;
+ TidStore *dead_items;
PVIndStats *indstats;
BufferUsage *buffer_usage;
WalUsage *wal_usage;
+ void *area_space;
+ dsa_area *dead_items_dsa;
bool *will_parallel_vacuum;
Size est_indstats_len;
Size est_shared_len;
- Size est_dead_items_len;
+ Size dsa_minsize = dsa_minimum_size();
int nindexes_mwm = 0;
int parallel_workers = 0;
int querylen;
@@ -283,9 +290,8 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_estimate_chunk(&pcxt->estimator, est_shared_len);
shm_toc_estimate_keys(&pcxt->estimator, 1);
- /* Estimate size for dead_items -- PARALLEL_VACUUM_KEY_DEAD_ITEMS */
- est_dead_items_len = vac_max_items_to_alloc_size(max_items);
- shm_toc_estimate_chunk(&pcxt->estimator, est_dead_items_len);
+ /* Initial size for dead tuple DSA -- PARALLEL_VACUUM_KEY_DEAD_ITEMS */
+ shm_toc_estimate_chunk(&pcxt->estimator, dsa_minsize);
shm_toc_estimate_keys(&pcxt->estimator, 1);
/*
@@ -351,6 +357,16 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_INDEX_STATS, indstats);
pvs->indstats = indstats;
+ /* Prepare DSA space for dead items */
+ area_space = shm_toc_allocate(pcxt->toc, dsa_minsize);
+ shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, area_space);
+ dead_items_dsa = dsa_create_in_place(area_space, dsa_minsize,
+ LWTRANCHE_PARALLEL_VACUUM_DSA,
+ pcxt->seg);
+ dead_items = TidStoreCreate(vac_work_mem, max_offset, dead_items_dsa);
+ pvs->dead_items = dead_items;
+ pvs->dead_items_area = dead_items_dsa;
+
/* Prepare shared information */
shared = (PVShared *) shm_toc_allocate(pcxt->toc, est_shared_len);
MemSet(shared, 0, est_shared_len);
@@ -360,6 +376,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
+ shared->dead_items_handle = TidStoreGetHandle(dead_items);
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
@@ -368,15 +385,6 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_SHARED, shared);
pvs->shared = shared;
- /* Prepare the dead_items space */
- dead_items = (VacDeadItems *) shm_toc_allocate(pcxt->toc,
- est_dead_items_len);
- dead_items->max_items = max_items;
- dead_items->num_items = 0;
- MemSet(dead_items->items, 0, sizeof(ItemPointerData) * max_items);
- shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, dead_items);
- pvs->dead_items = dead_items;
-
/*
* Allocate space for each worker's BufferUsage and WalUsage; no need to
* initialize
@@ -434,6 +442,9 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
istats[i] = NULL;
}
+ TidStoreDestroy(pvs->dead_items);
+ dsa_detach(pvs->dead_items_area);
+
DestroyParallelContext(pvs->pcxt);
ExitParallelMode();
@@ -442,7 +453,7 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
}
/* Returns the dead items space */
-VacDeadItems *
+TidStore *
parallel_vacuum_get_dead_items(ParallelVacuumState *pvs)
{
return pvs->dead_items;
@@ -940,7 +951,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
Relation *indrels;
PVIndStats *indstats;
PVShared *shared;
- VacDeadItems *dead_items;
+ TidStore *dead_items;
+ void *area_space;
+ dsa_area *dead_items_area;
BufferUsage *buffer_usage;
WalUsage *wal_usage;
int nindexes;
@@ -984,10 +997,10 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
PARALLEL_VACUUM_KEY_INDEX_STATS,
false);
- /* Set dead_items space */
- dead_items = (VacDeadItems *) shm_toc_lookup(toc,
- PARALLEL_VACUUM_KEY_DEAD_ITEMS,
- false);
+ /* Set dead items */
+ area_space = shm_toc_lookup(toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, false);
+ dead_items_area = dsa_attach_in_place(area_space, seg);
+ dead_items = TidStoreAttach(dead_items_area, shared->dead_items_handle);
/* Set cost-based vacuum delay */
VacuumCostActive = (VacuumCostDelay > 0);
@@ -1033,6 +1046,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
InstrEndParallelQuery(&buffer_usage[ParallelWorkerNumber],
&wal_usage[ParallelWorkerNumber]);
+ TidStoreDetach(dead_items);
+ dsa_detach(dead_items_area);
+
/* Pop the error context stack */
error_context_stack = errcallback.previous;
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index ff6149a179..a371f6fbba 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -3397,12 +3397,12 @@ check_autovacuum_work_mem(int *newval, void **extra, GucSource source)
return true;
/*
- * We clamp manually-set values to at least 1MB. Since
+ * We clamp manually-set values to at least 2MB. Since
* maintenance_work_mem is always set to at least this value, do the same
* here.
*/
- if (*newval < 1024)
- *newval = 1024;
+ if (*newval < 2048)
+ *newval = 2048;
return true;
}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 55b3a04097..c223a7dc94 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -192,6 +192,8 @@ static const char *const BuiltinTrancheNames[] = {
"LogicalRepLauncherDSA",
/* LWTRANCHE_LAUNCHER_HASH: */
"LogicalRepLauncherHash",
+ /* LWTRANCHE_PARALLEL_VACUUM_DSA: */
+ "ParallelVacuumDSA",
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 1c0583fe26..8a64614cd1 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2313,7 +2313,7 @@ struct config_int ConfigureNamesInt[] =
GUC_UNIT_KB
},
&maintenance_work_mem,
- 65536, 1024, MAX_KILOBYTES,
+ 65536, 2048, MAX_KILOBYTES,
NULL, NULL, NULL
},
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index e5add41352..b209d3cf84 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -23,8 +23,8 @@
#define PROGRESS_VACUUM_HEAP_BLKS_SCANNED 2
#define PROGRESS_VACUUM_HEAP_BLKS_VACUUMED 3
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
-#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
-#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_MAX_DEAD_TUPLE_BYTES 5
+#define PROGRESS_VACUUM_DEAD_TUPLE_BYTES 6
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 689dbb7702..a3ebb169ef 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -17,6 +17,7 @@
#include "access/htup.h"
#include "access/genam.h"
#include "access/parallel.h"
+#include "access/tidstore.h"
#include "catalog/pg_class.h"
#include "catalog/pg_statistic.h"
#include "catalog/pg_type.h"
@@ -276,21 +277,6 @@ struct VacuumCutoffs
MultiXactId MultiXactCutoff;
};
-/*
- * VacDeadItems stores TIDs whose index tuples are deleted by index vacuuming.
- */
-typedef struct VacDeadItems
-{
- int max_items; /* # slots allocated in array */
- int num_items; /* current # of entries */
-
- /* Sorted array of TIDs to delete from indexes */
- ItemPointerData items[FLEXIBLE_ARRAY_MEMBER];
-} VacDeadItems;
-
-#define MAXDEADITEMS(avail_mem) \
- (((avail_mem) - offsetof(VacDeadItems, items)) / sizeof(ItemPointerData))
-
/* GUC parameters */
extern PGDLLIMPORT int default_statistics_target; /* PGDLLIMPORT for PostGIS */
extern PGDLLIMPORT int vacuum_freeze_min_age;
@@ -339,18 +325,17 @@ extern Relation vacuum_open_relation(Oid relid, RangeVar *relation,
LOCKMODE lmode);
extern IndexBulkDeleteResult *vac_bulkdel_one_index(IndexVacuumInfo *ivinfo,
IndexBulkDeleteResult *istat,
- VacDeadItems *dead_items);
+ TidStore *dead_items);
extern IndexBulkDeleteResult *vac_cleanup_one_index(IndexVacuumInfo *ivinfo,
IndexBulkDeleteResult *istat);
-extern Size vac_max_items_to_alloc_size(int max_items);
/* in commands/vacuumparallel.c */
extern ParallelVacuumState *parallel_vacuum_init(Relation rel, Relation *indrels,
int nindexes, int nrequested_workers,
- int max_items, int elevel,
- BufferAccessStrategy bstrategy);
+ int vac_work_mem, int max_offset,
+ int elevel, BufferAccessStrategy bstrategy);
extern void parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats);
-extern VacDeadItems *parallel_vacuum_get_dead_items(ParallelVacuumState *pvs);
+extern TidStore *parallel_vacuum_get_dead_items(ParallelVacuumState *pvs);
extern void parallel_vacuum_bulkdel_all_indexes(ParallelVacuumState *pvs,
long num_table_tuples,
int num_index_scans);
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index 07002fdfbe..537b34b30c 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -207,6 +207,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_PGSTATS_DATA,
LWTRANCHE_LAUNCHER_DSA,
LWTRANCHE_LAUNCHER_HASH,
+ LWTRANCHE_PARALLEL_VACUUM_DSA,
LWTRANCHE_FIRST_USER_DEFINED
} BuiltinTrancheIds;
diff --git a/src/test/regress/expected/cluster.out b/src/test/regress/expected/cluster.out
index 2eec483eaa..e04f50726f 100644
--- a/src/test/regress/expected/cluster.out
+++ b/src/test/regress/expected/cluster.out
@@ -526,7 +526,7 @@ create index cluster_sort on clstr_4 (hundred, thousand, tenthous);
-- ensure we don't use the index in CLUSTER nor the checking SELECTs
set enable_indexscan = off;
-- Use external sort:
-set maintenance_work_mem = '1MB';
+set maintenance_work_mem = '2MB';
cluster clstr_4 using cluster_sort;
select * from
(select hundred, lag(hundred) over () as lhundred,
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index acfd9d1f4f..d320ad87dd 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -1214,7 +1214,7 @@ DROP TABLE unlogged_hash_table;
-- CREATE INDEX hash_ovfl_index ON hash_ovfl_heap USING hash (x int4_ops);
-- Test hash index build tuplesorting. Force hash tuplesort using low
-- maintenance_work_mem setting and fillfactor:
-SET maintenance_work_mem = '1MB';
+SET maintenance_work_mem = '2MB';
CREATE INDEX hash_tuplesort_idx ON tenk1 USING hash (stringu1 name_ops) WITH (fillfactor = 10);
EXPLAIN (COSTS OFF)
SELECT count(*) FROM tenk1 WHERE stringu1 = 'TVAAAA';
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index e953d1f515..ef46c2994f 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2032,8 +2032,8 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param3 AS heap_blks_scanned,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
- s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param6 AS max_dead_tuple_bytes,
+ s.param7 AS dead_tuple_bytes
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_recovery_prefetch| SELECT stats_reset,
diff --git a/src/test/regress/sql/cluster.sql b/src/test/regress/sql/cluster.sql
index a4cfaae807..a4cb5b98a5 100644
--- a/src/test/regress/sql/cluster.sql
+++ b/src/test/regress/sql/cluster.sql
@@ -258,7 +258,7 @@ create index cluster_sort on clstr_4 (hundred, thousand, tenthous);
set enable_indexscan = off;
-- Use external sort:
-set maintenance_work_mem = '1MB';
+set maintenance_work_mem = '2MB';
cluster clstr_4 using cluster_sort;
select * from
(select hundred, lag(hundred) over () as lhundred,
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index d49ce9f300..d6e2471b00 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -367,7 +367,7 @@ DROP TABLE unlogged_hash_table;
-- Test hash index build tuplesorting. Force hash tuplesort using low
-- maintenance_work_mem setting and fillfactor:
-SET maintenance_work_mem = '1MB';
+SET maintenance_work_mem = '2MB';
CREATE INDEX hash_tuplesort_idx ON tenk1 USING hash (stringu1 name_ops) WITH (fillfactor = 10);
EXPLAIN (COSTS OFF)
SELECT count(*) FROM tenk1 WHERE stringu1 = 'TVAAAA';
--
2.31.1
[application/octet-stream] v30-0004-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch (37.2K, ../../CAD21AoC+kjqkSnLLcbqtu=Td5HSx2H+iAwNHCeZcWqjy1w7k5g@mail.gmail.com/8-v30-0004-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch)
download | inline diff:
From 95bb8dc701efa4a5923a355880b60885dc18cfa3 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 23 Dec 2022 23:50:39 +0900
Subject: [PATCH v30 04/11] Add TIDStore, to store sets of TIDs
(ItemPointerData) efficiently.
The TIDStore is designed to store large sets of TIDs efficiently, and
is backed by the radix tree. A TID is encoded into 64-bit key and
value and inserted to the radix tree.
The TIDStore is not used for anything yet, aside from the test code,
but the follow up patch integrates the TIDStore with lazy vacuum,
reducing lazy vacuum memory usage and lifting the 1GB limit on its
size, by storing the list of dead TIDs more efficiently.
This includes a unit test module, in src/test/modules/test_tidstore.
---
doc/src/sgml/monitoring.sgml | 4 +
src/backend/access/common/Makefile | 1 +
src/backend/access/common/meson.build | 1 +
src/backend/access/common/tidstore.c | 710 ++++++++++++++++++
src/backend/storage/lmgr/lwlock.c | 2 +
src/include/access/tidstore.h | 50 ++
src/include/storage/lwlock.h | 1 +
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_tidstore/Makefile | 23 +
.../test_tidstore/expected/test_tidstore.out | 13 +
src/test/modules/test_tidstore/meson.build | 35 +
.../test_tidstore/sql/test_tidstore.sql | 7 +
.../test_tidstore/test_tidstore--1.0.sql | 8 +
.../modules/test_tidstore/test_tidstore.c | 228 ++++++
.../test_tidstore/test_tidstore.control | 4 +
16 files changed, 1089 insertions(+)
create mode 100644 src/backend/access/common/tidstore.c
create mode 100644 src/include/access/tidstore.h
create mode 100644 src/test/modules/test_tidstore/Makefile
create mode 100644 src/test/modules/test_tidstore/expected/test_tidstore.out
create mode 100644 src/test/modules/test_tidstore/meson.build
create mode 100644 src/test/modules/test_tidstore/sql/test_tidstore.sql
create mode 100644 src/test/modules/test_tidstore/test_tidstore--1.0.sql
create mode 100644 src/test/modules/test_tidstore/test_tidstore.c
create mode 100644 src/test/modules/test_tidstore/test_tidstore.control
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 6249bb50d0..97d588b1d8 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2203,6 +2203,10 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<entry>Waiting to access a shared TID bitmap during a parallel bitmap
index scan.</entry>
</row>
+ <row>
+ <entry><literal>SharedTidStore</literal></entry>
+ <entry>Waiting to access a shared TID store.</entry>
+ </row>
<row>
<entry><literal>SharedTupleStore</literal></entry>
<entry>Waiting to access a shared tuple store during parallel
diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile
index b9aff0ccfd..67b8cc6108 100644
--- a/src/backend/access/common/Makefile
+++ b/src/backend/access/common/Makefile
@@ -27,6 +27,7 @@ OBJS = \
syncscan.o \
toast_compression.o \
toast_internals.o \
+ tidstore.o \
tupconvert.o \
tupdesc.o
diff --git a/src/backend/access/common/meson.build b/src/backend/access/common/meson.build
index f5ac17b498..fce19c09ce 100644
--- a/src/backend/access/common/meson.build
+++ b/src/backend/access/common/meson.build
@@ -15,6 +15,7 @@ backend_sources += files(
'syncscan.c',
'toast_compression.c',
'toast_internals.c',
+ 'tidstore.c',
'tupconvert.c',
'tupdesc.c',
)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
new file mode 100644
index 0000000000..2d6f2b3ab9
--- /dev/null
+++ b/src/backend/access/common/tidstore.c
@@ -0,0 +1,710 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.c
+ * Tid (ItemPointerData) storage implementation.
+ *
+ * TidStore is a in-memory data structure to store tids (ItemPointerData).
+ * Internally, a tid is encoded as a pair of 64-bit key and 64-bit value,
+ * and stored in the radix tree.
+ *
+ * TidStore can be shared among parallel worker processes by passing DSA area
+ * to TidStoreCreate(). Other backends can attach to the shared TidStore by
+ * TidStoreAttach().
+ *
+ * Regarding the concurrency support, we use a single LWLock for the TidStore.
+ * The TidStore is exclusively locked when inserting encoded tids to the
+ * radix tree or when resetting itself. When searching on the TidStore or
+ * doing the iteration, it is not locked but the underlying radix tree is
+ * locked in shared mode.
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/access/common/tidstore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/tidstore.h"
+#include "miscadmin.h"
+#include "port/pg_bitutils.h"
+#include "storage/lwlock.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/*
+ * For encoding purposes, a tid is represented as a pair of 64-bit key and
+ * 64-bit value.
+ *
+ * First, we construct a 64-bit unsigned integer by combining the block
+ * number and the offset number. The number of bits used for the offset number
+ * is specified by max_off in TidStoreCreate(). We are frugal with the bits,
+ * because smaller keys could help keeping the radix tree shallow.
+ *
+ * For example, a tid of heap on a 8kB block uses the lowest 9 bits for
+ * the offset number and uses the next 32 bits for the block number. 9 bits
+ * are enough for the offset number, because MaxHeapTuplesPerPage < 2^9
+ * on 8kB blocks. That is, only 41 bits are used:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ *
+ * X = bits used for offset number
+ * Y = bits used for block number
+ * u = unused bit
+ * (high on the left, low on the right)
+ *
+ * Then, 64-bit value is the bitmap representation of the lowest 6 bits
+ * (LOWER_OFFSET_NBITS) of the integer, and 64-bit key consists of the
+ * upper 3 bits of the offset number and the block number, 35 bits in
+ * total:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ * |----| value
+ * |--------------------------------------| key
+ *
+ * The maximum height of the radix tree is 5 in this case.
+ *
+ * If the number of bits required for offset numbers fits in LOWER_OFFSET_NBITS,
+ * 64-bit value is the bitmap representation of the offset number, and the
+ * 64-bit key is the block number.
+ */
+typedef uint64 tidkey;
+typedef uint64 offsetbm;
+#define LOWER_OFFSET_NBITS 6 /* log(sizeof(offsetbm), 2) */
+#define LOWER_OFFSET_MASK ((1 << LOWER_OFFSET_NBITS) - 1)
+
+/* A magic value used to identify our TidStore. */
+#define TIDSTORE_MAGIC 0x826f6a10
+
+#define RT_PREFIX local_rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE tidkey
+#include "lib/radixtree.h"
+
+#define RT_PREFIX shared_rt
+#define RT_SHMEM
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE tidkey
+#include "lib/radixtree.h"
+
+/* The control object for a TidStore */
+typedef struct TidStoreControl
+{
+ /* the number of tids in the store */
+ int64 num_tids;
+
+ /* These values are never changed after creation */
+ size_t max_bytes; /* the maximum bytes a TidStore can use */
+ OffsetNumber max_off; /* the maximum offset number */
+ int max_off_nbits; /* the number of bits required for offset
+ * numbers */
+ int upper_off_nbits; /* the number of bits of offset numbers
+ * used in a key */
+
+ /* The below fields are used only in shared case */
+
+ uint32 magic;
+ LWLock lock;
+
+ /* handles for TidStore and radix tree */
+ TidStoreHandle handle;
+ shared_rt_handle tree_handle;
+} TidStoreControl;
+
+/* Per-backend state for a TidStore */
+struct TidStore
+{
+ /*
+ * Control object. This is allocated in DSA area 'area' in the shared
+ * case, otherwise in backend-local memory.
+ */
+ TidStoreControl *control;
+
+ /* Storage for Tids. Use either one depending on TidStoreIsShared() */
+ union
+ {
+ local_rt_radix_tree *local;
+ shared_rt_radix_tree *shared;
+ } tree;
+
+ /* DSA area for TidStore if used */
+ dsa_area *area;
+};
+#define TidStoreIsShared(ts) ((ts)->area != NULL)
+
+/* Iterator for TidStore */
+typedef struct TidStoreIter
+{
+ TidStore *ts;
+
+ /* iterator of radix tree. Use either one depending on TidStoreIsShared() */
+ union
+ {
+ shared_rt_iter *shared;
+ local_rt_iter *local;
+ } tree_iter;
+
+ /* we returned all tids? */
+ bool finished;
+
+ /* save for the next iteration */
+ tidkey next_tidkey;
+ offsetbm next_off_bitmap;
+
+ /*
+ * output for the caller. Must be last because variable-size.
+ */
+ TidStoreIterResult output;
+} TidStoreIter;
+
+static void iter_decode_key_off(TidStoreIter *iter, tidkey key, offsetbm off_bitmap);
+static inline BlockNumber key_get_blkno(TidStore *ts, tidkey key);
+static inline tidkey encode_blk_off(TidStore *ts, BlockNumber block,
+ OffsetNumber offset, offsetbm *off_bit);
+static inline tidkey encode_tid(TidStore *ts, ItemPointer tid, offsetbm *off_bit);
+
+/*
+ * Create a TidStore. The returned object is allocated in backend-local memory.
+ * The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
+ */
+TidStore *
+TidStoreCreate(size_t max_bytes, OffsetNumber max_off, dsa_area *area)
+{
+ TidStore *ts;
+
+ Assert(max_off <= MaxOffsetNumber);
+
+ ts = palloc0(sizeof(TidStore));
+
+ /*
+ * Create the radix tree for the main storage.
+ *
+ * Memory consumption depends on the number of stored tids, but also on the
+ * distribution of them, how the radix tree stores, and the memory management
+ * that backed the radix tree. The maximum bytes that a TidStore can
+ * use is specified by the max_bytes in TidStoreCreate(). We want the total
+ * amount of memory consumption by a TidStore not to exceed the max_bytes.
+ *
+ * In local TidStore cases, the radix tree uses slab allocators for each kind
+ * of node class. The most memory consuming case while adding Tids associated
+ * with one page (i.e. during TidStoreSetBlockOffsets()) is that we allocate a new
+ * slab block for a new radix tree node, which is approximately 70kB. Therefore,
+ * we deduct 70kB from the max_bytes.
+ *
+ * In shared cases, DSA allocates the memory segments big enough to follow
+ * a geometric series that approximately doubles the total DSA size (see
+ * make_new_segment() in dsa.c). We simulated the how DSA increases segment
+ * size and the simulation revealed, the 75% threshold for the maximum bytes
+ * perfectly works in case where the max_bytes is a power-of-2, and the 60%
+ * threshold works for other cases.
+ */
+ if (area != NULL)
+ {
+ dsa_pointer dp;
+ float ratio = ((max_bytes & (max_bytes - 1)) == 0) ? 0.75 : 0.6;
+
+ ts->tree.shared = shared_rt_create(CurrentMemoryContext, area,
+ LWTRANCHE_SHARED_TIDSTORE);
+
+ dp = dsa_allocate0(area, sizeof(TidStoreControl));
+ ts->control = (TidStoreControl *) dsa_get_address(area, dp);
+ ts->control->max_bytes = (size_t) (max_bytes * ratio);
+ ts->area = area;
+
+ ts->control->magic = TIDSTORE_MAGIC;
+ LWLockInitialize(&ts->control->lock, LWTRANCHE_SHARED_TIDSTORE);
+ ts->control->handle = dp;
+ ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+ }
+ else
+ {
+ ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+ ts->control = (TidStoreControl *) palloc0(sizeof(TidStoreControl));
+ ts->control->max_bytes = max_bytes - (70 * 1024);
+ }
+
+ ts->control->max_off = max_off;
+ ts->control->max_off_nbits = pg_ceil_log2_32(max_off);
+
+ if (ts->control->max_off_nbits < LOWER_OFFSET_NBITS)
+ ts->control->max_off_nbits = LOWER_OFFSET_NBITS;
+
+ ts->control->upper_off_nbits =
+ ts->control->max_off_nbits - LOWER_OFFSET_NBITS;
+
+ return ts;
+}
+
+/*
+ * Attach to the shared TidStore using a handle. The returned object is
+ * allocated in backend-local memory using the CurrentMemoryContext.
+ */
+TidStore *
+TidStoreAttach(dsa_area *area, TidStoreHandle handle)
+{
+ TidStore *ts;
+ dsa_pointer control;
+
+ Assert(area != NULL);
+ Assert(DsaPointerIsValid(handle));
+
+ /* create per-backend state */
+ ts = palloc0(sizeof(TidStore));
+
+ /* Find the control object in shared memory */
+ control = handle;
+
+ /* Set up the TidStore */
+ ts->control = (TidStoreControl *) dsa_get_address(area, control);
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ ts->tree.shared = shared_rt_attach(area, ts->control->tree_handle);
+ ts->area = area;
+
+ return ts;
+}
+
+/*
+ * Detach from a TidStore. This detaches from radix tree and frees the
+ * backend-local resources. The radix tree will continue to exist until
+ * it is either explicitly destroyed, or the area that backs it is returned
+ * to the operating system.
+ */
+void
+TidStoreDetach(TidStore *ts)
+{
+ Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+ shared_rt_detach(ts->tree.shared);
+ pfree(ts);
+}
+
+/*
+ * Destroy a TidStore, returning all memory.
+ *
+ * TODO: The caller must be certain that no other backend will attempt to
+ * access the TidStore before calling this function. Other backend must
+ * explicitly call TidStoreDetach() to free up backend-local memory associated
+ * with the TidStore. The backend that calls TidStoreDestroy() must not call
+ * TidStoreDetach().
+ */
+void
+TidStoreDestroy(TidStore *ts)
+{
+ if (TidStoreIsShared(ts))
+ {
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ /*
+ * Vandalize the control block to help catch programming error where
+ * other backends access the memory formerly occupied by this radix
+ * tree.
+ */
+ ts->control->magic = 0;
+ dsa_free(ts->area, ts->control->handle);
+ shared_rt_free(ts->tree.shared);
+ }
+ else
+ {
+ pfree(ts->control);
+ local_rt_free(ts->tree.local);
+ }
+
+ pfree(ts);
+}
+
+/*
+ * Forget all collected Tids. It's similar to TidStoreDestroy() but we don't free
+ * entire TidStore but recreate only the radix tree storage.
+ */
+void
+TidStoreReset(TidStore *ts)
+{
+ if (TidStoreIsShared(ts))
+ {
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+ /*
+ * Free the radix tree and return allocated DSA segments to
+ * the operating system.
+ */
+ shared_rt_free(ts->tree.shared);
+ dsa_trim(ts->area);
+
+ /* Recreate the radix tree */
+ ts->tree.shared = shared_rt_create(CurrentMemoryContext, ts->area,
+ LWTRANCHE_SHARED_TIDSTORE);
+
+ /* update the radix tree handle as we recreated it */
+ ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
+
+ LWLockRelease(&ts->control->lock);
+ }
+ else
+ {
+ local_rt_free(ts->tree.local);
+ ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
+ }
+}
+
+/*
+ * Set the given tids on the blkno to TidStore.
+ *
+ * NB: the offset numbers in offsets must be sorted in ascending order.
+ */
+void
+TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets)
+{
+ offsetbm *bitmaps;
+ tidkey key;
+ tidkey prev_key;
+ offsetbm off_bitmap = 0;
+ int idx;
+ const tidkey key_base = ((uint64) blkno) << ts->control->upper_off_nbits;
+ const int nkeys = UINT64CONST(1) << ts->control->upper_off_nbits;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+ Assert(BlockNumberIsValid(blkno));
+
+ bitmaps = palloc(sizeof(offsetbm) * nkeys);
+ key = prev_key = key_base;
+
+ for (int i = 0; i < num_offsets; i++)
+ {
+ offsetbm off_bit;
+
+ Assert(offsets[i] <= ts->control->max_off);
+
+ /* encode the tid to a key and partial offset */
+ key = encode_blk_off(ts, blkno, offsets[i], &off_bit);
+
+ /* make sure we scanned the line pointer array in order */
+ Assert(key >= prev_key);
+
+ if (key > prev_key)
+ {
+ idx = prev_key - key_base;
+ Assert(idx >= 0 && idx < nkeys);
+
+ /* write out offset bitmap for this key */
+ bitmaps[idx] = off_bitmap;
+
+ /* zero out any gaps up to the current key */
+ for (int empty_idx = idx + 1; empty_idx < key - key_base; empty_idx++)
+ bitmaps[empty_idx] = 0;
+
+ /* reset for current key -- the current offset will be handled below */
+ off_bitmap = 0;
+ prev_key = key;
+ }
+
+ off_bitmap |= off_bit;
+ }
+
+ /* save the final index for later */
+ idx = key - key_base;
+ /* write out last offset bitmap */
+ bitmaps[idx] = off_bitmap;
+
+ if (TidStoreIsShared(ts))
+ LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+ /* insert the calculated key-values to the tree */
+ for (int i = 0; i <= idx; i++)
+ {
+ if (bitmaps[i])
+ {
+ key = key_base + i;
+
+ if (TidStoreIsShared(ts))
+ shared_rt_set(ts->tree.shared, key, &bitmaps[i]);
+ else
+ local_rt_set(ts->tree.local, key, &bitmaps[i]);
+ }
+ }
+
+ /* update statistics */
+ ts->control->num_tids += num_offsets;
+
+ if (TidStoreIsShared(ts))
+ LWLockRelease(&ts->control->lock);
+
+ pfree(bitmaps);
+}
+
+/* Return true if the given tid is present in the TidStore */
+bool
+TidStoreIsMember(TidStore *ts, ItemPointer tid)
+{
+ tidkey key;
+ offsetbm off_bitmap = 0;
+ offsetbm off_bit;
+ bool found;
+
+ Assert(ItemPointerIsValid(tid));
+
+ key = encode_tid(ts, tid, &off_bit);
+
+ if (TidStoreIsShared(ts))
+ found = shared_rt_search(ts->tree.shared, key, &off_bitmap);
+ else
+ found = local_rt_search(ts->tree.local, key, &off_bitmap);
+
+ if (!found)
+ return false;
+
+ return (off_bitmap & off_bit) != 0;
+}
+
+/*
+ * Prepare to iterate through a TidStore. Since the radix tree is locked during
+ * the iteration, so TidStoreEndIterate() needs to be called when finished.
+ *
+ * The TidStoreIter struct is created in the caller's memory context.
+ *
+ * Concurrent updates during the iteration will be blocked when inserting a
+ * key-value to the radix tree.
+ */
+TidStoreIter *
+TidStoreBeginIterate(TidStore *ts)
+{
+ TidStoreIter *iter;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ iter = palloc0(sizeof(TidStoreIter) +
+ sizeof(OffsetNumber) * ts->control->max_off);
+ iter->ts = ts;
+
+ if (TidStoreIsShared(ts))
+ iter->tree_iter.shared = shared_rt_begin_iterate(ts->tree.shared);
+ else
+ iter->tree_iter.local = local_rt_begin_iterate(ts->tree.local);
+
+ /* If the TidStore is empty, there is no business */
+ if (TidStoreNumTids(ts) == 0)
+ iter->finished = true;
+
+ return iter;
+}
+
+static inline bool
+tidstore_iter(TidStoreIter *iter, tidkey *key, offsetbm *off_bitmap)
+{
+ if (TidStoreIsShared(iter->ts))
+ return shared_rt_iterate_next(iter->tree_iter.shared, key, off_bitmap);
+
+ return local_rt_iterate_next(iter->tree_iter.local, key, off_bitmap);
+}
+
+/*
+ * Scan the TidStore and return a pointer to TidStoreIterResult that has tids
+ * in one block. We return the block numbers in ascending order and the offset
+ * numbers in each result is also sorted in ascending order.
+ */
+TidStoreIterResult *
+TidStoreIterateNext(TidStoreIter *iter)
+{
+ tidkey key;
+ offsetbm off_bitmap = 0;
+ TidStoreIterResult *output = &(iter->output);
+
+ if (iter->finished)
+ return NULL;
+
+ /* Initialize the outputs */
+ output->blkno = InvalidBlockNumber;
+ output->num_offsets = 0;
+
+ /*
+ * Decode the key and offset bitmap that are collected in the previous
+ * time, if exists.
+ */
+ if (iter->next_off_bitmap > 0)
+ iter_decode_key_off(iter, iter->next_tidkey, iter->next_off_bitmap);
+
+ while (tidstore_iter(iter, &key, &off_bitmap))
+ {
+ BlockNumber blkno = key_get_blkno(iter->ts, key);
+ Assert(BlockNumberIsValid(blkno));
+
+ if (BlockNumberIsValid(output->blkno) && output->blkno != blkno)
+ {
+ /*
+ * We got tids for a different block. We return the collected
+ * tids so far, and remember the key-value for the next
+ * iteration.
+ */
+ iter->next_tidkey = key;
+ iter->next_off_bitmap = off_bitmap;
+ return output;
+ }
+
+ /* Collect tids decoded from the key and offset bitmap */
+ iter_decode_key_off(iter, key, off_bitmap);
+ }
+
+ iter->finished = true;
+ return output;
+}
+
+/*
+ * Finish an iteration over TidStore. This needs to be called after finishing
+ * or when existing an iteration.
+ */
+void
+TidStoreEndIterate(TidStoreIter *iter)
+{
+ if (TidStoreIsShared(iter->ts))
+ shared_rt_end_iterate(iter->tree_iter.shared);
+ else
+ local_rt_end_iterate(iter->tree_iter.local);
+
+ pfree(iter);
+}
+
+/* Return the number of tids we collected so far */
+int64
+TidStoreNumTids(TidStore *ts)
+{
+ int64 num_tids;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ if (!TidStoreIsShared(ts))
+ return ts->control->num_tids;
+
+ LWLockAcquire(&ts->control->lock, LW_SHARED);
+ num_tids = ts->control->num_tids;
+ LWLockRelease(&ts->control->lock);
+
+ Assert(num_tids >= 0);
+ return num_tids;
+}
+
+/* Return true if the current memory usage of TidStore exceeds the limit */
+bool
+TidStoreIsFull(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ return (TidStoreMemoryUsage(ts) > ts->control->max_bytes);
+}
+
+/* Return the maximum memory TidStore can use */
+size_t
+TidStoreMaxMemory(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ return ts->control->max_bytes;
+}
+
+/* Return the memory usage of TidStore */
+size_t
+TidStoreMemoryUsage(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ /*
+ * In the shared case, TidStoreControl and radix_tree are backed by the
+ * same DSA area and rt_memory_usage() returns the value including both.
+ * So we don't need to add the size of TidStoreControl separately.
+ */
+ if (TidStoreIsShared(ts))
+ return sizeof(TidStore) + shared_rt_memory_usage(ts->tree.shared);
+
+ return sizeof(TidStore) + sizeof(TidStore) + local_rt_memory_usage(ts->tree.local);
+}
+
+/*
+ * Get a handle that can be used by other processes to attach to this TidStore
+ */
+TidStoreHandle
+TidStoreGetHandle(TidStore *ts)
+{
+ Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+ return ts->control->handle;
+}
+
+/*
+ * Decode the key and offset bitmap to tids and store them to the iteration
+ * result.
+ */
+static void
+iter_decode_key_off(TidStoreIter *iter, tidkey key, offsetbm off_bitmap)
+{
+ TidStoreIterResult *output = (&iter->output);
+
+ while (off_bitmap)
+ {
+ uint64 compressed_tid;
+ OffsetNumber off;
+
+ compressed_tid = key << LOWER_OFFSET_NBITS;
+ compressed_tid |= pg_rightmost_one_pos64(off_bitmap);
+
+ off = compressed_tid & ((UINT64CONST(1) << iter->ts->control->max_off_nbits) - 1);
+
+ Assert(output->num_offsets < iter->ts->control->max_off);
+ output->offsets[output->num_offsets++] = off;
+
+ /* unset the rightmost bit */
+ off_bitmap &= ~pg_rightmost_one64(off_bitmap);
+ }
+
+ output->blkno = key_get_blkno(iter->ts, key);
+}
+
+/* Get block number from the given key */
+static inline BlockNumber
+key_get_blkno(TidStore *ts, tidkey key)
+{
+ return (BlockNumber) (key >> ts->control->upper_off_nbits);
+}
+
+/* Encode a tid to key and partial offset */
+static inline tidkey
+encode_tid(TidStore *ts, ItemPointer tid, offsetbm *off_bit)
+{
+ OffsetNumber offset = ItemPointerGetOffsetNumber(tid);
+ BlockNumber block = ItemPointerGetBlockNumber(tid);
+
+ return encode_blk_off(ts, block, offset, off_bit);
+}
+
+/* encode a block and offset to a key and partial offset */
+static inline tidkey
+encode_blk_off(TidStore *ts, BlockNumber block, OffsetNumber offset,
+ offsetbm *off_bit)
+{
+ tidkey key;
+ uint64 compressed_tid;
+ uint32 off_lower;
+
+ off_lower = offset & LOWER_OFFSET_MASK;
+ Assert(off_lower < (sizeof(offsetbm) * BITS_PER_BYTE));
+
+ *off_bit = UINT64CONST(1) << off_lower;
+ compressed_tid = offset | ((uint64) block << ts->control->max_off_nbits);
+ key = compressed_tid >> LOWER_OFFSET_NBITS;
+
+ return key;
+}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index d2ec396045..55b3a04097 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -176,6 +176,8 @@ static const char *const BuiltinTrancheNames[] = {
"SharedTupleStore",
/* LWTRANCHE_SHARED_TIDBITMAP: */
"SharedTidBitmap",
+ /* LWTRANCHE_SHARED_TIDSTORE: */
+ "SharedTidStore",
/* LWTRANCHE_PARALLEL_APPEND: */
"ParallelAppend",
/* LWTRANCHE_PER_XACT_PREDICATE_LIST: */
diff --git a/src/include/access/tidstore.h b/src/include/access/tidstore.h
new file mode 100644
index 0000000000..d1cc93cbb6
--- /dev/null
+++ b/src/include/access/tidstore.h
@@ -0,0 +1,50 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.h
+ * Tid storage.
+ *
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/tidstore.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef TIDSTORE_H
+#define TIDSTORE_H
+
+#include "storage/itemptr.h"
+#include "utils/dsa.h"
+
+typedef dsa_pointer TidStoreHandle;
+
+typedef struct TidStore TidStore;
+typedef struct TidStoreIter TidStoreIter;
+
+/* Result struct for TidStoreIterateNext */
+typedef struct TidStoreIterResult
+{
+ BlockNumber blkno;
+ int num_offsets;
+ OffsetNumber offsets[FLEXIBLE_ARRAY_MEMBER];
+} TidStoreIterResult;
+
+extern TidStore *TidStoreCreate(size_t max_bytes, OffsetNumber max_off, dsa_area *dsa);
+extern TidStore *TidStoreAttach(dsa_area *dsa, dsa_pointer handle);
+extern void TidStoreDetach(TidStore *ts);
+extern void TidStoreDestroy(TidStore *ts);
+extern void TidStoreReset(TidStore *ts);
+extern void TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets);
+extern bool TidStoreIsMember(TidStore *ts, ItemPointer tid);
+extern TidStoreIter * TidStoreBeginIterate(TidStore *ts);
+extern TidStoreIterResult *TidStoreIterateNext(TidStoreIter *iter);
+extern void TidStoreEndIterate(TidStoreIter *iter);
+extern int64 TidStoreNumTids(TidStore *ts);
+extern bool TidStoreIsFull(TidStore *ts);
+extern size_t TidStoreMaxMemory(TidStore *ts);
+extern size_t TidStoreMemoryUsage(TidStore *ts);
+extern TidStoreHandle TidStoreGetHandle(TidStore *ts);
+
+#endif /* TIDSTORE_H */
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index d2c7afb8f4..07002fdfbe 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -199,6 +199,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_PER_SESSION_RECORD_TYPMOD,
LWTRANCHE_SHARED_TUPLESTORE,
LWTRANCHE_SHARED_TIDBITMAP,
+ LWTRANCHE_SHARED_TIDSTORE,
LWTRANCHE_PARALLEL_APPEND,
LWTRANCHE_PER_XACT_PREDICATE_LIST,
LWTRANCHE_PGSTATS_DSA,
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 9659eb85d7..bddc16ada7 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
test_rls_hooks \
test_shm_mq \
test_slru \
+ test_tidstore \
unsafe_tests \
worker_spi
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 232cbdac80..c0d5645ad8 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -30,5 +30,6 @@ subdir('test_regex')
subdir('test_rls_hooks')
subdir('test_shm_mq')
subdir('test_slru')
+subdir('test_tidstore')
subdir('unsafe_tests')
subdir('worker_spi')
diff --git a/src/test/modules/test_tidstore/Makefile b/src/test/modules/test_tidstore/Makefile
new file mode 100644
index 0000000000..dab107d70c
--- /dev/null
+++ b/src/test/modules/test_tidstore/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_tidstore/Makefile
+
+MODULE_big = test_tidstore
+OBJS = \
+ $(WIN32RES) \
+ test_tidstore.o
+PGFILEDESC = "test_tidstore - test code for src/backend/access/common/tidstore.c"
+
+EXTENSION = test_tidstore
+DATA = test_tidstore--1.0.sql
+
+REGRESS = test_tidstore
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_tidstore
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_tidstore/expected/test_tidstore.out b/src/test/modules/test_tidstore/expected/test_tidstore.out
new file mode 100644
index 0000000000..7ff2f9af87
--- /dev/null
+++ b/src/test/modules/test_tidstore/expected/test_tidstore.out
@@ -0,0 +1,13 @@
+CREATE EXTENSION test_tidstore;
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
+NOTICE: testing empty tidstore
+NOTICE: testing basic operations
+ test_tidstore
+---------------
+
+(1 row)
+
diff --git a/src/test/modules/test_tidstore/meson.build b/src/test/modules/test_tidstore/meson.build
new file mode 100644
index 0000000000..31f2da7b61
--- /dev/null
+++ b/src/test/modules/test_tidstore/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_tidstore_sources = files(
+ 'test_tidstore.c',
+)
+
+if host_system == 'windows'
+ test_tidstore_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_tidstore',
+ '--FILEDESC', 'test_tidstore - test code for src/backend/access/common/tidstore.c',])
+endif
+
+test_tidstore = shared_module('test_tidstore',
+ test_tidstore_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += test_tidstore
+
+install_data(
+ 'test_tidstore.control',
+ 'test_tidstore--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'test_tidstore',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'test_tidstore',
+ ],
+ },
+}
diff --git a/src/test/modules/test_tidstore/sql/test_tidstore.sql b/src/test/modules/test_tidstore/sql/test_tidstore.sql
new file mode 100644
index 0000000000..03aea31815
--- /dev/null
+++ b/src/test/modules/test_tidstore/sql/test_tidstore.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_tidstore;
+
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
diff --git a/src/test/modules/test_tidstore/test_tidstore--1.0.sql b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
new file mode 100644
index 0000000000..47e9149900
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_tidstore/test_tidstore--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_tidstore" to load this file. \quit
+
+CREATE FUNCTION test_tidstore()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_tidstore/test_tidstore.c b/src/test/modules/test_tidstore/test_tidstore.c
new file mode 100644
index 0000000000..8659e6780e
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.c
@@ -0,0 +1,228 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_tidstore.c
+ * Test TidStore data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_tidstore/test_tidstore.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/tidstore.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "storage/lwlock.h"
+#include "utils/memutils.h"
+
+PG_MODULE_MAGIC;
+
+/* #define TEST_SHARED_TIDSTORE 1 */
+
+#define TEST_TIDSTORE_MAX_BYTES (2 * 1024 * 1024L) /* 2MB */
+
+PG_FUNCTION_INFO_V1(test_tidstore);
+
+static void
+check_tid(TidStore *ts, BlockNumber blkno, OffsetNumber off, bool expect)
+{
+ ItemPointerData tid;
+ bool found;
+
+ ItemPointerSet(&tid, blkno, off);
+
+ found = TidStoreIsMember(ts, &tid);
+
+ if (found != expect)
+ elog(ERROR, "TidStoreIsMember for TID (%u, %u) returned %d, expected %d",
+ blkno, off, found, expect);
+}
+
+static void
+test_basic(int max_offset)
+{
+#define TEST_TIDSTORE_NUM_BLOCKS 5
+#define TEST_TIDSTORE_NUM_OFFSETS 5
+
+ TidStore *ts;
+ TidStoreIter *iter;
+ TidStoreIterResult *iter_result;
+ BlockNumber blks[TEST_TIDSTORE_NUM_BLOCKS] = {
+ 0, MaxBlockNumber, MaxBlockNumber - 1, 1, MaxBlockNumber / 2,
+ };
+ BlockNumber blks_sorted[TEST_TIDSTORE_NUM_BLOCKS] = {
+ 0, 1, MaxBlockNumber / 2, MaxBlockNumber - 1, MaxBlockNumber
+ };
+ OffsetNumber offs[TEST_TIDSTORE_NUM_OFFSETS];
+ int blk_idx;
+
+#ifdef TEST_SHARED_TIDSTORE
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_tidstore");
+ dsa = dsa_create(tranche_id);
+
+ ts = TidStoreCreate(TEST_TIDSTORE_MAX_BYTES, max_offset, dsa);
+#else
+ ts = TidStoreCreate(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
+#endif
+
+ /* prepare the offset array */
+ offs[0] = FirstOffsetNumber;
+ offs[1] = FirstOffsetNumber + 1;
+ offs[2] = max_offset / 2;
+ offs[3] = max_offset - 1;
+ offs[4] = max_offset;
+
+ /* add tids */
+ for (int i = 0; i < TEST_TIDSTORE_NUM_BLOCKS; i++)
+ TidStoreSetBlockOffsets(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
+
+ /* lookup test */
+ for (OffsetNumber off = FirstOffsetNumber ; off < max_offset; off++)
+ {
+ bool expect = false;
+ for (int i = 0; i < TEST_TIDSTORE_NUM_OFFSETS; i++)
+ {
+ if (offs[i] == off)
+ {
+ expect = true;
+ break;
+ }
+ }
+
+ check_tid(ts, 0, off, expect);
+ check_tid(ts, 2, off, false);
+ check_tid(ts, MaxBlockNumber - 2, off, false);
+ check_tid(ts, MaxBlockNumber, off, expect);
+ }
+
+ /* test the number of tids */
+ if (TidStoreNumTids(ts) != (TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS))
+ elog(ERROR, "TidStoreNumTids returned " UINT64_FORMAT ", expected %d",
+ TidStoreNumTids(ts),
+ TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS);
+
+ /* iteration test */
+ iter = TidStoreBeginIterate(ts);
+ blk_idx = 0;
+ while ((iter_result = TidStoreIterateNext(iter)) != NULL)
+ {
+ /* check the returned block number */
+ if (blks_sorted[blk_idx] != iter_result->blkno)
+ elog(ERROR, "TidStoreIterateNext returned block number %u, expected %u",
+ iter_result->blkno, blks_sorted[blk_idx]);
+
+ /* check the returned offset numbers */
+ if (TEST_TIDSTORE_NUM_OFFSETS != iter_result->num_offsets)
+ elog(ERROR, "TidStoreIterateNext %u offsets, expected %u",
+ iter_result->num_offsets, TEST_TIDSTORE_NUM_OFFSETS);
+
+ for (int i = 0; i < iter_result->num_offsets; i++)
+ {
+ if (offs[i] != iter_result->offsets[i])
+ elog(ERROR, "TidStoreIterateNext offset number %u on block %u, expected %u",
+ iter_result->offsets[i], iter_result->blkno, offs[i]);
+ }
+
+ blk_idx++;
+ }
+
+ if (blk_idx != TEST_TIDSTORE_NUM_BLOCKS)
+ elog(ERROR, "TidStoreIterateNext returned %d blocks, expected %d",
+ blk_idx, TEST_TIDSTORE_NUM_BLOCKS);
+
+ /* remove all tids */
+ TidStoreReset(ts);
+
+ /* test the number of tids */
+ if (TidStoreNumTids(ts) != 0)
+ elog(ERROR, "TidStoreNumTids on empty store returned non-zero");
+
+ /* lookup test for empty store */
+ for (OffsetNumber off = FirstOffsetNumber ; off < MaxHeapTuplesPerPage;
+ off++)
+ {
+ check_tid(ts, 0, off, false);
+ check_tid(ts, 2, off, false);
+ check_tid(ts, MaxBlockNumber - 2, off, false);
+ check_tid(ts, MaxBlockNumber, off, false);
+ }
+
+ TidStoreDestroy(ts);
+
+#ifdef TEST_SHARED_TIDSTORE
+ dsa_detach(dsa);
+#endif
+}
+
+static void
+test_empty(void)
+{
+ TidStore *ts;
+ TidStoreIter *iter;
+ ItemPointerData tid;
+
+#ifdef TEST_SHARED_TIDSTORE
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_tidstore");
+ dsa = dsa_create(tranche_id);
+
+ ts = TidStoreCreate(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, dsa);
+#else
+ ts = TidStoreCreate(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
+#endif
+
+ elog(NOTICE, "testing empty tidstore");
+
+ ItemPointerSet(&tid, 0, FirstOffsetNumber);
+ if (TidStoreIsMember(ts, &tid))
+ elog(ERROR, "TidStoreIsMember for TID (%u,%u) on empty store returned true",
+ 0, FirstOffsetNumber);
+
+ ItemPointerSet(&tid, MaxBlockNumber, MaxOffsetNumber);
+ if (TidStoreIsMember(ts, &tid))
+ elog(ERROR, "TidStoreIsMember for TID (%u,%u) on empty store returned true",
+ MaxBlockNumber, MaxOffsetNumber);
+
+ if (TidStoreNumTids(ts) != 0)
+ elog(ERROR, "TidStoreNumTids on empty store returned non-zero");
+
+ if (TidStoreIsFull(ts))
+ elog(ERROR, "TidStoreIsFull on empty store returned true");
+
+ iter = TidStoreBeginIterate(ts);
+
+ if (TidStoreIterateNext(iter) != NULL)
+ elog(ERROR, "TidStoreIterateNext on empty store returned TIDs");
+
+ TidStoreEndIterate(iter);
+
+ TidStoreDestroy(ts);
+
+#ifdef TEST_SHARED_TIDSTORE
+ dsa_detach(dsa);
+#endif
+}
+
+Datum
+test_tidstore(PG_FUNCTION_ARGS)
+{
+ test_empty();
+
+ elog(NOTICE, "testing basic operations");
+ test_basic(MaxHeapTuplesPerPage);
+ test_basic(10);
+ test_basic(MaxHeapTuplesPerPage * 2);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_tidstore/test_tidstore.control b/src/test/modules/test_tidstore/test_tidstore.control
new file mode 100644
index 0000000000..9b6bd4638f
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.control
@@ -0,0 +1,4 @@
+comment = 'Test code for tidstore'
+default_version = '1.0'
+module_pathname = '$libdir/test_tidstore'
+relocatable = true
--
2.31.1
[application/octet-stream] v30-0005-Tool-for-measuring-radix-tree-and-tidstore-perfo.patch (24.7K, ../../CAD21AoC+kjqkSnLLcbqtu=Td5HSx2H+iAwNHCeZcWqjy1w7k5g@mail.gmail.com/9-v30-0005-Tool-for-measuring-radix-tree-and-tidstore-perfo.patch)
download | inline diff:
From 1feaf4249814a4bb7c5683649130b16cf3e5c754 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 16 Sep 2022 11:57:03 +0900
Subject: [PATCH v30 05/11] Tool for measuring radix tree and tidstore
performance
Includes Meson support, but commented out to avoid warnings
XXX: Not for commit
---
contrib/bench_radix_tree/Makefile | 21 +
.../bench_radix_tree--1.0.sql | 88 +++
contrib/bench_radix_tree/bench_radix_tree.c | 747 ++++++++++++++++++
.../bench_radix_tree/bench_radix_tree.control | 6 +
contrib/bench_radix_tree/expected/bench.out | 13 +
contrib/bench_radix_tree/meson.build | 33 +
contrib/bench_radix_tree/sql/bench.sql | 16 +
contrib/meson.build | 1 +
8 files changed, 925 insertions(+)
create mode 100644 contrib/bench_radix_tree/Makefile
create mode 100644 contrib/bench_radix_tree/bench_radix_tree--1.0.sql
create mode 100644 contrib/bench_radix_tree/bench_radix_tree.c
create mode 100644 contrib/bench_radix_tree/bench_radix_tree.control
create mode 100644 contrib/bench_radix_tree/expected/bench.out
create mode 100644 contrib/bench_radix_tree/meson.build
create mode 100644 contrib/bench_radix_tree/sql/bench.sql
diff --git a/contrib/bench_radix_tree/Makefile b/contrib/bench_radix_tree/Makefile
new file mode 100644
index 0000000000..952bb0ceae
--- /dev/null
+++ b/contrib/bench_radix_tree/Makefile
@@ -0,0 +1,21 @@
+# contrib/bench_radix_tree/Makefile
+
+MODULE_big = bench_radix_tree
+OBJS = \
+ bench_radix_tree.o
+
+EXTENSION = bench_radix_tree
+DATA = bench_radix_tree--1.0.sql
+
+REGRESS = bench_fixed_height
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/bench_radix_tree
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
new file mode 100644
index 0000000000..ad66265e23
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
@@ -0,0 +1,88 @@
+/* contrib/bench_radix_tree/bench_radix_tree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION bench_radix_tree" to load this file. \quit
+
+create function bench_shuffle_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_seq_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_load_random_int(
+cnt int8,
+OUT mem_allocated int8,
+OUT load_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_search_random_nodes(
+cnt int8,
+filter_str text DEFAULT NULL,
+OUT mem_allocated int8,
+OUT load_ms int8,
+OUT search_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C VOLATILE PARALLEL UNSAFE;
+
+create function bench_fixed_height_search(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT rt_search_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_node128_load(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_sparseload_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_tidstore_load(
+minblk int4,
+maxblk int4,
+OUT mem_allocated int8,
+OUT load_ms int8,
+OUT iter_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
new file mode 100644
index 0000000000..63e842395d
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -0,0 +1,747 @@
+/*-------------------------------------------------------------------------
+ *
+ * bench_radix_tree.c
+ *
+ * Copyright (c) 2016-2022, PostgreSQL Global Development Group
+ *
+ * contrib/bench_radix_tree/bench_radix_tree.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/tidstore.h"
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "funcapi.h"
+#include "lib/radixtree.h"
+#include <math.h>
+#include "miscadmin.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/timestamp.h"
+
+PG_MODULE_MAGIC;
+
+/* run benchmark also for binary-search case? */
+/* #define MEASURE_BINARY_SEARCH 1 */
+
+#define TIDS_PER_BLOCK_FOR_LOAD 30
+#define TIDS_PER_BLOCK_FOR_LOOKUP 50
+
+//#define RT_DEBUG
+#define RT_PREFIX rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE uint64
+// WIP: compiles with warnings because rt_attach is defined but not used
+// #define RT_SHMEM
+#include "lib/radixtree.h"
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+ return tree->ctl->num_keys;
+}
+
+
+PG_FUNCTION_INFO_V1(bench_seq_search);
+PG_FUNCTION_INFO_V1(bench_shuffle_search);
+PG_FUNCTION_INFO_V1(bench_load_random_int);
+PG_FUNCTION_INFO_V1(bench_fixed_height_search);
+PG_FUNCTION_INFO_V1(bench_search_random_nodes);
+PG_FUNCTION_INFO_V1(bench_node128_load);
+PG_FUNCTION_INFO_V1(bench_tidstore_load);
+
+static uint64
+tid_to_key_off(ItemPointer tid, uint32 *off)
+{
+ uint64 upper;
+ uint32 shift = pg_ceil_log2_32(MaxHeapTuplesPerPage);
+ int64 tid_i;
+
+ Assert(ItemPointerGetOffsetNumber(tid) < MaxHeapTuplesPerPage);
+
+ tid_i = ItemPointerGetOffsetNumber(tid);
+ tid_i |= (uint64) ItemPointerGetBlockNumber(tid) << shift;
+
+ /* log(sizeof(uint64) * BITS_PER_BYTE, 2) = log(64, 2) = 6 */
+ *off = tid_i & ((1 << 6) - 1);
+ upper = tid_i >> 6;
+ Assert(*off < (sizeof(uint64) * BITS_PER_BYTE));
+
+ Assert(*off < 64);
+
+ return upper;
+}
+
+static int
+shuffle_randrange(pg_prng_state *state, int lower, int upper)
+{
+ return (int) floor(pg_prng_double(state) * ((upper - lower) + 0.999999)) + lower;
+}
+
+/* Naive Fisher-Yates implementation*/
+static void
+shuffle_itemptrs(ItemPointer itemptr, uint64 nitems)
+{
+ /* reproducability */
+ pg_prng_state state;
+
+ pg_prng_seed(&state, 0);
+
+ for (int i = 0; i < nitems - 1; i++)
+ {
+ int j = shuffle_randrange(&state, i, nitems - 1);
+ ItemPointerData t = itemptr[j];
+
+ itemptr[j] = itemptr[i];
+ itemptr[i] = t;
+ }
+}
+
+static ItemPointer
+generate_tids(BlockNumber minblk, BlockNumber maxblk, int ntids_per_blk, uint64 *ntids_p,
+ bool random_block)
+{
+ ItemPointer tids;
+ uint64 maxitems;
+ uint64 ntids = 0;
+ pg_prng_state state;
+
+ maxitems = (maxblk - minblk + 1) * ntids_per_blk;
+ tids = MemoryContextAllocHuge(TopTransactionContext,
+ sizeof(ItemPointerData) * maxitems);
+
+ if (random_block)
+ pg_prng_seed(&state, 0x9E3779B185EBCA87);
+
+ for (BlockNumber blk = minblk; blk < maxblk; blk++)
+ {
+ if (random_block && !pg_prng_bool(&state))
+ continue;
+
+ for (OffsetNumber off = FirstOffsetNumber;
+ off <= ntids_per_blk; off++)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ ItemPointerSetBlockNumber(&(tids[ntids]), blk);
+ ItemPointerSetOffsetNumber(&(tids[ntids]), off);
+
+ ntids++;
+ }
+ }
+
+ *ntids_p = ntids;
+ return tids;
+}
+
+#ifdef MEASURE_BINARY_SEARCH
+static int
+vac_cmp_itemptr(const void *left, const void *right)
+{
+ BlockNumber lblk,
+ rblk;
+ OffsetNumber loff,
+ roff;
+
+ lblk = ItemPointerGetBlockNumber((ItemPointer) left);
+ rblk = ItemPointerGetBlockNumber((ItemPointer) right);
+
+ if (lblk < rblk)
+ return -1;
+ if (lblk > rblk)
+ return 1;
+
+ loff = ItemPointerGetOffsetNumber((ItemPointer) left);
+ roff = ItemPointerGetOffsetNumber((ItemPointer) right);
+
+ if (loff < roff)
+ return -1;
+ if (loff > roff)
+ return 1;
+
+ return 0;
+}
+#endif
+
+Datum
+bench_tidstore_load(PG_FUNCTION_ARGS)
+{
+ BlockNumber minblk = PG_GETARG_INT32(0);
+ BlockNumber maxblk = PG_GETARG_INT32(1);
+ TidStore *ts;
+ TidStoreIter *iter;
+ TidStoreIterResult *result;
+ OffsetNumber *offs;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_ms;
+ int64 iter_ms;
+ TupleDesc tupdesc;
+ Datum values[3];
+ bool nulls[3] = {false};
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ offs = palloc(sizeof(OffsetNumber) * TIDS_PER_BLOCK_FOR_LOAD);
+ for (int i = 0; i < TIDS_PER_BLOCK_FOR_LOAD; i++)
+ offs[i] = i + 1; /* FirstOffsetNumber is 1 */
+
+ ts = TidStoreCreate(1 * 1024L * 1024L * 1024L, MaxHeapTuplesPerPage, NULL);
+
+ /* load tids */
+ start_time = GetCurrentTimestamp();
+ for (BlockNumber blkno = minblk; blkno < maxblk; blkno++)
+ TidStoreSetBlockOffsets(ts, blkno, offs, TIDS_PER_BLOCK_FOR_LOAD);
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_ms = secs * 1000 + usecs / 1000;
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* iterate through tids */
+ iter = TidStoreBeginIterate(ts);
+ start_time = GetCurrentTimestamp();
+ while ((result = TidStoreIterateNext(iter)) != NULL)
+ ;
+ TidStoreEndIterate(iter);
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ iter_ms = secs * 1000 + usecs / 1000;
+
+ values[0] = Int64GetDatum(TidStoreMemoryUsage(ts));
+ values[1] = Int64GetDatum(load_ms);
+ values[2] = Int64GetDatum(iter_ms);
+
+ TidStoreDestroy(ts);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+static Datum
+bench_search(FunctionCallInfo fcinfo, bool shuffle)
+{
+ BlockNumber minblk = PG_GETARG_INT32(0);
+ BlockNumber maxblk = PG_GETARG_INT32(1);
+ bool random_block = PG_GETARG_BOOL(2);
+ rt_radix_tree *rt = NULL;
+ uint64 ntids;
+ uint64 key;
+ uint64 last_key = PG_UINT64_MAX;
+ uint64 val = 0;
+ ItemPointer tids;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_load_ms,
+ rt_search_ms;
+ Datum values[7];
+ bool nulls[7];
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ tids = generate_tids(minblk, maxblk, TIDS_PER_BLOCK_FOR_LOAD, &ntids, random_block);
+
+ /* measure the load time of the radix tree */
+ rt = rt_create(CurrentMemoryContext);
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ uint32 off;
+
+ CHECK_FOR_INTERRUPTS();
+
+ key = tid_to_key_off(tid, &off);
+
+ if (last_key != PG_UINT64_MAX && last_key != key)
+ {
+ rt_set(rt, last_key, &val);
+ val = 0;
+ }
+
+ last_key = key;
+ val |= (uint64) 1 << off;
+ }
+ if (last_key != PG_UINT64_MAX)
+ rt_set(rt, last_key, &val);
+
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_load_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+ if (shuffle)
+ shuffle_itemptrs(tids, ntids);
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* meaure the serach time of the radix tree */
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ uint32 off;
+ volatile bool ret; /* prevent calling rt_search from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ key = tid_to_key_off(tid, &off);
+
+ ret = rt_search(rt, key, &val);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_search_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int64GetDatum(rt_num_entries(rt));
+ values[1] = Int64GetDatum(rt_memory_usage(rt));
+ values[2] = Int64GetDatum(sizeof(ItemPointerData) * ntids);
+ values[3] = Int64GetDatum(rt_load_ms);
+ nulls[4] = true; /* ar_load_ms */
+ values[5] = Int64GetDatum(rt_search_ms);
+ nulls[6] = true; /* ar_search_ms */
+
+#ifdef MEASURE_BINARY_SEARCH
+ {
+ ItemPointer itemptrs = NULL;
+
+ int64 ar_load_ms,
+ ar_search_ms;
+
+ /* measure the load time of the array */
+ itemptrs = MemoryContextAllocHuge(CurrentMemoryContext,
+ sizeof(ItemPointerData) * ntids);
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointerSetBlockNumber(&(itemptrs[i]),
+ ItemPointerGetBlockNumber(&(tids[i])));
+ ItemPointerSetOffsetNumber(&(itemptrs[i]),
+ ItemPointerGetOffsetNumber(&(tids[i])));
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ ar_load_ms = secs * 1000 + usecs / 1000;
+
+ /* next, measure the serach time of the array */
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ volatile bool ret; /* prevent calling bsearch from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ ret = bsearch((void *) tid,
+ (void *) itemptrs,
+ ntids,
+ sizeof(ItemPointerData),
+ vac_cmp_itemptr);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ ar_search_ms = secs * 1000 + usecs / 1000;
+
+ /* set the result */
+ nulls[4] = false;
+ values[4] = Int64GetDatum(ar_load_ms);
+ nulls[6] = false;
+ values[6] = Int64GetDatum(ar_search_ms);
+ }
+#endif
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_seq_search(PG_FUNCTION_ARGS)
+{
+ return bench_search(fcinfo, false);
+}
+
+Datum
+bench_shuffle_search(PG_FUNCTION_ARGS)
+{
+ return bench_search(fcinfo, true);
+}
+
+Datum
+bench_load_random_int(PG_FUNCTION_ARGS)
+{
+ uint64 cnt = (uint64) PG_GETARG_INT64(0);
+ rt_radix_tree *rt;
+ pg_prng_state state;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_time_ms;
+ Datum values[2];
+ bool nulls[2];
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ pg_prng_seed(&state, 0);
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 key = pg_prng_uint64(&state);
+
+ rt_set(rt, key, &key);
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_time_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int64GetDatum(rt_memory_usage(rt));
+ values[1] = Int64GetDatum(load_time_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+/* copy of splitmix64() */
+static uint64
+hash64(uint64 x)
+{
+ x ^= x >> 30;
+ x *= UINT64CONST(0xbf58476d1ce4e5b9);
+ x ^= x >> 27;
+ x *= UINT64CONST(0x94d049bb133111eb);
+ x ^= x >> 31;
+ return x;
+}
+
+/* attempts to have a relatively even population of node kinds */
+Datum
+bench_search_random_nodes(PG_FUNCTION_ARGS)
+{
+ uint64 cnt = (uint64) PG_GETARG_INT64(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_time_ms;
+ int64 search_time_ms;
+ Datum values[3] = {0};
+ bool nulls[3] = {0};
+ /* from trial and error */
+ uint64 filter = (((uint64) 0x7F<<32) | (0x07<<24) | (0xFF<<16) | 0xFF);
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (!PG_ARGISNULL(1))
+ {
+ char *filter_str = text_to_cstring(PG_GETARG_TEXT_P(1));
+
+ if (sscanf(filter_str, "0x%lX", &filter) == 0)
+ elog(ERROR, "invalid filter string %s", filter_str);
+ }
+ elog(NOTICE, "bench with filter 0x%lX", filter);
+
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 hash = hash64(i);
+ uint64 key = hash & filter;
+
+ rt_set(rt, key, &key);
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_time_ms = secs * 1000 + usecs / 1000;
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 hash = hash64(i);
+ uint64 key = hash & filter;
+ uint64 val;
+ volatile bool ret; /* prevent calling rt_search from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ ret = rt_search(rt, key, &val);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ search_time_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ values[0] = Int64GetDatum(rt_memory_usage(rt));
+ values[1] = Int64GetDatum(load_time_ms);
+ values[2] = Int64GetDatum(search_time_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_fixed_height_search(PG_FUNCTION_ARGS)
+{
+ int fanout = PG_GETARG_INT32(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_load_ms,
+ rt_search_ms;
+ Datum values[5];
+ bool nulls[5];
+
+ /* test boundary between vector and iteration */
+ const int n_keys = 5 * 16 * 16 * 16 * 16;
+ uint64 r,
+ h,
+ i,
+ j,
+ k;
+ uint64 key_id;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+
+ key_id = 0;
+
+ /*
+ * lower nodes have limited fanout, the top is only limited by
+ * bits-per-byte
+ */
+ for (r = 1;; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ for (i = 1; i <= fanout; i++)
+ {
+ for (j = 1; j <= fanout; j++)
+ {
+ for (k = 1; k <= fanout; k++)
+ {
+ uint64 key;
+
+ key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+ CHECK_FOR_INTERRUPTS();
+
+ key_id++;
+ if (key_id > n_keys)
+ goto finish_set;
+
+ rt_set(rt, key, &key_id);
+ }
+ }
+ }
+ }
+ }
+finish_set:
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_load_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ /* meaure the search time of the radix tree */
+ start_time = GetCurrentTimestamp();
+
+ key_id = 0;
+ for (r = 1;; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ for (i = 1; i <= fanout; i++)
+ {
+ for (j = 1; j <= fanout; j++)
+ {
+ for (k = 1; k <= fanout; k++)
+ {
+ uint64 key,
+ val;
+
+ key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+ CHECK_FOR_INTERRUPTS();
+
+ key_id++;
+ if (key_id > n_keys)
+ goto finish_search;
+
+ rt_search(rt, key, &val);
+ }
+ }
+ }
+ }
+ }
+finish_search:
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_search_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int32GetDatum(fanout);
+ values[1] = Int64GetDatum(rt_num_entries(rt));
+ values[2] = Int64GetDatum(rt_memory_usage(rt));
+ values[3] = Int64GetDatum(rt_load_ms);
+ values[4] = Int64GetDatum(rt_search_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_node128_load(PG_FUNCTION_ARGS)
+{
+ int fanout = PG_GETARG_INT32(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_sparseload_ms;
+ Datum values[5];
+ bool nulls[5];
+
+ uint64 r,
+ h;
+ uint64 key_id;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ rt = rt_create(CurrentMemoryContext);
+
+ key_id = 0;
+
+ for (r = 1; r <= fanout; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (h);
+
+ key_id++;
+ rt_set(rt, key, &key_id);
+ }
+ }
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ /* measure sparse deletion and re-loading */
+ start_time = GetCurrentTimestamp();
+
+ for (int t = 0; t<10000; t++)
+ {
+ /* delete one key in each leaf */
+ for (r = 1; r <= fanout; r++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (fanout);
+
+ rt_delete(rt, key);
+ }
+
+ /* add them all back */
+ key_id = 0;
+ for (r = 1; r <= fanout; r++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (fanout);
+
+ key_id++;
+ rt_set(rt, key, &key_id);
+ }
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_sparseload_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int32GetDatum(fanout);
+ values[1] = Int64GetDatum(rt_num_entries(rt));
+ values[2] = Int64GetDatum(rt_memory_usage(rt));
+ values[3] = Int64GetDatum(rt_sparseload_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+/* to silence warnings about unused iter functions */
+static void pg_attribute_unused()
+stub_iter()
+{
+ rt_radix_tree *rt;
+ rt_iter *iter;
+ uint64 key = 1;
+ uint64 value = 1;
+
+ rt = rt_create(CurrentMemoryContext);
+
+ iter = rt_begin_iterate(rt);
+ rt_iterate_next(iter, &key, &value);
+ rt_end_iterate(iter);
+}
diff --git a/contrib/bench_radix_tree/bench_radix_tree.control b/contrib/bench_radix_tree/bench_radix_tree.control
new file mode 100644
index 0000000000..1d988e6c9a
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.control
@@ -0,0 +1,6 @@
+# bench_radix_tree extension
+comment = 'benchmark suits for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/bench_radix_tree'
+relocatable = true
+trusted = true
diff --git a/contrib/bench_radix_tree/expected/bench.out b/contrib/bench_radix_tree/expected/bench.out
new file mode 100644
index 0000000000..60c303892e
--- /dev/null
+++ b/contrib/bench_radix_tree/expected/bench.out
@@ -0,0 +1,13 @@
+create extension bench_radix_tree;
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/bench_radix_tree/meson.build b/contrib/bench_radix_tree/meson.build
new file mode 100644
index 0000000000..332c1ae7df
--- /dev/null
+++ b/contrib/bench_radix_tree/meson.build
@@ -0,0 +1,33 @@
+bench_radix_tree_sources = files(
+ 'bench_radix_tree.c',
+)
+
+if host_system == 'windows'
+ bench_radix_tree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'bench_radix_tree',
+ '--FILEDESC', 'bench_radix_tree - performance test code for radix tree',])
+endif
+
+bench_radix_tree = shared_module('bench_radix_tree',
+ bench_radix_tree_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += bench_radix_tree
+
+install_data(
+ 'bench_radix_tree.control',
+ 'bench_radix_tree--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'bench_radix_tree',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'bench_radix_tree',
+ ],
+ },
+}
diff --git a/contrib/bench_radix_tree/sql/bench.sql b/contrib/bench_radix_tree/sql/bench.sql
new file mode 100644
index 0000000000..a46018c9d4
--- /dev/null
+++ b/contrib/bench_radix_tree/sql/bench.sql
@@ -0,0 +1,16 @@
+create extension bench_radix_tree;
+
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/meson.build b/contrib/meson.build
index bd4a57c43c..421d469f8c 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,6 +12,7 @@ subdir('amcheck')
subdir('auth_delay')
subdir('auto_explain')
subdir('basic_archive')
+subdir('bench_radix_tree')
subdir('bloom')
subdir('basebackup_to_shell')
subdir('bool_plperl')
--
2.31.1
[application/octet-stream] v30-0002-Move-some-bitmap-logic-out-of-bitmapset.c.patch (6.1K, ../../CAD21AoC+kjqkSnLLcbqtu=Td5HSx2H+iAwNHCeZcWqjy1w7k5g@mail.gmail.com/10-v30-0002-Move-some-bitmap-logic-out-of-bitmapset.c.patch)
download | inline diff:
From 756f0a7a1f3e9030ddc68ae635baa25c4a310b4d Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 6 Dec 2022 13:39:41 +0700
Subject: [PATCH v30 02/11] Move some bitmap logic out of bitmapset.c
Add pg_rightmost_one32/64 functions. This functionality was previously
private to bitmapset.c as the RIGHTMOST_ONE macro. It has practical
use in other contexts, so move to pg_bitutils.h.
Also move the logic for selecting appropriate pg_bitutils functions
based on word size to bitmapset.h for wider visibility and add
appropriate selection for bmw_rightmost_one().
Since the previous macro relied on casting to signedbitmapword,
and the new functions do not, remove that typedef.
Design input and review by Tom Lane
Discussion: https://www.postgresql.org/message-id/CAFBsxsFW2JjTo58jtDB%2B3sZhxMx3t-3evew8%3DAcr%2BGGhC%2BkFaA%40mail.gmail.com
---
src/backend/nodes/bitmapset.c | 36 ++------------------------------
src/include/nodes/bitmapset.h | 16 ++++++++++++--
src/include/port/pg_bitutils.h | 31 +++++++++++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 -
4 files changed, 47 insertions(+), 37 deletions(-)
diff --git a/src/backend/nodes/bitmapset.c b/src/backend/nodes/bitmapset.c
index 98660524ad..fcd8e2ccbc 100644
--- a/src/backend/nodes/bitmapset.c
+++ b/src/backend/nodes/bitmapset.c
@@ -32,39 +32,7 @@
#define BITMAPSET_SIZE(nwords) \
(offsetof(Bitmapset, words) + (nwords) * sizeof(bitmapword))
-/*----------
- * This is a well-known cute trick for isolating the rightmost one-bit
- * in a word. It assumes two's complement arithmetic. Consider any
- * nonzero value, and focus attention on the rightmost one. The value is
- * then something like
- * xxxxxx10000
- * where x's are unspecified bits. The two's complement negative is formed
- * by inverting all the bits and adding one. Inversion gives
- * yyyyyy01111
- * where each y is the inverse of the corresponding x. Incrementing gives
- * yyyyyy10000
- * and then ANDing with the original value gives
- * 00000010000
- * This works for all cases except original value = zero, where of course
- * we get zero.
- *----------
- */
-#define RIGHTMOST_ONE(x) ((signedbitmapword) (x) & -((signedbitmapword) (x)))
-
-#define HAS_MULTIPLE_ONES(x) ((bitmapword) RIGHTMOST_ONE(x) != (x))
-
-/* Select appropriate bit-twiddling functions for bitmap word size */
-#if BITS_PER_BITMAPWORD == 32
-#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w)
-#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w)
-#define bmw_popcount(w) pg_popcount32(w)
-#elif BITS_PER_BITMAPWORD == 64
-#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w)
-#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w)
-#define bmw_popcount(w) pg_popcount64(w)
-#else
-#error "invalid BITS_PER_BITMAPWORD"
-#endif
+#define HAS_MULTIPLE_ONES(x) (bmw_rightmost_one(x) != (x))
/*
@@ -1013,7 +981,7 @@ bms_first_member(Bitmapset *a)
{
int result;
- w = RIGHTMOST_ONE(w);
+ w = bmw_rightmost_one(w);
a->words[wordnum] &= ~w;
result = wordnum * BITS_PER_BITMAPWORD;
diff --git a/src/include/nodes/bitmapset.h b/src/include/nodes/bitmapset.h
index 3d2225e1ae..5f9a511b4a 100644
--- a/src/include/nodes/bitmapset.h
+++ b/src/include/nodes/bitmapset.h
@@ -38,13 +38,11 @@ struct List;
#define BITS_PER_BITMAPWORD 64
typedef uint64 bitmapword; /* must be an unsigned type */
-typedef int64 signedbitmapword; /* must be the matching signed type */
#else
#define BITS_PER_BITMAPWORD 32
typedef uint32 bitmapword; /* must be an unsigned type */
-typedef int32 signedbitmapword; /* must be the matching signed type */
#endif
@@ -75,6 +73,20 @@ typedef enum
BMS_MULTIPLE /* >1 member */
} BMS_Membership;
+/* Select appropriate bit-twiddling functions for bitmap word size */
+#if BITS_PER_BITMAPWORD == 32
+#define bmw_rightmost_one(w) pg_rightmost_one32(w)
+#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w)
+#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w)
+#define bmw_popcount(w) pg_popcount32(w)
+#elif BITS_PER_BITMAPWORD == 64
+#define bmw_rightmost_one(w) pg_rightmost_one64(w)
+#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w)
+#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w)
+#define bmw_popcount(w) pg_popcount64(w)
+#else
+#error "invalid BITS_PER_BITMAPWORD"
+#endif
/*
* function prototypes in nodes/bitmapset.c
diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h
index 158ef73a2b..bf7588e075 100644
--- a/src/include/port/pg_bitutils.h
+++ b/src/include/port/pg_bitutils.h
@@ -32,6 +32,37 @@ extern PGDLLIMPORT const uint8 pg_leftmost_one_pos[256];
extern PGDLLIMPORT const uint8 pg_rightmost_one_pos[256];
extern PGDLLIMPORT const uint8 pg_number_of_ones[256];
+/*----------
+ * This is a well-known cute trick for isolating the rightmost one-bit
+ * in a word. It assumes two's complement arithmetic. Consider any
+ * nonzero value, and focus attention on the rightmost one. The value is
+ * then something like
+ * xxxxxx10000
+ * where x's are unspecified bits. The two's complement negative is formed
+ * by inverting all the bits and adding one. Inversion gives
+ * yyyyyy01111
+ * where each y is the inverse of the corresponding x. Incrementing gives
+ * yyyyyy10000
+ * and then ANDing with the original value gives
+ * 00000010000
+ * This works for all cases except original value = zero, where of course
+ * we get zero.
+ *----------
+ */
+static inline uint32
+pg_rightmost_one32(uint32 word)
+{
+ int32 result = (int32) word & -((int32) word);
+ return (uint32) result;
+}
+
+static inline uint64
+pg_rightmost_one64(uint64 word)
+{
+ int64 result = (int64) word & -((int64) word);
+ return (uint64) result;
+}
+
/*
* pg_leftmost_one_pos32
* Returns the position of the most significant set bit in "word",
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 86a9303bf5..4a5e776703 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3675,7 +3675,6 @@ shmem_request_hook_type
shmem_startup_hook_type
sig_atomic_t
sigjmp_buf
-signedbitmapword
sigset_t
size_t
slist_head
--
2.31.1
[application/octet-stream] v30-0003-Add-a-macro-templatized-radix-tree.patch (119.7K, ../../CAD21AoC+kjqkSnLLcbqtu=Td5HSx2H+iAwNHCeZcWqjy1w7k5g@mail.gmail.com/11-v30-0003-Add-a-macro-templatized-radix-tree.patch)
download | inline diff:
From 87b21d222bc9e2b8bdbd6cb7c880d1f5a5192242 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 14 Sep 2022 12:38:51 +0000
Subject: [PATCH v30 03/11] Add a macro templatized radix tree.
The radix tree data structure is implemented based on the idea from
the paper "The Adaptive Radix Tree: ARTful Indexing for Main-Memory
Databases" by Viktor Leis, Alfons Kemper, and Thomas Neumann,
2013. There are some optimizations that are proposed in the ART paper
not yet implemented, particularly path compression and lazy path
expansion. For a better performance, the radix trees have to be
adjusted to the individual user-case at compile-time. So the radix
tree is implemented using a macro-templatized header file, which
generates functions and types based on a prefix and other parameters.
The key of radix tree is 64-bit unsigned integer but the caller can
specify the type of the value. Our main innovation implemented
in our radix tree implementation compared to the ART paper is
decoupling the notion of size class from kind. The size classes within
a given node kind have the same underlying type, but a variable number
of children/values. Nodes of different kinds necessarily belong to
different size classes. Growing from one node kind to another requires
special code for each case, but growing from one size class to another
within the same kind is basically just allocate + memcpy.
The radix tree can be created also on a DSA area. To handle
concurrency, we use a single reader-writer lock for the radix
tree. The current locking mechanism is not optimized for high
concurrency with mixed read-write workloads. In the future it might be
worthwhile to replace it with the Optimistic Lock Coupling or ROWEX
mentioned in the paper "The ART of Practical Synchronization" by the
same authors as the ART paper, 2016.
Later patches use this infrastructure to use such radix tree for
storing dead tuple TIDs during lazy vacuum.There are possible cases
where this could be useful (e.g., replacement for hash table for
shared buffer).
This includes a unit test module, in src/test/modules/test_radixtree.
Discussion: https://postgr.es/m/CAD21AoAfOZvmfR0j8VmZorZjL7RhTiQdVttNuC4W-Shdc2a-AA@mail.gmail.com
---
src/backend/utils/mmgr/dsa.c | 12 +
src/include/lib/radixtree.h | 2523 +++++++++++++++++
src/include/lib/radixtree_delete_impl.h | 122 +
src/include/lib/radixtree_insert_impl.h | 328 +++
src/include/lib/radixtree_iter_impl.h | 144 +
src/include/lib/radixtree_search_impl.h | 138 +
src/include/utils/dsa.h | 1 +
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_radixtree/.gitignore | 4 +
src/test/modules/test_radixtree/Makefile | 23 +
src/test/modules/test_radixtree/README | 7 +
.../expected/test_radixtree.out | 38 +
src/test/modules/test_radixtree/meson.build | 35 +
.../test_radixtree/sql/test_radixtree.sql | 7 +
.../test_radixtree/test_radixtree--1.0.sql | 8 +
.../modules/test_radixtree/test_radixtree.c | 712 +++++
.../test_radixtree/test_radixtree.control | 4 +
src/tools/pginclude/cpluspluscheck | 6 +
src/tools/pginclude/headerscheck | 6 +
20 files changed, 4120 insertions(+)
create mode 100644 src/include/lib/radixtree.h
create mode 100644 src/include/lib/radixtree_delete_impl.h
create mode 100644 src/include/lib/radixtree_insert_impl.h
create mode 100644 src/include/lib/radixtree_iter_impl.h
create mode 100644 src/include/lib/radixtree_search_impl.h
create mode 100644 src/test/modules/test_radixtree/.gitignore
create mode 100644 src/test/modules/test_radixtree/Makefile
create mode 100644 src/test/modules/test_radixtree/README
create mode 100644 src/test/modules/test_radixtree/expected/test_radixtree.out
create mode 100644 src/test/modules/test_radixtree/meson.build
create mode 100644 src/test/modules/test_radixtree/sql/test_radixtree.sql
create mode 100644 src/test/modules/test_radixtree/test_radixtree--1.0.sql
create mode 100644 src/test/modules/test_radixtree/test_radixtree.c
create mode 100644 src/test/modules/test_radixtree/test_radixtree.control
diff --git a/src/backend/utils/mmgr/dsa.c b/src/backend/utils/mmgr/dsa.c
index f5a62061a3..80555aefff 100644
--- a/src/backend/utils/mmgr/dsa.c
+++ b/src/backend/utils/mmgr/dsa.c
@@ -1024,6 +1024,18 @@ dsa_set_size_limit(dsa_area *area, size_t limit)
LWLockRelease(DSA_AREA_LOCK(area));
}
+size_t
+dsa_get_total_size(dsa_area *area)
+{
+ size_t size;
+
+ LWLockAcquire(DSA_AREA_LOCK(area), LW_SHARED);
+ size = area->control->total_segment_size;
+ LWLockRelease(DSA_AREA_LOCK(area));
+
+ return size;
+}
+
/*
* Aggressively free all spare memory in the hope of returning DSM segments to
* the operating system.
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
new file mode 100644
index 0000000000..2e3963c3d5
--- /dev/null
+++ b/src/include/lib/radixtree.h
@@ -0,0 +1,2523 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree.h
+ * Template for adaptive radix tree.
+ *
+ * This module employs the idea from the paper "The Adaptive Radix Tree: ARTful
+ * Indexing for Main-Memory Databases" by Viktor Leis, Alfons Kemper, and Thomas
+ * Neumann, 2013. The radix tree uses adaptive node sizes, a small number of node
+ * types, each with a different numbers of elements. Depending on the number of
+ * children, the appropriate node type is used.
+ *
+ * WIP: notes about traditional radix tree trading off span vs height...
+ *
+ * There are two kinds of nodes, inner nodes and leaves. Inner nodes
+ * map partial keys to child pointers.
+ *
+ * The ART paper mentions three ways to implement leaves:
+ *
+ * "- Single-value leaves: The values are stored using an addi-
+ * tional leaf node type which stores one value.
+ * - Multi-value leaves: The values are stored in one of four
+ * different leaf node types, which mirror the structure of
+ * inner nodes, but contain values instead of pointers.
+ * - Combined pointer/value slots: If values fit into point-
+ * ers, no separate node types are necessary. Instead, each
+ * pointer storage location in an inner node can either
+ * store a pointer or a value."
+ *
+ * We chose "multi-value leaves" to avoid the additional pointer traversal
+ * required by "single-value leaves"
+ *
+ * For simplicity, the key is assumed to be 64-bit unsigned integer. The
+ * tree doesn't need to contain paths where the highest bytes of all keys
+ * are zero. That way, the tree's height adapts to the distribution of keys.
+ *
+ * TODO: In the future it might be worthwhile to offer configurability of
+ * leaf implementation for different use cases. Single-values leaves would
+ * give more flexibility in key type, including variable-length keys.
+ *
+ * There are some optimizations not yet implemented, particularly path
+ * compression and lazy path expansion.
+ *
+ * To handle concurrency, we use a single reader-writer lock for the radix
+ * tree. The radix tree is exclusively locked during write operations such
+ * as RT_SET() and RT_DELETE(), and shared locked during read operations
+ * such as RT_SEARCH(). An iteration also holds the shared lock on the radix
+ * tree until it is completed.
+ *
+ * TODO: The current locking mechanism is not optimized for high concurrency
+ * with mixed read-write workloads. In the future it might be worthwhile
+ * to replace it with the Optimistic Lock Coupling or ROWEX mentioned in
+ * the paper "The ART of Practical Synchronization" by the same authors as
+ * the ART paper, 2016.
+ *
+ * WIP: the radix tree nodes don't shrink.
+ *
+ * To generate a radix tree and associated functions for a use case several
+ * macros have to be #define'ed before this file is included. Including
+ * the file #undef's all those, so a new radix tree can be generated
+ * afterwards.
+ * The relevant parameters are:
+ * - RT_PREFIX - prefix for all symbol names generated. A prefix of 'foo'
+ * will result in radix tree type 'foo_radix_tree' and functions like
+ * 'foo_create'/'foo_free' and so forth.
+ * - RT_DECLARE - if defined function prototypes and type declarations are
+ * generated
+ * - RT_DEFINE - if defined function definitions are generated
+ * - RT_SCOPE - in which scope (e.g. extern, static inline) do function
+ * declarations reside
+ * - RT_VALUE_TYPE - the type of the value.
+ *
+ * Optional parameters:
+ * - RT_SHMEM - if defined, the radix tree is created in the DSA area
+ * so that multiple processes can access it simultaneously.
+ * - RT_DEBUG - if defined add stats tracking and debugging functions
+ *
+ * Interface
+ * ---------
+ *
+ * RT_CREATE - Create a new, empty radix tree
+ * RT_FREE - Free the radix tree
+ * RT_SEARCH - Search a key-value pair
+ * RT_SET - Set a key-value pair
+ * RT_BEGIN_ITERATE - Begin iterating through all key-value pairs
+ * RT_ITERATE_NEXT - Return next key-value pair, if any
+ * RT_END_ITERATE - End iteration
+ * RT_MEMORY_USAGE - Get the memory usage
+ *
+ * Interface for Shared Memory
+ * ---------
+ *
+ * RT_ATTACH - Attach to the radix tree
+ * RT_DETACH - Detach from the radix tree
+ * RT_GET_HANDLE - Return the handle of the radix tree
+ *
+ * Optional Interface
+ * ---------
+ *
+ * RT_DELETE - Delete a key-value pair. Declared/define if RT_USE_DELETE is defined
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/include/lib/radixtree.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/stringinfo.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "port/pg_bitutils.h"
+#include "port/simd.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/* helpers */
+#define RT_MAKE_PREFIX(a) CppConcat(a,_)
+#define RT_MAKE_NAME(name) RT_MAKE_NAME_(RT_MAKE_PREFIX(RT_PREFIX),name)
+#define RT_MAKE_NAME_(a,b) CppConcat(a,b)
+
+/* function declarations */
+#define RT_CREATE RT_MAKE_NAME(create)
+#define RT_FREE RT_MAKE_NAME(free)
+#define RT_SEARCH RT_MAKE_NAME(search)
+#ifdef RT_SHMEM
+#define RT_ATTACH RT_MAKE_NAME(attach)
+#define RT_DETACH RT_MAKE_NAME(detach)
+#define RT_GET_HANDLE RT_MAKE_NAME(get_handle)
+#endif
+#define RT_SET RT_MAKE_NAME(set)
+#define RT_BEGIN_ITERATE RT_MAKE_NAME(begin_iterate)
+#define RT_ITERATE_NEXT RT_MAKE_NAME(iterate_next)
+#define RT_END_ITERATE RT_MAKE_NAME(end_iterate)
+#ifdef RT_USE_DELETE
+#define RT_DELETE RT_MAKE_NAME(delete)
+#endif
+#define RT_MEMORY_USAGE RT_MAKE_NAME(memory_usage)
+#ifdef RT_DEBUG
+#define RT_DUMP RT_MAKE_NAME(dump)
+#define RT_DUMP_NODE RT_MAKE_NAME(dump_node)
+#define RT_DUMP_SEARCH RT_MAKE_NAME(dump_search)
+#define RT_STATS RT_MAKE_NAME(stats)
+#endif
+
+/* internal helper functions (no externally visible prototypes) */
+#define RT_NEW_ROOT RT_MAKE_NAME(new_root)
+#define RT_ALLOC_NODE RT_MAKE_NAME(alloc_node)
+#define RT_INIT_NODE RT_MAKE_NAME(init_node)
+#define RT_FREE_NODE RT_MAKE_NAME(free_node)
+#define RT_FREE_RECURSE RT_MAKE_NAME(free_recurse)
+#define RT_EXTEND_UP RT_MAKE_NAME(extend_up)
+#define RT_EXTEND_DOWN RT_MAKE_NAME(extend_down)
+#define RT_SWITCH_NODE_KIND RT_MAKE_NAME(grow_node_kind)
+#define RT_COPY_NODE RT_MAKE_NAME(copy_node)
+#define RT_REPLACE_NODE RT_MAKE_NAME(replace_node)
+#define RT_PTR_GET_LOCAL RT_MAKE_NAME(ptr_get_local)
+#define RT_PTR_ALLOC_IS_VALID RT_MAKE_NAME(ptr_stored_is_valid)
+#define RT_NODE_3_SEARCH_EQ RT_MAKE_NAME(node_3_search_eq)
+#define RT_NODE_32_SEARCH_EQ RT_MAKE_NAME(node_32_search_eq)
+#define RT_NODE_3_GET_INSERTPOS RT_MAKE_NAME(node_3_get_insertpos)
+#define RT_NODE_32_GET_INSERTPOS RT_MAKE_NAME(node_32_get_insertpos)
+#define RT_CHUNK_CHILDREN_ARRAY_SHIFT RT_MAKE_NAME(chunk_children_array_shift)
+#define RT_CHUNK_VALUES_ARRAY_SHIFT RT_MAKE_NAME(chunk_values_array_shift)
+#define RT_CHUNK_CHILDREN_ARRAY_DELETE RT_MAKE_NAME(chunk_children_array_delete)
+#define RT_CHUNK_VALUES_ARRAY_DELETE RT_MAKE_NAME(chunk_values_array_delete)
+#define RT_CHUNK_CHILDREN_ARRAY_COPY RT_MAKE_NAME(chunk_children_array_copy)
+#define RT_CHUNK_VALUES_ARRAY_COPY RT_MAKE_NAME(chunk_values_array_copy)
+#define RT_NODE_125_IS_CHUNK_USED RT_MAKE_NAME(node_125_is_chunk_used)
+#define RT_NODE_INNER_125_GET_CHILD RT_MAKE_NAME(node_inner_125_get_child)
+#define RT_NODE_LEAF_125_GET_VALUE RT_MAKE_NAME(node_leaf_125_get_value)
+#define RT_NODE_INNER_256_IS_CHUNK_USED RT_MAKE_NAME(node_inner_256_is_chunk_used)
+#define RT_NODE_LEAF_256_IS_CHUNK_USED RT_MAKE_NAME(node_leaf_256_is_chunk_used)
+#define RT_NODE_INNER_256_GET_CHILD RT_MAKE_NAME(node_inner_256_get_child)
+#define RT_NODE_LEAF_256_GET_VALUE RT_MAKE_NAME(node_leaf_256_get_value)
+#define RT_NODE_INNER_256_SET RT_MAKE_NAME(node_inner_256_set)
+#define RT_NODE_LEAF_256_SET RT_MAKE_NAME(node_leaf_256_set)
+#define RT_NODE_INNER_256_DELETE RT_MAKE_NAME(node_inner_256_delete)
+#define RT_NODE_LEAF_256_DELETE RT_MAKE_NAME(node_leaf_256_delete)
+#define RT_KEY_GET_SHIFT RT_MAKE_NAME(key_get_shift)
+#define RT_SHIFT_GET_MAX_VAL RT_MAKE_NAME(shift_get_max_val)
+#define RT_NODE_SEARCH_INNER RT_MAKE_NAME(node_search_inner)
+#define RT_NODE_SEARCH_LEAF RT_MAKE_NAME(node_search_leaf)
+#define RT_NODE_UPDATE_INNER RT_MAKE_NAME(node_update_inner)
+#define RT_NODE_DELETE_INNER RT_MAKE_NAME(node_delete_inner)
+#define RT_NODE_DELETE_LEAF RT_MAKE_NAME(node_delete_leaf)
+#define RT_NODE_INSERT_INNER RT_MAKE_NAME(node_insert_inner)
+#define RT_NODE_INSERT_LEAF RT_MAKE_NAME(node_insert_leaf)
+#define RT_NODE_INNER_ITERATE_NEXT RT_MAKE_NAME(node_inner_iterate_next)
+#define RT_NODE_LEAF_ITERATE_NEXT RT_MAKE_NAME(node_leaf_iterate_next)
+#define RT_ITER_SET_NODE_FROM RT_MAKE_NAME(iter_set_node_from)
+#define RT_ITER_UPDATE_KEY RT_MAKE_NAME(iter_update_key)
+#define RT_VERIFY_NODE RT_MAKE_NAME(verify_node)
+
+/* type declarations */
+#define RT_RADIX_TREE RT_MAKE_NAME(radix_tree)
+#define RT_RADIX_TREE_CONTROL RT_MAKE_NAME(radix_tree_control)
+#define RT_ITER RT_MAKE_NAME(iter)
+#ifdef RT_SHMEM
+#define RT_HANDLE RT_MAKE_NAME(handle)
+#endif
+#define RT_NODE RT_MAKE_NAME(node)
+#define RT_NODE_ITER RT_MAKE_NAME(node_iter)
+#define RT_NODE_BASE_3 RT_MAKE_NAME(node_base_3)
+#define RT_NODE_BASE_32 RT_MAKE_NAME(node_base_32)
+#define RT_NODE_BASE_125 RT_MAKE_NAME(node_base_125)
+#define RT_NODE_BASE_256 RT_MAKE_NAME(node_base_256)
+#define RT_NODE_INNER_3 RT_MAKE_NAME(node_inner_3)
+#define RT_NODE_INNER_32 RT_MAKE_NAME(node_inner_32)
+#define RT_NODE_INNER_125 RT_MAKE_NAME(node_inner_125)
+#define RT_NODE_INNER_256 RT_MAKE_NAME(node_inner_256)
+#define RT_NODE_LEAF_3 RT_MAKE_NAME(node_leaf_3)
+#define RT_NODE_LEAF_32 RT_MAKE_NAME(node_leaf_32)
+#define RT_NODE_LEAF_125 RT_MAKE_NAME(node_leaf_125)
+#define RT_NODE_LEAF_256 RT_MAKE_NAME(node_leaf_256)
+#define RT_SIZE_CLASS RT_MAKE_NAME(size_class)
+#define RT_SIZE_CLASS_ELEM RT_MAKE_NAME(size_class_elem)
+#define RT_SIZE_CLASS_INFO RT_MAKE_NAME(size_class_info)
+#define RT_CLASS_3 RT_MAKE_NAME(class_3)
+#define RT_CLASS_32_MIN RT_MAKE_NAME(class_32_min)
+#define RT_CLASS_32_MAX RT_MAKE_NAME(class_32_max)
+#define RT_CLASS_125 RT_MAKE_NAME(class_125)
+#define RT_CLASS_256 RT_MAKE_NAME(class_256)
+
+/* generate forward declarations necessary to use the radix tree */
+#ifdef RT_DECLARE
+
+typedef struct RT_RADIX_TREE RT_RADIX_TREE;
+typedef struct RT_ITER RT_ITER;
+
+#ifdef RT_SHMEM
+typedef dsa_pointer RT_HANDLE;
+#endif
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id);
+RT_SCOPE RT_RADIX_TREE * RT_ATTACH(dsa_area *dsa, dsa_pointer dp);
+RT_SCOPE void RT_DETACH(RT_RADIX_TREE *tree);
+RT_SCOPE RT_HANDLE RT_GET_HANDLE(RT_RADIX_TREE *tree);
+#else
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx);
+#endif
+RT_SCOPE void RT_FREE(RT_RADIX_TREE *tree);
+
+RT_SCOPE bool RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p);
+RT_SCOPE bool RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p);
+#ifdef RT_USE_DELETE
+RT_SCOPE bool RT_DELETE(RT_RADIX_TREE *tree, uint64 key);
+#endif
+
+RT_SCOPE RT_ITER * RT_BEGIN_ITERATE(RT_RADIX_TREE *tree);
+RT_SCOPE bool RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p);
+RT_SCOPE void RT_END_ITERATE(RT_ITER *iter);
+
+RT_SCOPE uint64 RT_MEMORY_USAGE(RT_RADIX_TREE *tree);
+
+#ifdef RT_DEBUG
+RT_SCOPE void RT_DUMP(RT_RADIX_TREE *tree);
+RT_SCOPE void RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key);
+RT_SCOPE void RT_STATS(RT_RADIX_TREE *tree);
+#endif
+
+#endif /* RT_DECLARE */
+
+
+/* generate implementation of the radix tree */
+#ifdef RT_DEFINE
+
+/* The number of bits encoded in one tree level */
+#define RT_NODE_SPAN BITS_PER_BYTE
+
+/* The number of maximum slots in the node */
+#define RT_NODE_MAX_SLOTS (1 << RT_NODE_SPAN)
+
+/* Mask for extracting a chunk from the key */
+#define RT_CHUNK_MASK ((1 << RT_NODE_SPAN) - 1)
+
+/* Maximum shift the radix tree uses */
+#define RT_MAX_SHIFT RT_KEY_GET_SHIFT(UINT64_MAX)
+
+/* Tree level the radix tree uses */
+#define RT_MAX_LEVEL ((sizeof(uint64) * BITS_PER_BYTE) / RT_NODE_SPAN)
+
+/*
+ * Number of bits necessary for isset array in the slot-index node.
+ * Since bitmapword can be 64 bits, the only values that make sense
+ * here are 64 and 128.
+ */
+#define RT_SLOT_IDX_LIMIT (RT_NODE_MAX_SLOTS / 2)
+
+/* Invalid index used in node-125 */
+#define RT_INVALID_SLOT_IDX 0xFF
+
+/* Get a chunk from the key */
+#define RT_GET_KEY_CHUNK(key, shift) ((uint8) (((key) >> (shift)) & RT_CHUNK_MASK))
+
+/* For accessing bitmaps */
+#define RT_BM_IDX(x) ((x) / BITS_PER_BITMAPWORD)
+#define RT_BM_BIT(x) ((x) % BITS_PER_BITMAPWORD)
+
+/*
+ * Node kinds
+ *
+ * The different node kinds are what make the tree "adaptive".
+ *
+ * Each node kind is associated with a different datatype and different
+ * search/set/delete/iterate algorithms adapted for its size. The largest
+ * kind, node256 is basically the same as a traditional radix tree,
+ * and would be most wasteful of memory when sparsely populated. The
+ * smaller nodes expend some additional CPU time to enable a smaller
+ * memory footprint.
+ *
+ * XXX There are 4 node kinds, and this should never be increased,
+ * for several reasons:
+ * 1. With 5 or more kinds, gcc tends to use a jump table for switch
+ * statements.
+ * 2. The 4 kinds can be represented with 2 bits, so we have the option
+ * in the future to tag the node pointer with the kind, even on
+ * platforms with 32-bit pointers. This might speed up node traversal
+ * in trees with highly random node kinds.
+ * 3. We can have multiple size classes per node kind.
+ */
+#define RT_NODE_KIND_3 0x00
+#define RT_NODE_KIND_32 0x01
+#define RT_NODE_KIND_125 0x02
+#define RT_NODE_KIND_256 0x03
+#define RT_NODE_KIND_COUNT 4
+
+/*
+ * Calculate the slab blocksize so that we can allocate at least 32 chunks
+ * from the block.
+ */
+#define RT_SLAB_BLOCK_SIZE(size) \
+ Max((SLAB_DEFAULT_BLOCK_SIZE / (size)) * (size), (size) * 32)
+
+/* Common type for all nodes types */
+typedef struct RT_NODE
+{
+ /*
+ * Number of children. We use uint16 to be able to indicate 256 children
+ * at the fanout of 8.
+ */
+ uint16 count;
+
+ /*
+ * Max capacity for the current size class. Storing this in the
+ * node enables multiple size classes per node kind.
+ * Technically, kinds with a single size class don't need this, so we could
+ * keep this in the individual base types, but the code is simpler this way.
+ * Note: node256 is unique in that it cannot possibly have more than a
+ * single size class, so for that kind we store zero, and uint8 is
+ * sufficient for other kinds.
+ */
+ uint8 fanout;
+
+ /*
+ * Shift indicates which part of the key space is represented by this
+ * node. That is, the key is shifted by 'shift' and the lowest
+ * RT_NODE_SPAN bits are then represented in chunk.
+ */
+ uint8 shift;
+
+ /* Node kind, one per search/set algorithm */
+ uint8 kind;
+} RT_NODE;
+
+
+#define RT_PTR_LOCAL RT_NODE *
+
+#ifdef RT_SHMEM
+#define RT_PTR_ALLOC dsa_pointer
+#else
+#define RT_PTR_ALLOC RT_PTR_LOCAL
+#endif
+
+
+#ifdef RT_SHMEM
+#define RT_INVALID_PTR_ALLOC InvalidDsaPointer
+#else
+#define RT_INVALID_PTR_ALLOC NULL
+#endif
+
+#ifdef RT_SHMEM
+#define RT_LOCK_EXCLUSIVE(tree) LWLockAcquire(&tree->ctl->lock, LW_EXCLUSIVE)
+#define RT_LOCK_SHARED(tree) LWLockAcquire(&tree->ctl->lock, LW_SHARED)
+#define RT_UNLOCK(tree) LWLockRelease(&tree->ctl->lock);
+#else
+#define RT_LOCK_EXCLUSIVE(tree) ((void) 0)
+#define RT_LOCK_SHARED(tree) ((void) 0)
+#define RT_UNLOCK(tree) ((void) 0)
+#endif
+
+/*
+ * Inner nodes and leaf nodes have analogous structure. To distinguish
+ * them at runtime, we take advantage of the fact that the key chunk
+ * is accessed by shifting: Inner tree nodes (shift > 0), store the
+ * pointer to its child node in the slot. In leaf nodes (shift == 0),
+ * the slot contains the value corresponding to the key.
+ */
+#define RT_NODE_IS_LEAF(n) (((RT_PTR_LOCAL) (n))->shift == 0)
+
+#define RT_NODE_MUST_GROW(node) \
+ ((node)->base.n.count == (node)->base.n.fanout)
+
+/*
+ * Base type of each node kinds for leaf and inner nodes.
+ * The base types must be a be able to accommodate the largest size
+ * class for variable-sized node kinds.
+ */
+typedef struct RT_NODE_BASE_3
+{
+ RT_NODE n;
+
+ /* 3 children, for key chunks */
+ uint8 chunks[3];
+} RT_NODE_BASE_3;
+
+typedef struct RT_NODE_BASE_32
+{
+ RT_NODE n;
+
+ /* 32 children, for key chunks */
+ uint8 chunks[32];
+} RT_NODE_BASE_32;
+
+/*
+ * node-125 uses slot_idx array, an array of RT_NODE_MAX_SLOTS length
+ * to store indexes into a second array that contains the values (or
+ * child pointers).
+ */
+typedef struct RT_NODE_BASE_125
+{
+ RT_NODE n;
+
+ /* The index of slots for each fanout */
+ uint8 slot_idxs[RT_NODE_MAX_SLOTS];
+
+ /* bitmap to track which slots are in use */
+ bitmapword isset[RT_BM_IDX(RT_SLOT_IDX_LIMIT)];
+} RT_NODE_BASE_125;
+
+typedef struct RT_NODE_BASE_256
+{
+ RT_NODE n;
+} RT_NODE_BASE_256;
+
+/*
+ * Inner and leaf nodes.
+ *
+ * These are separate because the value type might be different than
+ * something fitting into a pointer-width type.
+ */
+typedef struct RT_NODE_INNER_3
+{
+ RT_NODE_BASE_3 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_3;
+
+typedef struct RT_NODE_LEAF_3
+{
+ RT_NODE_BASE_3 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_3;
+
+typedef struct RT_NODE_INNER_32
+{
+ RT_NODE_BASE_32 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_32;
+
+typedef struct RT_NODE_LEAF_32
+{
+ RT_NODE_BASE_32 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_32;
+
+typedef struct RT_NODE_INNER_125
+{
+ RT_NODE_BASE_125 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_125;
+
+typedef struct RT_NODE_LEAF_125
+{
+ RT_NODE_BASE_125 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_125;
+
+/*
+ * node-256 is the largest node type. This node has an array
+ * for directly storing values (or child pointers in inner nodes).
+ * Unlike other node kinds, it's array size is by definition
+ * fixed.
+ */
+typedef struct RT_NODE_INNER_256
+{
+ RT_NODE_BASE_256 base;
+
+ /* Slots for 256 children */
+ RT_PTR_ALLOC children[RT_NODE_MAX_SLOTS];
+} RT_NODE_INNER_256;
+
+typedef struct RT_NODE_LEAF_256
+{
+ RT_NODE_BASE_256 base;
+
+ /*
+ * Unlike with inner256, zero is a valid value here, so we use a
+ * bitmap to track which slots are in use.
+ */
+ bitmapword isset[RT_BM_IDX(RT_NODE_MAX_SLOTS)];
+
+ /* Slots for 256 values */
+ RT_VALUE_TYPE values[RT_NODE_MAX_SLOTS];
+} RT_NODE_LEAF_256;
+
+/*
+ * Node size classes
+ *
+ * Nodes of different kinds necessarily belong to different size classes.
+ * The main innovation in our implementation compared to the ART paper
+ * is decoupling the notion of size class from kind.
+ *
+ * The size classes within a given node kind have the same underlying
+ * type, but a variable number of children/values. This is possible
+ * because the base type contains small fixed data structures that
+ * work the same way regardless of how full the node is. We store the
+ * node's allocated capacity in the "fanout" member of RT_NODE, to allow
+ * runtime introspection.
+ *
+ * Growing from one node kind to another requires special code for each
+ * case, but growing from one size class to another within the same kind
+ * is basically just allocate + memcpy.
+ *
+ * The size classes have been chosen so that inner nodes on platforms
+ * with 64-bit pointers (and leaf nodes when using a 64-bit key) are
+ * equal to or slightly smaller than some DSA size class.
+ */
+typedef enum RT_SIZE_CLASS
+{
+ RT_CLASS_3 = 0,
+ RT_CLASS_32_MIN,
+ RT_CLASS_32_MAX,
+ RT_CLASS_125,
+ RT_CLASS_256
+} RT_SIZE_CLASS;
+
+/* Information for each size class */
+typedef struct RT_SIZE_CLASS_ELEM
+{
+ const char *name;
+ int fanout;
+
+ /* slab chunk size */
+ Size inner_size;
+ Size leaf_size;
+} RT_SIZE_CLASS_ELEM;
+
+static const RT_SIZE_CLASS_ELEM RT_SIZE_CLASS_INFO[] = {
+ [RT_CLASS_3] = {
+ .name = "radix tree node 3",
+ .fanout = 3,
+ .inner_size = sizeof(RT_NODE_INNER_3) + 3 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_3) + 3 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_32_MIN] = {
+ .name = "radix tree node 15",
+ .fanout = 15,
+ .inner_size = sizeof(RT_NODE_INNER_32) + 15 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_32) + 15 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_32_MAX] = {
+ .name = "radix tree node 32",
+ .fanout = 32,
+ .inner_size = sizeof(RT_NODE_INNER_32) + 32 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_32) + 32 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_125] = {
+ .name = "radix tree node 125",
+ .fanout = 125,
+ .inner_size = sizeof(RT_NODE_INNER_125) + 125 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_125) + 125 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_256] = {
+ .name = "radix tree node 256",
+ .fanout = 256,
+ .inner_size = sizeof(RT_NODE_INNER_256),
+ .leaf_size = sizeof(RT_NODE_LEAF_256),
+ },
+};
+
+#define RT_SIZE_CLASS_COUNT lengthof(RT_SIZE_CLASS_INFO)
+
+#ifdef RT_SHMEM
+/* A magic value used to identify our radix tree */
+#define RT_RADIX_TREE_MAGIC 0x54A48167
+#endif
+
+/* Contains the actual tree and ancillary info */
+typedef struct RT_RADIX_TREE_CONTROL
+{
+#ifdef RT_SHMEM
+ RT_HANDLE handle;
+ uint32 magic;
+ LWLock lock;
+#endif
+
+ RT_PTR_ALLOC root;
+ uint64 max_val;
+ uint64 num_keys;
+
+ /* statistics */
+#ifdef RT_DEBUG
+ int32 cnt[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE_CONTROL;
+
+/* Entry point for allocating and accessing the tree */
+typedef struct RT_RADIX_TREE
+{
+ MemoryContext context;
+
+ /* pointing to either local memory or DSA */
+ RT_RADIX_TREE_CONTROL *ctl;
+
+#ifdef RT_SHMEM
+ dsa_area *dsa;
+#else
+ MemoryContextData *inner_slabs[RT_SIZE_CLASS_COUNT];
+ MemoryContextData *leaf_slabs[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE;
+
+/*
+ * Iteration support.
+ *
+ * Iterating the radix tree returns each pair of key and value in the ascending
+ * order of the key.
+ *
+ * RT_NODE_ITER is the struct for iteration of one radix tree node.
+ *
+ * RT_ITER is the struct for iteration of the radix tree, and uses RT_NODE_ITER
+ * for each level to track the iteration within the node.
+ */
+typedef struct RT_NODE_ITER
+{
+ /*
+ * Local pointer to the node we are iterating over.
+ *
+ * Since the radix tree doesn't support the shared iteration among multiple
+ * processes, we use RT_PTR_LOCAL rather than RT_PTR_ALLOC.
+ */
+ RT_PTR_LOCAL node;
+
+ /*
+ * The next index of the chunk array in RT_NODE_KIND_3 and
+ * RT_NODE_KIND_32 nodes, or the next chunk in RT_NODE_KIND_125 and
+ * RT_NODE_KIND_256 nodes. 0 for the initial value.
+ */
+ int idx;
+} RT_NODE_ITER;
+
+typedef struct RT_ITER
+{
+ RT_RADIX_TREE *tree;
+
+ /* Track the nodes for each level. level = 0 is for a leaf node */
+ RT_NODE_ITER node_iters[RT_MAX_LEVEL];
+ int top_level;
+
+ /* The key constructed during the iteration */
+ uint64 key;
+} RT_ITER;
+
+
+static void RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_PTR_ALLOC child);
+static bool RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_VALUE_TYPE *value_p);
+
+/* verification (available only with assertion) */
+static void RT_VERIFY_NODE(RT_PTR_LOCAL node);
+
+/* Get the local address of an allocated node */
+static inline RT_PTR_LOCAL
+RT_PTR_GET_LOCAL(RT_RADIX_TREE *tree, RT_PTR_ALLOC node)
+{
+#ifdef RT_SHMEM
+ return dsa_get_address(tree->dsa, (dsa_pointer) node);
+#else
+ return node;
+#endif
+}
+
+static inline bool
+RT_PTR_ALLOC_IS_VALID(RT_PTR_ALLOC ptr)
+{
+#ifdef RT_SHMEM
+ return DsaPointerIsValid(ptr);
+#else
+ return PointerIsValid(ptr);
+#endif
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_3_SEARCH_EQ(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+ int idx = -1;
+
+ for (int i = 0; i < node->n.count; i++)
+ {
+ if (node->chunks[i] == chunk)
+ {
+ idx = i;
+ break;
+ }
+ }
+
+ return idx;
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_3_GET_INSERTPOS(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+ int idx;
+
+ for (idx = 0; idx < node->n.count; idx++)
+ {
+ if (node->chunks[idx] >= chunk)
+ break;
+ }
+
+ return idx;
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_32_SEARCH_EQ(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+ int count = node->n.count;
+#ifndef USE_NO_SIMD
+ Vector8 spread_chunk;
+ Vector8 haystack1;
+ Vector8 haystack2;
+ Vector8 cmp1;
+ Vector8 cmp2;
+ uint32 bitfield;
+ int index_simd = -1;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+ int index = -1;
+
+ for (int i = 0; i < count; i++)
+ {
+ if (node->chunks[i] == chunk)
+ {
+ index = i;
+ break;
+ }
+ }
+#endif
+
+#ifndef USE_NO_SIMD
+ /* replicate the search key */
+ spread_chunk = vector8_broadcast(chunk);
+
+ /* compare to all 32 keys stored in the node */
+ vector8_load(&haystack1, &node->chunks[0]);
+ vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+ cmp1 = vector8_eq(spread_chunk, haystack1);
+ cmp2 = vector8_eq(spread_chunk, haystack2);
+
+ /* convert comparison to a bitfield */
+ bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+
+ /* mask off invalid entries */
+ bitfield &= ((UINT64CONST(1) << count) - 1);
+
+ /* convert bitfield to index by counting trailing zeros */
+ if (bitfield)
+ index_simd = pg_rightmost_one_pos32(bitfield);
+
+ Assert(index_simd == index);
+ return index_simd;
+#else
+ return index;
+#endif
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_32_GET_INSERTPOS(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+ int count = node->n.count;
+#ifndef USE_NO_SIMD
+ Vector8 spread_chunk;
+ Vector8 haystack1;
+ Vector8 haystack2;
+ Vector8 cmp1;
+ Vector8 cmp2;
+ Vector8 min1;
+ Vector8 min2;
+ uint32 bitfield;
+ int index_simd;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+ int index;
+
+ for (index = 0; index < count; index++)
+ {
+ /*
+ * This is coded with '>=' to match what we can do with SIMD,
+ * with an assert to keep us honest.
+ */
+ if (node->chunks[index] >= chunk)
+ {
+ Assert(node->chunks[index] != chunk);
+ break;
+ }
+ }
+#endif
+
+#ifndef USE_NO_SIMD
+ /*
+ * This is a bit more complicated than RT_NODE_32_SEARCH_EQ(), because
+ * no unsigned uint8 comparison instruction exists, at least for SSE2. So
+ * we need to play some trickery using vector8_min() to effectively get
+ * >=. There'll never be any equal elements in current uses, but that's
+ * what we get here...
+ */
+ spread_chunk = vector8_broadcast(chunk);
+ vector8_load(&haystack1, &node->chunks[0]);
+ vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+ min1 = vector8_min(spread_chunk, haystack1);
+ min2 = vector8_min(spread_chunk, haystack2);
+ cmp1 = vector8_eq(spread_chunk, min1);
+ cmp2 = vector8_eq(spread_chunk, min2);
+ bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+ bitfield &= ((UINT64CONST(1) << count) - 1);
+
+ if (bitfield)
+ index_simd = pg_rightmost_one_pos32(bitfield);
+ else
+ index_simd = count;
+
+ Assert(index_simd == index);
+ return index_simd;
+#else
+ return index;
+#endif
+}
+
+
+/*
+ * Functions to manipulate both chunks array and children/values array.
+ * These are used for node-3 and node-32.
+ */
+
+/* Shift the elements right at 'idx' by one */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_SHIFT(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+ memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+ memmove(&(children[idx + 1]), &(children[idx]), sizeof(RT_PTR_ALLOC) * (count - idx));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_SHIFT(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+ memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+ memmove(&(values[idx + 1]), &(values[idx]), sizeof(RT_VALUE_TYPE) * (count - idx));
+}
+
+/* Delete the element at 'idx' */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_DELETE(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+ memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+ memmove(&(children[idx]), &(children[idx + 1]), sizeof(RT_PTR_ALLOC) * (count - idx - 1));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_DELETE(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+ memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+ memmove(&(values[idx]), &(values[idx + 1]), sizeof(RT_VALUE_TYPE) * (count - idx - 1));
+}
+
+/* Copy both chunks and children/values arrays */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_COPY(uint8 *src_chunks, RT_PTR_ALLOC *src_children,
+ uint8 *dst_chunks, RT_PTR_ALLOC *dst_children)
+{
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const Size chunk_size = sizeof(uint8) * fanout;
+ const Size children_size = sizeof(RT_PTR_ALLOC) * fanout;
+
+ memcpy(dst_chunks, src_chunks, chunk_size);
+ memcpy(dst_children, src_children, children_size);
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_COPY(uint8 *src_chunks, RT_VALUE_TYPE *src_values,
+ uint8 *dst_chunks, RT_VALUE_TYPE *dst_values)
+{
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const Size chunk_size = sizeof(uint8) * fanout;
+ const Size values_size = sizeof(RT_VALUE_TYPE) * fanout;
+
+ memcpy(dst_chunks, src_chunks, chunk_size);
+ memcpy(dst_values, src_values, values_size);
+}
+
+/* Functions to manipulate inner and leaf node-125 */
+
+/* Does the given chunk in the node has the value? */
+static inline bool
+RT_NODE_125_IS_CHUNK_USED(RT_NODE_BASE_125 *node, uint8 chunk)
+{
+ return node->slot_idxs[chunk] != RT_INVALID_SLOT_IDX;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_125_GET_CHILD(RT_NODE_INNER_125 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ return node->children[node->base.slot_idxs[chunk]];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_125_GET_VALUE(RT_NODE_LEAF_125 *node, uint8 chunk)
+{
+ Assert(RT_NODE_IS_LEAF(node));
+ Assert(((RT_NODE_BASE_125 *) node)->slot_idxs[chunk] != RT_INVALID_SLOT_IDX);
+ return node->values[node->base.slot_idxs[chunk]];
+}
+
+/* Functions to manipulate inner and leaf node-256 */
+
+/* Return true if the slot corresponding to the given chunk is in use */
+static inline bool
+RT_NODE_INNER_256_IS_CHUNK_USED(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ return node->children[chunk] != RT_INVALID_PTR_ALLOC;
+}
+
+static inline bool
+RT_NODE_LEAF_256_IS_CHUNK_USED(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ return (node->isset[idx] & ((bitmapword) 1 << bitnum)) != 0;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_256_GET_CHILD(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ Assert(RT_NODE_INNER_256_IS_CHUNK_USED(node, chunk));
+ return node->children[chunk];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_256_GET_VALUE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ Assert(RT_NODE_IS_LEAF(node));
+ Assert(RT_NODE_LEAF_256_IS_CHUNK_USED(node, chunk));
+ return node->values[chunk];
+}
+
+/* Set the child in the node-256 */
+static inline void
+RT_NODE_INNER_256_SET(RT_NODE_INNER_256 *node, uint8 chunk, RT_PTR_ALLOC child)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ node->children[chunk] = child;
+}
+
+/* Set the value in the node-256 */
+static inline void
+RT_NODE_LEAF_256_SET(RT_NODE_LEAF_256 *node, uint8 chunk, RT_VALUE_TYPE value)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ node->isset[idx] |= ((bitmapword) 1 << bitnum);
+ node->values[chunk] = value;
+}
+
+/* Set the slot at the given chunk position */
+static inline void
+RT_NODE_INNER_256_DELETE(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ node->children[chunk] = RT_INVALID_PTR_ALLOC;
+}
+
+static inline void
+RT_NODE_LEAF_256_DELETE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ node->isset[idx] &= ~((bitmapword) 1 << bitnum);
+}
+
+/*
+ * Return the largest shift that will allowing storing the given key.
+ */
+static inline int
+RT_KEY_GET_SHIFT(uint64 key)
+{
+ if (key == 0)
+ return 0;
+ else
+ return (pg_leftmost_one_pos64(key) / RT_NODE_SPAN) * RT_NODE_SPAN;
+}
+
+/*
+ * Return the max value that can be stored in the tree with the given shift.
+ */
+static uint64
+RT_SHIFT_GET_MAX_VAL(int shift)
+{
+ if (shift == RT_MAX_SHIFT)
+ return UINT64_MAX;
+
+ return (UINT64CONST(1) << (shift + RT_NODE_SPAN)) - 1;
+}
+
+/*
+ * Allocate a new node with the given node kind.
+ */
+static RT_PTR_ALLOC
+RT_ALLOC_NODE(RT_RADIX_TREE *tree, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+ RT_PTR_ALLOC allocnode;
+ size_t allocsize;
+
+ if (is_leaf)
+ allocsize = RT_SIZE_CLASS_INFO[size_class].leaf_size;
+ else
+ allocsize = RT_SIZE_CLASS_INFO[size_class].inner_size;
+
+#ifdef RT_SHMEM
+ allocnode = dsa_allocate(tree->dsa, allocsize);
+#else
+ if (is_leaf)
+ allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->leaf_slabs[size_class],
+ allocsize);
+ else
+ allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->inner_slabs[size_class],
+ allocsize);
+#endif
+
+#ifdef RT_DEBUG
+ /* update the statistics */
+ tree->ctl->cnt[size_class]++;
+#endif
+
+ return allocnode;
+}
+
+/* Initialize the node contents */
+static inline void
+RT_INIT_NODE(RT_PTR_LOCAL node, uint8 kind, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+ if (is_leaf)
+ MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].leaf_size);
+ else
+ MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].inner_size);
+
+ node->kind = kind;
+
+ if (kind == RT_NODE_KIND_256)
+ /* See comment for the RT_NODE type */
+ Assert(node->fanout == 0);
+ else
+ node->fanout = RT_SIZE_CLASS_INFO[size_class].fanout;
+
+ /* Initialize slot_idxs to invalid values */
+ if (kind == RT_NODE_KIND_125)
+ {
+ RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+
+ memset(n125->slot_idxs, RT_INVALID_SLOT_IDX, sizeof(n125->slot_idxs));
+ }
+}
+
+/*
+ * Create a new node as the root. Subordinate nodes will be created during
+ * the insertion.
+ */
+static pg_noinline void
+RT_NEW_ROOT(RT_RADIX_TREE *tree, uint64 key)
+{
+ int shift = RT_KEY_GET_SHIFT(key);
+ bool is_leaf = shift == 0;
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(newnode, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ newnode->shift = shift;
+ tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(shift);
+ tree->ctl->root = allocnode;
+}
+
+static inline void
+RT_COPY_NODE(RT_PTR_LOCAL newnode, RT_PTR_LOCAL oldnode)
+{
+ newnode->shift = oldnode->shift;
+ newnode->count = oldnode->count;
+}
+
+/*
+ * Given a new allocated node and an old node, initialize the new
+ * node with the necessary fields and return its local pointer.
+ */
+static inline RT_PTR_LOCAL
+RT_SWITCH_NODE_KIND(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, RT_PTR_LOCAL node,
+ uint8 new_kind, uint8 new_class, bool is_leaf)
+{
+ RT_PTR_LOCAL newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(newnode, new_kind, new_class, is_leaf);
+ RT_COPY_NODE(newnode, node);
+
+ return newnode;
+}
+
+/* Free the given node */
+static void
+RT_FREE_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode)
+{
+ /* If we're deleting the root node, make the tree empty */
+ if (tree->ctl->root == allocnode)
+ {
+ tree->ctl->root = RT_INVALID_PTR_ALLOC;
+ tree->ctl->max_val = 0;
+ }
+
+#ifdef RT_DEBUG
+ {
+ int i;
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+ /* update the statistics */
+ for (i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ if (node->fanout == RT_SIZE_CLASS_INFO[i].fanout)
+ break;
+ }
+
+ /* fanout of node256 is intentionally 0 */
+ if (i == RT_SIZE_CLASS_COUNT)
+ i = RT_CLASS_256;
+
+ tree->ctl->cnt[i]--;
+ Assert(tree->ctl->cnt[i] >= 0);
+ }
+#endif
+
+#ifdef RT_SHMEM
+ dsa_free(tree->dsa, allocnode);
+#else
+ pfree(allocnode);
+#endif
+}
+
+/* Update the parent's pointer when growing a node */
+static inline void
+RT_NODE_UPDATE_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC new_child)
+{
+#define RT_ACTION_UPDATE
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+#undef RT_ACTION_UPDATE
+}
+
+/*
+ * Replace old_child with new_child, and free the old one.
+ */
+static inline void
+RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
+ RT_PTR_ALLOC stored_old_child, RT_PTR_LOCAL old_child,
+ RT_PTR_ALLOC new_child, uint64 key)
+{
+#ifdef USE_ASSERT_CHECKING
+ RT_PTR_LOCAL new = RT_PTR_GET_LOCAL(tree, new_child);
+
+ Assert(old_child->shift == new->shift);
+ Assert(old_child->count == new->count);
+#endif
+
+ if (parent == old_child)
+ {
+ /* Replace the root node with the new larger node */
+ tree->ctl->root = new_child;
+ }
+ else
+ RT_NODE_UPDATE_INNER(parent, key, new_child);
+
+ RT_FREE_NODE(tree, stored_old_child);
+}
+
+/*
+ * The radix tree doesn't have sufficient height. Extend the radix tree so
+ * it can store the key.
+ */
+static pg_noinline void
+RT_EXTEND_UP(RT_RADIX_TREE *tree, uint64 key)
+{
+ int target_shift;
+ RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+ int shift = root->shift + RT_NODE_SPAN;
+
+ target_shift = RT_KEY_GET_SHIFT(key);
+
+ /* Grow tree from 'shift' to 'target_shift' */
+ while (shift <= target_shift)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL node;
+ RT_NODE_INNER_3 *n3;
+
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, true);
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(node, RT_NODE_KIND_3, RT_CLASS_3, true);
+ node->shift = shift;
+ node->count = 1;
+
+ n3 = (RT_NODE_INNER_3 *) node;
+ n3->base.chunks[0] = 0;
+ n3->children[0] = tree->ctl->root;
+
+ /* Update the root */
+ tree->ctl->root = allocnode;
+
+ shift += RT_NODE_SPAN;
+ }
+
+ tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(target_shift);
+}
+
+/*
+ * The radix tree doesn't have inner and leaf nodes for given key-value pair.
+ * Insert inner and leaf nodes from 'node' to bottom.
+ */
+static pg_noinline void
+RT_EXTEND_DOWN(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
+ RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node)
+{
+ int shift = node->shift;
+
+ Assert(RT_PTR_GET_LOCAL(tree, stored_node) == node);
+
+ while (shift >= RT_NODE_SPAN)
+ {
+ RT_PTR_ALLOC allocchild;
+ RT_PTR_LOCAL newchild;
+ int newshift = shift - RT_NODE_SPAN;
+ bool is_leaf = newshift == 0;
+
+ allocchild = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ newchild = RT_PTR_GET_LOCAL(tree, allocchild);
+ RT_INIT_NODE(newchild, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ newchild->shift = newshift;
+ RT_NODE_INSERT_INNER(tree, parent, stored_node, node, key, allocchild);
+
+ parent = node;
+ node = newchild;
+ stored_node = allocchild;
+ shift -= RT_NODE_SPAN;
+ }
+
+ RT_NODE_INSERT_LEAF(tree, parent, stored_node, node, key, value_p);
+ tree->ctl->num_keys++;
+}
+
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the child
+ * pointer is set to child_p.
+ */
+static inline bool
+RT_NODE_SEARCH_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC *child_p)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the pointer
+ * to the value is set to value_p.
+ */
+static inline bool
+RT_NODE_SEARCH_LEAF(RT_PTR_LOCAL node, uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_INNER(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_LEAF(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+#endif
+
+/*
+ * Insert "child" into "node".
+ *
+ * "parent" is the parent of "node", so the grandparent of the child.
+ * If the node we're inserting into needs to grow, we update the parent's
+ * child pointer with the pointer to the new larger node.
+ */
+static void
+RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_PTR_ALLOC child)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/* Like RT_NODE_INSERT_INNER, but for leaf nodes */
+static bool
+RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Create the radix tree in the given memory context and return it.
+ */
+RT_SCOPE RT_RADIX_TREE *
+#ifdef RT_SHMEM
+RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id)
+#else
+RT_CREATE(MemoryContext ctx)
+#endif
+{
+ RT_RADIX_TREE *tree;
+ MemoryContext old_ctx;
+#ifdef RT_SHMEM
+ dsa_pointer dp;
+#endif
+
+ old_ctx = MemoryContextSwitchTo(ctx);
+
+ tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+ tree->context = ctx;
+
+#ifdef RT_SHMEM
+ tree->dsa = dsa;
+ dp = dsa_allocate0(dsa, sizeof(RT_RADIX_TREE_CONTROL));
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, dp);
+ tree->ctl->handle = dp;
+ tree->ctl->magic = RT_RADIX_TREE_MAGIC;
+ LWLockInitialize(&tree->ctl->lock, tranche_id);
+#else
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) palloc0(sizeof(RT_RADIX_TREE_CONTROL));
+
+ /* Create a slab context for each size class */
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ RT_SIZE_CLASS_ELEM size_class = RT_SIZE_CLASS_INFO[i];
+ size_t inner_blocksize = RT_SLAB_BLOCK_SIZE(size_class.inner_size);
+ size_t leaf_blocksize = RT_SLAB_BLOCK_SIZE(size_class.leaf_size);
+
+ tree->inner_slabs[i] = SlabContextCreate(ctx,
+ size_class.name,
+ inner_blocksize,
+ size_class.inner_size);
+ tree->leaf_slabs[i] = SlabContextCreate(ctx,
+ size_class.name,
+ leaf_blocksize,
+ size_class.leaf_size);
+ }
+#endif
+
+ tree->ctl->root = RT_INVALID_PTR_ALLOC;
+
+ MemoryContextSwitchTo(old_ctx);
+
+ return tree;
+}
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE *
+RT_ATTACH(dsa_area *dsa, RT_HANDLE handle)
+{
+ RT_RADIX_TREE *tree;
+ dsa_pointer control;
+
+ tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+
+ /* Find the control object in shard memory */
+ control = handle;
+
+ tree->dsa = dsa;
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, control);
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+ return tree;
+}
+
+RT_SCOPE void
+RT_DETACH(RT_RADIX_TREE *tree)
+{
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ pfree(tree);
+}
+
+RT_SCOPE RT_HANDLE
+RT_GET_HANDLE(RT_RADIX_TREE *tree)
+{
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ return tree->ctl->handle;
+}
+
+/*
+ * Recursively free all nodes allocated to the DSA area.
+ */
+static void
+RT_FREE_RECURSE(RT_RADIX_TREE *tree, RT_PTR_ALLOC ptr)
+{
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, ptr);
+
+ check_stack_depth();
+ CHECK_FOR_INTERRUPTS();
+
+ /* The leaf node doesn't have child pointers */
+ if (RT_NODE_IS_LEAF(node))
+ {
+ dsa_free(tree->dsa, ptr);
+ return;
+ }
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+ for (int i = 0; i < n3->base.n.count; i++)
+ RT_FREE_RECURSE(tree, n3->children[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+ for (int i = 0; i < n32->base.n.count; i++)
+ RT_FREE_RECURSE(tree, n32->children[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) node;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+ continue;
+
+ RT_FREE_RECURSE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+ }
+
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ RT_FREE_RECURSE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+ }
+
+ break;
+ }
+ }
+
+ /* Free the inner node */
+ dsa_free(tree->dsa, ptr);
+}
+#endif
+
+/*
+ * Free the given radix tree.
+ */
+RT_SCOPE void
+RT_FREE(RT_RADIX_TREE *tree)
+{
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+ /* Free all memory used for radix tree nodes */
+ if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ RT_FREE_RECURSE(tree, tree->ctl->root);
+
+ /*
+ * Vandalize the control block to help catch programming error where
+ * other backends access the memory formerly occupied by this radix tree.
+ */
+ tree->ctl->magic = 0;
+ dsa_free(tree->dsa, tree->ctl->handle);
+#else
+ pfree(tree->ctl);
+
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ MemoryContextDelete(tree->inner_slabs[i]);
+ MemoryContextDelete(tree->leaf_slabs[i]);
+ }
+#endif
+
+ pfree(tree);
+}
+
+/*
+ * Set key to value. If the entry already exists, we update its value to 'value'
+ * and return true. Returns false if entry doesn't yet exist.
+ */
+RT_SCOPE bool
+RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+ int shift;
+ bool updated;
+ RT_PTR_LOCAL parent;
+ RT_PTR_ALLOC stored_child;
+ RT_PTR_LOCAL child;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ RT_LOCK_EXCLUSIVE(tree);
+
+ /* Empty tree, create the root */
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ RT_NEW_ROOT(tree, key);
+
+ /* Extend the tree if necessary */
+ if (key > tree->ctl->max_val)
+ RT_EXTEND_UP(tree, key);
+
+ stored_child = tree->ctl->root;
+ parent = RT_PTR_GET_LOCAL(tree, stored_child);
+ shift = parent->shift;
+
+ /* Descend the tree until we reach a leaf node */
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC new_child = RT_INVALID_PTR_ALLOC;
+
+ child = RT_PTR_GET_LOCAL(tree, stored_child);
+
+ if (RT_NODE_IS_LEAF(child))
+ break;
+
+ if (!RT_NODE_SEARCH_INNER(child, key, &new_child))
+ {
+ RT_EXTEND_DOWN(tree, key, value_p, parent, stored_child, child);
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ parent = child;
+ stored_child = new_child;
+ shift -= RT_NODE_SPAN;
+ }
+
+ updated = RT_NODE_INSERT_LEAF(tree, parent, stored_child, child, key, value_p);
+
+ /* Update the statistics */
+ if (!updated)
+ tree->ctl->num_keys++;
+
+ RT_UNLOCK(tree);
+ return updated;
+}
+
+/*
+ * Search the given key in the radix tree. Return true if there is the key,
+ * otherwise return false. On success, we set the value to *value_p so it must
+ * not be NULL.
+ */
+RT_SCOPE bool
+RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+ RT_PTR_LOCAL node;
+ int shift;
+ bool found;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+ Assert(value_p != NULL);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ node = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+ shift = node->shift;
+
+ /* Descend the tree until a leaf node */
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+ if (RT_NODE_IS_LEAF(node))
+ break;
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ node = RT_PTR_GET_LOCAL(tree, child);
+ shift -= RT_NODE_SPAN;
+ }
+
+ found = RT_NODE_SEARCH_LEAF(node, key, value_p);
+
+ RT_UNLOCK(tree);
+ return found;
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Delete the given key from the radix tree. Return true if the key is found (and
+ * deleted), otherwise do nothing and return false.
+ */
+RT_SCOPE bool
+RT_DELETE(RT_RADIX_TREE *tree, uint64 key)
+{
+ RT_PTR_LOCAL node;
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_ALLOC stack[RT_MAX_LEVEL] = {0};
+ int shift;
+ int level;
+ bool deleted;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ RT_LOCK_EXCLUSIVE(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ /*
+ * Descend the tree to search the key while building a stack of nodes we
+ * visited.
+ */
+ allocnode = tree->ctl->root;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift = node->shift;
+ level = -1;
+ while (shift > 0)
+ {
+ RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+ /* Push the current node to the stack */
+ stack[++level] = allocnode;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ allocnode = child;
+ shift -= RT_NODE_SPAN;
+ }
+
+ /* Delete the key from the leaf node if exists */
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ deleted = RT_NODE_DELETE_LEAF(node, key);
+
+ if (!deleted)
+ {
+ /* no key is found in the leaf node */
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ /* Found the key to delete. Update the statistics */
+ tree->ctl->num_keys--;
+
+ /*
+ * Return if the leaf node still has keys and we don't need to delete the
+ * node.
+ */
+ if (node->count > 0)
+ {
+ RT_UNLOCK(tree);
+ return true;
+ }
+
+ /* Free the empty leaf node */
+ RT_FREE_NODE(tree, allocnode);
+
+ /* Delete the key in inner nodes recursively */
+ while (level >= 0)
+ {
+ allocnode = stack[level--];
+
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ deleted = RT_NODE_DELETE_INNER(node, key);
+ Assert(deleted);
+
+ /* If the node didn't become empty, we stop deleting the key */
+ if (node->count > 0)
+ break;
+
+ /* The node became empty */
+ RT_FREE_NODE(tree, allocnode);
+ }
+
+ RT_UNLOCK(tree);
+ return true;
+}
+#endif
+
+/*
+ * Scan the inner node and return the next child node if exist, otherwise
+ * return NULL.
+ */
+static inline RT_PTR_LOCAL
+RT_NODE_INNER_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Scan the leaf node, and return true and the next value is set to value_p
+ * if exists. Otherwise return false.
+ */
+static inline bool
+RT_NODE_LEAF_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter,
+ RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * While descending the radix tree from the 'from' node to the bottom, we
+ * set the next node to iterate for each level.
+ */
+static void
+RT_ITER_SET_NODE_FROM(RT_ITER *iter, RT_PTR_LOCAL from)
+{
+ int level = from->shift / RT_NODE_SPAN;
+ RT_PTR_LOCAL node = from;
+
+ for (;;)
+ {
+ RT_NODE_ITER *node_iter = &(iter->node_iters[level--]);
+
+#ifdef USE_ASSERT_CHECKING
+ if (node_iter->node)
+ {
+ /* We must have finished the iteration on the previous node */
+ if (RT_NODE_IS_LEAF(node_iter->node))
+ {
+ uint64 dummy;
+ Assert(!RT_NODE_LEAF_ITERATE_NEXT(iter, node_iter, &dummy));
+ }
+ else
+ Assert(!RT_NODE_INNER_ITERATE_NEXT(iter, node_iter));
+ }
+#endif
+
+ /* Set the node to the node iterator of this level */
+ node_iter->node = node;
+ node_iter->idx = 0;
+
+ if (RT_NODE_IS_LEAF(node))
+ {
+ /* We will visit the leaf node when RT_ITERATE_NEXT() */
+ break;
+ }
+
+ /*
+ * Get the first child node from the node, which corresponds to the
+ * lowest chunk within the node.
+ */
+ node = RT_NODE_INNER_ITERATE_NEXT(iter, node_iter);
+
+ /* The first child must be found */
+ Assert(node);
+ }
+}
+
+/*
+ * Create and return the iterator for the given radix tree.
+ *
+ * The radix tree is locked in shared mode during the iteration, so
+ * RT_END_ITERATE needs to be called when finished to release the lock.
+ */
+RT_SCOPE RT_ITER *
+RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
+{
+ RT_ITER *iter;
+ RT_PTR_LOCAL root;
+
+ iter = (RT_ITER *) MemoryContextAllocZero(tree->context,
+ sizeof(RT_ITER));
+ iter->tree = tree;
+
+ RT_LOCK_SHARED(tree);
+
+ /* empty tree */
+ if (!iter->tree->ctl->root)
+ return iter;
+
+ root = RT_PTR_GET_LOCAL(tree, iter->tree->ctl->root);
+ iter->top_level = root->shift / RT_NODE_SPAN;
+
+ /*
+ * Set the next node to iterate for each level from the level of the
+ * root node.
+ */
+ RT_ITER_SET_NODE_FROM(iter, root);
+
+ return iter;
+}
+
+/*
+ * Return true with setting key_p and value_p if there is next key. Otherwise
+ * return false.
+ */
+RT_SCOPE bool
+RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p)
+{
+ Assert(value_p != NULL);
+
+ /* Empty tree */
+ if (!iter->tree->ctl->root)
+ return false;
+
+ for (;;)
+ {
+ RT_PTR_LOCAL child = NULL;
+
+ /* Get the next chunk of the leaf node */
+ if (RT_NODE_LEAF_ITERATE_NEXT(iter, &(iter->node_iters[0]), value_p))
+ {
+ *key_p = iter->key;
+ return true;
+ }
+
+ /*
+ * We've visited all values in the leaf node, so advance all inner node
+ * iterators by visiting inner nodes from the level = 1 until we find the
+ * next inner node that has a child node.
+ */
+ for (int level = 1; level <= iter->top_level; level++)
+ {
+ child = RT_NODE_INNER_ITERATE_NEXT(iter, &(iter->node_iters[level]));
+
+ if (child)
+ break;
+ }
+
+ /* We've visited all nodes, so the iteration finished */
+ if (!child)
+ break;
+
+ /*
+ * Found the new child node. We update the next node to iterate for each
+ * level from the level of this child node.
+ */
+ RT_ITER_SET_NODE_FROM(iter, child);
+
+ /* Find key-value from the leaf node again */
+ }
+
+ return false;
+}
+
+/*
+ * Terminate the iteration and release the lock.
+ *
+ * This function needs to be called after finishing or when exiting an
+ * iteration.
+ */
+RT_SCOPE void
+RT_END_ITERATE(RT_ITER *iter)
+{
+#ifdef RT_SHMEM
+ Assert(LWLockHeldByMe(&iter->tree->ctl->lock));
+#endif
+
+ RT_UNLOCK(iter->tree);
+ pfree(iter);
+}
+
+/*
+ * Return the statistics of the amount of memory used by the radix tree.
+ */
+RT_SCOPE uint64
+RT_MEMORY_USAGE(RT_RADIX_TREE *tree)
+{
+ Size total = 0;
+
+ RT_LOCK_SHARED(tree);
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ total = dsa_get_total_size(tree->dsa);
+#else
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ total += MemoryContextMemAllocated(tree->inner_slabs[i], true);
+ total += MemoryContextMemAllocated(tree->leaf_slabs[i], true);
+ }
+#endif
+
+ RT_UNLOCK(tree);
+ return total;
+}
+
+/*
+ * Verify the radix tree node.
+ */
+static void
+RT_VERIFY_NODE(RT_PTR_LOCAL node)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(node->count >= 0);
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE_BASE_3 *n3 = (RT_NODE_BASE_3 *) node;
+
+ for (int i = 1; i < n3->n.count; i++)
+ Assert(n3->chunks[i - 1] < n3->chunks[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE_BASE_32 *n32 = (RT_NODE_BASE_32 *) node;
+
+ for (int i = 1; i < n32->n.count; i++)
+ Assert(n32->chunks[i - 1] < n32->chunks[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+ int cnt = 0;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ uint8 slot = n125->slot_idxs[i];
+ int idx = RT_BM_IDX(slot);
+ int bitnum = RT_BM_BIT(slot);
+
+ if (!RT_NODE_125_IS_CHUNK_USED(n125, i))
+ continue;
+
+ /* Check if the corresponding slot is used */
+ Assert(slot < node->fanout);
+ Assert((n125->isset[idx] & ((bitmapword) 1 << bitnum)) != 0);
+
+ cnt++;
+ }
+
+ Assert(n125->n.count == cnt);
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+ int cnt = 0;
+
+ for (int i = 0; i < RT_BM_IDX(RT_NODE_MAX_SLOTS); i++)
+ cnt += bmw_popcount(n256->isset[i]);
+
+ /* Check if the number of used chunk matches */
+ Assert(n256->base.n.count == cnt);
+
+ break;
+ }
+ }
+ }
+#endif
+}
+
+/***************** DEBUG FUNCTIONS *****************/
+#ifdef RT_DEBUG
+
+#define RT_UINT64_FORMAT_HEX "%" INT64_MODIFIER "X"
+
+RT_SCOPE void
+RT_STATS(RT_RADIX_TREE *tree)
+{
+ RT_LOCK_SHARED(tree);
+
+ fprintf(stderr, "max_val = " UINT64_FORMAT "\n", tree->ctl->max_val);
+ fprintf(stderr, "num_keys = " UINT64_FORMAT "\n", tree->ctl->num_keys);
+
+#ifdef RT_SHMEM
+ fprintf(stderr, "handle = " UINT64_FORMAT "\n", tree->ctl->handle);
+#endif
+
+ if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+
+ fprintf(stderr, "height = %d, n3 = %u, n15 = %u, n32 = %u, n125 = %u, n256 = %u\n",
+ root->shift / RT_NODE_SPAN,
+ tree->ctl->cnt[RT_CLASS_3],
+ tree->ctl->cnt[RT_CLASS_32_MIN],
+ tree->ctl->cnt[RT_CLASS_32_MAX],
+ tree->ctl->cnt[RT_CLASS_125],
+ tree->ctl->cnt[RT_CLASS_256]);
+ }
+
+ RT_UNLOCK(tree);
+}
+
+static void
+RT_DUMP_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, int level,
+ bool recurse, StringInfo buf)
+{
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+ StringInfoData spaces;
+
+ initStringInfo(&spaces);
+ appendStringInfoSpaces(&spaces, (level * 4) + 1);
+
+ appendStringInfo(buf, "%s%s[%s] kind %d, fanout %d, count %u, shift %u:\n",
+ spaces.data,
+ level == 0 ? "" : "-> ",
+ RT_NODE_IS_LEAF(node) ? "LEAF" : "INNR",
+ (node->kind == RT_NODE_KIND_3) ? 3 :
+ (node->kind == RT_NODE_KIND_32) ? 32 :
+ (node->kind == RT_NODE_KIND_125) ? 125 : 256,
+ node->fanout == 0 ? 256 : node->fanout,
+ node->count, node->shift);
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ for (int i = 0; i < node->count; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_3 *n3 = (RT_NODE_LEAF_3 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+ spaces.data, i, n3->base.chunks[i]);
+ }
+ else
+ {
+ RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X",
+ spaces.data, i, n3->base.chunks[i]);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, n3->children[i], level + 1,
+ recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ for (int i = 0; i < node->count; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_32 *n32 = (RT_NODE_LEAF_32 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+ spaces.data, i, n32->base.chunks[i]);
+ }
+ else
+ {
+ RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X",
+ spaces.data, i, n32->base.chunks[i]);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, n32->children[i], level + 1,
+ recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_BASE_125 *b125 = (RT_NODE_BASE_125 *) node;
+ char *sep = "";
+
+ appendStringInfo(buf, "%sslot_idxs: ", spaces.data);
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+ continue;
+
+ appendStringInfo(buf, "%s[%d]=%d ",
+ sep, i, b125->slot_idxs[i]);
+ sep = ",";
+ }
+
+ appendStringInfo(buf, "\n%sisset-bitmap: ", spaces.data);
+ for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+ appendStringInfo(buf, "%X ", ((uint8 *) b125->isset)[i]);
+ appendStringInfo(buf, "\n");
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+ continue;
+
+ if (RT_NODE_IS_LEAF(node))
+ appendStringInfo(buf, "%schunk 0x%X\n",
+ spaces.data, i);
+ else
+ {
+ RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) b125;
+
+ appendStringInfo(buf, "%schunk 0x%X",
+ spaces.data, i);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i),
+ level + 1, recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+ appendStringInfo(buf, "%sisset-bitmap: ", spaces.data);
+ for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+ appendStringInfo(buf, "%X ", ((uint8 *) n256->isset)[i]);
+ appendStringInfo(buf, "\n");
+ }
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ appendStringInfo(buf, "%schunk 0x%X\n",
+ spaces.data, i);
+ }
+ else
+ {
+ RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ appendStringInfo(buf, "%schunk 0x%X",
+ spaces.data, i);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i),
+ level + 1, recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ }
+}
+
+RT_SCOPE void
+RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key)
+{
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL node;
+ StringInfoData buf;
+ int shift;
+ int level = 0;
+
+ RT_STATS(tree);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "empty tree\n");
+ return;
+ }
+
+ if (key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "key " UINT64_FORMAT "(0x" RT_UINT64_FORMAT_HEX ") is larger than max val\n",
+ key, key);
+ return;
+ }
+
+ initStringInfo(&buf);
+ allocnode = tree->ctl->root;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift = node->shift;
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC child;
+
+ RT_DUMP_NODE(tree, allocnode, level, false, &buf);
+
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_VALUE_TYPE dummy;
+
+ /* We reached at a leaf node, find the corresponding slot */
+ RT_NODE_SEARCH_LEAF(node, key, &dummy);
+
+ break;
+ }
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ break;
+
+ allocnode = child;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift -= RT_NODE_SPAN;
+ level++;
+ }
+ RT_UNLOCK(tree);
+
+ fprintf(stderr, "%s", buf.data);
+}
+
+RT_SCOPE void
+RT_DUMP(RT_RADIX_TREE *tree)
+{
+ StringInfoData buf;
+
+ RT_STATS(tree);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "empty tree\n");
+ return;
+ }
+
+ initStringInfo(&buf);
+
+ RT_DUMP_NODE(tree, tree->ctl->root, 0, true, &buf);
+ RT_UNLOCK(tree);
+
+ fprintf(stderr, "%s",buf.data);
+}
+#endif
+
+#endif /* RT_DEFINE */
+
+
+/* undefine external parameters, so next radix tree can be defined */
+#undef RT_PREFIX
+#undef RT_SCOPE
+#undef RT_DECLARE
+#undef RT_DEFINE
+#undef RT_VALUE_TYPE
+
+/* locally declared macros */
+#undef RT_MAKE_PREFIX
+#undef RT_MAKE_NAME
+#undef RT_MAKE_NAME_
+#undef RT_NODE_SPAN
+#undef RT_NODE_MAX_SLOTS
+#undef RT_CHUNK_MASK
+#undef RT_MAX_SHIFT
+#undef RT_MAX_LEVEL
+#undef RT_GET_KEY_CHUNK
+#undef RT_BM_IDX
+#undef RT_BM_BIT
+#undef RT_LOCK_EXCLUSIVE
+#undef RT_LOCK_SHARED
+#undef RT_UNLOCK
+#undef RT_NODE_IS_LEAF
+#undef RT_NODE_MUST_GROW
+#undef RT_NODE_KIND_COUNT
+#undef RT_SIZE_CLASS_COUNT
+#undef RT_SLOT_IDX_LIMIT
+#undef RT_INVALID_SLOT_IDX
+#undef RT_SLAB_BLOCK_SIZE
+#undef RT_RADIX_TREE_MAGIC
+#undef RT_UINT64_FORMAT_HEX
+
+/* type declarations */
+#undef RT_RADIX_TREE
+#undef RT_RADIX_TREE_CONTROL
+#undef RT_PTR_LOCAL
+#undef RT_PTR_ALLOC
+#undef RT_INVALID_PTR_ALLOC
+#undef RT_HANDLE
+#undef RT_ITER
+#undef RT_NODE
+#undef RT_NODE_ITER
+#undef RT_NODE_KIND_3
+#undef RT_NODE_KIND_32
+#undef RT_NODE_KIND_125
+#undef RT_NODE_KIND_256
+#undef RT_NODE_BASE_3
+#undef RT_NODE_BASE_32
+#undef RT_NODE_BASE_125
+#undef RT_NODE_BASE_256
+#undef RT_NODE_INNER_3
+#undef RT_NODE_INNER_32
+#undef RT_NODE_INNER_125
+#undef RT_NODE_INNER_256
+#undef RT_NODE_LEAF_3
+#undef RT_NODE_LEAF_32
+#undef RT_NODE_LEAF_125
+#undef RT_NODE_LEAF_256
+#undef RT_SIZE_CLASS
+#undef RT_SIZE_CLASS_ELEM
+#undef RT_SIZE_CLASS_INFO
+#undef RT_CLASS_3
+#undef RT_CLASS_32_MIN
+#undef RT_CLASS_32_MAX
+#undef RT_CLASS_125
+#undef RT_CLASS_256
+
+/* function declarations */
+#undef RT_CREATE
+#undef RT_FREE
+#undef RT_ATTACH
+#undef RT_DETACH
+#undef RT_GET_HANDLE
+#undef RT_SEARCH
+#undef RT_SET
+#undef RT_BEGIN_ITERATE
+#undef RT_ITERATE_NEXT
+#undef RT_END_ITERATE
+#undef RT_USE_DELETE
+#undef RT_DELETE
+#undef RT_MEMORY_USAGE
+#undef RT_DUMP
+#undef RT_DUMP_NODE
+#undef RT_DUMP_SEARCH
+#undef RT_STATS
+
+/* internal helper functions */
+#undef RT_NEW_ROOT
+#undef RT_ALLOC_NODE
+#undef RT_INIT_NODE
+#undef RT_FREE_NODE
+#undef RT_FREE_RECURSE
+#undef RT_EXTEND_UP
+#undef RT_EXTEND_DOWN
+#undef RT_SWITCH_NODE_KIND
+#undef RT_COPY_NODE
+#undef RT_REPLACE_NODE
+#undef RT_PTR_GET_LOCAL
+#undef RT_PTR_ALLOC_IS_VALID
+#undef RT_NODE_3_SEARCH_EQ
+#undef RT_NODE_32_SEARCH_EQ
+#undef RT_NODE_3_GET_INSERTPOS
+#undef RT_NODE_32_GET_INSERTPOS
+#undef RT_CHUNK_CHILDREN_ARRAY_SHIFT
+#undef RT_CHUNK_VALUES_ARRAY_SHIFT
+#undef RT_CHUNK_CHILDREN_ARRAY_DELETE
+#undef RT_CHUNK_VALUES_ARRAY_DELETE
+#undef RT_CHUNK_CHILDREN_ARRAY_COPY
+#undef RT_CHUNK_VALUES_ARRAY_COPY
+#undef RT_NODE_125_IS_CHUNK_USED
+#undef RT_NODE_INNER_125_GET_CHILD
+#undef RT_NODE_LEAF_125_GET_VALUE
+#undef RT_NODE_INNER_256_IS_CHUNK_USED
+#undef RT_NODE_LEAF_256_IS_CHUNK_USED
+#undef RT_NODE_INNER_256_GET_CHILD
+#undef RT_NODE_LEAF_256_GET_VALUE
+#undef RT_NODE_INNER_256_SET
+#undef RT_NODE_LEAF_256_SET
+#undef RT_NODE_INNER_256_DELETE
+#undef RT_NODE_LEAF_256_DELETE
+#undef RT_KEY_GET_SHIFT
+#undef RT_SHIFT_GET_MAX_VAL
+#undef RT_NODE_SEARCH_INNER
+#undef RT_NODE_SEARCH_LEAF
+#undef RT_NODE_UPDATE_INNER
+#undef RT_NODE_DELETE_INNER
+#undef RT_NODE_DELETE_LEAF
+#undef RT_NODE_INSERT_INNER
+#undef RT_NODE_INSERT_LEAF
+#undef RT_NODE_INNER_ITERATE_NEXT
+#undef RT_NODE_LEAF_ITERATE_NEXT
+#undef RT_RT_ITER_SET_NODE_FROM
+#undef RT_VERIFY_NODE
+
+#undef RT_DEBUG
diff --git a/src/include/lib/radixtree_delete_impl.h b/src/include/lib/radixtree_delete_impl.h
new file mode 100644
index 0000000000..5f6dda1f12
--- /dev/null
+++ b/src/include/lib/radixtree_delete_impl.h
@@ -0,0 +1,122 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_delete_impl.h
+ * Common implementation for deletion in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ * TODO: Shrink nodes when deletion would allow them to fit in a smaller
+ * size class.
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_delete_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_DELETE(n3->base.chunks, n3->values,
+ n3->base.n.count, idx);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_DELETE(n3->base.chunks, n3->children,
+ n3->base.n.count, idx);
+#endif
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_DELETE(n32->base.chunks, n32->values,
+ n32->base.n.count, idx);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_DELETE(n32->base.chunks, n32->children,
+ n32->base.n.count, idx);
+#endif
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+ int idx;
+ int bitnum;
+
+ if (slotpos == RT_INVALID_SLOT_IDX)
+ return false;
+
+ idx = RT_BM_IDX(slotpos);
+ bitnum = RT_BM_BIT(slotpos);
+ n125->base.isset[idx] &= ~((bitmapword) 1 << bitnum);
+ n125->base.slot_idxs[chunk] = RT_INVALID_SLOT_IDX;
+
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_DELETE(n256, chunk);
+#else
+ RT_NODE_INNER_256_DELETE(n256, chunk);
+#endif
+ break;
+ }
+ }
+
+ /* update statistics */
+ node->count--;
+
+ return true;
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
new file mode 100644
index 0000000000..d56e58dcac
--- /dev/null
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -0,0 +1,328 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_insert_impl.h
+ * Common implementation for insertion in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_insert_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ const bool is_leaf = true;
+ bool chunk_exists = false;
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+ const bool is_leaf = false;
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ int idx = RT_NODE_3_SEARCH_EQ(&n3->base, chunk);
+
+ if (idx != -1)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n3->values[idx] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n3)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE32_TYPE *new32;
+ const uint8 new_kind = RT_NODE_KIND_32;
+ const RT_SIZE_CLASS new_class = RT_CLASS_32_MIN;
+
+ /* grow node from 3 to 32 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_COPY(n3->base.chunks, n3->values,
+ new32->base.chunks, new32->values);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_COPY(n3->base.chunks, n3->children,
+ new32->base.chunks, new32->children);
+#endif
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
+ int count = n3->base.n.count;
+
+ /* shift chunks and children */
+ if (insertpos < count)
+ {
+ Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_SHIFT(n3->base.chunks, n3->values,
+ count, insertpos);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_SHIFT(n3->base.chunks, n3->children,
+ count, insertpos);
+#endif
+ }
+
+ n3->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+ n3->values[insertpos] = *value_p;
+#else
+ n3->children[insertpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_32:
+ {
+ const RT_SIZE_CLASS_ELEM class32_max = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX];
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ int idx = RT_NODE_32_SEARCH_EQ(&n32->base, chunk);
+
+ if (idx != -1)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n32->values[idx] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n32)) &&
+ n32->base.n.fanout < class32_max.fanout)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ const RT_SIZE_CLASS_ELEM class32_min = RT_SIZE_CLASS_INFO[RT_CLASS_32_MIN];
+ const RT_SIZE_CLASS new_class = RT_CLASS_32_MAX;
+
+ Assert(n32->base.n.fanout == class32_min.fanout);
+
+ /* grow to the next size class of this kind */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ n32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ memcpy(newnode, node, class32_min.leaf_size);
+#else
+ memcpy(newnode, node, class32_min.inner_size);
+#endif
+ newnode->fanout = class32_max.fanout;
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+
+ if (unlikely(RT_NODE_MUST_GROW(n32)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE125_TYPE *new125;
+ const uint8 new_kind = RT_NODE_KIND_125;
+ const RT_SIZE_CLASS new_class = RT_CLASS_125;
+
+ Assert(n32->base.n.fanout == class32_max.fanout);
+
+ /* grow node from 32 to 125 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new125 = (RT_NODE125_TYPE *) newnode;
+
+ for (int i = 0; i < class32_max.fanout; i++)
+ {
+ new125->base.slot_idxs[n32->base.chunks[i]] = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ new125->values[i] = n32->values[i];
+#else
+ new125->children[i] = n32->children[i];
+#endif
+ }
+
+ /*
+ * Since we just copied a dense array, we can set the bits
+ * using a single store, provided the length of that array
+ * is at most the number of bits in a bitmapword.
+ */
+ Assert(class32_max.fanout <= sizeof(bitmapword) * BITS_PER_BYTE);
+ new125->base.isset[0] = (bitmapword) (((uint64) 1 << class32_max.fanout) - 1);
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
+ int count = n32->base.n.count;
+
+ if (insertpos < count)
+ {
+ Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_SHIFT(n32->base.chunks, n32->values,
+ count, insertpos);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_SHIFT(n32->base.chunks, n32->children,
+ count, insertpos);
+#endif
+ }
+
+ n32->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+ n32->values[insertpos] = *value_p;
+#else
+ n32->children[insertpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos;
+ int cnt = 0;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ slotpos = n125->base.slot_idxs[chunk];
+ if (slotpos != RT_INVALID_SLOT_IDX)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n125->values[slotpos] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n125)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE256_TYPE *new256;
+ const uint8 new_kind = RT_NODE_KIND_256;
+ const RT_SIZE_CLASS new_class = RT_CLASS_256;
+
+ /* grow node from 125 to 256 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new256 = (RT_NODE256_TYPE *) newnode;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS && cnt < n125->base.n.count; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+ continue;
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_SET(new256, i, RT_NODE_LEAF_125_GET_VALUE(n125, i));
+#else
+ RT_NODE_INNER_256_SET(new256, i, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+ cnt++;
+ }
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int idx;
+ bitmapword inverse;
+
+ /* get the first word with at least one bit not set */
+ for (idx = 0; idx < RT_BM_IDX(RT_SLOT_IDX_LIMIT); idx++)
+ {
+ if (n125->base.isset[idx] < ~((bitmapword) 0))
+ break;
+ }
+
+ /* To get the first unset bit in X, get the first set bit in ~X */
+ inverse = ~(n125->base.isset[idx]);
+ slotpos = idx * BITS_PER_BITMAPWORD;
+ slotpos += bmw_rightmost_one_pos(inverse);
+ Assert(slotpos < node->fanout);
+
+ /* mark the slot used */
+ n125->base.isset[idx] |= bmw_rightmost_one(inverse);
+ n125->base.slot_idxs[chunk] = slotpos;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ n125->values[slotpos] = *value_p;
+#else
+ n125->children[slotpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ chunk_exists = RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk);
+ Assert(chunk_exists || node->count < RT_NODE_MAX_SLOTS);
+ RT_NODE_LEAF_256_SET(n256, chunk, *value_p);
+#else
+ Assert(node->count < RT_NODE_MAX_SLOTS);
+ RT_NODE_INNER_256_SET(n256, chunk, child);
+#endif
+ break;
+ }
+ }
+
+ /* Update statistics */
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!chunk_exists)
+ node->count++;
+#else
+ node->count++;
+#endif
+
+ /*
+ * Done. Finally, verify the chunk and value is inserted or replaced
+ * properly in the node.
+ */
+ RT_VERIFY_NODE(node);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ return chunk_exists;
+#else
+ return;
+#endif
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_iter_impl.h b/src/include/lib/radixtree_iter_impl.h
new file mode 100644
index 0000000000..5c1034768e
--- /dev/null
+++ b/src/include/lib/radixtree_iter_impl.h
@@ -0,0 +1,144 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_iter_impl.h
+ * Common implementation for iteration in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_iter_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 key_chunk = 0;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(value_p != NULL);
+ Assert(RT_NODE_IS_LEAF(node_iter->node));
+#else
+ RT_PTR_LOCAL child = NULL;
+
+ Assert(!RT_NODE_IS_LEAF(node_iter->node));
+#endif
+
+#ifdef RT_SHMEM
+ Assert(iter->tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ switch (node_iter->node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node_iter->node;
+
+ if (node_iter->idx >= n3->base.n.count)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n3->values[node_iter->idx];
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, n3->children[node_iter->idx]);
+#endif
+ key_chunk = n3->base.chunks[node_iter->idx];
+ node_iter->idx++;
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node_iter->node;
+
+ if (node_iter->idx >= n32->base.n.count)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n32->values[node_iter->idx];
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, n32->children[node_iter->idx]);
+#endif
+ key_chunk = n32->base.chunks[node_iter->idx];
+ node_iter->idx++;
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node_iter->node;
+ int chunk;
+
+ for (chunk = node_iter->idx; chunk < RT_NODE_MAX_SLOTS; chunk++)
+ {
+ if (RT_NODE_125_IS_CHUNK_USED((RT_NODE_BASE_125 *) n125, chunk))
+ break;
+ }
+
+ if (chunk >= RT_NODE_MAX_SLOTS)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_125_GET_VALUE(n125, chunk);
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_125_GET_CHILD(n125, chunk));
+#endif
+ key_chunk = chunk;
+ node_iter->idx = chunk + 1;
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node_iter->node;
+ int chunk;
+
+ for (chunk = node_iter->idx; chunk < RT_NODE_MAX_SLOTS; chunk++)
+ {
+#ifdef RT_NODE_LEVEL_LEAF
+ if (RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ break;
+ }
+
+ if (chunk >= RT_NODE_MAX_SLOTS)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_256_GET_VALUE(n256, chunk);
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_256_GET_CHILD(n256, chunk));
+#endif
+ key_chunk = chunk;
+ node_iter->idx = chunk + 1;
+ break;
+ }
+ }
+
+ /* Update the part of the key */
+ iter->key &= ~(((uint64) RT_CHUNK_MASK) << node_iter->node->shift);
+ iter->key |= (((uint64) key_chunk) << node_iter->node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ return true;
+#else
+ return child;
+#endif
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_search_impl.h b/src/include/lib/radixtree_search_impl.h
new file mode 100644
index 0000000000..a8925c75d0
--- /dev/null
+++ b/src/include/lib/radixtree_search_impl.h
@@ -0,0 +1,138 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_search_impl.h
+ * Common implementation for search in leaf and inner nodes, plus
+ * update for inner nodes only.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_search_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(value_p != NULL);
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+#ifndef RT_ACTION_UPDATE
+ Assert(child_p != NULL);
+#endif
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+#ifdef RT_ACTION_UPDATE
+ Assert(idx >= 0);
+ n3->children[idx] = new_child;
+#else
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n3->values[idx];
+#else
+ *child_p = n3->children[idx];
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+#ifdef RT_ACTION_UPDATE
+ Assert(idx >= 0);
+ n32->children[idx] = new_child;
+#else
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n32->values[idx];
+#else
+ *child_p = n32->children[idx];
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+
+#ifdef RT_ACTION_UPDATE
+ Assert(slotpos != RT_INVALID_SLOT_IDX);
+ n125->children[slotpos] = new_child;
+#else
+ if (slotpos == RT_INVALID_SLOT_IDX)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_125_GET_VALUE(n125, chunk);
+#else
+ *child_p = RT_NODE_INNER_125_GET_CHILD(n125, chunk);
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_ACTION_UPDATE
+ RT_NODE_INNER_256_SET(n256, chunk, new_child);
+#else
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_256_GET_VALUE(n256, chunk);
+#else
+ *child_p = RT_NODE_INNER_256_GET_CHILD(n256, chunk);
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ }
+
+#ifdef RT_ACTION_UPDATE
+ return;
+#else
+ return true;
+#endif /* RT_ACTION_UPDATE */
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/utils/dsa.h b/src/include/utils/dsa.h
index 3ce4ee300a..2af215484f 100644
--- a/src/include/utils/dsa.h
+++ b/src/include/utils/dsa.h
@@ -121,6 +121,7 @@ extern dsa_handle dsa_get_handle(dsa_area *area);
extern dsa_pointer dsa_allocate_extended(dsa_area *area, size_t size, int flags);
extern void dsa_free(dsa_area *area, dsa_pointer dp);
extern void *dsa_get_address(dsa_area *area, dsa_pointer dp);
+extern size_t dsa_get_total_size(dsa_area *area);
extern void dsa_trim(dsa_area *area);
extern void dsa_dump(dsa_area *area);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index c629cbe383..9659eb85d7 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -28,6 +28,7 @@ SUBDIRS = \
test_pg_db_role_setting \
test_pg_dump \
test_predtest \
+ test_radixtree \
test_rbtree \
test_regex \
test_rls_hooks \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 1baa6b558d..232cbdac80 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -24,6 +24,7 @@ subdir('test_parser')
subdir('test_pg_db_role_setting')
subdir('test_pg_dump')
subdir('test_predtest')
+subdir('test_radixtree')
subdir('test_rbtree')
subdir('test_regex')
subdir('test_rls_hooks')
diff --git a/src/test/modules/test_radixtree/.gitignore b/src/test/modules/test_radixtree/.gitignore
new file mode 100644
index 0000000000..5dcb3ff972
--- /dev/null
+++ b/src/test/modules/test_radixtree/.gitignore
@@ -0,0 +1,4 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
diff --git a/src/test/modules/test_radixtree/Makefile b/src/test/modules/test_radixtree/Makefile
new file mode 100644
index 0000000000..da06b93da3
--- /dev/null
+++ b/src/test/modules/test_radixtree/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_radixtree/Makefile
+
+MODULE_big = test_radixtree
+OBJS = \
+ $(WIN32RES) \
+ test_radixtree.o
+PGFILEDESC = "test_radixtree - test code for src/backend/lib/radixtree.c"
+
+EXTENSION = test_radixtree
+DATA = test_radixtree--1.0.sql
+
+REGRESS = test_radixtree
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_radixtree
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_radixtree/README b/src/test/modules/test_radixtree/README
new file mode 100644
index 0000000000..a8b271869a
--- /dev/null
+++ b/src/test/modules/test_radixtree/README
@@ -0,0 +1,7 @@
+test_integerset contains unit tests for testing the integer set implementation
+in src/backend/lib/integerset.c.
+
+The tests verify the correctness of the implementation, but they can also be
+used as a micro-benchmark. If you set the 'intset_test_stats' flag in
+test_integerset.c, the tests will print extra information about execution time
+and memory usage.
diff --git a/src/test/modules/test_radixtree/expected/test_radixtree.out b/src/test/modules/test_radixtree/expected/test_radixtree.out
new file mode 100644
index 0000000000..7ad1ce3605
--- /dev/null
+++ b/src/test/modules/test_radixtree/expected/test_radixtree.out
@@ -0,0 +1,38 @@
+CREATE EXTENSION test_radixtree;
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
+NOTICE: testing basic operations with leaf node 3
+NOTICE: testing basic operations with inner node 3
+NOTICE: testing basic operations with leaf node 15
+NOTICE: testing basic operations with inner node 15
+NOTICE: testing basic operations with leaf node 32
+NOTICE: testing basic operations with inner node 32
+NOTICE: testing basic operations with leaf node 125
+NOTICE: testing basic operations with inner node 125
+NOTICE: testing basic operations with leaf node 256
+NOTICE: testing basic operations with inner node 256
+NOTICE: testing radix tree node types with shift "0"
+NOTICE: testing radix tree node types with shift "8"
+NOTICE: testing radix tree node types with shift "16"
+NOTICE: testing radix tree node types with shift "24"
+NOTICE: testing radix tree node types with shift "32"
+NOTICE: testing radix tree node types with shift "40"
+NOTICE: testing radix tree node types with shift "48"
+NOTICE: testing radix tree node types with shift "56"
+NOTICE: testing radix tree with pattern "all ones"
+NOTICE: testing radix tree with pattern "alternating bits"
+NOTICE: testing radix tree with pattern "clusters of ten"
+NOTICE: testing radix tree with pattern "clusters of hundred"
+NOTICE: testing radix tree with pattern "one-every-64k"
+NOTICE: testing radix tree with pattern "sparse"
+NOTICE: testing radix tree with pattern "single values, distance > 2^32"
+NOTICE: testing radix tree with pattern "clusters, distance > 2^32"
+NOTICE: testing radix tree with pattern "clusters, distance > 2^60"
+ test_radixtree
+----------------
+
+(1 row)
+
diff --git a/src/test/modules/test_radixtree/meson.build b/src/test/modules/test_radixtree/meson.build
new file mode 100644
index 0000000000..6add06bbdb
--- /dev/null
+++ b/src/test/modules/test_radixtree/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_radixtree_sources = files(
+ 'test_radixtree.c',
+)
+
+if host_system == 'windows'
+ test_radixtree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_radixtree',
+ '--FILEDESC', 'test_radixtree - test code for src/include//lib/radixtree.h',])
+endif
+
+test_radixtree = shared_module('test_radixtree',
+ test_radixtree_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += test_radixtree
+
+install_data(
+ 'test_radixtree.control',
+ 'test_radixtree--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'test_radixtree',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'test_radixtree',
+ ],
+ },
+}
diff --git a/src/test/modules/test_radixtree/sql/test_radixtree.sql b/src/test/modules/test_radixtree/sql/test_radixtree.sql
new file mode 100644
index 0000000000..41ece5e9f5
--- /dev/null
+++ b/src/test/modules/test_radixtree/sql/test_radixtree.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_radixtree;
+
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
diff --git a/src/test/modules/test_radixtree/test_radixtree--1.0.sql b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
new file mode 100644
index 0000000000..074a5a7ea7
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_radixtree/test_radixtree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_radixtree" to load this file. \quit
+
+CREATE FUNCTION test_radixtree()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
new file mode 100644
index 0000000000..5a169854d9
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -0,0 +1,712 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_radixtree.c
+ * Test radixtree set data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_radixtree/test_radixtree.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "storage/lwlock.h"
+#include "utils/memutils.h"
+#include "utils/timestamp.h"
+
+#define UINT64_HEX_FORMAT "%" INT64_MODIFIER "X"
+
+/*
+ * The tests pass with uint32, but build with warnings because the string
+ * format expects uint64.
+ */
+typedef uint64 TestValueType;
+
+/*
+ * If you enable this, the "pattern" tests will print information about
+ * how long populating, probing, and iterating the test set takes, and
+ * how much memory the test set consumed. That can be used as
+ * micro-benchmark of various operations and input patterns (you might
+ * want to increase the number of values used in each of the test, if
+ * you do that, to reduce noise).
+ *
+ * The information is printed to the server's stderr, mostly because
+ * that's where MemoryContextStats() output goes.
+ */
+static const bool rt_test_stats = false;
+
+/*
+ * XXX: should we expose and use RT_SIZE_CLASS and RT_SIZE_CLASS_INFO?
+ */
+static int rt_node_class_fanouts[] = {
+ 3, /* RT_CLASS_3 */
+ 15, /* RT_CLASS_32_MIN */
+ 32, /* RT_CLASS_32_MAX */
+ 125, /* RT_CLASS_125 */
+ 256 /* RT_CLASS_256 */
+};
+/*
+ * A struct to define a pattern of integers, for use with the test_pattern()
+ * function.
+ */
+typedef struct
+{
+ char *test_name; /* short name of the test, for humans */
+ char *pattern_str; /* a bit pattern */
+ uint64 spacing; /* pattern repeats at this interval */
+ uint64 num_values; /* number of integers to set in total */
+} test_spec;
+
+/* Test patterns borrowed from test_integerset.c */
+static const test_spec test_specs[] = {
+ {
+ "all ones", "1111111111",
+ 10, 1000000
+ },
+ {
+ "alternating bits", "0101010101",
+ 10, 1000000
+ },
+ {
+ "clusters of ten", "1111111111",
+ 10000, 1000000
+ },
+ {
+ "clusters of hundred",
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111",
+ 10000, 1000000
+ },
+ {
+ "one-every-64k", "1",
+ 65536, 1000000
+ },
+ {
+ "sparse", "100000000000000000000000000000001",
+ 10000000, 1000000
+ },
+ {
+ "single values, distance > 2^32", "1",
+ UINT64CONST(10000000000), 100000
+ },
+ {
+ "clusters, distance > 2^32", "10101010",
+ UINT64CONST(10000000000), 1000000
+ },
+ {
+ "clusters, distance > 2^60", "10101010",
+ UINT64CONST(2000000000000000000),
+ 23 /* can't be much higher than this, or we
+ * overflow uint64 */
+ }
+};
+
+/* define the radix tree implementation to test */
+#define RT_PREFIX rt
+#define RT_SCOPE
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE TestValueType
+/* #define RT_SHMEM */
+#include "lib/radixtree.h"
+
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+ return tree->ctl->num_keys;
+}
+
+PG_MODULE_MAGIC;
+
+PG_FUNCTION_INFO_V1(test_radixtree);
+
+static void
+test_empty(void)
+{
+ rt_radix_tree *radixtree;
+ rt_iter *iter;
+ TestValueType dummy;
+ uint64 key;
+ TestValueType val;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
+ dsa = dsa_create(tranche_id);
+
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ if (rt_search(radixtree, 0, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_search(radixtree, 1, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_search(radixtree, PG_UINT64_MAX, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_delete(radixtree, 0))
+ elog(ERROR, "rt_delete on empty tree returned true");
+
+ if (rt_num_entries(radixtree) != 0)
+ elog(ERROR, "rt_num_entries on empty tree return non-zero");
+
+ iter = rt_begin_iterate(radixtree);
+
+ if (rt_iterate_next(iter, &key, &val))
+ elog(ERROR, "rt_itereate_next on empty tree returned true");
+
+ rt_end_iterate(iter);
+
+ rt_free(radixtree);
+
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+static void
+test_basic(int children, bool test_inner)
+{
+ rt_radix_tree *radixtree;
+ uint64 *keys;
+ int shift = test_inner ? 8 : 0;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing basic operations with %s node %d",
+ test_inner ? "inner" : "leaf", children);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ /* prepare keys in order like 1, 32, 2, 31, 2, ... */
+ keys = palloc(sizeof(uint64) * children);
+ for (int i = 0; i < children; i++)
+ {
+ if (i % 2 == 0)
+ keys[i] = (uint64) ((i / 2) + 1) << shift;
+ else
+ keys[i] = (uint64) (children - (i / 2)) << shift;
+ }
+
+ /* insert keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (rt_set(radixtree, keys[i], (TestValueType*) &keys[i]))
+ elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+ }
+
+ /* look up keys */
+ for (int i = 0; i < children; i++)
+ {
+ TestValueType value;
+
+ if (!rt_search(radixtree, keys[i], &value))
+ elog(ERROR, "could not find key 0x" UINT64_HEX_FORMAT, keys[i]);
+ if (value != (TestValueType) keys[i])
+ elog(ERROR, "rt_search returned 0x" UINT64_HEX_FORMAT ", expected " UINT64_HEX_FORMAT,
+ value, (TestValueType) keys[i]);
+ }
+
+ /* update keys */
+ for (int i = 0; i < children; i++)
+ {
+ TestValueType update = keys[i] + 1;
+ if (!rt_set(radixtree, keys[i], (TestValueType*) &update))
+ elog(ERROR, "could not update key 0x" UINT64_HEX_FORMAT, keys[i]);
+ }
+
+ /* repeat deleting and inserting keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (!rt_delete(radixtree, keys[i]))
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, keys[i]);
+ if (rt_set(radixtree, keys[i], (TestValueType*) &keys[i]))
+ elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+ }
+
+ pfree(keys);
+ rt_free(radixtree);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Check if keys from start to end with the shift exist in the tree.
+ */
+static void
+check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end)
+{
+ for (int i = start; i <= end; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ TestValueType val;
+
+ if (!rt_search(radixtree, key, &val))
+ elog(ERROR, "key 0x" UINT64_HEX_FORMAT " is not found on node-%d",
+ key, end);
+ if (val != (TestValueType) key)
+ elog(ERROR, "rt_search with key 0x" UINT64_HEX_FORMAT " returns 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+ key, val, key);
+ }
+}
+
+/*
+ * Insert 256 key-value pairs, and check if keys are properly inserted on each
+ * node class.
+ */
+/* Test keys [0, 256) */
+#define NODE_TYPE_TEST_KEY_MIN 0
+#define NODE_TYPE_TEST_KEY_MAX 256
+static void
+test_node_types_insert_asc(rt_radix_tree *radixtree, uint8 shift)
+{
+ uint64 num_entries;
+ int node_class_idx = 0;
+ uint64 key_checked = 0;
+
+ for (int i = NODE_TYPE_TEST_KEY_MIN; i < NODE_TYPE_TEST_KEY_MAX; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_set(radixtree, key, (TestValueType *) &key);
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " is found", key);
+
+ /*
+ * After filling all slots in each node type, check if the values
+ * are stored properly.
+ */
+ if ((i + 1) == rt_node_class_fanouts[node_class_idx])
+ {
+ check_search_on_node(radixtree, shift, key_checked, i);
+ key_checked = i;
+ node_class_idx++;
+ }
+ }
+
+ num_entries = rt_num_entries(radixtree);
+ if (num_entries != 256)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+/*
+ * Similar to test_node_types_insert_asc(), but inserts keys in descending order.
+ */
+static void
+test_node_types_insert_desc(rt_radix_tree *radixtree, uint8 shift)
+{
+ uint64 num_entries;
+ int node_class_idx = 0;
+ uint64 key_checked = NODE_TYPE_TEST_KEY_MAX - 1;
+
+ for (int i = NODE_TYPE_TEST_KEY_MAX - 1; i >= NODE_TYPE_TEST_KEY_MIN; i--)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_set(radixtree, key, (TestValueType *) &key);
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " is found", key);
+
+ if ((i + 1) == rt_node_class_fanouts[node_class_idx])
+ {
+ check_search_on_node(radixtree, shift, i, key_checked);
+ key_checked = i;
+ node_class_idx++;
+ }
+ }
+
+ num_entries = rt_num_entries(radixtree);
+ if (num_entries != 256)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+static void
+test_node_types_delete(rt_radix_tree *radixtree, uint8 shift)
+{
+ uint64 num_entries;
+
+ for (int i = NODE_TYPE_TEST_KEY_MIN; i < NODE_TYPE_TEST_KEY_MAX; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_delete(radixtree, key);
+
+ if (!found)
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, key);
+ }
+
+ num_entries = rt_num_entries(radixtree);
+
+ /* The tree must be empty */
+ if (num_entries != 0)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+/*
+ * Test for inserting and deleting key-value pairs to each node type at the given shift
+ * level.
+ */
+static void
+test_node_types(uint8 shift)
+{
+ rt_radix_tree *radixtree;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing radix tree node types with shift \"%d\"", shift);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ /*
+ * Insert and search entries for every node type at the 'shift' level,
+ * then delete all entries to make it empty, and insert and search entries
+ * again.
+ */
+ test_node_types_insert_asc(radixtree, shift);
+ test_node_types_delete(radixtree, shift);
+ test_node_types_insert_desc(radixtree, shift);
+
+ rt_free(radixtree);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Test with a repeating pattern, defined by the 'spec'.
+ */
+static void
+test_pattern(const test_spec * spec)
+{
+ rt_radix_tree *radixtree;
+ rt_iter *iter;
+ MemoryContext radixtree_ctx;
+ TimestampTz starttime;
+ TimestampTz endtime;
+ uint64 n;
+ uint64 last_int;
+ uint64 ndeleted;
+ uint64 nbefore;
+ uint64 nafter;
+ int patternlen;
+ uint64 *pattern_values;
+ uint64 pattern_num_values;
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing radix tree with pattern \"%s\"", spec->test_name);
+ if (rt_test_stats)
+ fprintf(stderr, "-----\ntesting radix tree with pattern \"%s\"\n", spec->test_name);
+
+ /* Pre-process the pattern, creating an array of integers from it. */
+ patternlen = strlen(spec->pattern_str);
+ pattern_values = palloc(patternlen * sizeof(uint64));
+ pattern_num_values = 0;
+ for (int i = 0; i < patternlen; i++)
+ {
+ if (spec->pattern_str[i] == '1')
+ pattern_values[pattern_num_values++] = i;
+ }
+
+ /*
+ * Allocate the radix tree.
+ *
+ * Allocate it in a separate memory context, so that we can print its
+ * memory usage easily.
+ */
+ radixtree_ctx = AllocSetContextCreate(CurrentMemoryContext,
+ "radixtree test",
+ ALLOCSET_SMALL_SIZES);
+ MemoryContextSetIdentifier(radixtree_ctx, spec->test_name);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(radixtree_ctx, dsa, tranche_id);
+#else
+ radixtree = rt_create(radixtree_ctx);
+#endif
+
+
+ /*
+ * Add values to the set.
+ */
+ starttime = GetCurrentTimestamp();
+
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ uint64 x = 0;
+
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ bool found;
+
+ x = last_int + pattern_values[i];
+
+ found = rt_set(radixtree, x, (TestValueType*) &x);
+
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " found", x);
+
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+
+ endtime = GetCurrentTimestamp();
+
+ if (rt_test_stats)
+ fprintf(stderr, "added " UINT64_FORMAT " values in %d ms\n",
+ spec->num_values, (int) (endtime - starttime) / 1000);
+
+ /*
+ * Print stats on the amount of memory used.
+ *
+ * We print the usage reported by rt_memory_usage(), as well as the stats
+ * from the memory context. They should be in the same ballpark, but it's
+ * hard to automate testing that, so if you're making changes to the
+ * implementation, just observe that manually.
+ */
+ if (rt_test_stats)
+ {
+ uint64 mem_usage;
+
+ /*
+ * Also print memory usage as reported by rt_memory_usage(). It
+ * should be in the same ballpark as the usage reported by
+ * MemoryContextStats().
+ */
+ mem_usage = rt_memory_usage(radixtree);
+ fprintf(stderr, "rt_memory_usage() reported " UINT64_FORMAT " (%0.2f bytes / integer)\n",
+ mem_usage, (double) mem_usage / spec->num_values);
+
+ MemoryContextStats(radixtree_ctx);
+ }
+
+ /* Check that rt_num_entries works */
+ n = rt_num_entries(radixtree);
+ if (n != spec->num_values)
+ elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT, n, spec->num_values);
+
+ /*
+ * Test random-access probes with rt_search()
+ */
+ starttime = GetCurrentTimestamp();
+
+ for (n = 0; n < 100000; n++)
+ {
+ bool found;
+ bool expected;
+ uint64 x;
+ TestValueType v;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit values that are actually in the set.
+ */
+ x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+ /* Do we expect this value to be present in the set? */
+ if (x >= last_int)
+ expected = false;
+ else
+ {
+ uint64 idx = x % spec->spacing;
+
+ if (idx >= patternlen)
+ expected = false;
+ else if (spec->pattern_str[idx] == '1')
+ expected = true;
+ else
+ expected = false;
+ }
+
+ /* Is it present according to rt_search() ? */
+ found = rt_search(radixtree, x, &v);
+
+ if (found != expected)
+ elog(ERROR, "mismatch at 0x" UINT64_HEX_FORMAT ": %d vs %d", x, found, expected);
+ if (found && (v != (TestValueType) x))
+ elog(ERROR, "found 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+ v, x);
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "probed " UINT64_FORMAT " values in %d ms\n",
+ n, (int) (endtime - starttime) / 1000);
+
+ /*
+ * Test iterator
+ */
+ starttime = GetCurrentTimestamp();
+
+ iter = rt_begin_iterate(radixtree);
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ uint64 expected = last_int + pattern_values[i];
+ uint64 x;
+ TestValueType val;
+
+ if (!rt_iterate_next(iter, &x, &val))
+ break;
+
+ if (x != expected)
+ elog(ERROR,
+ "iterate returned wrong key; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d",
+ x, expected, i);
+ if (val != (TestValueType) expected)
+ elog(ERROR,
+ "iterate returned wrong value; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d", x, expected, i);
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "iterated " UINT64_FORMAT " values in %d ms\n",
+ n, (int) (endtime - starttime) / 1000);
+
+ rt_end_iterate(iter);
+
+ if (n < spec->num_values)
+ elog(ERROR, "iterator stopped short after " UINT64_FORMAT " entries, expected " UINT64_FORMAT, n, spec->num_values);
+ if (n > spec->num_values)
+ elog(ERROR, "iterator returned " UINT64_FORMAT " entries, " UINT64_FORMAT " was expected", n, spec->num_values);
+
+ /*
+ * Test random-access probes with rt_delete()
+ */
+ starttime = GetCurrentTimestamp();
+
+ nbefore = rt_num_entries(radixtree);
+ ndeleted = 0;
+ for (n = 0; n < 1; n++)
+ {
+ bool found;
+ uint64 x;
+ TestValueType v;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit values that are actually in the set.
+ */
+ x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+ /* Is it present according to rt_search() ? */
+ found = rt_search(radixtree, x, &v);
+
+ if (!found)
+ continue;
+
+ /* If the key is found, delete it and check again */
+ if (!rt_delete(radixtree, x))
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, x);
+ if (rt_search(radixtree, x, &v))
+ elog(ERROR, "found deleted key 0x" UINT64_HEX_FORMAT, x);
+ if (rt_delete(radixtree, x))
+ elog(ERROR, "deleted already-deleted key 0x" UINT64_HEX_FORMAT, x);
+
+ ndeleted++;
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "deleted " UINT64_FORMAT " values in %d ms\n",
+ ndeleted, (int) (endtime - starttime) / 1000);
+
+ nafter = rt_num_entries(radixtree);
+
+ /* Check that rt_num_entries works */
+ if ((nbefore - ndeleted) != nafter)
+ elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT "after " UINT64_FORMAT " deletion",
+ nafter, (nbefore - ndeleted), ndeleted);
+
+ rt_free(radixtree);
+ MemoryContextDelete(radixtree_ctx);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+Datum
+test_radixtree(PG_FUNCTION_ARGS)
+{
+ test_empty();
+
+ for (int i = 0; i < lengthof(rt_node_class_fanouts); i++)
+ {
+ test_basic(rt_node_class_fanouts[i], false);
+ test_basic(rt_node_class_fanouts[i], true);
+ }
+
+ for (int shift = 0; shift <= (64 - 8); shift += 8)
+ test_node_types(shift);
+
+ /* Test different test patterns, with lots of entries */
+ for (int i = 0; i < lengthof(test_specs); i++)
+ test_pattern(&test_specs[i]);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_radixtree/test_radixtree.control b/src/test/modules/test_radixtree/test_radixtree.control
new file mode 100644
index 0000000000..e53f2a3e0c
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.control
@@ -0,0 +1,4 @@
+comment = 'Test code for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/test_radixtree'
+relocatable = true
diff --git a/src/tools/pginclude/cpluspluscheck b/src/tools/pginclude/cpluspluscheck
index 2c5042eb41..14b37e8eef 100755
--- a/src/tools/pginclude/cpluspluscheck
+++ b/src/tools/pginclude/cpluspluscheck
@@ -101,6 +101,12 @@ do
test "$f" = src/include/nodes/nodetags.h && continue
test "$f" = src/backend/nodes/nodetags.h && continue
+ # radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+ test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+ test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+ test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+ test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
# These files are not meant to be included standalone, because
# they contain lists that might have multiple use-cases.
test "$f" = src/include/access/rmgrlist.h && continue
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index abbba7aa63..d4d2f1da03 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -96,6 +96,12 @@ do
test "$f" = src/include/nodes/nodetags.h && continue
test "$f" = src/backend/nodes/nodetags.h && continue
+ # radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+ test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+ test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+ test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+ test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
# These files are not meant to be included standalone, because
# they contain lists that might have multiple use-cases.
test "$f" = src/include/access/rmgrlist.h && continue
--
2.31.1
[application/octet-stream] v30-0001-Introduce-helper-SIMD-functions-for-small-byte-a.patch (2.9K, ../../CAD21AoC+kjqkSnLLcbqtu=Td5HSx2H+iAwNHCeZcWqjy1w7k5g@mail.gmail.com/12-v30-0001-Introduce-helper-SIMD-functions-for-small-byte-a.patch)
download | inline diff:
From 22b578551e15e829e6649784eac8ec66d4a455c3 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 24 Oct 2022 14:07:09 +0900
Subject: [PATCH v30 01/11] Introduce helper SIMD functions for small byte
arrays
vector8_min - helper for emulating ">=" semantics
vector8_highbit_mask - used to turn the result of a vector
comparison into a bitmask
Masahiko Sawada
Reviewed by Nathan Bossart, additional adjustments by me
Discussion: https://www.postgresql.org/message-id/CAD21AoDap240WDDdUDE0JMpCmuMMnGajrKrkCRxM7zn9Xk3JRA%40mail.gmail.com
---
src/include/port/simd.h | 47 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)
diff --git a/src/include/port/simd.h b/src/include/port/simd.h
index 1fa6c3bc6c..dfae14e463 100644
--- a/src/include/port/simd.h
+++ b/src/include/port/simd.h
@@ -79,6 +79,7 @@ static inline bool vector8_has_le(const Vector8 v, const uint8 c);
static inline bool vector8_is_highbit_set(const Vector8 v);
#ifndef USE_NO_SIMD
static inline bool vector32_is_highbit_set(const Vector32 v);
+static inline uint32 vector8_highbit_mask(const Vector8 v);
#endif
/* arithmetic operations */
@@ -96,6 +97,7 @@ static inline Vector8 vector8_ssub(const Vector8 v1, const Vector8 v2);
*/
#ifndef USE_NO_SIMD
static inline Vector8 vector8_eq(const Vector8 v1, const Vector8 v2);
+static inline Vector8 vector8_min(const Vector8 v1, const Vector8 v2);
static inline Vector32 vector32_eq(const Vector32 v1, const Vector32 v2);
#endif
@@ -299,6 +301,36 @@ vector32_is_highbit_set(const Vector32 v)
}
#endif /* ! USE_NO_SIMD */
+/*
+ * Return a bitmask formed from the high-bit of each element.
+ */
+#ifndef USE_NO_SIMD
+static inline uint32
+vector8_highbit_mask(const Vector8 v)
+{
+#ifdef USE_SSE2
+ return (uint32) _mm_movemask_epi8(v);
+#elif defined(USE_NEON)
+ /*
+ * Note: There is a faster way to do this, but it returns a uint64 and
+ * and if the caller wanted to extract the bit position using CTZ,
+ * it would have to divide that result by 4.
+ */
+ static const uint8 mask[16] = {
+ 1 << 0, 1 << 1, 1 << 2, 1 << 3,
+ 1 << 4, 1 << 5, 1 << 6, 1 << 7,
+ 1 << 0, 1 << 1, 1 << 2, 1 << 3,
+ 1 << 4, 1 << 5, 1 << 6, 1 << 7,
+ };
+
+ uint8x16_t masked = vandq_u8(vld1q_u8(mask), (uint8x16_t) vshrq_n_s8(v, 7));
+ uint8x16_t maskedhi = vextq_u8(masked, masked, 8);
+
+ return (uint32) vaddvq_u16((uint16x8_t) vzip1q_u8(masked, maskedhi));
+#endif
+}
+#endif /* ! USE_NO_SIMD */
+
/*
* Return the bitwise OR of the inputs
*/
@@ -372,4 +404,19 @@ vector32_eq(const Vector32 v1, const Vector32 v2)
}
#endif /* ! USE_NO_SIMD */
+/*
+ * Given two vectors, return a vector with the minimum element of each.
+ */
+#ifndef USE_NO_SIMD
+static inline Vector8
+vector8_min(const Vector8 v1, const Vector8 v2)
+{
+#ifdef USE_SSE2
+ return _mm_min_epu8(v1, v2);
+#elif defined(USE_NEON)
+ return vminq_u8(v1, v2);
+#endif
+}
+#endif /* ! USE_NO_SIMD */
+
#endif /* SIMD_H */
--
2.31.1
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-03-10 06:42 ` John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: John Naylor @ 2023-03-10 06:42 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Thu, Mar 9, 2023 at 1:51 PM Masahiko Sawada <[email protected]>
wrote:
> I've attached the new version patches. I merged improvements and fixes
> I did in the v29 patch.
I haven't yet had a chance to look at those closely, since I've had to
devote time to other commitments. I remember I wasn't particularly
impressed that v29-0008 mixed my requested name-casing changes with a bunch
of other random things. Separating those out would be an obvious way to
make it easier for me to look at, whenever I can get back to this. I need
to look at the iteration changes as well, in addition to testing memory
measurement (thanks for the new results, they look encouraging).
> Apart from the memory measurement stuff, I've done another todo item
> on my list; adding min max classes for node3 and node125. I've done
This didn't help us move us closer to something committable the first time
you coded this without making sure it was a good idea. It's still not
helping and arguably makes it worse. To be fair, I did speak positively
about _considering_ additional size classes some months ago, but that has a
very obvious maintenance cost, something we can least afford right now.
I'm frankly baffled you thought this was important enough to work on again,
yet thought it was a waste of time to try to prove to ourselves that
autovacuum in a realistic, non-deterministic workload gave the same answer
as the current tid lookup. Even if we had gone that far, it doesn't seem
like a good idea to add non-essential code to critical paths right now.
We're rapidly running out of time, and we're at the point in the cycle
where it's impossible to get meaningful review from anyone not already
intimately familiar with the patch series. I only want to see progress on
addressing possible (especially architectural) objections from the
community, because if they don't notice them now, they surely will after
commit. I have my own list of possible objections as well as bikeshedding
points, which I'll clean up and share next week. I plan to invite Andres to
look at that list and give his impressions, because it's a lot quicker than
reading the patches. Based on that, I'll hopefully be able to decide
whether we have enough time to address any feedback and do remaining
polishing in time for feature freeze.
I'd suggest sharing your todo list in the meanwhile, it'd be good to
discuss what's worth doing and what is not.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-03-10 14:30 ` Masahiko Sawada <[email protected]>
2023-03-10 15:26 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-11 15:54 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
0 siblings, 2 replies; 78+ messages in thread
From: Masahiko Sawada @ 2023-03-10 14:30 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Fri, Mar 10, 2023 at 3:42 PM John Naylor
<[email protected]> wrote:
>
> On Thu, Mar 9, 2023 at 1:51 PM Masahiko Sawada <[email protected]> wrote:
>
> > I've attached the new version patches. I merged improvements and fixes
> > I did in the v29 patch.
>
> I haven't yet had a chance to look at those closely, since I've had to devote time to other commitments. I remember I wasn't particularly impressed that v29-0008 mixed my requested name-casing changes with a bunch of other random things. Separating those out would be an obvious way to make it easier for me to look at, whenever I can get back to this. I need to look at the iteration changes as well, in addition to testing memory measurement (thanks for the new results, they look encouraging).
Okay, I'll separate them again.
>
> > Apart from the memory measurement stuff, I've done another todo item
> > on my list; adding min max classes for node3 and node125. I've done
>
> This didn't help us move us closer to something committable the first time you coded this without making sure it was a good idea. It's still not helping and arguably makes it worse. To be fair, I did speak positively about _considering_ additional size classes some months ago, but that has a very obvious maintenance cost, something we can least afford right now.
>
> I'm frankly baffled you thought this was important enough to work on again, yet thought it was a waste of time to try to prove to ourselves that autovacuum in a realistic, non-deterministic workload gave the same answer as the current tid lookup. Even if we had gone that far, it doesn't seem like a good idea to add non-essential code to critical paths right now.
I didn't think that proving tidstore and the current tid lookup return
the same result was a waste of time. I've shared a patch to do that in
tidstore before. I agreed not to add it to the tree but we can test
that using this patch. Actually I've done a test that ran pgbench
workload for a few days.
IIUC it's still important to consider whether to have node1 since it
could be a good alternative for the path compression. The prototype
also implemented it. Of course we can leave it for future improvement.
But considering this item with the performance tests helps us to prove
our decoupling approach is promising.
> We're rapidly running out of time, and we're at the point in the cycle where it's impossible to get meaningful review from anyone not already intimately familiar with the patch series. I only want to see progress on addressing possible (especially architectural) objections from the community, because if they don't notice them now, they surely will after commit.
Right, we've been making many design decisions. Some of them are
agreed just between you and me and some are agreed with other hackers.
There are some irrevertible design decisions due to the remaining
time.
> I have my own list of possible objections as well as bikeshedding points, which I'll clean up and share next week.
Thanks.
> I plan to invite Andres to look at that list and give his impressions, because it's a lot quicker than reading the patches. Based on that, I'll hopefully be able to decide whether we have enough time to address any feedback and do remaining polishing in time for feature freeze.
>
> I'd suggest sharing your todo list in the meanwhile, it'd be good to discuss what's worth doing and what is not.
Apart from more rounds of reviews and tests, my todo items that need
discussion and possibly implementation are:
* The memory measurement in radix trees and the memory limit in
tidstores. I've implemented it in v30-0007 through 0009 but we need to
review it. This is the highest priority for me.
* Additional size classes. It's important for an alternative of path
compression as well as supporting our decoupling approach. Middle
priority.
* Node shrinking support. Low priority.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-03-10 15:26 ` Masahiko Sawada <[email protected]>
2023-04-17 15:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
1 sibling, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-03-10 15:26 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Fri, Mar 10, 2023 at 11:30 PM Masahiko Sawada <[email protected]> wrote:
>
> On Fri, Mar 10, 2023 at 3:42 PM John Naylor
> <[email protected]> wrote:
> >
> > On Thu, Mar 9, 2023 at 1:51 PM Masahiko Sawada <[email protected]> wrote:
> >
> > > I've attached the new version patches. I merged improvements and fixes
> > > I did in the v29 patch.
> >
> > I haven't yet had a chance to look at those closely, since I've had to devote time to other commitments. I remember I wasn't particularly impressed that v29-0008 mixed my requested name-casing changes with a bunch of other random things. Separating those out would be an obvious way to make it easier for me to look at, whenever I can get back to this. I need to look at the iteration changes as well, in addition to testing memory measurement (thanks for the new results, they look encouraging).
>
> Okay, I'll separate them again.
Attached new patch series. In addition to separate them again, I've
fixed a conflict with HEAD.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] v31-0013-Add-min-and-max-classes-for-node3-and-node125.patch (11.9K, ../../CAD21AoDBmD5q=eO+K=gyuVt53XvwpJ2dgxPwrtZ-eVOjVmtJjg@mail.gmail.com/2-v31-0013-Add-min-and-max-classes-for-node3-and-node125.patch)
download | inline diff:
From 1b43002d25137699d0e13158d821a8550e757348 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Thu, 9 Mar 2023 11:42:17 +0900
Subject: [PATCH v31 13/14] Add min and max classes for node3 and node125.
---
src/include/lib/radixtree.h | 70 +++++++++++++------
src/include/lib/radixtree_insert_impl.h | 56 ++++++++++++++-
.../expected/test_radixtree.out | 4 ++
.../modules/test_radixtree/test_radixtree.c | 6 +-
4 files changed, 110 insertions(+), 26 deletions(-)
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index f7812eb12a..1759c909b6 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -225,10 +225,12 @@
#define RT_SIZE_CLASS RT_MAKE_NAME(size_class)
#define RT_SIZE_CLASS_ELEM RT_MAKE_NAME(size_class_elem)
#define RT_SIZE_CLASS_INFO RT_MAKE_NAME(size_class_info)
-#define RT_CLASS_3 RT_MAKE_NAME(class_3)
+#define RT_CLASS_3_MIN RT_MAKE_NAME(class_3_min)
+#define RT_CLASS_3_MAX RT_MAKE_NAME(class_3_max)
#define RT_CLASS_32_MIN RT_MAKE_NAME(class_32_min)
#define RT_CLASS_32_MAX RT_MAKE_NAME(class_32_max)
-#define RT_CLASS_125 RT_MAKE_NAME(class_125)
+#define RT_CLASS_125_MIN RT_MAKE_NAME(class_125_min)
+#define RT_CLASS_125_MAX RT_MAKE_NAME(class_125_max)
#define RT_CLASS_256 RT_MAKE_NAME(class_256)
/* generate forward declarations necessary to use the radix tree */
@@ -561,10 +563,12 @@ typedef struct RT_NODE_LEAF_256
*/
typedef enum RT_SIZE_CLASS
{
- RT_CLASS_3 = 0,
+ RT_CLASS_3_MIN = 0,
+ RT_CLASS_3_MAX,
RT_CLASS_32_MIN,
RT_CLASS_32_MAX,
- RT_CLASS_125,
+ RT_CLASS_125_MIN,
+ RT_CLASS_125_MAX,
RT_CLASS_256
} RT_SIZE_CLASS;
@@ -580,7 +584,13 @@ typedef struct RT_SIZE_CLASS_ELEM
} RT_SIZE_CLASS_ELEM;
static const RT_SIZE_CLASS_ELEM RT_SIZE_CLASS_INFO[] = {
- [RT_CLASS_3] = {
+ [RT_CLASS_3_MIN] = {
+ .name = "radix tree node 1",
+ .fanout = 1,
+ .inner_size = sizeof(RT_NODE_INNER_3) + 1 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_3) + 1 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_3_MAX] = {
.name = "radix tree node 3",
.fanout = 3,
.inner_size = sizeof(RT_NODE_INNER_3) + 3 * sizeof(RT_PTR_ALLOC),
@@ -598,7 +608,13 @@ static const RT_SIZE_CLASS_ELEM RT_SIZE_CLASS_INFO[] = {
.inner_size = sizeof(RT_NODE_INNER_32) + 32 * sizeof(RT_PTR_ALLOC),
.leaf_size = sizeof(RT_NODE_LEAF_32) + 32 * sizeof(RT_VALUE_TYPE),
},
- [RT_CLASS_125] = {
+ [RT_CLASS_125_MIN] = {
+ .name = "radix tree node 125",
+ .fanout = 61,
+ .inner_size = sizeof(RT_NODE_INNER_125) + 61 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_125) + 61 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_125_MAX] = {
.name = "radix tree node 125",
.fanout = 125,
.inner_size = sizeof(RT_NODE_INNER_125) + 125 * sizeof(RT_PTR_ALLOC),
@@ -934,7 +950,7 @@ static inline void
RT_CHUNK_CHILDREN_ARRAY_COPY(uint8 *src_chunks, RT_PTR_ALLOC *src_children,
uint8 *dst_chunks, RT_PTR_ALLOC *dst_children)
{
- const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3_MAX].fanout;
const Size chunk_size = sizeof(uint8) * fanout;
const Size children_size = sizeof(RT_PTR_ALLOC) * fanout;
@@ -946,7 +962,7 @@ static inline void
RT_CHUNK_VALUES_ARRAY_COPY(uint8 *src_chunks, RT_VALUE_TYPE *src_values,
uint8 *dst_chunks, RT_VALUE_TYPE *dst_values)
{
- const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3_MAX].fanout;
const Size chunk_size = sizeof(uint8) * fanout;
const Size values_size = sizeof(RT_VALUE_TYPE) * fanout;
@@ -1152,9 +1168,9 @@ RT_NEW_ROOT(RT_RADIX_TREE *tree, uint64 key)
RT_PTR_ALLOC allocnode;
RT_PTR_LOCAL newnode;
- allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3_MIN, is_leaf);
newnode = RT_PTR_GET_LOCAL(tree, allocnode);
- RT_INIT_NODE(newnode, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ RT_INIT_NODE(newnode, RT_NODE_KIND_3, RT_CLASS_3_MIN, is_leaf);
newnode->shift = shift;
tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(shift);
tree->ctl->root = allocnode;
@@ -1188,17 +1204,21 @@ static inline Size
RT_FANOUT_GET_NODE_SIZE(int fanout, bool is_leaf)
{
const Size fanout_inner_node_size[] = {
- [3] = RT_SIZE_CLASS_INFO[RT_CLASS_3].inner_size,
+ [1] = RT_SIZE_CLASS_INFO[RT_CLASS_3_MIN].inner_size,
+ [3] = RT_SIZE_CLASS_INFO[RT_CLASS_3_MAX].inner_size,
[15] = RT_SIZE_CLASS_INFO[RT_CLASS_32_MIN].inner_size,
[32] = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX].inner_size,
- [125] = RT_SIZE_CLASS_INFO[RT_CLASS_125].inner_size,
+ [61] = RT_SIZE_CLASS_INFO[RT_CLASS_125_MIN].inner_size,
+ [125] = RT_SIZE_CLASS_INFO[RT_CLASS_125_MAX].inner_size,
[256] = RT_SIZE_CLASS_INFO[RT_CLASS_256].inner_size,
};
const Size fanout_leaf_node_size[] = {
- [3] = RT_SIZE_CLASS_INFO[RT_CLASS_3].leaf_size,
+ [1] = RT_SIZE_CLASS_INFO[RT_CLASS_3_MIN].leaf_size,
+ [3] = RT_SIZE_CLASS_INFO[RT_CLASS_3_MAX].leaf_size,
[15] = RT_SIZE_CLASS_INFO[RT_CLASS_32_MIN].leaf_size,
[32] = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX].leaf_size,
- [125] = RT_SIZE_CLASS_INFO[RT_CLASS_125].leaf_size,
+ [61] = RT_SIZE_CLASS_INFO[RT_CLASS_125_MIN].leaf_size,
+ [125] = RT_SIZE_CLASS_INFO[RT_CLASS_125_MAX].leaf_size,
[256] = RT_SIZE_CLASS_INFO[RT_CLASS_256].leaf_size,
};
Size node_size;
@@ -1337,9 +1357,9 @@ RT_EXTEND_UP(RT_RADIX_TREE *tree, uint64 key)
RT_PTR_LOCAL node;
RT_NODE_INNER_3 *n3;
- allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, true);
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3_MIN, true);
node = RT_PTR_GET_LOCAL(tree, allocnode);
- RT_INIT_NODE(node, RT_NODE_KIND_3, RT_CLASS_3, true);
+ RT_INIT_NODE(node, RT_NODE_KIND_3, RT_CLASS_3_MIN, true);
node->shift = shift;
node->count = 1;
@@ -1375,9 +1395,9 @@ RT_EXTEND_DOWN(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_L
int newshift = shift - RT_NODE_SPAN;
bool is_leaf = newshift == 0;
- allocchild = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ allocchild = RT_ALLOC_NODE(tree, RT_CLASS_3_MIN, is_leaf);
newchild = RT_PTR_GET_LOCAL(tree, allocchild);
- RT_INIT_NODE(newchild, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ RT_INIT_NODE(newchild, RT_NODE_KIND_3, RT_CLASS_3_MIN, is_leaf);
newchild->shift = newshift;
RT_NODE_INSERT_INNER(tree, parent, stored_node, node, key, allocchild);
@@ -2177,12 +2197,14 @@ RT_STATS(RT_RADIX_TREE *tree)
{
RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
- fprintf(stderr, "height = %d, n3 = %u, n15 = %u, n32 = %u, n125 = %u, n256 = %u\n",
+ fprintf(stderr, "height = %d, n1 = %u, n3 = %u, n15 = %u, n32 = %u, n61 = %u, n125 = %u, n256 = %u\n",
root->shift / RT_NODE_SPAN,
- tree->ctl->cnt[RT_CLASS_3],
+ tree->ctl->cnt[RT_CLASS_3_MIN],
+ tree->ctl->cnt[RT_CLASS_3_MAX],
tree->ctl->cnt[RT_CLASS_32_MIN],
tree->ctl->cnt[RT_CLASS_32_MAX],
- tree->ctl->cnt[RT_CLASS_125],
+ tree->ctl->cnt[RT_CLASS_125_MIN],
+ tree->ctl->cnt[RT_CLASS_125_MAX],
tree->ctl->cnt[RT_CLASS_256]);
}
@@ -2519,10 +2541,12 @@ RT_DUMP(RT_RADIX_TREE *tree)
#undef RT_SIZE_CLASS
#undef RT_SIZE_CLASS_ELEM
#undef RT_SIZE_CLASS_INFO
-#undef RT_CLASS_3
+#undef RT_CLASS_3_MIN
+#undef RT_CLASS_3_MAX
#undef RT_CLASS_32_MIN
#undef RT_CLASS_32_MAX
-#undef RT_CLASS_125
+#undef RT_CLASS_125_MIN
+#undef RT_CLASS_125_MAX
#undef RT_CLASS_256
/* function declarations */
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
index d56e58dcac..d10093dfba 100644
--- a/src/include/lib/radixtree_insert_impl.h
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -42,6 +42,7 @@
{
case RT_NODE_KIND_3:
{
+ const RT_SIZE_CLASS_ELEM class3_max = RT_SIZE_CLASS_INFO[RT_CLASS_3_MAX];
RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
#ifdef RT_NODE_LEVEL_LEAF
@@ -55,6 +56,32 @@
break;
}
#endif
+ if (unlikely(RT_NODE_MUST_GROW(n3)) &&
+ n3->base.n.fanout < class3_max.fanout)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ const RT_SIZE_CLASS_ELEM class3_min = RT_SIZE_CLASS_INFO[RT_CLASS_3_MIN];
+ const RT_SIZE_CLASS new_class = RT_CLASS_3_MAX;
+
+ Assert(n3->base.n.fanout == class3_min.fanout);
+
+ /* grow to the next size class of this kind */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ n3 = (RT_NODE3_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ memcpy(newnode, node, class3_min.leaf_size);
+#else
+ memcpy(newnode, node, class3_min.inner_size);
+#endif
+ newnode->fanout = class3_max.fanout;
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+
if (unlikely(RT_NODE_MUST_GROW(n3)))
{
RT_PTR_ALLOC allocnode;
@@ -154,7 +181,7 @@
RT_PTR_LOCAL newnode;
RT_NODE125_TYPE *new125;
const uint8 new_kind = RT_NODE_KIND_125;
- const RT_SIZE_CLASS new_class = RT_CLASS_125;
+ const RT_SIZE_CLASS new_class = RT_CLASS_125_MIN;
Assert(n32->base.n.fanout == class32_max.fanout);
@@ -213,6 +240,7 @@
/* FALLTHROUGH */
case RT_NODE_KIND_125:
{
+ const RT_SIZE_CLASS_ELEM class125_max = RT_SIZE_CLASS_INFO[RT_CLASS_125_MAX];
RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
int slotpos;
int cnt = 0;
@@ -227,6 +255,32 @@
break;
}
#endif
+ if (unlikely(RT_NODE_MUST_GROW(n125)) &&
+ n125->base.n.fanout < class125_max.fanout)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ const RT_SIZE_CLASS_ELEM class125_min = RT_SIZE_CLASS_INFO[RT_CLASS_125_MIN];
+ const RT_SIZE_CLASS new_class = RT_CLASS_125_MAX;
+
+ Assert(n125->base.n.fanout == class125_min.fanout);
+
+ /* grow to the next size class of this kind */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ n125 = (RT_NODE125_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ memcpy(newnode, node, class125_min.leaf_size);
+#else
+ memcpy(newnode, node, class125_min.inner_size);
+#endif
+ newnode->fanout = class125_max.fanout;
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+
if (unlikely(RT_NODE_MUST_GROW(n125)))
{
RT_PTR_ALLOC allocnode;
diff --git a/src/test/modules/test_radixtree/expected/test_radixtree.out b/src/test/modules/test_radixtree/expected/test_radixtree.out
index 7ad1ce3605..f2b1d7e4f8 100644
--- a/src/test/modules/test_radixtree/expected/test_radixtree.out
+++ b/src/test/modules/test_radixtree/expected/test_radixtree.out
@@ -4,12 +4,16 @@ CREATE EXTENSION test_radixtree;
-- an error if something fails.
--
SELECT test_radixtree();
+NOTICE: testing basic operations with leaf node 1
+NOTICE: testing basic operations with inner node 1
NOTICE: testing basic operations with leaf node 3
NOTICE: testing basic operations with inner node 3
NOTICE: testing basic operations with leaf node 15
NOTICE: testing basic operations with inner node 15
NOTICE: testing basic operations with leaf node 32
NOTICE: testing basic operations with inner node 32
+NOTICE: testing basic operations with leaf node 61
+NOTICE: testing basic operations with inner node 61
NOTICE: testing basic operations with leaf node 125
NOTICE: testing basic operations with inner node 125
NOTICE: testing basic operations with leaf node 256
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
index 19d286d84b..4f38b6e3de 100644
--- a/src/test/modules/test_radixtree/test_radixtree.c
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -47,10 +47,12 @@ static const bool rt_test_stats = false;
* XXX: should we expose and use RT_SIZE_CLASS and RT_SIZE_CLASS_INFO?
*/
static int rt_node_class_fanouts[] = {
- 3, /* RT_CLASS_3 */
+ 1, /* RT_CLASS_3_MIN */
+ 3, /* RT_CLASS_3_MAX */
15, /* RT_CLASS_32_MIN */
32, /* RT_CLASS_32_MAX */
- 125, /* RT_CLASS_125 */
+ 61, /* RT_CLASS_125_MIN */
+ 125, /* RT_CLASS_125_MAX */
256 /* RT_CLASS_256 */
};
/*
--
2.31.1
[application/octet-stream] v31-0011-Remove-the-max-memory-deduction-from-TidStore.patch (3.9K, ../../CAD21AoDBmD5q=eO+K=gyuVt53XvwpJ2dgxPwrtZ-eVOjVmtJjg@mail.gmail.com/3-v31-0011-Remove-the-max-memory-deduction-from-TidStore.patch)
download | inline diff:
From e86e43b93fb901aacd8d2b69aa53ad896c5b5e1c Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 8 Mar 2023 15:08:58 +0900
Subject: [PATCH v31 11/14] Remove the max memory deduction from TidStore.
---
src/backend/access/common/tidstore.c | 43 +++++++---------------------
1 file changed, 10 insertions(+), 33 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 9360520482..ee73759648 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -82,6 +82,7 @@ typedef uint64 offsetbm;
#define RT_SCOPE static
#define RT_DECLARE
#define RT_DEFINE
+#define RT_MEASURE_MEMORY_USAGE
#define RT_VALUE_TYPE tidkey
#include "lib/radixtree.h"
@@ -90,6 +91,7 @@ typedef uint64 offsetbm;
#define RT_SCOPE static
#define RT_DECLARE
#define RT_DEFINE
+#define RT_MEASURE_MEMORY_USAGE
#define RT_VALUE_TYPE tidkey
#include "lib/radixtree.h"
@@ -180,39 +182,15 @@ TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
ts = palloc0(sizeof(TidStore));
- /*
- * Create the radix tree for the main storage.
- *
- * Memory consumption depends on the number of stored tids, but also on the
- * distribution of them, how the radix tree stores, and the memory management
- * that backed the radix tree. The maximum bytes that a TidStore can
- * use is specified by the max_bytes in TidStoreCreate(). We want the total
- * amount of memory consumption by a TidStore not to exceed the max_bytes.
- *
- * In local TidStore cases, the radix tree uses slab allocators for each kind
- * of node class. The most memory consuming case while adding Tids associated
- * with one page (i.e. during TidStoreSetBlockOffsets()) is that we allocate a new
- * slab block for a new radix tree node, which is approximately 70kB. Therefore,
- * we deduct 70kB from the max_bytes.
- *
- * In shared cases, DSA allocates the memory segments big enough to follow
- * a geometric series that approximately doubles the total DSA size (see
- * make_new_segment() in dsa.c). We simulated the how DSA increases segment
- * size and the simulation revealed, the 75% threshold for the maximum bytes
- * perfectly works in case where the max_bytes is a power-of-2, and the 60%
- * threshold works for other cases.
- */
if (area != NULL)
{
dsa_pointer dp;
- float ratio = ((max_bytes & (max_bytes - 1)) == 0) ? 0.75 : 0.6;
ts->tree.shared = shared_rt_create(CurrentMemoryContext, area,
LWTRANCHE_SHARED_TIDSTORE);
dp = dsa_allocate0(area, sizeof(TidStoreControl));
ts->control = (TidStoreControl *) dsa_get_address(area, dp);
- ts->control->max_bytes = (size_t) (max_bytes * ratio);
ts->area = area;
ts->control->magic = TIDSTORE_MAGIC;
@@ -223,11 +201,15 @@ TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
else
{
ts->tree.local = local_rt_create(CurrentMemoryContext);
-
ts->control = (TidStoreControl *) palloc0(sizeof(TidStoreControl));
- ts->control->max_bytes = max_bytes - (70 * 1024);
}
+ /*
+ * max_bytes is forced to be at least 64KB, the current minimum valid value
+ * for the work_mem GUC.
+ */
+ ts->control->max_bytes = Max(64 * 1024L, max_bytes);
+
ts->control->max_off = max_off;
ts->control->max_off_nbits = pg_ceil_log2_32(max_off);
@@ -331,14 +313,8 @@ TidStoreReset(TidStore *ts)
LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
- /*
- * Free the radix tree and return allocated DSA segments to
- * the operating system.
- */
- shared_rt_free(ts->tree.shared);
- dsa_trim(ts->area);
-
/* Recreate the radix tree */
+ shared_rt_free(ts->tree.shared);
ts->tree.shared = shared_rt_create(CurrentMemoryContext, ts->area,
LWTRANCHE_SHARED_TIDSTORE);
@@ -352,6 +328,7 @@ TidStoreReset(TidStore *ts)
}
else
{
+ /* Recreate the radix tree */
local_rt_free(ts->tree.local);
ts->tree.local = local_rt_create(CurrentMemoryContext);
--
2.31.1
[application/octet-stream] v31-0009-Review-vacuum-integration.patch (13.2K, ../../CAD21AoDBmD5q=eO+K=gyuVt53XvwpJ2dgxPwrtZ-eVOjVmtJjg@mail.gmail.com/4-v31-0009-Review-vacuum-integration.patch)
download | inline diff:
From c1c126e0f4e9f5eeb642bd892bd40948a41b8aae Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 17 Feb 2023 00:04:37 +0900
Subject: [PATCH v31 09/14] Review vacuum integration.
---
doc/src/sgml/monitoring.sgml | 2 +-
src/backend/access/heap/vacuumlazy.c | 61 +++++++++++++--------------
src/backend/commands/vacuum.c | 4 +-
src/backend/commands/vacuumparallel.c | 25 +++++------
4 files changed, 46 insertions(+), 46 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 47b346d36c..61e163636a 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -7181,7 +7181,7 @@ FROM pg_stat_get_backend_idset() AS backendid;
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>num_dead_tuple_bytes</structfield> <type>bigint</type>
+ <structfield>dead_tuple_bytes</structfield> <type>bigint</type>
</para>
<para>
Amount of dead tuple data collected since the last index vacuum cycle.
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index b4e40423a8..edb9079124 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -10,11 +10,10 @@
* of dead TIDs at once.
*
* We are willing to use at most maintenance_work_mem (or perhaps
- * autovacuum_work_mem) memory space to keep track of dead TIDs. We initially
- * create a TidStore with the maximum bytes that can be used by the TidStore.
- * If the TidStore is full, we must call lazy_vacuum to vacuum indexes (and to
- * vacuum the pages that we've pruned). This frees up the memory space dedicated
- * to storing dead TIDs.
+ * autovacuum_work_mem) memory space to keep track of dead TIDs. If the
+ * TidStore is full, we must call lazy_vacuum to vacuum indexes (and to vacuum
+ * the pages that we've pruned). This frees up the memory space dedicated to
+ * to store dead TIDs.
*
* In practice VACUUM will often complete its initial pass over the target
* heap relation without ever running out of space to store TIDs. This means
@@ -844,7 +843,7 @@ lazy_scan_heap(LVRelState *vacrel)
/* Report that we're scanning the heap, advertising total # of blocks */
initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
initprog_val[1] = rel_pages;
- initprog_val[2] = tidstore_max_memory(vacrel->dead_items);
+ initprog_val[2] = TidStoreMaxMemory(vacrel->dead_items);
pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
/* Set up an initial range of skippable blocks using the visibility map */
@@ -911,7 +910,7 @@ lazy_scan_heap(LVRelState *vacrel)
* dead_items TIDs, pause and do a cycle of vacuuming before we tackle
* this page.
*/
- if (tidstore_is_full(vacrel->dead_items))
+ if (TidStoreIsFull(vacrel->dead_items))
{
/*
* Before beginning index vacuuming, we release any pin we may
@@ -1080,16 +1079,16 @@ lazy_scan_heap(LVRelState *vacrel)
* with prunestate-driven visibility map and FSM steps (just like
* the two-pass strategy).
*/
- Assert(tidstore_num_tids(dead_items) == 0);
+ Assert(TidStoreNumTids(dead_items) == 0);
}
else if (prunestate.num_offsets > 0)
{
/* Save details of the LP_DEAD items from the page in dead_items */
- tidstore_add_tids(dead_items, blkno, prunestate.deadoffsets,
- prunestate.num_offsets);
+ TidStoreSetBlockOffsets(dead_items, blkno, prunestate.deadoffsets,
+ prunestate.num_offsets);
pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
- tidstore_memory_usage(dead_items));
+ TidStoreMemoryUsage(dead_items));
}
/*
@@ -1260,7 +1259,7 @@ lazy_scan_heap(LVRelState *vacrel)
* Do index vacuuming (call each index's ambulkdelete routine), then do
* related heap vacuuming
*/
- if (tidstore_num_tids(dead_items) > 0)
+ if (TidStoreNumTids(dead_items) > 0)
lazy_vacuum(vacrel);
/*
@@ -2127,10 +2126,10 @@ lazy_scan_noprune(LVRelState *vacrel,
*/
vacrel->lpdead_item_pages++;
- tidstore_add_tids(dead_items, blkno, deadoffsets, lpdead_items);
+ TidStoreSetBlockOffsets(dead_items, blkno, deadoffsets, lpdead_items);
pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
- tidstore_memory_usage(dead_items));
+ TidStoreMemoryUsage(dead_items));
vacrel->lpdead_items += lpdead_items;
@@ -2179,7 +2178,7 @@ lazy_vacuum(LVRelState *vacrel)
if (!vacrel->do_index_vacuuming)
{
Assert(!vacrel->do_index_cleanup);
- tidstore_reset(vacrel->dead_items);
+ TidStoreReset(vacrel->dead_items);
return;
}
@@ -2208,7 +2207,7 @@ lazy_vacuum(LVRelState *vacrel)
BlockNumber threshold;
Assert(vacrel->num_index_scans == 0);
- Assert(vacrel->lpdead_items == tidstore_num_tids(vacrel->dead_items));
+ Assert(vacrel->lpdead_items == TidStoreNumTids(vacrel->dead_items));
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2236,7 +2235,7 @@ lazy_vacuum(LVRelState *vacrel)
*/
threshold = (double) vacrel->rel_pages * BYPASS_THRESHOLD_PAGES;
bypass = (vacrel->lpdead_item_pages < threshold) &&
- tidstore_memory_usage(vacrel->dead_items) < (32L * 1024L * 1024L);
+ TidStoreMemoryUsage(vacrel->dead_items) < (32L * 1024L * 1024L);
}
if (bypass)
@@ -2281,7 +2280,7 @@ lazy_vacuum(LVRelState *vacrel)
* Forget the LP_DEAD items that we just vacuumed (or just decided to not
* vacuum)
*/
- tidstore_reset(vacrel->dead_items);
+ TidStoreReset(vacrel->dead_items);
}
/*
@@ -2354,7 +2353,7 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
* place).
*/
Assert(vacrel->num_index_scans > 0 ||
- tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items);
+ TidStoreNumTids(vacrel->dead_items) == vacrel->lpdead_items);
Assert(allindexes || vacrel->failsafe_active);
/*
@@ -2394,7 +2393,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
Buffer vmbuffer = InvalidBuffer;
LVSavedErrInfo saved_err_info;
TidStoreIter *iter;
- TidStoreIterResult *result;
+ TidStoreIterResult *iter_result;
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2409,8 +2408,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
VACUUM_ERRCB_PHASE_VACUUM_HEAP,
InvalidBlockNumber, InvalidOffsetNumber);
- iter = tidstore_begin_iterate(vacrel->dead_items);
- while ((result = tidstore_iterate_next(iter)) != NULL)
+ iter = TidStoreBeginIterate(vacrel->dead_items);
+ while ((iter_result = TidStoreIterateNext(iter)) != NULL)
{
BlockNumber blkno;
Buffer buf;
@@ -2419,7 +2418,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
vacuum_delay_point();
- blkno = result->blkno;
+ blkno = iter_result->blkno;
vacrel->blkno = blkno;
/*
@@ -2433,8 +2432,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
buf = ReadBufferExtended(vacrel->rel, MAIN_FORKNUM, blkno, RBM_NORMAL,
vacrel->bstrategy);
LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
- lazy_vacuum_heap_page(vacrel, blkno, result->offsets, result->num_offsets,
- buf, vmbuffer);
+ lazy_vacuum_heap_page(vacrel, blkno, iter_result->offsets,
+ iter_result->num_offsets, buf, vmbuffer);
/* Now that we've vacuumed the page, record its available space */
page = BufferGetPage(buf);
@@ -2444,7 +2443,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
vacuumed_pages++;
}
- tidstore_end_iterate(iter);
+ TidStoreEndIterate(iter);
vacrel->blkno = InvalidBlockNumber;
if (BufferIsValid(vmbuffer))
@@ -2455,12 +2454,12 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
* the second heap pass. No more, no less.
*/
Assert(vacrel->num_index_scans > 1 ||
- (tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items &&
+ (TidStoreNumTids(vacrel->dead_items) == vacrel->lpdead_items &&
vacuumed_pages == vacrel->lpdead_item_pages));
ereport(DEBUG2,
- (errmsg("table \"%s\": removed " UINT64_FORMAT "dead item identifiers in %u pages",
- vacrel->relname, tidstore_num_tids(vacrel->dead_items),
+ (errmsg("table \"%s\": removed " INT64_FORMAT "dead item identifiers in %u pages",
+ vacrel->relname, TidStoreNumTids(vacrel->dead_items),
vacuumed_pages)));
/* Revert to the previous phase information for error traceback */
@@ -3118,8 +3117,8 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
}
/* Serial VACUUM case */
- vacrel->dead_items = tidstore_create(vac_work_mem, MaxHeapTuplesPerPage,
- NULL);
+ vacrel->dead_items = TidStoreCreate(vac_work_mem, MaxHeapTuplesPerPage,
+ NULL);
}
/*
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 785b825bbc..afedb87941 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -2335,7 +2335,7 @@ vac_bulkdel_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat,
ereport(ivinfo->message_level,
(errmsg("scanned index \"%s\" to remove " UINT64_FORMAT " row versions",
RelationGetRelationName(ivinfo->index),
- tidstore_num_tids(dead_items))));
+ TidStoreNumTids(dead_items))));
return istat;
}
@@ -2376,5 +2376,5 @@ vac_tid_reaped(ItemPointer itemptr, void *state)
{
TidStore *dead_items = (TidStore *) state;
- return tidstore_lookup_tid(dead_items, itemptr);
+ return TidStoreIsMember(dead_items, itemptr);
}
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index d653683693..9225daf3ab 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -9,11 +9,12 @@
* In a parallel vacuum, we perform both index bulk deletion and index cleanup
* with parallel worker processes. Individual indexes are processed by one
* vacuum process. ParalleVacuumState contains shared information as well as
- * the shared TidStore. We launch parallel worker processes at the start of
- * parallel index bulk-deletion and index cleanup and once all indexes are
- * processed, the parallel worker processes exit. Each time we process indexes
- * in parallel, the parallel context is re-initialized so that the same DSM can
- * be used for multiple passes of index bulk-deletion and index cleanup.
+ * the memory space for storing dead items allocated in the DSA area. We
+ * launch parallel worker processes at the start of parallel index
+ * bulk-deletion and index cleanup and once all indexes are processed, the
+ * parallel worker processes exit. Each time we process indexes in parallel,
+ * the parallel context is re-initialized so that the same DSM can be used for
+ * multiple passes of index bulk-deletion and index cleanup.
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -104,7 +105,7 @@ typedef struct PVShared
pg_atomic_uint32 idx;
/* Handle of the shared TidStore */
- tidstore_handle dead_items_handle;
+ TidStoreHandle dead_items_handle;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -289,7 +290,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_estimate_chunk(&pcxt->estimator, est_shared_len);
shm_toc_estimate_keys(&pcxt->estimator, 1);
- /* Estimate size for dead tuple DSA -- PARALLEL_VACUUM_KEY_DSA */
+ /* Initial size for dead tuple DSA -- PARALLEL_VACUUM_KEY_DEAD_ITEMS */
shm_toc_estimate_chunk(&pcxt->estimator, dsa_minsize);
shm_toc_estimate_keys(&pcxt->estimator, 1);
@@ -362,7 +363,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
dead_items_dsa = dsa_create_in_place(area_space, dsa_minsize,
LWTRANCHE_PARALLEL_VACUUM_DSA,
pcxt->seg);
- dead_items = tidstore_create(vac_work_mem, max_offset, dead_items_dsa);
+ dead_items = TidStoreCreate(vac_work_mem, max_offset, dead_items_dsa);
pvs->dead_items = dead_items;
pvs->dead_items_area = dead_items_dsa;
@@ -375,7 +376,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
- shared->dead_items_handle = tidstore_get_handle(dead_items);
+ shared->dead_items_handle = TidStoreGetHandle(dead_items);
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
@@ -441,7 +442,7 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
istats[i] = NULL;
}
- tidstore_destroy(pvs->dead_items);
+ TidStoreDestroy(pvs->dead_items);
dsa_detach(pvs->dead_items_area);
DestroyParallelContext(pvs->pcxt);
@@ -999,7 +1000,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
/* Set dead items */
area_space = shm_toc_lookup(toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, false);
dead_items_area = dsa_attach_in_place(area_space, seg);
- dead_items = tidstore_attach(dead_items_area, shared->dead_items_handle);
+ dead_items = TidStoreAttach(dead_items_area, shared->dead_items_handle);
/* Set cost-based vacuum delay */
VacuumCostActive = (VacuumCostDelay > 0);
@@ -1045,7 +1046,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
InstrEndParallelQuery(&buffer_usage[ParallelWorkerNumber],
&wal_usage[ParallelWorkerNumber]);
- tidstore_detach(pvs.dead_items);
+ TidStoreDetach(dead_items);
dsa_detach(dead_items_area);
/* Pop the error context stack */
--
2.31.1
[application/octet-stream] v31-0007-Review-radix-tree.patch (22.5K, ../../CAD21AoDBmD5q=eO+K=gyuVt53XvwpJ2dgxPwrtZ-eVOjVmtJjg@mail.gmail.com/5-v31-0007-Review-radix-tree.patch)
download | inline diff:
From 2c280fb3697501c70e4ce43808e3a5175bbc5eb2 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 20 Feb 2023 11:28:50 +0900
Subject: [PATCH v31 07/14] Review radix tree.
Mainly improve the iteration codes and comments.
---
src/include/lib/radixtree.h | 169 +++++++++---------
src/include/lib/radixtree_iter_impl.h | 85 ++++-----
.../expected/test_radixtree.out | 6 +-
.../modules/test_radixtree/test_radixtree.c | 103 +++++++----
4 files changed, 197 insertions(+), 166 deletions(-)
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index e546bd705c..8bea606c62 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -83,7 +83,7 @@
* RT_SET - Set a key-value pair
* RT_BEGIN_ITERATE - Begin iterating through all key-value pairs
* RT_ITERATE_NEXT - Return next key-value pair, if any
- * RT_END_ITER - End iteration
+ * RT_END_ITERATE - End iteration
* RT_MEMORY_USAGE - Get the memory usage
*
* Interface for Shared Memory
@@ -152,8 +152,8 @@
#define RT_INIT_NODE RT_MAKE_NAME(init_node)
#define RT_FREE_NODE RT_MAKE_NAME(free_node)
#define RT_FREE_RECURSE RT_MAKE_NAME(free_recurse)
-#define RT_EXTEND RT_MAKE_NAME(extend)
-#define RT_SET_EXTEND RT_MAKE_NAME(set_extend)
+#define RT_EXTEND_UP RT_MAKE_NAME(extend_up)
+#define RT_EXTEND_DOWN RT_MAKE_NAME(extend_down)
#define RT_SWITCH_NODE_KIND RT_MAKE_NAME(grow_node_kind)
#define RT_COPY_NODE RT_MAKE_NAME(copy_node)
#define RT_REPLACE_NODE RT_MAKE_NAME(replace_node)
@@ -191,7 +191,7 @@
#define RT_NODE_INSERT_LEAF RT_MAKE_NAME(node_insert_leaf)
#define RT_NODE_INNER_ITERATE_NEXT RT_MAKE_NAME(node_inner_iterate_next)
#define RT_NODE_LEAF_ITERATE_NEXT RT_MAKE_NAME(node_leaf_iterate_next)
-#define RT_UPDATE_ITER_STACK RT_MAKE_NAME(update_iter_stack)
+#define RT_ITER_SET_NODE_FROM RT_MAKE_NAME(iter_set_node_from)
#define RT_ITER_UPDATE_KEY RT_MAKE_NAME(iter_update_key)
#define RT_VERIFY_NODE RT_MAKE_NAME(verify_node)
@@ -612,7 +612,6 @@ static const RT_SIZE_CLASS_ELEM RT_SIZE_CLASS_INFO[] = {
#endif
/* Contains the actual tree and ancillary info */
-// WIP: this name is a bit strange
typedef struct RT_RADIX_TREE_CONTROL
{
#ifdef RT_SHMEM
@@ -651,36 +650,40 @@ typedef struct RT_RADIX_TREE
* Iteration support.
*
* Iterating the radix tree returns each pair of key and value in the ascending
- * order of the key. To support this, the we iterate nodes of each level.
+ * order of the key.
*
- * RT_NODE_ITER struct is used to track the iteration within a node.
+ * RT_NODE_ITER is the struct for iteration of one radix tree node.
*
* RT_ITER is the struct for iteration of the radix tree, and uses RT_NODE_ITER
- * in order to track the iteration of each level. During iteration, we also
- * construct the key whenever updating the node iteration information, e.g., when
- * advancing the current index within the node or when moving to the next node
- * at the same level.
- *
- * XXX: Currently we allow only one process to do iteration. Therefore, rt_node_iter
- * has the local pointers to nodes, rather than RT_PTR_ALLOC.
- * We need either a safeguard to disallow other processes to begin the iteration
- * while one process is doing or to allow multiple processes to do the iteration.
+ * for each level to track the iteration within the node.
*/
typedef struct RT_NODE_ITER
{
- RT_PTR_LOCAL node; /* current node being iterated */
- int current_idx; /* current position. -1 for initial value */
+ /*
+ * Local pointer to the node we are iterating over.
+ *
+ * Since the radix tree doesn't support the shared iteration among multiple
+ * processes, we use RT_PTR_LOCAL rather than RT_PTR_ALLOC.
+ */
+ RT_PTR_LOCAL node;
+
+ /*
+ * The next index of the chunk array in RT_NODE_KIND_3 and
+ * RT_NODE_KIND_32 nodes, or the next chunk in RT_NODE_KIND_125 and
+ * RT_NODE_KIND_256 nodes. 0 for the initial value.
+ */
+ int idx;
} RT_NODE_ITER;
typedef struct RT_ITER
{
RT_RADIX_TREE *tree;
- /* Track the iteration on nodes of each level */
- RT_NODE_ITER stack[RT_MAX_LEVEL];
- int stack_len;
+ /* Track the nodes for each level. level = 0 is for a leaf node */
+ RT_NODE_ITER node_iters[RT_MAX_LEVEL];
+ int top_level;
- /* The key is constructed during iteration */
+ /* The key constructed during the iteration */
uint64 key;
} RT_ITER;
@@ -1243,7 +1246,7 @@ RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
* it can store the key.
*/
static pg_noinline void
-RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
+RT_EXTEND_UP(RT_RADIX_TREE *tree, uint64 key)
{
int target_shift;
RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
@@ -1282,7 +1285,7 @@ RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
* Insert inner and leaf nodes from 'node' to bottom.
*/
static pg_noinline void
-RT_SET_EXTEND(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
+RT_EXTEND_DOWN(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node)
{
int shift = node->shift;
@@ -1613,7 +1616,7 @@ RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
/* Extend the tree if necessary */
if (key > tree->ctl->max_val)
- RT_EXTEND(tree, key);
+ RT_EXTEND_UP(tree, key);
stored_child = tree->ctl->root;
parent = RT_PTR_GET_LOCAL(tree, stored_child);
@@ -1631,7 +1634,7 @@ RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
if (!RT_NODE_SEARCH_INNER(child, key, &new_child))
{
- RT_SET_EXTEND(tree, key, value_p, parent, stored_child, child);
+ RT_EXTEND_DOWN(tree, key, value_p, parent, stored_child, child);
RT_UNLOCK(tree);
return false;
}
@@ -1805,16 +1808,9 @@ RT_DELETE(RT_RADIX_TREE *tree, uint64 key)
}
#endif
-static inline void
-RT_ITER_UPDATE_KEY(RT_ITER *iter, uint8 chunk, uint8 shift)
-{
- iter->key &= ~(((uint64) RT_CHUNK_MASK) << shift);
- iter->key |= (((uint64) chunk) << shift);
-}
-
/*
- * Advance the slot in the inner node. Return the child if exists, otherwise
- * null.
+ * Scan the inner node and return the next child node if exist, otherwise
+ * return NULL.
*/
static inline RT_PTR_LOCAL
RT_NODE_INNER_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter)
@@ -1825,8 +1821,8 @@ RT_NODE_INNER_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter)
}
/*
- * Advance the slot in the leaf node. On success, return true and the value
- * is set to value_p, otherwise return false.
+ * Scan the leaf node, and return true and the next value is set to value_p
+ * if exists. Otherwise return false.
*/
static inline bool
RT_NODE_LEAF_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter,
@@ -1838,29 +1834,50 @@ RT_NODE_LEAF_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter,
}
/*
- * Update each node_iter for inner nodes in the iterator node stack.
+ * While descending the radix tree from the 'from' node to the bottom, we
+ * set the next node to iterate for each level.
*/
static void
-RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
+RT_ITER_SET_NODE_FROM(RT_ITER *iter, RT_PTR_LOCAL from)
{
- int level = from;
- RT_PTR_LOCAL node = from_node;
+ int level = from->shift / RT_NODE_SPAN;
+ RT_PTR_LOCAL node = from;
for (;;)
{
- RT_NODE_ITER *node_iter = &(iter->stack[level--]);
+ RT_NODE_ITER *node_iter = &(iter->node_iters[level--]);
+
+#ifdef USE_ASSERT_CHECKING
+ if (node_iter->node)
+ {
+ /* We must have finished the iteration on the previous node */
+ if (RT_NODE_IS_LEAF(node_iter->node))
+ {
+ uint64 dummy;
+ Assert(!RT_NODE_LEAF_ITERATE_NEXT(iter, node_iter, &dummy));
+ }
+ else
+ Assert(!RT_NODE_INNER_ITERATE_NEXT(iter, node_iter));
+ }
+#endif
+ /* Set the node to the node iterator of this level */
node_iter->node = node;
- node_iter->current_idx = -1;
+ node_iter->idx = 0;
- /* We don't advance the leaf node iterator here */
if (RT_NODE_IS_LEAF(node))
- return;
+ {
+ /* We will visit the leaf node when RT_ITERATE_NEXT() */
+ break;
+ }
- /* Advance to the next slot in the inner node */
+ /*
+ * Get the first child node from the node, which corresponds to the
+ * lowest chunk within the node.
+ */
node = RT_NODE_INNER_ITERATE_NEXT(iter, node_iter);
- /* We must find the first children in the node */
+ /* The first child must be found */
Assert(node);
}
}
@@ -1874,14 +1891,11 @@ RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
RT_SCOPE RT_ITER *
RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
{
- MemoryContext old_ctx;
RT_ITER *iter;
RT_PTR_LOCAL root;
- int top_level;
- old_ctx = MemoryContextSwitchTo(tree->context);
-
- iter = (RT_ITER *) palloc0(sizeof(RT_ITER));
+ iter = (RT_ITER *) MemoryContextAllocZero(tree->context,
+ sizeof(RT_ITER));
iter->tree = tree;
RT_LOCK_SHARED(tree);
@@ -1891,16 +1905,13 @@ RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
return iter;
root = RT_PTR_GET_LOCAL(tree, iter->tree->ctl->root);
- top_level = root->shift / RT_NODE_SPAN;
- iter->stack_len = top_level;
+ iter->top_level = root->shift / RT_NODE_SPAN;
/*
- * Descend to the left most leaf node from the root. The key is being
- * constructed while descending to the leaf.
+ * Set the next node to iterate for each level from the level of the
+ * root node.
*/
- RT_UPDATE_ITER_STACK(iter, root, top_level);
-
- MemoryContextSwitchTo(old_ctx);
+ RT_ITER_SET_NODE_FROM(iter, root);
return iter;
}
@@ -1912,6 +1923,8 @@ RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
RT_SCOPE bool
RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p)
{
+ Assert(value_p != NULL);
+
/* Empty tree */
if (!iter->tree->ctl->root)
return false;
@@ -1919,43 +1932,38 @@ RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p)
for (;;)
{
RT_PTR_LOCAL child = NULL;
- RT_VALUE_TYPE value;
- int level;
- bool found;
-
- /* Advance the leaf node iterator to get next key-value pair */
- found = RT_NODE_LEAF_ITERATE_NEXT(iter, &(iter->stack[0]), &value);
- if (found)
+ /* Get the next chunk of the leaf node */
+ if (RT_NODE_LEAF_ITERATE_NEXT(iter, &(iter->node_iters[0]), value_p))
{
*key_p = iter->key;
- *value_p = value;
return true;
}
/*
- * We've visited all values in the leaf node, so advance inner node
- * iterators from the level=1 until we find the next child node.
+ * We've visited all values in the leaf node, so advance all inner node
+ * iterators by visiting inner nodes from the level = 1 until we find the
+ * next inner node that has a child node.
*/
- for (level = 1; level <= iter->stack_len; level++)
+ for (int level = 1; level <= iter->top_level; level++)
{
- child = RT_NODE_INNER_ITERATE_NEXT(iter, &(iter->stack[level]));
+ child = RT_NODE_INNER_ITERATE_NEXT(iter, &(iter->node_iters[level]));
if (child)
break;
}
- /* the iteration finished */
+ /* We've visited all nodes, so the iteration finished */
if (!child)
- return false;
+ break;
/*
- * Set the node to the node iterator and update the iterator stack
- * from this node.
+ * Found the new child node. We update the next node to iterate for each
+ * level from the level of this child node.
*/
- RT_UPDATE_ITER_STACK(iter, child, level - 1);
+ RT_ITER_SET_NODE_FROM(iter, child);
- /* Node iterators are updated, so try again from the leaf */
+ /* Find key-value from the leaf node again */
}
return false;
@@ -2470,8 +2478,8 @@ RT_DUMP(RT_RADIX_TREE *tree)
#undef RT_INIT_NODE
#undef RT_FREE_NODE
#undef RT_FREE_RECURSE
-#undef RT_EXTEND
-#undef RT_SET_EXTEND
+#undef RT_EXTEND_UP
+#undef RT_EXTEND_DOWN
#undef RT_SWITCH_NODE_KIND
#undef RT_COPY_NODE
#undef RT_REPLACE_NODE
@@ -2509,8 +2517,7 @@ RT_DUMP(RT_RADIX_TREE *tree)
#undef RT_NODE_INSERT_LEAF
#undef RT_NODE_INNER_ITERATE_NEXT
#undef RT_NODE_LEAF_ITERATE_NEXT
-#undef RT_UPDATE_ITER_STACK
-#undef RT_ITER_UPDATE_KEY
+#undef RT_RT_ITER_SET_NODE_FROM
#undef RT_VERIFY_NODE
#undef RT_DEBUG
diff --git a/src/include/lib/radixtree_iter_impl.h b/src/include/lib/radixtree_iter_impl.h
index 98c78eb237..5c1034768e 100644
--- a/src/include/lib/radixtree_iter_impl.h
+++ b/src/include/lib/radixtree_iter_impl.h
@@ -27,12 +27,10 @@
#error node level must be either inner or leaf
#endif
- bool found = false;
- uint8 key_chunk;
+ uint8 key_chunk = 0;
#ifdef RT_NODE_LEVEL_LEAF
- RT_VALUE_TYPE value;
-
+ Assert(value_p != NULL);
Assert(RT_NODE_IS_LEAF(node_iter->node));
#else
RT_PTR_LOCAL child = NULL;
@@ -50,99 +48,92 @@
{
RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node_iter->node;
- node_iter->current_idx++;
- if (node_iter->current_idx >= n3->base.n.count)
- break;
+ if (node_iter->idx >= n3->base.n.count)
+ return false;
+
#ifdef RT_NODE_LEVEL_LEAF
- value = n3->values[node_iter->current_idx];
+ *value_p = n3->values[node_iter->idx];
#else
- child = RT_PTR_GET_LOCAL(iter->tree, n3->children[node_iter->current_idx]);
+ child = RT_PTR_GET_LOCAL(iter->tree, n3->children[node_iter->idx]);
#endif
- key_chunk = n3->base.chunks[node_iter->current_idx];
- found = true;
+ key_chunk = n3->base.chunks[node_iter->idx];
+ node_iter->idx++;
break;
}
case RT_NODE_KIND_32:
{
RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node_iter->node;
- node_iter->current_idx++;
- if (node_iter->current_idx >= n32->base.n.count)
- break;
+ if (node_iter->idx >= n32->base.n.count)
+ return false;
#ifdef RT_NODE_LEVEL_LEAF
- value = n32->values[node_iter->current_idx];
+ *value_p = n32->values[node_iter->idx];
#else
- child = RT_PTR_GET_LOCAL(iter->tree, n32->children[node_iter->current_idx]);
+ child = RT_PTR_GET_LOCAL(iter->tree, n32->children[node_iter->idx]);
#endif
- key_chunk = n32->base.chunks[node_iter->current_idx];
- found = true;
+ key_chunk = n32->base.chunks[node_iter->idx];
+ node_iter->idx++;
break;
}
case RT_NODE_KIND_125:
{
RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node_iter->node;
- int i;
+ int chunk;
- for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ for (chunk = node_iter->idx; chunk < RT_NODE_MAX_SLOTS; chunk++)
{
- if (RT_NODE_125_IS_CHUNK_USED((RT_NODE_BASE_125 *) n125, i))
+ if (RT_NODE_125_IS_CHUNK_USED((RT_NODE_BASE_125 *) n125, chunk))
break;
}
- if (i >= RT_NODE_MAX_SLOTS)
- break;
+ if (chunk >= RT_NODE_MAX_SLOTS)
+ return false;
- node_iter->current_idx = i;
#ifdef RT_NODE_LEVEL_LEAF
- value = RT_NODE_LEAF_125_GET_VALUE(n125, i);
+ *value_p = RT_NODE_LEAF_125_GET_VALUE(n125, chunk);
#else
- child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_125_GET_CHILD(n125, chunk));
#endif
- key_chunk = i;
- found = true;
+ key_chunk = chunk;
+ node_iter->idx = chunk + 1;
break;
}
case RT_NODE_KIND_256:
{
RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node_iter->node;
- int i;
+ int chunk;
- for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ for (chunk = node_iter->idx; chunk < RT_NODE_MAX_SLOTS; chunk++)
{
#ifdef RT_NODE_LEVEL_LEAF
- if (RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+ if (RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
#else
- if (RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ if (RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
#endif
break;
}
- if (i >= RT_NODE_MAX_SLOTS)
- break;
+ if (chunk >= RT_NODE_MAX_SLOTS)
+ return false;
- node_iter->current_idx = i;
#ifdef RT_NODE_LEVEL_LEAF
- value = RT_NODE_LEAF_256_GET_VALUE(n256, i);
+ *value_p = RT_NODE_LEAF_256_GET_VALUE(n256, chunk);
#else
- child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_256_GET_CHILD(n256, chunk));
#endif
- key_chunk = i;
- found = true;
+ key_chunk = chunk;
+ node_iter->idx = chunk + 1;
break;
}
}
- if (found)
- {
- RT_ITER_UPDATE_KEY(iter, key_chunk, node_iter->node->shift);
-#ifdef RT_NODE_LEVEL_LEAF
- *value_p = value;
-#endif
- }
+ /* Update the part of the key */
+ iter->key &= ~(((uint64) RT_CHUNK_MASK) << node_iter->node->shift);
+ iter->key |= (((uint64) key_chunk) << node_iter->node->shift);
#ifdef RT_NODE_LEVEL_LEAF
- return found;
+ return true;
#else
return child;
#endif
diff --git a/src/test/modules/test_radixtree/expected/test_radixtree.out b/src/test/modules/test_radixtree/expected/test_radixtree.out
index ce645cb8b5..7ad1ce3605 100644
--- a/src/test/modules/test_radixtree/expected/test_radixtree.out
+++ b/src/test/modules/test_radixtree/expected/test_radixtree.out
@@ -4,8 +4,10 @@ CREATE EXTENSION test_radixtree;
-- an error if something fails.
--
SELECT test_radixtree();
-NOTICE: testing basic operations with leaf node 4
-NOTICE: testing basic operations with inner node 4
+NOTICE: testing basic operations with leaf node 3
+NOTICE: testing basic operations with inner node 3
+NOTICE: testing basic operations with leaf node 15
+NOTICE: testing basic operations with inner node 15
NOTICE: testing basic operations with leaf node 32
NOTICE: testing basic operations with inner node 32
NOTICE: testing basic operations with leaf node 125
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
index afe53382f3..5a169854d9 100644
--- a/src/test/modules/test_radixtree/test_radixtree.c
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -43,12 +43,15 @@ typedef uint64 TestValueType;
*/
static const bool rt_test_stats = false;
-static int rt_node_kind_fanouts[] = {
- 0,
- 4, /* RT_NODE_KIND_4 */
- 32, /* RT_NODE_KIND_32 */
- 125, /* RT_NODE_KIND_125 */
- 256 /* RT_NODE_KIND_256 */
+/*
+ * XXX: should we expose and use RT_SIZE_CLASS and RT_SIZE_CLASS_INFO?
+ */
+static int rt_node_class_fanouts[] = {
+ 3, /* RT_CLASS_3 */
+ 15, /* RT_CLASS_32_MIN */
+ 32, /* RT_CLASS_32_MAX */
+ 125, /* RT_CLASS_125 */
+ 256 /* RT_CLASS_256 */
};
/*
* A struct to define a pattern of integers, for use with the test_pattern()
@@ -260,10 +263,9 @@ test_basic(int children, bool test_inner)
* Check if keys from start to end with the shift exist in the tree.
*/
static void
-check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end,
- int incr)
+check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end)
{
- for (int i = start; i < end; i++)
+ for (int i = start; i <= end; i++)
{
uint64 key = ((uint64) i << shift);
TestValueType val;
@@ -277,22 +279,26 @@ check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end,
}
}
+/*
+ * Insert 256 key-value pairs, and check if keys are properly inserted on each
+ * node class.
+ */
+/* Test keys [0, 256) */
+#define NODE_TYPE_TEST_KEY_MIN 0
+#define NODE_TYPE_TEST_KEY_MAX 256
static void
-test_node_types_insert(rt_radix_tree *radixtree, uint8 shift, bool insert_asc)
+test_node_types_insert_asc(rt_radix_tree *radixtree, uint8 shift)
{
- uint64 num_entries;
- int ninserted = 0;
- int start = insert_asc ? 0 : 256;
- int incr = insert_asc ? 1 : -1;
- int end = insert_asc ? 256 : 0;
- int node_kind_idx = 1;
+ uint64 num_entries;
+ int node_class_idx = 0;
+ uint64 key_checked = 0;
- for (int i = start; i != end; i += incr)
+ for (int i = NODE_TYPE_TEST_KEY_MIN; i < NODE_TYPE_TEST_KEY_MAX; i++)
{
uint64 key = ((uint64) i << shift);
bool found;
- found = rt_set(radixtree, key, (TestValueType*) &key);
+ found = rt_set(radixtree, key, (TestValueType *) &key);
if (found)
elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " is found", key);
@@ -300,24 +306,49 @@ test_node_types_insert(rt_radix_tree *radixtree, uint8 shift, bool insert_asc)
* After filling all slots in each node type, check if the values
* are stored properly.
*/
- if (ninserted == rt_node_kind_fanouts[node_kind_idx] - 1)
+ if ((i + 1) == rt_node_class_fanouts[node_class_idx])
{
- int check_start = insert_asc
- ? rt_node_kind_fanouts[node_kind_idx - 1]
- : rt_node_kind_fanouts[node_kind_idx];
- int check_end = insert_asc
- ? rt_node_kind_fanouts[node_kind_idx]
- : rt_node_kind_fanouts[node_kind_idx - 1];
-
- check_search_on_node(radixtree, shift, check_start, check_end, incr);
- node_kind_idx++;
+ check_search_on_node(radixtree, shift, key_checked, i);
+ key_checked = i;
+ node_class_idx++;
}
-
- ninserted++;
}
num_entries = rt_num_entries(radixtree);
+ if (num_entries != 256)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+/*
+ * Similar to test_node_types_insert_asc(), but inserts keys in descending order.
+ */
+static void
+test_node_types_insert_desc(rt_radix_tree *radixtree, uint8 shift)
+{
+ uint64 num_entries;
+ int node_class_idx = 0;
+ uint64 key_checked = NODE_TYPE_TEST_KEY_MAX - 1;
+
+ for (int i = NODE_TYPE_TEST_KEY_MAX - 1; i >= NODE_TYPE_TEST_KEY_MIN; i--)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_set(radixtree, key, (TestValueType *) &key);
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " is found", key);
+ if ((i + 1) == rt_node_class_fanouts[node_class_idx])
+ {
+ check_search_on_node(radixtree, shift, i, key_checked);
+ key_checked = i;
+ node_class_idx++;
+ }
+ }
+
+ num_entries = rt_num_entries(radixtree);
if (num_entries != 256)
elog(ERROR,
"rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
@@ -329,7 +360,7 @@ test_node_types_delete(rt_radix_tree *radixtree, uint8 shift)
{
uint64 num_entries;
- for (int i = 0; i < 256; i++)
+ for (int i = NODE_TYPE_TEST_KEY_MIN; i < NODE_TYPE_TEST_KEY_MAX; i++)
{
uint64 key = ((uint64) i << shift);
bool found;
@@ -379,9 +410,9 @@ test_node_types(uint8 shift)
* then delete all entries to make it empty, and insert and search entries
* again.
*/
- test_node_types_insert(radixtree, shift, true);
+ test_node_types_insert_asc(radixtree, shift);
test_node_types_delete(radixtree, shift);
- test_node_types_insert(radixtree, shift, false);
+ test_node_types_insert_desc(radixtree, shift);
rt_free(radixtree);
#ifdef RT_SHMEM
@@ -664,10 +695,10 @@ test_radixtree(PG_FUNCTION_ARGS)
{
test_empty();
- for (int i = 1; i < lengthof(rt_node_kind_fanouts); i++)
+ for (int i = 0; i < lengthof(rt_node_class_fanouts); i++)
{
- test_basic(rt_node_kind_fanouts[i], false);
- test_basic(rt_node_kind_fanouts[i], true);
+ test_basic(rt_node_class_fanouts[i], false);
+ test_basic(rt_node_class_fanouts[i], true);
}
for (int shift = 0; shift <= (64 - 8); shift += 8)
--
2.31.1
[application/octet-stream] v31-0014-Revert-building-benchmark-module-for-CI.patch (694B, ../../CAD21AoDBmD5q=eO+K=gyuVt53XvwpJ2dgxPwrtZ-eVOjVmtJjg@mail.gmail.com/6-v31-0014-Revert-building-benchmark-module-for-CI.patch)
download | inline diff:
From 7bae7b13e777c826c542ac33766ad8358672d9cc Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 14 Feb 2023 19:31:34 +0700
Subject: [PATCH v31 14/14] Revert building benchmark module for CI
---
contrib/meson.build | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/contrib/meson.build b/contrib/meson.build
index 421d469f8c..52253de793 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,7 +12,7 @@ subdir('amcheck')
subdir('auth_delay')
subdir('auto_explain')
subdir('basic_archive')
-subdir('bench_radix_tree')
+#subdir('bench_radix_tree')
subdir('bloom')
subdir('basebackup_to_shell')
subdir('bool_plperl')
--
2.31.1
[application/octet-stream] v31-0005-Tool-for-measuring-radix-tree-and-tidstore-perfo.patch (24.7K, ../../CAD21AoDBmD5q=eO+K=gyuVt53XvwpJ2dgxPwrtZ-eVOjVmtJjg@mail.gmail.com/7-v31-0005-Tool-for-measuring-radix-tree-and-tidstore-perfo.patch)
download | inline diff:
From bc5b4650377c4dcb4f108013a5638d6f17cd13ef Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 16 Sep 2022 11:57:03 +0900
Subject: [PATCH v31 05/14] Tool for measuring radix tree and tidstore
performance
Includes Meson support, but commented out to avoid warnings
XXX: Not for commit
---
contrib/bench_radix_tree/Makefile | 21 +
.../bench_radix_tree--1.0.sql | 88 +++
contrib/bench_radix_tree/bench_radix_tree.c | 747 ++++++++++++++++++
.../bench_radix_tree/bench_radix_tree.control | 6 +
contrib/bench_radix_tree/expected/bench.out | 13 +
contrib/bench_radix_tree/meson.build | 33 +
contrib/bench_radix_tree/sql/bench.sql | 16 +
contrib/meson.build | 1 +
8 files changed, 925 insertions(+)
create mode 100644 contrib/bench_radix_tree/Makefile
create mode 100644 contrib/bench_radix_tree/bench_radix_tree--1.0.sql
create mode 100644 contrib/bench_radix_tree/bench_radix_tree.c
create mode 100644 contrib/bench_radix_tree/bench_radix_tree.control
create mode 100644 contrib/bench_radix_tree/expected/bench.out
create mode 100644 contrib/bench_radix_tree/meson.build
create mode 100644 contrib/bench_radix_tree/sql/bench.sql
diff --git a/contrib/bench_radix_tree/Makefile b/contrib/bench_radix_tree/Makefile
new file mode 100644
index 0000000000..952bb0ceae
--- /dev/null
+++ b/contrib/bench_radix_tree/Makefile
@@ -0,0 +1,21 @@
+# contrib/bench_radix_tree/Makefile
+
+MODULE_big = bench_radix_tree
+OBJS = \
+ bench_radix_tree.o
+
+EXTENSION = bench_radix_tree
+DATA = bench_radix_tree--1.0.sql
+
+REGRESS = bench_fixed_height
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/bench_radix_tree
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
new file mode 100644
index 0000000000..ad66265e23
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
@@ -0,0 +1,88 @@
+/* contrib/bench_radix_tree/bench_radix_tree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION bench_radix_tree" to load this file. \quit
+
+create function bench_shuffle_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_seq_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_load_random_int(
+cnt int8,
+OUT mem_allocated int8,
+OUT load_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_search_random_nodes(
+cnt int8,
+filter_str text DEFAULT NULL,
+OUT mem_allocated int8,
+OUT load_ms int8,
+OUT search_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C VOLATILE PARALLEL UNSAFE;
+
+create function bench_fixed_height_search(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT rt_search_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_node128_load(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_sparseload_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_tidstore_load(
+minblk int4,
+maxblk int4,
+OUT mem_allocated int8,
+OUT load_ms int8,
+OUT iter_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
new file mode 100644
index 0000000000..6e5149e2c4
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -0,0 +1,747 @@
+/*-------------------------------------------------------------------------
+ *
+ * bench_radix_tree.c
+ *
+ * Copyright (c) 2016-2022, PostgreSQL Global Development Group
+ *
+ * contrib/bench_radix_tree/bench_radix_tree.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/tidstore.h"
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "funcapi.h"
+#include "lib/radixtree.h"
+#include <math.h>
+#include "miscadmin.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/timestamp.h"
+
+PG_MODULE_MAGIC;
+
+/* run benchmark also for binary-search case? */
+/* #define MEASURE_BINARY_SEARCH 1 */
+
+#define TIDS_PER_BLOCK_FOR_LOAD 30
+#define TIDS_PER_BLOCK_FOR_LOOKUP 50
+
+//#define RT_DEBUG
+#define RT_PREFIX rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE uint64
+// WIP: compiles with warnings because rt_attach is defined but not used
+// #define RT_SHMEM
+#include "lib/radixtree.h"
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+ return tree->ctl->num_keys;
+}
+
+
+PG_FUNCTION_INFO_V1(bench_seq_search);
+PG_FUNCTION_INFO_V1(bench_shuffle_search);
+PG_FUNCTION_INFO_V1(bench_load_random_int);
+PG_FUNCTION_INFO_V1(bench_fixed_height_search);
+PG_FUNCTION_INFO_V1(bench_search_random_nodes);
+PG_FUNCTION_INFO_V1(bench_node128_load);
+PG_FUNCTION_INFO_V1(bench_tidstore_load);
+
+static uint64
+tid_to_key_off(ItemPointer tid, uint32 *off)
+{
+ uint64 upper;
+ uint32 shift = pg_ceil_log2_32(MaxHeapTuplesPerPage);
+ int64 tid_i;
+
+ Assert(ItemPointerGetOffsetNumber(tid) < MaxHeapTuplesPerPage);
+
+ tid_i = ItemPointerGetOffsetNumber(tid);
+ tid_i |= (uint64) ItemPointerGetBlockNumber(tid) << shift;
+
+ /* log(sizeof(uint64) * BITS_PER_BYTE, 2) = log(64, 2) = 6 */
+ *off = tid_i & ((1 << 6) - 1);
+ upper = tid_i >> 6;
+ Assert(*off < (sizeof(uint64) * BITS_PER_BYTE));
+
+ Assert(*off < 64);
+
+ return upper;
+}
+
+static int
+shuffle_randrange(pg_prng_state *state, int lower, int upper)
+{
+ return (int) floor(pg_prng_double(state) * ((upper - lower) + 0.999999)) + lower;
+}
+
+/* Naive Fisher-Yates implementation*/
+static void
+shuffle_itemptrs(ItemPointer itemptr, uint64 nitems)
+{
+ /* reproducability */
+ pg_prng_state state;
+
+ pg_prng_seed(&state, 0);
+
+ for (int i = 0; i < nitems - 1; i++)
+ {
+ int j = shuffle_randrange(&state, i, nitems - 1);
+ ItemPointerData t = itemptr[j];
+
+ itemptr[j] = itemptr[i];
+ itemptr[i] = t;
+ }
+}
+
+static ItemPointer
+generate_tids(BlockNumber minblk, BlockNumber maxblk, int ntids_per_blk, uint64 *ntids_p,
+ bool random_block)
+{
+ ItemPointer tids;
+ uint64 maxitems;
+ uint64 ntids = 0;
+ pg_prng_state state;
+
+ maxitems = (maxblk - minblk + 1) * ntids_per_blk;
+ tids = MemoryContextAllocHuge(TopTransactionContext,
+ sizeof(ItemPointerData) * maxitems);
+
+ if (random_block)
+ pg_prng_seed(&state, 0x9E3779B185EBCA87);
+
+ for (BlockNumber blk = minblk; blk < maxblk; blk++)
+ {
+ if (random_block && !pg_prng_bool(&state))
+ continue;
+
+ for (OffsetNumber off = FirstOffsetNumber;
+ off <= ntids_per_blk; off++)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ ItemPointerSetBlockNumber(&(tids[ntids]), blk);
+ ItemPointerSetOffsetNumber(&(tids[ntids]), off);
+
+ ntids++;
+ }
+ }
+
+ *ntids_p = ntids;
+ return tids;
+}
+
+#ifdef MEASURE_BINARY_SEARCH
+static int
+vac_cmp_itemptr(const void *left, const void *right)
+{
+ BlockNumber lblk,
+ rblk;
+ OffsetNumber loff,
+ roff;
+
+ lblk = ItemPointerGetBlockNumber((ItemPointer) left);
+ rblk = ItemPointerGetBlockNumber((ItemPointer) right);
+
+ if (lblk < rblk)
+ return -1;
+ if (lblk > rblk)
+ return 1;
+
+ loff = ItemPointerGetOffsetNumber((ItemPointer) left);
+ roff = ItemPointerGetOffsetNumber((ItemPointer) right);
+
+ if (loff < roff)
+ return -1;
+ if (loff > roff)
+ return 1;
+
+ return 0;
+}
+#endif
+
+Datum
+bench_tidstore_load(PG_FUNCTION_ARGS)
+{
+ BlockNumber minblk = PG_GETARG_INT32(0);
+ BlockNumber maxblk = PG_GETARG_INT32(1);
+ TidStore *ts;
+ TidStoreIter *iter;
+ TidStoreIterResult *result;
+ OffsetNumber *offs;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_ms;
+ int64 iter_ms;
+ TupleDesc tupdesc;
+ Datum values[3];
+ bool nulls[3] = {false};
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ offs = palloc(sizeof(OffsetNumber) * TIDS_PER_BLOCK_FOR_LOAD);
+ for (int i = 0; i < TIDS_PER_BLOCK_FOR_LOAD; i++)
+ offs[i] = i + 1; /* FirstOffsetNumber is 1 */
+
+ ts = tidstore_create(1 * 1024L * 1024L * 1024L, MaxHeapTuplesPerPage, NULL);
+
+ /* load tids */
+ start_time = GetCurrentTimestamp();
+ for (BlockNumber blkno = minblk; blkno < maxblk; blkno++)
+ tidstore_add_tids(ts, blkno, offs, TIDS_PER_BLOCK_FOR_LOAD);
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_ms = secs * 1000 + usecs / 1000;
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* iterate through tids */
+ iter = tidstore_begin_iterate(ts);
+ start_time = GetCurrentTimestamp();
+ while ((result = tidstore_iterate_next(iter)) != NULL)
+ ;
+ tidstore_end_iterate(iter);
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ iter_ms = secs * 1000 + usecs / 1000;
+
+ values[0] = Int64GetDatum(tidstore_memory_usage(ts));
+ values[1] = Int64GetDatum(load_ms);
+ values[2] = Int64GetDatum(iter_ms);
+
+ tidstore_destroy(ts);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+static Datum
+bench_search(FunctionCallInfo fcinfo, bool shuffle)
+{
+ BlockNumber minblk = PG_GETARG_INT32(0);
+ BlockNumber maxblk = PG_GETARG_INT32(1);
+ bool random_block = PG_GETARG_BOOL(2);
+ rt_radix_tree *rt = NULL;
+ uint64 ntids;
+ uint64 key;
+ uint64 last_key = PG_UINT64_MAX;
+ uint64 val = 0;
+ ItemPointer tids;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_load_ms,
+ rt_search_ms;
+ Datum values[7];
+ bool nulls[7];
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ tids = generate_tids(minblk, maxblk, TIDS_PER_BLOCK_FOR_LOAD, &ntids, random_block);
+
+ /* measure the load time of the radix tree */
+ rt = rt_create(CurrentMemoryContext);
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ uint32 off;
+
+ CHECK_FOR_INTERRUPTS();
+
+ key = tid_to_key_off(tid, &off);
+
+ if (last_key != PG_UINT64_MAX && last_key != key)
+ {
+ rt_set(rt, last_key, &val);
+ val = 0;
+ }
+
+ last_key = key;
+ val |= (uint64) 1 << off;
+ }
+ if (last_key != PG_UINT64_MAX)
+ rt_set(rt, last_key, &val);
+
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_load_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+ if (shuffle)
+ shuffle_itemptrs(tids, ntids);
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* meaure the serach time of the radix tree */
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ uint32 off;
+ volatile bool ret; /* prevent calling rt_search from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ key = tid_to_key_off(tid, &off);
+
+ ret = rt_search(rt, key, &val);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_search_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int64GetDatum(rt_num_entries(rt));
+ values[1] = Int64GetDatum(rt_memory_usage(rt));
+ values[2] = Int64GetDatum(sizeof(ItemPointerData) * ntids);
+ values[3] = Int64GetDatum(rt_load_ms);
+ nulls[4] = true; /* ar_load_ms */
+ values[5] = Int64GetDatum(rt_search_ms);
+ nulls[6] = true; /* ar_search_ms */
+
+#ifdef MEASURE_BINARY_SEARCH
+ {
+ ItemPointer itemptrs = NULL;
+
+ int64 ar_load_ms,
+ ar_search_ms;
+
+ /* measure the load time of the array */
+ itemptrs = MemoryContextAllocHuge(CurrentMemoryContext,
+ sizeof(ItemPointerData) * ntids);
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointerSetBlockNumber(&(itemptrs[i]),
+ ItemPointerGetBlockNumber(&(tids[i])));
+ ItemPointerSetOffsetNumber(&(itemptrs[i]),
+ ItemPointerGetOffsetNumber(&(tids[i])));
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ ar_load_ms = secs * 1000 + usecs / 1000;
+
+ /* next, measure the serach time of the array */
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ volatile bool ret; /* prevent calling bsearch from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ ret = bsearch((void *) tid,
+ (void *) itemptrs,
+ ntids,
+ sizeof(ItemPointerData),
+ vac_cmp_itemptr);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ ar_search_ms = secs * 1000 + usecs / 1000;
+
+ /* set the result */
+ nulls[4] = false;
+ values[4] = Int64GetDatum(ar_load_ms);
+ nulls[6] = false;
+ values[6] = Int64GetDatum(ar_search_ms);
+ }
+#endif
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_seq_search(PG_FUNCTION_ARGS)
+{
+ return bench_search(fcinfo, false);
+}
+
+Datum
+bench_shuffle_search(PG_FUNCTION_ARGS)
+{
+ return bench_search(fcinfo, true);
+}
+
+Datum
+bench_load_random_int(PG_FUNCTION_ARGS)
+{
+ uint64 cnt = (uint64) PG_GETARG_INT64(0);
+ rt_radix_tree *rt;
+ pg_prng_state state;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_time_ms;
+ Datum values[2];
+ bool nulls[2];
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ pg_prng_seed(&state, 0);
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 key = pg_prng_uint64(&state);
+
+ rt_set(rt, key, &key);
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_time_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int64GetDatum(rt_memory_usage(rt));
+ values[1] = Int64GetDatum(load_time_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+/* copy of splitmix64() */
+static uint64
+hash64(uint64 x)
+{
+ x ^= x >> 30;
+ x *= UINT64CONST(0xbf58476d1ce4e5b9);
+ x ^= x >> 27;
+ x *= UINT64CONST(0x94d049bb133111eb);
+ x ^= x >> 31;
+ return x;
+}
+
+/* attempts to have a relatively even population of node kinds */
+Datum
+bench_search_random_nodes(PG_FUNCTION_ARGS)
+{
+ uint64 cnt = (uint64) PG_GETARG_INT64(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_time_ms;
+ int64 search_time_ms;
+ Datum values[3] = {0};
+ bool nulls[3] = {0};
+ /* from trial and error */
+ uint64 filter = (((uint64) 0x7F<<32) | (0x07<<24) | (0xFF<<16) | 0xFF);
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (!PG_ARGISNULL(1))
+ {
+ char *filter_str = text_to_cstring(PG_GETARG_TEXT_P(1));
+
+ if (sscanf(filter_str, "0x%lX", &filter) == 0)
+ elog(ERROR, "invalid filter string %s", filter_str);
+ }
+ elog(NOTICE, "bench with filter 0x%lX", filter);
+
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 hash = hash64(i);
+ uint64 key = hash & filter;
+
+ rt_set(rt, key, &key);
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_time_ms = secs * 1000 + usecs / 1000;
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 hash = hash64(i);
+ uint64 key = hash & filter;
+ uint64 val;
+ volatile bool ret; /* prevent calling rt_search from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ ret = rt_search(rt, key, &val);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ search_time_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ values[0] = Int64GetDatum(rt_memory_usage(rt));
+ values[1] = Int64GetDatum(load_time_ms);
+ values[2] = Int64GetDatum(search_time_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_fixed_height_search(PG_FUNCTION_ARGS)
+{
+ int fanout = PG_GETARG_INT32(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_load_ms,
+ rt_search_ms;
+ Datum values[5];
+ bool nulls[5];
+
+ /* test boundary between vector and iteration */
+ const int n_keys = 5 * 16 * 16 * 16 * 16;
+ uint64 r,
+ h,
+ i,
+ j,
+ k;
+ uint64 key_id;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+
+ key_id = 0;
+
+ /*
+ * lower nodes have limited fanout, the top is only limited by
+ * bits-per-byte
+ */
+ for (r = 1;; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ for (i = 1; i <= fanout; i++)
+ {
+ for (j = 1; j <= fanout; j++)
+ {
+ for (k = 1; k <= fanout; k++)
+ {
+ uint64 key;
+
+ key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+ CHECK_FOR_INTERRUPTS();
+
+ key_id++;
+ if (key_id > n_keys)
+ goto finish_set;
+
+ rt_set(rt, key, &key_id);
+ }
+ }
+ }
+ }
+ }
+finish_set:
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_load_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ /* meaure the search time of the radix tree */
+ start_time = GetCurrentTimestamp();
+
+ key_id = 0;
+ for (r = 1;; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ for (i = 1; i <= fanout; i++)
+ {
+ for (j = 1; j <= fanout; j++)
+ {
+ for (k = 1; k <= fanout; k++)
+ {
+ uint64 key,
+ val;
+
+ key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+ CHECK_FOR_INTERRUPTS();
+
+ key_id++;
+ if (key_id > n_keys)
+ goto finish_search;
+
+ rt_search(rt, key, &val);
+ }
+ }
+ }
+ }
+ }
+finish_search:
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_search_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int32GetDatum(fanout);
+ values[1] = Int64GetDatum(rt_num_entries(rt));
+ values[2] = Int64GetDatum(rt_memory_usage(rt));
+ values[3] = Int64GetDatum(rt_load_ms);
+ values[4] = Int64GetDatum(rt_search_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_node128_load(PG_FUNCTION_ARGS)
+{
+ int fanout = PG_GETARG_INT32(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_sparseload_ms;
+ Datum values[5];
+ bool nulls[5];
+
+ uint64 r,
+ h;
+ uint64 key_id;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ rt = rt_create(CurrentMemoryContext);
+
+ key_id = 0;
+
+ for (r = 1; r <= fanout; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (h);
+
+ key_id++;
+ rt_set(rt, key, &key_id);
+ }
+ }
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ /* measure sparse deletion and re-loading */
+ start_time = GetCurrentTimestamp();
+
+ for (int t = 0; t<10000; t++)
+ {
+ /* delete one key in each leaf */
+ for (r = 1; r <= fanout; r++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (fanout);
+
+ rt_delete(rt, key);
+ }
+
+ /* add them all back */
+ key_id = 0;
+ for (r = 1; r <= fanout; r++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (fanout);
+
+ key_id++;
+ rt_set(rt, key, &key_id);
+ }
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_sparseload_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int32GetDatum(fanout);
+ values[1] = Int64GetDatum(rt_num_entries(rt));
+ values[2] = Int64GetDatum(rt_memory_usage(rt));
+ values[3] = Int64GetDatum(rt_sparseload_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+/* to silence warnings about unused iter functions */
+static void pg_attribute_unused()
+stub_iter()
+{
+ rt_radix_tree *rt;
+ rt_iter *iter;
+ uint64 key = 1;
+ uint64 value = 1;
+
+ rt = rt_create(CurrentMemoryContext);
+
+ iter = rt_begin_iterate(rt);
+ rt_iterate_next(iter, &key, &value);
+ rt_end_iterate(iter);
+}
\ No newline at end of file
diff --git a/contrib/bench_radix_tree/bench_radix_tree.control b/contrib/bench_radix_tree/bench_radix_tree.control
new file mode 100644
index 0000000000..1d988e6c9a
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.control
@@ -0,0 +1,6 @@
+# bench_radix_tree extension
+comment = 'benchmark suits for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/bench_radix_tree'
+relocatable = true
+trusted = true
diff --git a/contrib/bench_radix_tree/expected/bench.out b/contrib/bench_radix_tree/expected/bench.out
new file mode 100644
index 0000000000..60c303892e
--- /dev/null
+++ b/contrib/bench_radix_tree/expected/bench.out
@@ -0,0 +1,13 @@
+create extension bench_radix_tree;
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/bench_radix_tree/meson.build b/contrib/bench_radix_tree/meson.build
new file mode 100644
index 0000000000..332c1ae7df
--- /dev/null
+++ b/contrib/bench_radix_tree/meson.build
@@ -0,0 +1,33 @@
+bench_radix_tree_sources = files(
+ 'bench_radix_tree.c',
+)
+
+if host_system == 'windows'
+ bench_radix_tree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'bench_radix_tree',
+ '--FILEDESC', 'bench_radix_tree - performance test code for radix tree',])
+endif
+
+bench_radix_tree = shared_module('bench_radix_tree',
+ bench_radix_tree_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += bench_radix_tree
+
+install_data(
+ 'bench_radix_tree.control',
+ 'bench_radix_tree--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'bench_radix_tree',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'bench_radix_tree',
+ ],
+ },
+}
diff --git a/contrib/bench_radix_tree/sql/bench.sql b/contrib/bench_radix_tree/sql/bench.sql
new file mode 100644
index 0000000000..a46018c9d4
--- /dev/null
+++ b/contrib/bench_radix_tree/sql/bench.sql
@@ -0,0 +1,16 @@
+create extension bench_radix_tree;
+
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/meson.build b/contrib/meson.build
index bd4a57c43c..421d469f8c 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,6 +12,7 @@ subdir('amcheck')
subdir('auth_delay')
subdir('auto_explain')
subdir('basic_archive')
+subdir('bench_radix_tree')
subdir('bloom')
subdir('basebackup_to_shell')
subdir('bool_plperl')
--
2.31.1
[application/octet-stream] v31-0008-Review-TidStore.patch (32.1K, ../../CAD21AoDBmD5q=eO+K=gyuVt53XvwpJ2dgxPwrtZ-eVOjVmtJjg@mail.gmail.com/8-v31-0008-Review-TidStore.patch)
download | inline diff:
From 6842622ec10cf702fd062caccb091ce5ecbe56b5 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Thu, 16 Feb 2023 23:45:39 +0900
Subject: [PATCH v31 08/14] Review TidStore.
---
src/backend/access/common/tidstore.c | 340 +++++++++---------
src/include/access/tidstore.h | 37 +-
.../modules/test_tidstore/test_tidstore.c | 68 ++--
3 files changed, 234 insertions(+), 211 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 8c05e60d92..9360520482 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -3,18 +3,19 @@
* tidstore.c
* Tid (ItemPointerData) storage implementation.
*
- * This module provides a in-memory data structure to store Tids (ItemPointer).
- * Internally, a tid is encoded as a pair of 64-bit key and 64-bit value, and
- * stored in the radix tree.
+ * TidStore is a in-memory data structure to store tids (ItemPointerData).
+ * Internally, a tid is encoded as a pair of 64-bit key and 64-bit value,
+ * and stored in the radix tree.
*
- * A TidStore can be shared among parallel worker processes by passing DSA area
- * to tidstore_create(). Other backends can attach to the shared TidStore by
- * tidstore_attach().
+ * TidStore can be shared among parallel worker processes by passing DSA area
+ * to TidStoreCreate(). Other backends can attach to the shared TidStore by
+ * TidStoreAttach().
*
- * Regarding the concurrency, it basically relies on the concurrency support in
- * the radix tree, but we acquires the lock on a TidStore in some cases, for
- * example, when to reset the store and when to access the number tids in the
- * store (num_tids).
+ * Regarding the concurrency support, we use a single LWLock for the TidStore.
+ * The TidStore is exclusively locked when inserting encoded tids to the
+ * radix tree or when resetting itself. When searching on the TidStore or
+ * doing the iteration, it is not locked but the underlying radix tree is
+ * locked in shared mode.
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -34,16 +35,18 @@
#include "utils/memutils.h"
/*
- * For encoding purposes, tids are represented as a pair of 64-bit key and
- * 64-bit value. First, we construct 64-bit unsigned integer by combining
- * the block number and the offset number. The number of bits used for the
- * offset number is specified by max_offsets in tidstore_create(). We are
- * frugal with the bits, because smaller keys could help keeping the radix
- * tree shallow.
+ * For encoding purposes, a tid is represented as a pair of 64-bit key and
+ * 64-bit value.
*
- * For example, a tid of heap with 8kB blocks uses the lowest 9 bits for
- * the offset number and uses the next 32 bits for the block number. That
- * is, only 41 bits are used:
+ * First, we construct a 64-bit unsigned integer by combining the block
+ * number and the offset number. The number of bits used for the offset number
+ * is specified by max_off in TidStoreCreate(). We are frugal with the bits,
+ * because smaller keys could help keeping the radix tree shallow.
+ *
+ * For example, a tid of heap on a 8kB block uses the lowest 9 bits for
+ * the offset number and uses the next 32 bits for the block number. 9 bits
+ * are enough for the offset number, because MaxHeapTuplesPerPage < 2^9
+ * on 8kB blocks. That is, only 41 bits are used:
*
* uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
*
@@ -52,30 +55,34 @@
* u = unused bit
* (high on the left, low on the right)
*
- * 9 bits are enough for the offset number, because MaxHeapTuplesPerPage < 2^9
- * on 8kB blocks.
- *
- * The 64-bit value is the bitmap representation of the lowest 6 bits
- * (TIDSTORE_VALUE_NBITS) of the integer, and the rest 35 bits are used
- * as the key:
+ * Then, 64-bit value is the bitmap representation of the lowest 6 bits
+ * (LOWER_OFFSET_NBITS) of the integer, and 64-bit key consists of the
+ * upper 3 bits of the offset number and the block number, 35 bits in
+ * total:
*
* uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
* |----| value
- * |---------------------------------------------| key
+ * |--------------------------------------| key
*
* The maximum height of the radix tree is 5 in this case.
+ *
+ * If the number of bits required for offset numbers fits in LOWER_OFFSET_NBITS,
+ * 64-bit value is the bitmap representation of the offset number, and the
+ * 64-bit key is the block number.
*/
-#define TIDSTORE_VALUE_NBITS 6 /* log(64, 2) */
-#define TIDSTORE_OFFSET_MASK ((1 << TIDSTORE_VALUE_NBITS) - 1)
+typedef uint64 tidkey;
+typedef uint64 offsetbm;
+#define LOWER_OFFSET_NBITS 6 /* log(sizeof(offsetbm), 2) */
+#define LOWER_OFFSET_MASK ((1 << LOWER_OFFSET_NBITS) - 1)
-/* A magic value used to identify our TidStores. */
+/* A magic value used to identify our TidStore. */
#define TIDSTORE_MAGIC 0x826f6a10
#define RT_PREFIX local_rt
#define RT_SCOPE static
#define RT_DECLARE
#define RT_DEFINE
-#define RT_VALUE_TYPE uint64
+#define RT_VALUE_TYPE tidkey
#include "lib/radixtree.h"
#define RT_PREFIX shared_rt
@@ -83,7 +90,7 @@
#define RT_SCOPE static
#define RT_DECLARE
#define RT_DEFINE
-#define RT_VALUE_TYPE uint64
+#define RT_VALUE_TYPE tidkey
#include "lib/radixtree.h"
/* The control object for a TidStore */
@@ -94,10 +101,10 @@ typedef struct TidStoreControl
/* These values are never changed after creation */
size_t max_bytes; /* the maximum bytes a TidStore can use */
- int max_offset; /* the maximum offset number */
- int offset_nbits; /* the number of bits required for an offset
- * number */
- int offset_key_nbits; /* the number of bits of an offset number
+ int max_off; /* the maximum offset number */
+ int max_off_nbits; /* the number of bits required for offset
+ * numbers */
+ int upper_off_nbits; /* the number of bits of offset numbers
* used in a key */
/* The below fields are used only in shared case */
@@ -106,7 +113,7 @@ typedef struct TidStoreControl
LWLock lock;
/* handles for TidStore and radix tree */
- tidstore_handle handle;
+ TidStoreHandle handle;
shared_rt_handle tree_handle;
} TidStoreControl;
@@ -147,24 +154,27 @@ typedef struct TidStoreIter
bool finished;
/* save for the next iteration */
- uint64 next_key;
- uint64 next_val;
+ tidkey next_tidkey;
+ offsetbm next_off_bitmap;
- /* output for the caller */
- TidStoreIterResult result;
+ /*
+ * output for the caller. Must be last because variable-size.
+ */
+ TidStoreIterResult output;
} TidStoreIter;
-static void tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val);
-static inline BlockNumber key_get_blkno(TidStore *ts, uint64 key);
-static inline uint64 encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit);
-static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit);
+static void iter_decode_key_off(TidStoreIter *iter, tidkey key, offsetbm off_bitmap);
+static inline BlockNumber key_get_blkno(TidStore *ts, tidkey key);
+static inline tidkey encode_blk_off(TidStore *ts, BlockNumber block,
+ OffsetNumber offset, offsetbm *off_bit);
+static inline tidkey encode_tid(TidStore *ts, ItemPointer tid, offsetbm *off_bit);
/*
* Create a TidStore. The returned object is allocated in backend-local memory.
* The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
*/
TidStore *
-tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
+TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
{
TidStore *ts;
@@ -176,12 +186,12 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
* Memory consumption depends on the number of stored tids, but also on the
* distribution of them, how the radix tree stores, and the memory management
* that backed the radix tree. The maximum bytes that a TidStore can
- * use is specified by the max_bytes in tidstore_create(). We want the total
+ * use is specified by the max_bytes in TidStoreCreate(). We want the total
* amount of memory consumption by a TidStore not to exceed the max_bytes.
*
* In local TidStore cases, the radix tree uses slab allocators for each kind
* of node class. The most memory consuming case while adding Tids associated
- * with one page (i.e. during tidstore_add_tids()) is that we allocate a new
+ * with one page (i.e. during TidStoreSetBlockOffsets()) is that we allocate a new
* slab block for a new radix tree node, which is approximately 70kB. Therefore,
* we deduct 70kB from the max_bytes.
*
@@ -202,7 +212,7 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
dp = dsa_allocate0(area, sizeof(TidStoreControl));
ts->control = (TidStoreControl *) dsa_get_address(area, dp);
- ts->control->max_bytes = (uint64) (max_bytes * ratio);
+ ts->control->max_bytes = (size_t) (max_bytes * ratio);
ts->area = area;
ts->control->magic = TIDSTORE_MAGIC;
@@ -218,14 +228,14 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
ts->control->max_bytes = max_bytes - (70 * 1024);
}
- ts->control->max_offset = max_offset;
- ts->control->offset_nbits = pg_ceil_log2_32(max_offset);
+ ts->control->max_off = max_off;
+ ts->control->max_off_nbits = pg_ceil_log2_32(max_off);
- if (ts->control->offset_nbits < TIDSTORE_VALUE_NBITS)
- ts->control->offset_nbits = TIDSTORE_VALUE_NBITS;
+ if (ts->control->max_off_nbits < LOWER_OFFSET_NBITS)
+ ts->control->max_off_nbits = LOWER_OFFSET_NBITS;
- ts->control->offset_key_nbits =
- ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
+ ts->control->upper_off_nbits =
+ ts->control->max_off_nbits - LOWER_OFFSET_NBITS;
return ts;
}
@@ -235,7 +245,7 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
* allocated in backend-local memory using the CurrentMemoryContext.
*/
TidStore *
-tidstore_attach(dsa_area *area, tidstore_handle handle)
+TidStoreAttach(dsa_area *area, TidStoreHandle handle)
{
TidStore *ts;
dsa_pointer control;
@@ -266,7 +276,7 @@ tidstore_attach(dsa_area *area, tidstore_handle handle)
* to the operating system.
*/
void
-tidstore_detach(TidStore *ts)
+TidStoreDetach(TidStore *ts)
{
Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
@@ -279,12 +289,12 @@ tidstore_detach(TidStore *ts)
*
* TODO: The caller must be certain that no other backend will attempt to
* access the TidStore before calling this function. Other backend must
- * explicitly call tidstore_detach to free up backend-local memory associated
- * with the TidStore. The backend that calls tidstore_destroy must not call
- * tidstore_detach.
+ * explicitly call TidStoreDetach() to free up backend-local memory associated
+ * with the TidStore. The backend that calls TidStoreDestroy() must not call
+ * TidStoreDetach().
*/
void
-tidstore_destroy(TidStore *ts)
+TidStoreDestroy(TidStore *ts)
{
if (TidStoreIsShared(ts))
{
@@ -309,11 +319,11 @@ tidstore_destroy(TidStore *ts)
}
/*
- * Forget all collected Tids. It's similar to tidstore_destroy but we don't free
+ * Forget all collected Tids. It's similar to TidStoreDestroy() but we don't free
* entire TidStore but recreate only the radix tree storage.
*/
void
-tidstore_reset(TidStore *ts)
+TidStoreReset(TidStore *ts)
{
if (TidStoreIsShared(ts))
{
@@ -350,30 +360,34 @@ tidstore_reset(TidStore *ts)
}
}
-/* Add Tids on a block to TidStore */
+/*
+ * Set the given tids on the blkno to TidStore.
+ *
+ * NB: the offset numbers in offsets must be sorted in ascending order.
+ */
void
-tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
- int num_offsets)
+TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets)
{
- uint64 *values;
- uint64 key;
- uint64 prev_key;
- uint64 off_bitmap = 0;
+ offsetbm *bitmaps;
+ tidkey key;
+ tidkey prev_key;
+ offsetbm off_bitmap = 0;
int idx;
- const uint64 key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
- const int nkeys = UINT64CONST(1) << ts->control->offset_key_nbits;
+ const tidkey key_base = ((uint64) blkno) << ts->control->upper_off_nbits;
+ const int nkeys = UINT64CONST(1) << ts->control->upper_off_nbits;
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
- values = palloc(sizeof(uint64) * nkeys);
+ bitmaps = palloc(sizeof(offsetbm) * nkeys);
key = prev_key = key_base;
for (int i = 0; i < num_offsets; i++)
{
- uint64 off_bit;
+ offsetbm off_bit;
/* encode the tid to a key and partial offset */
- key = encode_key_off(ts, blkno, offsets[i], &off_bit);
+ key = encode_blk_off(ts, blkno, offsets[i], &off_bit);
/* make sure we scanned the line pointer array in order */
Assert(key >= prev_key);
@@ -384,11 +398,11 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
Assert(idx >= 0 && idx < nkeys);
/* write out offset bitmap for this key */
- values[idx] = off_bitmap;
+ bitmaps[idx] = off_bitmap;
/* zero out any gaps up to the current key */
for (int empty_idx = idx + 1; empty_idx < key - key_base; empty_idx++)
- values[empty_idx] = 0;
+ bitmaps[empty_idx] = 0;
/* reset for current key -- the current offset will be handled below */
off_bitmap = 0;
@@ -401,7 +415,7 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
/* save the final index for later */
idx = key - key_base;
/* write out last offset bitmap */
- values[idx] = off_bitmap;
+ bitmaps[idx] = off_bitmap;
if (TidStoreIsShared(ts))
LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
@@ -409,14 +423,14 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
/* insert the calculated key-values to the tree */
for (int i = 0; i <= idx; i++)
{
- if (values[i])
+ if (bitmaps[i])
{
key = key_base + i;
if (TidStoreIsShared(ts))
- shared_rt_set(ts->tree.shared, key, &values[i]);
+ shared_rt_set(ts->tree.shared, key, &bitmaps[i]);
else
- local_rt_set(ts->tree.local, key, &values[i]);
+ local_rt_set(ts->tree.local, key, &bitmaps[i]);
}
}
@@ -426,70 +440,70 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
if (TidStoreIsShared(ts))
LWLockRelease(&ts->control->lock);
- pfree(values);
+ pfree(bitmaps);
}
/* Return true if the given tid is present in the TidStore */
bool
-tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
+TidStoreIsMember(TidStore *ts, ItemPointer tid)
{
- uint64 key;
- uint64 val = 0;
- uint64 off_bit;
+ tidkey key;
+ offsetbm off_bitmap = 0;
+ offsetbm off_bit;
bool found;
- key = tid_to_key_off(ts, tid, &off_bit);
+ key = encode_tid(ts, tid, &off_bit);
if (TidStoreIsShared(ts))
- found = shared_rt_search(ts->tree.shared, key, &val);
+ found = shared_rt_search(ts->tree.shared, key, &off_bitmap);
else
- found = local_rt_search(ts->tree.local, key, &val);
+ found = local_rt_search(ts->tree.local, key, &off_bitmap);
if (!found)
return false;
- return (val & off_bit) != 0;
+ return (off_bitmap & off_bit) != 0;
}
/*
- * Prepare to iterate through a TidStore. Since the radix tree is locked during the
- * iteration, so tidstore_end_iterate() needs to called when finished.
+ * Prepare to iterate through a TidStore. Since the radix tree is locked during
+ * the iteration, so TidStoreEndIterate() needs to be called when finished.
+ *
+ * The TidStoreIter struct is created in the caller's memory context.
*
* Concurrent updates during the iteration will be blocked when inserting a
* key-value to the radix tree.
*/
TidStoreIter *
-tidstore_begin_iterate(TidStore *ts)
+TidStoreBeginIterate(TidStore *ts)
{
TidStoreIter *iter;
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
- iter = palloc0(sizeof(TidStoreIter));
+ iter = palloc0(sizeof(TidStoreIter) +
+ sizeof(OffsetNumber) * ts->control->max_off);
iter->ts = ts;
- iter->result.blkno = InvalidBlockNumber;
- iter->result.offsets = palloc(sizeof(OffsetNumber) * ts->control->max_offset);
-
if (TidStoreIsShared(ts))
iter->tree_iter.shared = shared_rt_begin_iterate(ts->tree.shared);
else
iter->tree_iter.local = local_rt_begin_iterate(ts->tree.local);
/* If the TidStore is empty, there is no business */
- if (tidstore_num_tids(ts) == 0)
+ if (TidStoreNumTids(ts) == 0)
iter->finished = true;
return iter;
}
static inline bool
-tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
+tidstore_iter(TidStoreIter *iter, tidkey *key, offsetbm *off_bitmap)
{
if (TidStoreIsShared(iter->ts))
- return shared_rt_iterate_next(iter->tree_iter.shared, key, val);
+ return shared_rt_iterate_next(iter->tree_iter.shared, key, off_bitmap);
- return local_rt_iterate_next(iter->tree_iter.local, key, val);
+ return local_rt_iterate_next(iter->tree_iter.local, key, off_bitmap);
}
/*
@@ -498,45 +512,48 @@ tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
* numbers in each result is also sorted in ascending order.
*/
TidStoreIterResult *
-tidstore_iterate_next(TidStoreIter *iter)
+TidStoreIterateNext(TidStoreIter *iter)
{
- uint64 key;
- uint64 val;
- TidStoreIterResult *result = &(iter->result);
+ tidkey key;
+ offsetbm off_bitmap = 0;
+ TidStoreIterResult *output = &(iter->output);
if (iter->finished)
return NULL;
- if (BlockNumberIsValid(result->blkno))
- {
- /* Process the previously collected key-value */
- result->num_offsets = 0;
- tidstore_iter_extract_tids(iter, iter->next_key, iter->next_val);
- }
+ /* Initialize the outputs */
+ output->blkno = InvalidBlockNumber;
+ output->num_offsets = 0;
- while (tidstore_iter_kv(iter, &key, &val))
- {
- BlockNumber blkno;
+ /*
+ * Decode the key and offset bitmap that are collected in the previous
+ * time, if exists.
+ */
+ if (iter->next_off_bitmap > 0)
+ iter_decode_key_off(iter, iter->next_tidkey, iter->next_off_bitmap);
- blkno = key_get_blkno(iter->ts, key);
+ while (tidstore_iter(iter, &key, &off_bitmap))
+ {
+ BlockNumber blkno = key_get_blkno(iter->ts, key);
- if (BlockNumberIsValid(result->blkno) && result->blkno != blkno)
+ if (BlockNumberIsValid(output->blkno) && output->blkno != blkno)
{
/*
- * We got a key-value pair for a different block. So return the
- * collected tids, and remember the key-value for the next iteration.
+ * We got tids for a different block. We return the collected
+ * tids so far, and remember the key-value for the next
+ * iteration.
*/
- iter->next_key = key;
- iter->next_val = val;
- return result;
+ iter->next_tidkey = key;
+ iter->next_off_bitmap = off_bitmap;
+ return output;
}
- /* Collect tids extracted from the key-value pair */
- tidstore_iter_extract_tids(iter, key, val);
+ /* Collect tids decoded from the key and offset bitmap */
+ iter_decode_key_off(iter, key, off_bitmap);
}
iter->finished = true;
- return result;
+ return output;
}
/*
@@ -544,22 +561,21 @@ tidstore_iterate_next(TidStoreIter *iter)
* or when existing an iteration.
*/
void
-tidstore_end_iterate(TidStoreIter *iter)
+TidStoreEndIterate(TidStoreIter *iter)
{
if (TidStoreIsShared(iter->ts))
shared_rt_end_iterate(iter->tree_iter.shared);
else
local_rt_end_iterate(iter->tree_iter.local);
- pfree(iter->result.offsets);
pfree(iter);
}
/* Return the number of tids we collected so far */
int64
-tidstore_num_tids(TidStore *ts)
+TidStoreNumTids(TidStore *ts)
{
- uint64 num_tids;
+ int64 num_tids;
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
@@ -575,16 +591,16 @@ tidstore_num_tids(TidStore *ts)
/* Return true if the current memory usage of TidStore exceeds the limit */
bool
-tidstore_is_full(TidStore *ts)
+TidStoreIsFull(TidStore *ts)
{
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
- return (tidstore_memory_usage(ts) > ts->control->max_bytes);
+ return (TidStoreMemoryUsage(ts) > ts->control->max_bytes);
}
/* Return the maximum memory TidStore can use */
size_t
-tidstore_max_memory(TidStore *ts)
+TidStoreMaxMemory(TidStore *ts)
{
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
@@ -593,7 +609,7 @@ tidstore_max_memory(TidStore *ts)
/* Return the memory usage of TidStore */
size_t
-tidstore_memory_usage(TidStore *ts)
+TidStoreMemoryUsage(TidStore *ts)
{
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
@@ -611,71 +627,75 @@ tidstore_memory_usage(TidStore *ts)
/*
* Get a handle that can be used by other processes to attach to this TidStore
*/
-tidstore_handle
-tidstore_get_handle(TidStore *ts)
+TidStoreHandle
+TidStoreGetHandle(TidStore *ts)
{
Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
return ts->control->handle;
}
-/* Extract tids from the given key-value pair */
+/*
+ * Decode the key and offset bitmap to tids and store them to the iteration
+ * result.
+ */
static void
-tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
+iter_decode_key_off(TidStoreIter *iter, tidkey key, offsetbm off_bitmap)
{
- TidStoreIterResult *result = (&iter->result);
+ TidStoreIterResult *output = (&iter->output);
- while (val)
+ while (off_bitmap)
{
- uint64 tid_i;
+ uint64 compressed_tid;
OffsetNumber off;
- tid_i = key << TIDSTORE_VALUE_NBITS;
- tid_i |= pg_rightmost_one_pos64(val);
+ compressed_tid = key << LOWER_OFFSET_NBITS;
+ compressed_tid |= pg_rightmost_one_pos64(off_bitmap);
- off = tid_i & ((UINT64CONST(1) << iter->ts->control->offset_nbits) - 1);
+ off = compressed_tid & ((UINT64CONST(1) << iter->ts->control->max_off_nbits) - 1);
- Assert(result->num_offsets < iter->ts->control->max_offset);
- result->offsets[result->num_offsets++] = off;
+ Assert(output->num_offsets < iter->ts->control->max_off);
+ output->offsets[output->num_offsets++] = off;
/* unset the rightmost bit */
- val &= ~pg_rightmost_one64(val);
+ off_bitmap &= ~pg_rightmost_one64(off_bitmap);
}
- result->blkno = key_get_blkno(iter->ts, key);
+ output->blkno = key_get_blkno(iter->ts, key);
}
/* Get block number from the given key */
static inline BlockNumber
-key_get_blkno(TidStore *ts, uint64 key)
+key_get_blkno(TidStore *ts, tidkey key)
{
- return (BlockNumber) (key >> ts->control->offset_key_nbits);
+ return (BlockNumber) (key >> ts->control->upper_off_nbits);
}
-/* Encode a tid to key and offset */
-static inline uint64
-tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit)
+/* Encode a tid to key and partial offset */
+static inline tidkey
+encode_tid(TidStore *ts, ItemPointer tid, offsetbm *off_bit)
{
- uint32 offset = ItemPointerGetOffsetNumber(tid);
+ OffsetNumber offset = ItemPointerGetOffsetNumber(tid);
BlockNumber block = ItemPointerGetBlockNumber(tid);
- return encode_key_off(ts, block, offset, off_bit);
+ return encode_blk_off(ts, block, offset, off_bit);
}
/* encode a block and offset to a key and partial offset */
-static inline uint64
-encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit)
+static inline tidkey
+encode_blk_off(TidStore *ts, BlockNumber block, OffsetNumber offset,
+ offsetbm *off_bit)
{
- uint64 key;
- uint64 tid_i;
+ tidkey key;
+ uint64 compressed_tid;
uint32 off_lower;
- off_lower = offset & TIDSTORE_OFFSET_MASK;
- Assert(off_lower < (sizeof(uint64) * BITS_PER_BYTE));
+ off_lower = offset & LOWER_OFFSET_MASK;
+ Assert(off_lower < (sizeof(offsetbm) * BITS_PER_BYTE));
*off_bit = UINT64CONST(1) << off_lower;
- tid_i = offset | ((uint64) block << ts->control->offset_nbits);
- key = tid_i >> TIDSTORE_VALUE_NBITS;
+ compressed_tid = offset | ((uint64) block << ts->control->max_off_nbits);
+ key = compressed_tid >> LOWER_OFFSET_NBITS;
return key;
}
diff --git a/src/include/access/tidstore.h b/src/include/access/tidstore.h
index a35a52124a..66f0fdd482 100644
--- a/src/include/access/tidstore.h
+++ b/src/include/access/tidstore.h
@@ -17,33 +17,34 @@
#include "storage/itemptr.h"
#include "utils/dsa.h"
-typedef dsa_pointer tidstore_handle;
+typedef dsa_pointer TidStoreHandle;
typedef struct TidStore TidStore;
typedef struct TidStoreIter TidStoreIter;
+/* Result struct for TidStoreIterateNext */
typedef struct TidStoreIterResult
{
BlockNumber blkno;
- OffsetNumber *offsets;
int num_offsets;
+ OffsetNumber offsets[FLEXIBLE_ARRAY_MEMBER];
} TidStoreIterResult;
-extern TidStore *tidstore_create(size_t max_bytes, int max_offset, dsa_area *dsa);
-extern TidStore *tidstore_attach(dsa_area *dsa, dsa_pointer handle);
-extern void tidstore_detach(TidStore *ts);
-extern void tidstore_destroy(TidStore *ts);
-extern void tidstore_reset(TidStore *ts);
-extern void tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
- int num_offsets);
-extern bool tidstore_lookup_tid(TidStore *ts, ItemPointer tid);
-extern TidStoreIter * tidstore_begin_iterate(TidStore *ts);
-extern TidStoreIterResult *tidstore_iterate_next(TidStoreIter *iter);
-extern void tidstore_end_iterate(TidStoreIter *iter);
-extern int64 tidstore_num_tids(TidStore *ts);
-extern bool tidstore_is_full(TidStore *ts);
-extern size_t tidstore_max_memory(TidStore *ts);
-extern size_t tidstore_memory_usage(TidStore *ts);
-extern tidstore_handle tidstore_get_handle(TidStore *ts);
+extern TidStore *TidStoreCreate(size_t max_bytes, int max_off, dsa_area *dsa);
+extern TidStore *TidStoreAttach(dsa_area *dsa, dsa_pointer handle);
+extern void TidStoreDetach(TidStore *ts);
+extern void TidStoreDestroy(TidStore *ts);
+extern void TidStoreReset(TidStore *ts);
+extern void TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets);
+extern bool TidStoreIsMember(TidStore *ts, ItemPointer tid);
+extern TidStoreIter * TidStoreBeginIterate(TidStore *ts);
+extern TidStoreIterResult *TidStoreIterateNext(TidStoreIter *iter);
+extern void TidStoreEndIterate(TidStoreIter *iter);
+extern int64 TidStoreNumTids(TidStore *ts);
+extern bool TidStoreIsFull(TidStore *ts);
+extern size_t TidStoreMaxMemory(TidStore *ts);
+extern size_t TidStoreMemoryUsage(TidStore *ts);
+extern TidStoreHandle TidStoreGetHandle(TidStore *ts);
#endif /* TIDSTORE_H */
diff --git a/src/test/modules/test_tidstore/test_tidstore.c b/src/test/modules/test_tidstore/test_tidstore.c
index 9a1217f833..8659e6780e 100644
--- a/src/test/modules/test_tidstore/test_tidstore.c
+++ b/src/test/modules/test_tidstore/test_tidstore.c
@@ -37,10 +37,10 @@ check_tid(TidStore *ts, BlockNumber blkno, OffsetNumber off, bool expect)
ItemPointerSet(&tid, blkno, off);
- found = tidstore_lookup_tid(ts, &tid);
+ found = TidStoreIsMember(ts, &tid);
if (found != expect)
- elog(ERROR, "lookup TID (%u, %u) returned %d, expected %d",
+ elog(ERROR, "TidStoreIsMember for TID (%u, %u) returned %d, expected %d",
blkno, off, found, expect);
}
@@ -69,9 +69,9 @@ test_basic(int max_offset)
LWLockRegisterTranche(tranche_id, "test_tidstore");
dsa = dsa_create(tranche_id);
- ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, dsa);
+ ts = TidStoreCreate(TEST_TIDSTORE_MAX_BYTES, max_offset, dsa);
#else
- ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
+ ts = TidStoreCreate(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
#endif
/* prepare the offset array */
@@ -83,7 +83,7 @@ test_basic(int max_offset)
/* add tids */
for (int i = 0; i < TEST_TIDSTORE_NUM_BLOCKS; i++)
- tidstore_add_tids(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
+ TidStoreSetBlockOffsets(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
/* lookup test */
for (OffsetNumber off = FirstOffsetNumber ; off < max_offset; off++)
@@ -105,30 +105,30 @@ test_basic(int max_offset)
}
/* test the number of tids */
- if (tidstore_num_tids(ts) != (TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS))
- elog(ERROR, "tidstore_num_tids returned " UINT64_FORMAT ", expected %d",
- tidstore_num_tids(ts),
+ if (TidStoreNumTids(ts) != (TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS))
+ elog(ERROR, "TidStoreNumTids returned " UINT64_FORMAT ", expected %d",
+ TidStoreNumTids(ts),
TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS);
/* iteration test */
- iter = tidstore_begin_iterate(ts);
+ iter = TidStoreBeginIterate(ts);
blk_idx = 0;
- while ((iter_result = tidstore_iterate_next(iter)) != NULL)
+ while ((iter_result = TidStoreIterateNext(iter)) != NULL)
{
/* check the returned block number */
if (blks_sorted[blk_idx] != iter_result->blkno)
- elog(ERROR, "tidstore_iterate_next returned block number %u, expected %u",
+ elog(ERROR, "TidStoreIterateNext returned block number %u, expected %u",
iter_result->blkno, blks_sorted[blk_idx]);
/* check the returned offset numbers */
if (TEST_TIDSTORE_NUM_OFFSETS != iter_result->num_offsets)
- elog(ERROR, "tidstore_iterate_next returned %u offsets, expected %u",
+ elog(ERROR, "TidStoreIterateNext %u offsets, expected %u",
iter_result->num_offsets, TEST_TIDSTORE_NUM_OFFSETS);
for (int i = 0; i < iter_result->num_offsets; i++)
{
if (offs[i] != iter_result->offsets[i])
- elog(ERROR, "tidstore_iterate_next returned offset number %u on block %u, expected %u",
+ elog(ERROR, "TidStoreIterateNext offset number %u on block %u, expected %u",
iter_result->offsets[i], iter_result->blkno, offs[i]);
}
@@ -136,15 +136,15 @@ test_basic(int max_offset)
}
if (blk_idx != TEST_TIDSTORE_NUM_BLOCKS)
- elog(ERROR, "tidstore_iterate_next returned %d blocks, expected %d",
+ elog(ERROR, "TidStoreIterateNext returned %d blocks, expected %d",
blk_idx, TEST_TIDSTORE_NUM_BLOCKS);
/* remove all tids */
- tidstore_reset(ts);
+ TidStoreReset(ts);
/* test the number of tids */
- if (tidstore_num_tids(ts) != 0)
- elog(ERROR, "tidstore_num_tids on empty store returned non-zero");
+ if (TidStoreNumTids(ts) != 0)
+ elog(ERROR, "TidStoreNumTids on empty store returned non-zero");
/* lookup test for empty store */
for (OffsetNumber off = FirstOffsetNumber ; off < MaxHeapTuplesPerPage;
@@ -156,7 +156,7 @@ test_basic(int max_offset)
check_tid(ts, MaxBlockNumber, off, false);
}
- tidstore_destroy(ts);
+ TidStoreDestroy(ts);
#ifdef TEST_SHARED_TIDSTORE
dsa_detach(dsa);
@@ -177,36 +177,37 @@ test_empty(void)
LWLockRegisterTranche(tranche_id, "test_tidstore");
dsa = dsa_create(tranche_id);
- ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, dsa);
+ ts = TidStoreCreate(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, dsa);
#else
- ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
+ ts = TidStoreCreate(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
#endif
elog(NOTICE, "testing empty tidstore");
ItemPointerSet(&tid, 0, FirstOffsetNumber);
- if (tidstore_lookup_tid(ts, &tid))
- elog(ERROR, "tidstore_lookup_tid for (0,1) on empty store returned true");
+ if (TidStoreIsMember(ts, &tid))
+ elog(ERROR, "TidStoreIsMember for TID (%u,%u) on empty store returned true",
+ 0, FirstOffsetNumber);
ItemPointerSet(&tid, MaxBlockNumber, MaxOffsetNumber);
- if (tidstore_lookup_tid(ts, &tid))
- elog(ERROR, "tidstore_lookup_tid for (%u,%u) on empty store returned true",
+ if (TidStoreIsMember(ts, &tid))
+ elog(ERROR, "TidStoreIsMember for TID (%u,%u) on empty store returned true",
MaxBlockNumber, MaxOffsetNumber);
- if (tidstore_num_tids(ts) != 0)
- elog(ERROR, "tidstore_num_entries on empty store returned non-zero");
+ if (TidStoreNumTids(ts) != 0)
+ elog(ERROR, "TidStoreNumTids on empty store returned non-zero");
- if (tidstore_is_full(ts))
- elog(ERROR, "tidstore_is_full on empty store returned true");
+ if (TidStoreIsFull(ts))
+ elog(ERROR, "TidStoreIsFull on empty store returned true");
- iter = tidstore_begin_iterate(ts);
+ iter = TidStoreBeginIterate(ts);
- if (tidstore_iterate_next(iter) != NULL)
- elog(ERROR, "tidstore_iterate_next on empty store returned TIDs");
+ if (TidStoreIterateNext(iter) != NULL)
+ elog(ERROR, "TidStoreIterateNext on empty store returned TIDs");
- tidstore_end_iterate(iter);
+ TidStoreEndIterate(iter);
- tidstore_destroy(ts);
+ TidStoreDestroy(ts);
#ifdef TEST_SHARED_TIDSTORE
dsa_detach(dsa);
@@ -221,6 +222,7 @@ test_tidstore(PG_FUNCTION_ARGS)
elog(NOTICE, "testing basic operations");
test_basic(MaxHeapTuplesPerPage);
test_basic(10);
+ test_basic(MaxHeapTuplesPerPage * 2);
PG_RETURN_VOID();
}
--
2.31.1
[application/octet-stream] v31-0003-Add-radixtree-template.patch (117.0K, ../../CAD21AoDBmD5q=eO+K=gyuVt53XvwpJ2dgxPwrtZ-eVOjVmtJjg@mail.gmail.com/9-v31-0003-Add-radixtree-template.patch)
download | inline diff:
From 014d2f9a13af4e9f57ff2f8e44fba61c71ecec66 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 14 Sep 2022 12:38:51 +0000
Subject: [PATCH v31 03/14] Add radixtree template
WIP: commit message based on template comments
---
src/backend/utils/mmgr/dsa.c | 12 +
src/include/lib/radixtree.h | 2516 +++++++++++++++++
src/include/lib/radixtree_delete_impl.h | 122 +
src/include/lib/radixtree_insert_impl.h | 328 +++
src/include/lib/radixtree_iter_impl.h | 153 +
src/include/lib/radixtree_search_impl.h | 138 +
src/include/utils/dsa.h | 1 +
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_radixtree/.gitignore | 4 +
src/test/modules/test_radixtree/Makefile | 23 +
src/test/modules/test_radixtree/README | 7 +
.../expected/test_radixtree.out | 36 +
src/test/modules/test_radixtree/meson.build | 35 +
.../test_radixtree/sql/test_radixtree.sql | 7 +
.../test_radixtree/test_radixtree--1.0.sql | 8 +
.../modules/test_radixtree/test_radixtree.c | 681 +++++
.../test_radixtree/test_radixtree.control | 4 +
src/tools/pginclude/cpluspluscheck | 6 +
src/tools/pginclude/headerscheck | 6 +
20 files changed, 4089 insertions(+)
create mode 100644 src/include/lib/radixtree.h
create mode 100644 src/include/lib/radixtree_delete_impl.h
create mode 100644 src/include/lib/radixtree_insert_impl.h
create mode 100644 src/include/lib/radixtree_iter_impl.h
create mode 100644 src/include/lib/radixtree_search_impl.h
create mode 100644 src/test/modules/test_radixtree/.gitignore
create mode 100644 src/test/modules/test_radixtree/Makefile
create mode 100644 src/test/modules/test_radixtree/README
create mode 100644 src/test/modules/test_radixtree/expected/test_radixtree.out
create mode 100644 src/test/modules/test_radixtree/meson.build
create mode 100644 src/test/modules/test_radixtree/sql/test_radixtree.sql
create mode 100644 src/test/modules/test_radixtree/test_radixtree--1.0.sql
create mode 100644 src/test/modules/test_radixtree/test_radixtree.c
create mode 100644 src/test/modules/test_radixtree/test_radixtree.control
diff --git a/src/backend/utils/mmgr/dsa.c b/src/backend/utils/mmgr/dsa.c
index f5a62061a3..80555aefff 100644
--- a/src/backend/utils/mmgr/dsa.c
+++ b/src/backend/utils/mmgr/dsa.c
@@ -1024,6 +1024,18 @@ dsa_set_size_limit(dsa_area *area, size_t limit)
LWLockRelease(DSA_AREA_LOCK(area));
}
+size_t
+dsa_get_total_size(dsa_area *area)
+{
+ size_t size;
+
+ LWLockAcquire(DSA_AREA_LOCK(area), LW_SHARED);
+ size = area->control->total_segment_size;
+ LWLockRelease(DSA_AREA_LOCK(area));
+
+ return size;
+}
+
/*
* Aggressively free all spare memory in the hope of returning DSM segments to
* the operating system.
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
new file mode 100644
index 0000000000..e546bd705c
--- /dev/null
+++ b/src/include/lib/radixtree.h
@@ -0,0 +1,2516 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree.h
+ * Template for adaptive radix tree.
+ *
+ * This module employs the idea from the paper "The Adaptive Radix Tree: ARTful
+ * Indexing for Main-Memory Databases" by Viktor Leis, Alfons Kemper, and Thomas
+ * Neumann, 2013. The radix tree uses adaptive node sizes, a small number of node
+ * types, each with a different numbers of elements. Depending on the number of
+ * children, the appropriate node type is used.
+ *
+ * WIP: notes about traditional radix tree trading off span vs height...
+ *
+ * There are two kinds of nodes, inner nodes and leaves. Inner nodes
+ * map partial keys to child pointers.
+ *
+ * The ART paper mentions three ways to implement leaves:
+ *
+ * "- Single-value leaves: The values are stored using an addi-
+ * tional leaf node type which stores one value.
+ * - Multi-value leaves: The values are stored in one of four
+ * different leaf node types, which mirror the structure of
+ * inner nodes, but contain values instead of pointers.
+ * - Combined pointer/value slots: If values fit into point-
+ * ers, no separate node types are necessary. Instead, each
+ * pointer storage location in an inner node can either
+ * store a pointer or a value."
+ *
+ * We chose "multi-value leaves" to avoid the additional pointer traversal
+ * required by "single-value leaves"
+ *
+ * For simplicity, the key is assumed to be 64-bit unsigned integer. The
+ * tree doesn't need to contain paths where the highest bytes of all keys
+ * are zero. That way, the tree's height adapts to the distribution of keys.
+ *
+ * TODO: In the future it might be worthwhile to offer configurability of
+ * leaf implementation for different use cases. Single-values leaves would
+ * give more flexibility in key type, including variable-length keys.
+ *
+ * There are some optimizations not yet implemented, particularly path
+ * compression and lazy path expansion.
+ *
+ * To handle concurrency, we use a single reader-writer lock for the radix
+ * tree. The radix tree is exclusively locked during write operations such
+ * as RT_SET() and RT_DELETE(), and shared locked during read operations
+ * such as RT_SEARCH(). An iteration also holds the shared lock on the radix
+ * tree until it is completed.
+ *
+ * TODO: The current locking mechanism is not optimized for high concurrency
+ * with mixed read-write workloads. In the future it might be worthwhile
+ * to replace it with the Optimistic Lock Coupling or ROWEX mentioned in
+ * the paper "The ART of Practical Synchronization" by the same authors as
+ * the ART paper, 2016.
+ *
+ * WIP: the radix tree nodes don't shrink.
+ *
+ * To generate a radix tree and associated functions for a use case several
+ * macros have to be #define'ed before this file is included. Including
+ * the file #undef's all those, so a new radix tree can be generated
+ * afterwards.
+ * The relevant parameters are:
+ * - RT_PREFIX - prefix for all symbol names generated. A prefix of 'foo'
+ * will result in radix tree type 'foo_radix_tree' and functions like
+ * 'foo_create'/'foo_free' and so forth.
+ * - RT_DECLARE - if defined function prototypes and type declarations are
+ * generated
+ * - RT_DEFINE - if defined function definitions are generated
+ * - RT_SCOPE - in which scope (e.g. extern, static inline) do function
+ * declarations reside
+ * - RT_VALUE_TYPE - the type of the value.
+ *
+ * Optional parameters:
+ * - RT_SHMEM - if defined, the radix tree is created in the DSA area
+ * so that multiple processes can access it simultaneously.
+ * - RT_DEBUG - if defined add stats tracking and debugging functions
+ *
+ * Interface
+ * ---------
+ *
+ * RT_CREATE - Create a new, empty radix tree
+ * RT_FREE - Free the radix tree
+ * RT_SEARCH - Search a key-value pair
+ * RT_SET - Set a key-value pair
+ * RT_BEGIN_ITERATE - Begin iterating through all key-value pairs
+ * RT_ITERATE_NEXT - Return next key-value pair, if any
+ * RT_END_ITER - End iteration
+ * RT_MEMORY_USAGE - Get the memory usage
+ *
+ * Interface for Shared Memory
+ * ---------
+ *
+ * RT_ATTACH - Attach to the radix tree
+ * RT_DETACH - Detach from the radix tree
+ * RT_GET_HANDLE - Return the handle of the radix tree
+ *
+ * Optional Interface
+ * ---------
+ *
+ * RT_DELETE - Delete a key-value pair. Declared/define if RT_USE_DELETE is defined
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/include/lib/radixtree.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/stringinfo.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "port/pg_bitutils.h"
+#include "port/simd.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/* helpers */
+#define RT_MAKE_PREFIX(a) CppConcat(a,_)
+#define RT_MAKE_NAME(name) RT_MAKE_NAME_(RT_MAKE_PREFIX(RT_PREFIX),name)
+#define RT_MAKE_NAME_(a,b) CppConcat(a,b)
+
+/* function declarations */
+#define RT_CREATE RT_MAKE_NAME(create)
+#define RT_FREE RT_MAKE_NAME(free)
+#define RT_SEARCH RT_MAKE_NAME(search)
+#ifdef RT_SHMEM
+#define RT_ATTACH RT_MAKE_NAME(attach)
+#define RT_DETACH RT_MAKE_NAME(detach)
+#define RT_GET_HANDLE RT_MAKE_NAME(get_handle)
+#endif
+#define RT_SET RT_MAKE_NAME(set)
+#define RT_BEGIN_ITERATE RT_MAKE_NAME(begin_iterate)
+#define RT_ITERATE_NEXT RT_MAKE_NAME(iterate_next)
+#define RT_END_ITERATE RT_MAKE_NAME(end_iterate)
+#ifdef RT_USE_DELETE
+#define RT_DELETE RT_MAKE_NAME(delete)
+#endif
+#define RT_MEMORY_USAGE RT_MAKE_NAME(memory_usage)
+#ifdef RT_DEBUG
+#define RT_DUMP RT_MAKE_NAME(dump)
+#define RT_DUMP_NODE RT_MAKE_NAME(dump_node)
+#define RT_DUMP_SEARCH RT_MAKE_NAME(dump_search)
+#define RT_STATS RT_MAKE_NAME(stats)
+#endif
+
+/* internal helper functions (no externally visible prototypes) */
+#define RT_NEW_ROOT RT_MAKE_NAME(new_root)
+#define RT_ALLOC_NODE RT_MAKE_NAME(alloc_node)
+#define RT_INIT_NODE RT_MAKE_NAME(init_node)
+#define RT_FREE_NODE RT_MAKE_NAME(free_node)
+#define RT_FREE_RECURSE RT_MAKE_NAME(free_recurse)
+#define RT_EXTEND RT_MAKE_NAME(extend)
+#define RT_SET_EXTEND RT_MAKE_NAME(set_extend)
+#define RT_SWITCH_NODE_KIND RT_MAKE_NAME(grow_node_kind)
+#define RT_COPY_NODE RT_MAKE_NAME(copy_node)
+#define RT_REPLACE_NODE RT_MAKE_NAME(replace_node)
+#define RT_PTR_GET_LOCAL RT_MAKE_NAME(ptr_get_local)
+#define RT_PTR_ALLOC_IS_VALID RT_MAKE_NAME(ptr_stored_is_valid)
+#define RT_NODE_3_SEARCH_EQ RT_MAKE_NAME(node_3_search_eq)
+#define RT_NODE_32_SEARCH_EQ RT_MAKE_NAME(node_32_search_eq)
+#define RT_NODE_3_GET_INSERTPOS RT_MAKE_NAME(node_3_get_insertpos)
+#define RT_NODE_32_GET_INSERTPOS RT_MAKE_NAME(node_32_get_insertpos)
+#define RT_CHUNK_CHILDREN_ARRAY_SHIFT RT_MAKE_NAME(chunk_children_array_shift)
+#define RT_CHUNK_VALUES_ARRAY_SHIFT RT_MAKE_NAME(chunk_values_array_shift)
+#define RT_CHUNK_CHILDREN_ARRAY_DELETE RT_MAKE_NAME(chunk_children_array_delete)
+#define RT_CHUNK_VALUES_ARRAY_DELETE RT_MAKE_NAME(chunk_values_array_delete)
+#define RT_CHUNK_CHILDREN_ARRAY_COPY RT_MAKE_NAME(chunk_children_array_copy)
+#define RT_CHUNK_VALUES_ARRAY_COPY RT_MAKE_NAME(chunk_values_array_copy)
+#define RT_NODE_125_IS_CHUNK_USED RT_MAKE_NAME(node_125_is_chunk_used)
+#define RT_NODE_INNER_125_GET_CHILD RT_MAKE_NAME(node_inner_125_get_child)
+#define RT_NODE_LEAF_125_GET_VALUE RT_MAKE_NAME(node_leaf_125_get_value)
+#define RT_NODE_INNER_256_IS_CHUNK_USED RT_MAKE_NAME(node_inner_256_is_chunk_used)
+#define RT_NODE_LEAF_256_IS_CHUNK_USED RT_MAKE_NAME(node_leaf_256_is_chunk_used)
+#define RT_NODE_INNER_256_GET_CHILD RT_MAKE_NAME(node_inner_256_get_child)
+#define RT_NODE_LEAF_256_GET_VALUE RT_MAKE_NAME(node_leaf_256_get_value)
+#define RT_NODE_INNER_256_SET RT_MAKE_NAME(node_inner_256_set)
+#define RT_NODE_LEAF_256_SET RT_MAKE_NAME(node_leaf_256_set)
+#define RT_NODE_INNER_256_DELETE RT_MAKE_NAME(node_inner_256_delete)
+#define RT_NODE_LEAF_256_DELETE RT_MAKE_NAME(node_leaf_256_delete)
+#define RT_KEY_GET_SHIFT RT_MAKE_NAME(key_get_shift)
+#define RT_SHIFT_GET_MAX_VAL RT_MAKE_NAME(shift_get_max_val)
+#define RT_NODE_SEARCH_INNER RT_MAKE_NAME(node_search_inner)
+#define RT_NODE_SEARCH_LEAF RT_MAKE_NAME(node_search_leaf)
+#define RT_NODE_UPDATE_INNER RT_MAKE_NAME(node_update_inner)
+#define RT_NODE_DELETE_INNER RT_MAKE_NAME(node_delete_inner)
+#define RT_NODE_DELETE_LEAF RT_MAKE_NAME(node_delete_leaf)
+#define RT_NODE_INSERT_INNER RT_MAKE_NAME(node_insert_inner)
+#define RT_NODE_INSERT_LEAF RT_MAKE_NAME(node_insert_leaf)
+#define RT_NODE_INNER_ITERATE_NEXT RT_MAKE_NAME(node_inner_iterate_next)
+#define RT_NODE_LEAF_ITERATE_NEXT RT_MAKE_NAME(node_leaf_iterate_next)
+#define RT_UPDATE_ITER_STACK RT_MAKE_NAME(update_iter_stack)
+#define RT_ITER_UPDATE_KEY RT_MAKE_NAME(iter_update_key)
+#define RT_VERIFY_NODE RT_MAKE_NAME(verify_node)
+
+/* type declarations */
+#define RT_RADIX_TREE RT_MAKE_NAME(radix_tree)
+#define RT_RADIX_TREE_CONTROL RT_MAKE_NAME(radix_tree_control)
+#define RT_ITER RT_MAKE_NAME(iter)
+#ifdef RT_SHMEM
+#define RT_HANDLE RT_MAKE_NAME(handle)
+#endif
+#define RT_NODE RT_MAKE_NAME(node)
+#define RT_NODE_ITER RT_MAKE_NAME(node_iter)
+#define RT_NODE_BASE_3 RT_MAKE_NAME(node_base_3)
+#define RT_NODE_BASE_32 RT_MAKE_NAME(node_base_32)
+#define RT_NODE_BASE_125 RT_MAKE_NAME(node_base_125)
+#define RT_NODE_BASE_256 RT_MAKE_NAME(node_base_256)
+#define RT_NODE_INNER_3 RT_MAKE_NAME(node_inner_3)
+#define RT_NODE_INNER_32 RT_MAKE_NAME(node_inner_32)
+#define RT_NODE_INNER_125 RT_MAKE_NAME(node_inner_125)
+#define RT_NODE_INNER_256 RT_MAKE_NAME(node_inner_256)
+#define RT_NODE_LEAF_3 RT_MAKE_NAME(node_leaf_3)
+#define RT_NODE_LEAF_32 RT_MAKE_NAME(node_leaf_32)
+#define RT_NODE_LEAF_125 RT_MAKE_NAME(node_leaf_125)
+#define RT_NODE_LEAF_256 RT_MAKE_NAME(node_leaf_256)
+#define RT_SIZE_CLASS RT_MAKE_NAME(size_class)
+#define RT_SIZE_CLASS_ELEM RT_MAKE_NAME(size_class_elem)
+#define RT_SIZE_CLASS_INFO RT_MAKE_NAME(size_class_info)
+#define RT_CLASS_3 RT_MAKE_NAME(class_3)
+#define RT_CLASS_32_MIN RT_MAKE_NAME(class_32_min)
+#define RT_CLASS_32_MAX RT_MAKE_NAME(class_32_max)
+#define RT_CLASS_125 RT_MAKE_NAME(class_125)
+#define RT_CLASS_256 RT_MAKE_NAME(class_256)
+
+/* generate forward declarations necessary to use the radix tree */
+#ifdef RT_DECLARE
+
+typedef struct RT_RADIX_TREE RT_RADIX_TREE;
+typedef struct RT_ITER RT_ITER;
+
+#ifdef RT_SHMEM
+typedef dsa_pointer RT_HANDLE;
+#endif
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id);
+RT_SCOPE RT_RADIX_TREE * RT_ATTACH(dsa_area *dsa, dsa_pointer dp);
+RT_SCOPE void RT_DETACH(RT_RADIX_TREE *tree);
+RT_SCOPE RT_HANDLE RT_GET_HANDLE(RT_RADIX_TREE *tree);
+#else
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx);
+#endif
+RT_SCOPE void RT_FREE(RT_RADIX_TREE *tree);
+
+RT_SCOPE bool RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p);
+RT_SCOPE bool RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p);
+#ifdef RT_USE_DELETE
+RT_SCOPE bool RT_DELETE(RT_RADIX_TREE *tree, uint64 key);
+#endif
+
+RT_SCOPE RT_ITER * RT_BEGIN_ITERATE(RT_RADIX_TREE *tree);
+RT_SCOPE bool RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p);
+RT_SCOPE void RT_END_ITERATE(RT_ITER *iter);
+
+RT_SCOPE uint64 RT_MEMORY_USAGE(RT_RADIX_TREE *tree);
+
+#ifdef RT_DEBUG
+RT_SCOPE void RT_DUMP(RT_RADIX_TREE *tree);
+RT_SCOPE void RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key);
+RT_SCOPE void RT_STATS(RT_RADIX_TREE *tree);
+#endif
+
+#endif /* RT_DECLARE */
+
+
+/* generate implementation of the radix tree */
+#ifdef RT_DEFINE
+
+/* The number of bits encoded in one tree level */
+#define RT_NODE_SPAN BITS_PER_BYTE
+
+/* The number of maximum slots in the node */
+#define RT_NODE_MAX_SLOTS (1 << RT_NODE_SPAN)
+
+/* Mask for extracting a chunk from the key */
+#define RT_CHUNK_MASK ((1 << RT_NODE_SPAN) - 1)
+
+/* Maximum shift the radix tree uses */
+#define RT_MAX_SHIFT RT_KEY_GET_SHIFT(UINT64_MAX)
+
+/* Tree level the radix tree uses */
+#define RT_MAX_LEVEL ((sizeof(uint64) * BITS_PER_BYTE) / RT_NODE_SPAN)
+
+/*
+ * Number of bits necessary for isset array in the slot-index node.
+ * Since bitmapword can be 64 bits, the only values that make sense
+ * here are 64 and 128.
+ */
+#define RT_SLOT_IDX_LIMIT (RT_NODE_MAX_SLOTS / 2)
+
+/* Invalid index used in node-125 */
+#define RT_INVALID_SLOT_IDX 0xFF
+
+/* Get a chunk from the key */
+#define RT_GET_KEY_CHUNK(key, shift) ((uint8) (((key) >> (shift)) & RT_CHUNK_MASK))
+
+/* For accessing bitmaps */
+#define RT_BM_IDX(x) ((x) / BITS_PER_BITMAPWORD)
+#define RT_BM_BIT(x) ((x) % BITS_PER_BITMAPWORD)
+
+/*
+ * Node kinds
+ *
+ * The different node kinds are what make the tree "adaptive".
+ *
+ * Each node kind is associated with a different datatype and different
+ * search/set/delete/iterate algorithms adapted for its size. The largest
+ * kind, node256 is basically the same as a traditional radix tree,
+ * and would be most wasteful of memory when sparsely populated. The
+ * smaller nodes expend some additional CPU time to enable a smaller
+ * memory footprint.
+ *
+ * XXX There are 4 node kinds, and this should never be increased,
+ * for several reasons:
+ * 1. With 5 or more kinds, gcc tends to use a jump table for switch
+ * statements.
+ * 2. The 4 kinds can be represented with 2 bits, so we have the option
+ * in the future to tag the node pointer with the kind, even on
+ * platforms with 32-bit pointers. This might speed up node traversal
+ * in trees with highly random node kinds.
+ * 3. We can have multiple size classes per node kind.
+ */
+#define RT_NODE_KIND_3 0x00
+#define RT_NODE_KIND_32 0x01
+#define RT_NODE_KIND_125 0x02
+#define RT_NODE_KIND_256 0x03
+#define RT_NODE_KIND_COUNT 4
+
+/*
+ * Calculate the slab blocksize so that we can allocate at least 32 chunks
+ * from the block.
+ */
+#define RT_SLAB_BLOCK_SIZE(size) \
+ Max((SLAB_DEFAULT_BLOCK_SIZE / (size)) * (size), (size) * 32)
+
+/* Common type for all nodes types */
+typedef struct RT_NODE
+{
+ /*
+ * Number of children. We use uint16 to be able to indicate 256 children
+ * at the fanout of 8.
+ */
+ uint16 count;
+
+ /*
+ * Max capacity for the current size class. Storing this in the
+ * node enables multiple size classes per node kind.
+ * Technically, kinds with a single size class don't need this, so we could
+ * keep this in the individual base types, but the code is simpler this way.
+ * Note: node256 is unique in that it cannot possibly have more than a
+ * single size class, so for that kind we store zero, and uint8 is
+ * sufficient for other kinds.
+ */
+ uint8 fanout;
+
+ /*
+ * Shift indicates which part of the key space is represented by this
+ * node. That is, the key is shifted by 'shift' and the lowest
+ * RT_NODE_SPAN bits are then represented in chunk.
+ */
+ uint8 shift;
+
+ /* Node kind, one per search/set algorithm */
+ uint8 kind;
+} RT_NODE;
+
+
+#define RT_PTR_LOCAL RT_NODE *
+
+#ifdef RT_SHMEM
+#define RT_PTR_ALLOC dsa_pointer
+#else
+#define RT_PTR_ALLOC RT_PTR_LOCAL
+#endif
+
+
+#ifdef RT_SHMEM
+#define RT_INVALID_PTR_ALLOC InvalidDsaPointer
+#else
+#define RT_INVALID_PTR_ALLOC NULL
+#endif
+
+#ifdef RT_SHMEM
+#define RT_LOCK_EXCLUSIVE(tree) LWLockAcquire(&tree->ctl->lock, LW_EXCLUSIVE)
+#define RT_LOCK_SHARED(tree) LWLockAcquire(&tree->ctl->lock, LW_SHARED)
+#define RT_UNLOCK(tree) LWLockRelease(&tree->ctl->lock);
+#else
+#define RT_LOCK_EXCLUSIVE(tree) ((void) 0)
+#define RT_LOCK_SHARED(tree) ((void) 0)
+#define RT_UNLOCK(tree) ((void) 0)
+#endif
+
+/*
+ * Inner nodes and leaf nodes have analogous structure. To distinguish
+ * them at runtime, we take advantage of the fact that the key chunk
+ * is accessed by shifting: Inner tree nodes (shift > 0), store the
+ * pointer to its child node in the slot. In leaf nodes (shift == 0),
+ * the slot contains the value corresponding to the key.
+ */
+#define RT_NODE_IS_LEAF(n) (((RT_PTR_LOCAL) (n))->shift == 0)
+
+#define RT_NODE_MUST_GROW(node) \
+ ((node)->base.n.count == (node)->base.n.fanout)
+
+/*
+ * Base type of each node kinds for leaf and inner nodes.
+ * The base types must be a be able to accommodate the largest size
+ * class for variable-sized node kinds.
+ */
+typedef struct RT_NODE_BASE_3
+{
+ RT_NODE n;
+
+ /* 3 children, for key chunks */
+ uint8 chunks[3];
+} RT_NODE_BASE_3;
+
+typedef struct RT_NODE_BASE_32
+{
+ RT_NODE n;
+
+ /* 32 children, for key chunks */
+ uint8 chunks[32];
+} RT_NODE_BASE_32;
+
+/*
+ * node-125 uses slot_idx array, an array of RT_NODE_MAX_SLOTS length
+ * to store indexes into a second array that contains the values (or
+ * child pointers).
+ */
+typedef struct RT_NODE_BASE_125
+{
+ RT_NODE n;
+
+ /* The index of slots for each fanout */
+ uint8 slot_idxs[RT_NODE_MAX_SLOTS];
+
+ /* bitmap to track which slots are in use */
+ bitmapword isset[RT_BM_IDX(RT_SLOT_IDX_LIMIT)];
+} RT_NODE_BASE_125;
+
+typedef struct RT_NODE_BASE_256
+{
+ RT_NODE n;
+} RT_NODE_BASE_256;
+
+/*
+ * Inner and leaf nodes.
+ *
+ * These are separate because the value type might be different than
+ * something fitting into a pointer-width type.
+ */
+typedef struct RT_NODE_INNER_3
+{
+ RT_NODE_BASE_3 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_3;
+
+typedef struct RT_NODE_LEAF_3
+{
+ RT_NODE_BASE_3 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_3;
+
+typedef struct RT_NODE_INNER_32
+{
+ RT_NODE_BASE_32 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_32;
+
+typedef struct RT_NODE_LEAF_32
+{
+ RT_NODE_BASE_32 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_32;
+
+typedef struct RT_NODE_INNER_125
+{
+ RT_NODE_BASE_125 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_125;
+
+typedef struct RT_NODE_LEAF_125
+{
+ RT_NODE_BASE_125 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_125;
+
+/*
+ * node-256 is the largest node type. This node has an array
+ * for directly storing values (or child pointers in inner nodes).
+ * Unlike other node kinds, it's array size is by definition
+ * fixed.
+ */
+typedef struct RT_NODE_INNER_256
+{
+ RT_NODE_BASE_256 base;
+
+ /* Slots for 256 children */
+ RT_PTR_ALLOC children[RT_NODE_MAX_SLOTS];
+} RT_NODE_INNER_256;
+
+typedef struct RT_NODE_LEAF_256
+{
+ RT_NODE_BASE_256 base;
+
+ /*
+ * Unlike with inner256, zero is a valid value here, so we use a
+ * bitmap to track which slots are in use.
+ */
+ bitmapword isset[RT_BM_IDX(RT_NODE_MAX_SLOTS)];
+
+ /* Slots for 256 values */
+ RT_VALUE_TYPE values[RT_NODE_MAX_SLOTS];
+} RT_NODE_LEAF_256;
+
+/*
+ * Node size classes
+ *
+ * Nodes of different kinds necessarily belong to different size classes.
+ * The main innovation in our implementation compared to the ART paper
+ * is decoupling the notion of size class from kind.
+ *
+ * The size classes within a given node kind have the same underlying
+ * type, but a variable number of children/values. This is possible
+ * because the base type contains small fixed data structures that
+ * work the same way regardless of how full the node is. We store the
+ * node's allocated capacity in the "fanout" member of RT_NODE, to allow
+ * runtime introspection.
+ *
+ * Growing from one node kind to another requires special code for each
+ * case, but growing from one size class to another within the same kind
+ * is basically just allocate + memcpy.
+ *
+ * The size classes have been chosen so that inner nodes on platforms
+ * with 64-bit pointers (and leaf nodes when using a 64-bit key) are
+ * equal to or slightly smaller than some DSA size class.
+ */
+typedef enum RT_SIZE_CLASS
+{
+ RT_CLASS_3 = 0,
+ RT_CLASS_32_MIN,
+ RT_CLASS_32_MAX,
+ RT_CLASS_125,
+ RT_CLASS_256
+} RT_SIZE_CLASS;
+
+/* Information for each size class */
+typedef struct RT_SIZE_CLASS_ELEM
+{
+ const char *name;
+ int fanout;
+
+ /* slab chunk size */
+ Size inner_size;
+ Size leaf_size;
+} RT_SIZE_CLASS_ELEM;
+
+static const RT_SIZE_CLASS_ELEM RT_SIZE_CLASS_INFO[] = {
+ [RT_CLASS_3] = {
+ .name = "radix tree node 3",
+ .fanout = 3,
+ .inner_size = sizeof(RT_NODE_INNER_3) + 3 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_3) + 3 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_32_MIN] = {
+ .name = "radix tree node 15",
+ .fanout = 15,
+ .inner_size = sizeof(RT_NODE_INNER_32) + 15 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_32) + 15 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_32_MAX] = {
+ .name = "radix tree node 32",
+ .fanout = 32,
+ .inner_size = sizeof(RT_NODE_INNER_32) + 32 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_32) + 32 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_125] = {
+ .name = "radix tree node 125",
+ .fanout = 125,
+ .inner_size = sizeof(RT_NODE_INNER_125) + 125 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_125) + 125 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_256] = {
+ .name = "radix tree node 256",
+ .fanout = 256,
+ .inner_size = sizeof(RT_NODE_INNER_256),
+ .leaf_size = sizeof(RT_NODE_LEAF_256),
+ },
+};
+
+#define RT_SIZE_CLASS_COUNT lengthof(RT_SIZE_CLASS_INFO)
+
+#ifdef RT_SHMEM
+/* A magic value used to identify our radix tree */
+#define RT_RADIX_TREE_MAGIC 0x54A48167
+#endif
+
+/* Contains the actual tree and ancillary info */
+// WIP: this name is a bit strange
+typedef struct RT_RADIX_TREE_CONTROL
+{
+#ifdef RT_SHMEM
+ RT_HANDLE handle;
+ uint32 magic;
+ LWLock lock;
+#endif
+
+ RT_PTR_ALLOC root;
+ uint64 max_val;
+ uint64 num_keys;
+
+ /* statistics */
+#ifdef RT_DEBUG
+ int32 cnt[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE_CONTROL;
+
+/* Entry point for allocating and accessing the tree */
+typedef struct RT_RADIX_TREE
+{
+ MemoryContext context;
+
+ /* pointing to either local memory or DSA */
+ RT_RADIX_TREE_CONTROL *ctl;
+
+#ifdef RT_SHMEM
+ dsa_area *dsa;
+#else
+ MemoryContextData *inner_slabs[RT_SIZE_CLASS_COUNT];
+ MemoryContextData *leaf_slabs[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE;
+
+/*
+ * Iteration support.
+ *
+ * Iterating the radix tree returns each pair of key and value in the ascending
+ * order of the key. To support this, the we iterate nodes of each level.
+ *
+ * RT_NODE_ITER struct is used to track the iteration within a node.
+ *
+ * RT_ITER is the struct for iteration of the radix tree, and uses RT_NODE_ITER
+ * in order to track the iteration of each level. During iteration, we also
+ * construct the key whenever updating the node iteration information, e.g., when
+ * advancing the current index within the node or when moving to the next node
+ * at the same level.
+ *
+ * XXX: Currently we allow only one process to do iteration. Therefore, rt_node_iter
+ * has the local pointers to nodes, rather than RT_PTR_ALLOC.
+ * We need either a safeguard to disallow other processes to begin the iteration
+ * while one process is doing or to allow multiple processes to do the iteration.
+ */
+typedef struct RT_NODE_ITER
+{
+ RT_PTR_LOCAL node; /* current node being iterated */
+ int current_idx; /* current position. -1 for initial value */
+} RT_NODE_ITER;
+
+typedef struct RT_ITER
+{
+ RT_RADIX_TREE *tree;
+
+ /* Track the iteration on nodes of each level */
+ RT_NODE_ITER stack[RT_MAX_LEVEL];
+ int stack_len;
+
+ /* The key is constructed during iteration */
+ uint64 key;
+} RT_ITER;
+
+
+static void RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_PTR_ALLOC child);
+static bool RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_VALUE_TYPE *value_p);
+
+/* verification (available only with assertion) */
+static void RT_VERIFY_NODE(RT_PTR_LOCAL node);
+
+/* Get the local address of an allocated node */
+static inline RT_PTR_LOCAL
+RT_PTR_GET_LOCAL(RT_RADIX_TREE *tree, RT_PTR_ALLOC node)
+{
+#ifdef RT_SHMEM
+ return dsa_get_address(tree->dsa, (dsa_pointer) node);
+#else
+ return node;
+#endif
+}
+
+static inline bool
+RT_PTR_ALLOC_IS_VALID(RT_PTR_ALLOC ptr)
+{
+#ifdef RT_SHMEM
+ return DsaPointerIsValid(ptr);
+#else
+ return PointerIsValid(ptr);
+#endif
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_3_SEARCH_EQ(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+ int idx = -1;
+
+ for (int i = 0; i < node->n.count; i++)
+ {
+ if (node->chunks[i] == chunk)
+ {
+ idx = i;
+ break;
+ }
+ }
+
+ return idx;
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_3_GET_INSERTPOS(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+ int idx;
+
+ for (idx = 0; idx < node->n.count; idx++)
+ {
+ if (node->chunks[idx] >= chunk)
+ break;
+ }
+
+ return idx;
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_32_SEARCH_EQ(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+ int count = node->n.count;
+#ifndef USE_NO_SIMD
+ Vector8 spread_chunk;
+ Vector8 haystack1;
+ Vector8 haystack2;
+ Vector8 cmp1;
+ Vector8 cmp2;
+ uint32 bitfield;
+ int index_simd = -1;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+ int index = -1;
+
+ for (int i = 0; i < count; i++)
+ {
+ if (node->chunks[i] == chunk)
+ {
+ index = i;
+ break;
+ }
+ }
+#endif
+
+#ifndef USE_NO_SIMD
+ /* replicate the search key */
+ spread_chunk = vector8_broadcast(chunk);
+
+ /* compare to all 32 keys stored in the node */
+ vector8_load(&haystack1, &node->chunks[0]);
+ vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+ cmp1 = vector8_eq(spread_chunk, haystack1);
+ cmp2 = vector8_eq(spread_chunk, haystack2);
+
+ /* convert comparison to a bitfield */
+ bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+
+ /* mask off invalid entries */
+ bitfield &= ((UINT64CONST(1) << count) - 1);
+
+ /* convert bitfield to index by counting trailing zeros */
+ if (bitfield)
+ index_simd = pg_rightmost_one_pos32(bitfield);
+
+ Assert(index_simd == index);
+ return index_simd;
+#else
+ return index;
+#endif
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_32_GET_INSERTPOS(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+ int count = node->n.count;
+#ifndef USE_NO_SIMD
+ Vector8 spread_chunk;
+ Vector8 haystack1;
+ Vector8 haystack2;
+ Vector8 cmp1;
+ Vector8 cmp2;
+ Vector8 min1;
+ Vector8 min2;
+ uint32 bitfield;
+ int index_simd;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+ int index;
+
+ for (index = 0; index < count; index++)
+ {
+ /*
+ * This is coded with '>=' to match what we can do with SIMD,
+ * with an assert to keep us honest.
+ */
+ if (node->chunks[index] >= chunk)
+ {
+ Assert(node->chunks[index] != chunk);
+ break;
+ }
+ }
+#endif
+
+#ifndef USE_NO_SIMD
+ /*
+ * This is a bit more complicated than RT_NODE_32_SEARCH_EQ(), because
+ * no unsigned uint8 comparison instruction exists, at least for SSE2. So
+ * we need to play some trickery using vector8_min() to effectively get
+ * >=. There'll never be any equal elements in current uses, but that's
+ * what we get here...
+ */
+ spread_chunk = vector8_broadcast(chunk);
+ vector8_load(&haystack1, &node->chunks[0]);
+ vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+ min1 = vector8_min(spread_chunk, haystack1);
+ min2 = vector8_min(spread_chunk, haystack2);
+ cmp1 = vector8_eq(spread_chunk, min1);
+ cmp2 = vector8_eq(spread_chunk, min2);
+ bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+ bitfield &= ((UINT64CONST(1) << count) - 1);
+
+ if (bitfield)
+ index_simd = pg_rightmost_one_pos32(bitfield);
+ else
+ index_simd = count;
+
+ Assert(index_simd == index);
+ return index_simd;
+#else
+ return index;
+#endif
+}
+
+
+/*
+ * Functions to manipulate both chunks array and children/values array.
+ * These are used for node-3 and node-32.
+ */
+
+/* Shift the elements right at 'idx' by one */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_SHIFT(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+ memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+ memmove(&(children[idx + 1]), &(children[idx]), sizeof(RT_PTR_ALLOC) * (count - idx));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_SHIFT(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+ memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+ memmove(&(values[idx + 1]), &(values[idx]), sizeof(RT_VALUE_TYPE) * (count - idx));
+}
+
+/* Delete the element at 'idx' */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_DELETE(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+ memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+ memmove(&(children[idx]), &(children[idx + 1]), sizeof(RT_PTR_ALLOC) * (count - idx - 1));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_DELETE(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+ memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+ memmove(&(values[idx]), &(values[idx + 1]), sizeof(RT_VALUE_TYPE) * (count - idx - 1));
+}
+
+/* Copy both chunks and children/values arrays */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_COPY(uint8 *src_chunks, RT_PTR_ALLOC *src_children,
+ uint8 *dst_chunks, RT_PTR_ALLOC *dst_children)
+{
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const Size chunk_size = sizeof(uint8) * fanout;
+ const Size children_size = sizeof(RT_PTR_ALLOC) * fanout;
+
+ memcpy(dst_chunks, src_chunks, chunk_size);
+ memcpy(dst_children, src_children, children_size);
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_COPY(uint8 *src_chunks, RT_VALUE_TYPE *src_values,
+ uint8 *dst_chunks, RT_VALUE_TYPE *dst_values)
+{
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const Size chunk_size = sizeof(uint8) * fanout;
+ const Size values_size = sizeof(RT_VALUE_TYPE) * fanout;
+
+ memcpy(dst_chunks, src_chunks, chunk_size);
+ memcpy(dst_values, src_values, values_size);
+}
+
+/* Functions to manipulate inner and leaf node-125 */
+
+/* Does the given chunk in the node has the value? */
+static inline bool
+RT_NODE_125_IS_CHUNK_USED(RT_NODE_BASE_125 *node, uint8 chunk)
+{
+ return node->slot_idxs[chunk] != RT_INVALID_SLOT_IDX;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_125_GET_CHILD(RT_NODE_INNER_125 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ return node->children[node->base.slot_idxs[chunk]];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_125_GET_VALUE(RT_NODE_LEAF_125 *node, uint8 chunk)
+{
+ Assert(RT_NODE_IS_LEAF(node));
+ Assert(((RT_NODE_BASE_125 *) node)->slot_idxs[chunk] != RT_INVALID_SLOT_IDX);
+ return node->values[node->base.slot_idxs[chunk]];
+}
+
+/* Functions to manipulate inner and leaf node-256 */
+
+/* Return true if the slot corresponding to the given chunk is in use */
+static inline bool
+RT_NODE_INNER_256_IS_CHUNK_USED(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ return node->children[chunk] != RT_INVALID_PTR_ALLOC;
+}
+
+static inline bool
+RT_NODE_LEAF_256_IS_CHUNK_USED(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ return (node->isset[idx] & ((bitmapword) 1 << bitnum)) != 0;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_256_GET_CHILD(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ Assert(RT_NODE_INNER_256_IS_CHUNK_USED(node, chunk));
+ return node->children[chunk];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_256_GET_VALUE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ Assert(RT_NODE_IS_LEAF(node));
+ Assert(RT_NODE_LEAF_256_IS_CHUNK_USED(node, chunk));
+ return node->values[chunk];
+}
+
+/* Set the child in the node-256 */
+static inline void
+RT_NODE_INNER_256_SET(RT_NODE_INNER_256 *node, uint8 chunk, RT_PTR_ALLOC child)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ node->children[chunk] = child;
+}
+
+/* Set the value in the node-256 */
+static inline void
+RT_NODE_LEAF_256_SET(RT_NODE_LEAF_256 *node, uint8 chunk, RT_VALUE_TYPE value)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ node->isset[idx] |= ((bitmapword) 1 << bitnum);
+ node->values[chunk] = value;
+}
+
+/* Set the slot at the given chunk position */
+static inline void
+RT_NODE_INNER_256_DELETE(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ node->children[chunk] = RT_INVALID_PTR_ALLOC;
+}
+
+static inline void
+RT_NODE_LEAF_256_DELETE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ node->isset[idx] &= ~((bitmapword) 1 << bitnum);
+}
+
+/*
+ * Return the largest shift that will allowing storing the given key.
+ */
+static inline int
+RT_KEY_GET_SHIFT(uint64 key)
+{
+ if (key == 0)
+ return 0;
+ else
+ return (pg_leftmost_one_pos64(key) / RT_NODE_SPAN) * RT_NODE_SPAN;
+}
+
+/*
+ * Return the max value that can be stored in the tree with the given shift.
+ */
+static uint64
+RT_SHIFT_GET_MAX_VAL(int shift)
+{
+ if (shift == RT_MAX_SHIFT)
+ return UINT64_MAX;
+
+ return (UINT64CONST(1) << (shift + RT_NODE_SPAN)) - 1;
+}
+
+/*
+ * Allocate a new node with the given node kind.
+ */
+static RT_PTR_ALLOC
+RT_ALLOC_NODE(RT_RADIX_TREE *tree, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+ RT_PTR_ALLOC allocnode;
+ size_t allocsize;
+
+ if (is_leaf)
+ allocsize = RT_SIZE_CLASS_INFO[size_class].leaf_size;
+ else
+ allocsize = RT_SIZE_CLASS_INFO[size_class].inner_size;
+
+#ifdef RT_SHMEM
+ allocnode = dsa_allocate(tree->dsa, allocsize);
+#else
+ if (is_leaf)
+ allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->leaf_slabs[size_class],
+ allocsize);
+ else
+ allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->inner_slabs[size_class],
+ allocsize);
+#endif
+
+#ifdef RT_DEBUG
+ /* update the statistics */
+ tree->ctl->cnt[size_class]++;
+#endif
+
+ return allocnode;
+}
+
+/* Initialize the node contents */
+static inline void
+RT_INIT_NODE(RT_PTR_LOCAL node, uint8 kind, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+ if (is_leaf)
+ MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].leaf_size);
+ else
+ MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].inner_size);
+
+ node->kind = kind;
+
+ if (kind == RT_NODE_KIND_256)
+ /* See comment for the RT_NODE type */
+ Assert(node->fanout == 0);
+ else
+ node->fanout = RT_SIZE_CLASS_INFO[size_class].fanout;
+
+ /* Initialize slot_idxs to invalid values */
+ if (kind == RT_NODE_KIND_125)
+ {
+ RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+
+ memset(n125->slot_idxs, RT_INVALID_SLOT_IDX, sizeof(n125->slot_idxs));
+ }
+}
+
+/*
+ * Create a new node as the root. Subordinate nodes will be created during
+ * the insertion.
+ */
+static pg_noinline void
+RT_NEW_ROOT(RT_RADIX_TREE *tree, uint64 key)
+{
+ int shift = RT_KEY_GET_SHIFT(key);
+ bool is_leaf = shift == 0;
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(newnode, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ newnode->shift = shift;
+ tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(shift);
+ tree->ctl->root = allocnode;
+}
+
+static inline void
+RT_COPY_NODE(RT_PTR_LOCAL newnode, RT_PTR_LOCAL oldnode)
+{
+ newnode->shift = oldnode->shift;
+ newnode->count = oldnode->count;
+}
+
+/*
+ * Given a new allocated node and an old node, initalize the new
+ * node with the necessary fields and return its local pointer.
+ */
+static inline RT_PTR_LOCAL
+RT_SWITCH_NODE_KIND(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, RT_PTR_LOCAL node,
+ uint8 new_kind, uint8 new_class, bool is_leaf)
+{
+ RT_PTR_LOCAL newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(newnode, new_kind, new_class, is_leaf);
+ RT_COPY_NODE(newnode, node);
+
+ return newnode;
+}
+
+/* Free the given node */
+static void
+RT_FREE_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode)
+{
+ /* If we're deleting the root node, make the tree empty */
+ if (tree->ctl->root == allocnode)
+ {
+ tree->ctl->root = RT_INVALID_PTR_ALLOC;
+ tree->ctl->max_val = 0;
+ }
+
+#ifdef RT_DEBUG
+ {
+ int i;
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+ /* update the statistics */
+ for (i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ if (node->fanout == RT_SIZE_CLASS_INFO[i].fanout)
+ break;
+ }
+
+ /* fanout of node256 is intentionally 0 */
+ if (i == RT_SIZE_CLASS_COUNT)
+ i = RT_CLASS_256;
+
+ tree->ctl->cnt[i]--;
+ Assert(tree->ctl->cnt[i] >= 0);
+ }
+#endif
+
+#ifdef RT_SHMEM
+ dsa_free(tree->dsa, allocnode);
+#else
+ pfree(allocnode);
+#endif
+}
+
+/* Update the parent's pointer when growing a node */
+static inline void
+RT_NODE_UPDATE_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC new_child)
+{
+#define RT_ACTION_UPDATE
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+#undef RT_ACTION_UPDATE
+}
+
+/*
+ * Replace old_child with new_child, and free the old one.
+ */
+static inline void
+RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
+ RT_PTR_ALLOC stored_old_child, RT_PTR_LOCAL old_child,
+ RT_PTR_ALLOC new_child, uint64 key)
+{
+#ifdef USE_ASSERT_CHECKING
+ RT_PTR_LOCAL new = RT_PTR_GET_LOCAL(tree, new_child);
+
+ Assert(old_child->shift == new->shift);
+ Assert(old_child->count == new->count);
+#endif
+
+ if (parent == old_child)
+ {
+ /* Replace the root node with the new larger node */
+ tree->ctl->root = new_child;
+ }
+ else
+ RT_NODE_UPDATE_INNER(parent, key, new_child);
+
+ RT_FREE_NODE(tree, stored_old_child);
+}
+
+/*
+ * The radix tree doesn't have sufficient height. Extend the radix tree so
+ * it can store the key.
+ */
+static pg_noinline void
+RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
+{
+ int target_shift;
+ RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+ int shift = root->shift + RT_NODE_SPAN;
+
+ target_shift = RT_KEY_GET_SHIFT(key);
+
+ /* Grow tree from 'shift' to 'target_shift' */
+ while (shift <= target_shift)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL node;
+ RT_NODE_INNER_3 *n3;
+
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, true);
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(node, RT_NODE_KIND_3, RT_CLASS_3, true);
+ node->shift = shift;
+ node->count = 1;
+
+ n3 = (RT_NODE_INNER_3 *) node;
+ n3->base.chunks[0] = 0;
+ n3->children[0] = tree->ctl->root;
+
+ /* Update the root */
+ tree->ctl->root = allocnode;
+
+ shift += RT_NODE_SPAN;
+ }
+
+ tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(target_shift);
+}
+
+/*
+ * The radix tree doesn't have inner and leaf nodes for given key-value pair.
+ * Insert inner and leaf nodes from 'node' to bottom.
+ */
+static pg_noinline void
+RT_SET_EXTEND(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
+ RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node)
+{
+ int shift = node->shift;
+
+ Assert(RT_PTR_GET_LOCAL(tree, stored_node) == node);
+
+ while (shift >= RT_NODE_SPAN)
+ {
+ RT_PTR_ALLOC allocchild;
+ RT_PTR_LOCAL newchild;
+ int newshift = shift - RT_NODE_SPAN;
+ bool is_leaf = newshift == 0;
+
+ allocchild = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ newchild = RT_PTR_GET_LOCAL(tree, allocchild);
+ RT_INIT_NODE(newchild, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ newchild->shift = newshift;
+ RT_NODE_INSERT_INNER(tree, parent, stored_node, node, key, allocchild);
+
+ parent = node;
+ node = newchild;
+ stored_node = allocchild;
+ shift -= RT_NODE_SPAN;
+ }
+
+ RT_NODE_INSERT_LEAF(tree, parent, stored_node, node, key, value_p);
+ tree->ctl->num_keys++;
+}
+
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the child
+ * pointer is set to child_p.
+ */
+static inline bool
+RT_NODE_SEARCH_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC *child_p)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the pointer
+ * to the value is set to value_p.
+ */
+static inline bool
+RT_NODE_SEARCH_LEAF(RT_PTR_LOCAL node, uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_INNER(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_LEAF(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+#endif
+
+/*
+ * Insert "child" into "node".
+ *
+ * "parent" is the parent of "node", so the grandparent of the child.
+ * If the node we're inserting into needs to grow, we update the parent's
+ * child pointer with the pointer to the new larger node.
+ */
+static void
+RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_PTR_ALLOC child)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/* Like RT_NODE_INSERT_INNER, but for leaf nodes */
+static bool
+RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Create the radix tree in the given memory context and return it.
+ */
+RT_SCOPE RT_RADIX_TREE *
+#ifdef RT_SHMEM
+RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id)
+#else
+RT_CREATE(MemoryContext ctx)
+#endif
+{
+ RT_RADIX_TREE *tree;
+ MemoryContext old_ctx;
+#ifdef RT_SHMEM
+ dsa_pointer dp;
+#endif
+
+ old_ctx = MemoryContextSwitchTo(ctx);
+
+ tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+ tree->context = ctx;
+
+#ifdef RT_SHMEM
+ tree->dsa = dsa;
+ dp = dsa_allocate0(dsa, sizeof(RT_RADIX_TREE_CONTROL));
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, dp);
+ tree->ctl->handle = dp;
+ tree->ctl->magic = RT_RADIX_TREE_MAGIC;
+ LWLockInitialize(&tree->ctl->lock, tranche_id);
+#else
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) palloc0(sizeof(RT_RADIX_TREE_CONTROL));
+
+ /* Create a slab context for each size class */
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ RT_SIZE_CLASS_ELEM size_class = RT_SIZE_CLASS_INFO[i];
+ size_t inner_blocksize = RT_SLAB_BLOCK_SIZE(size_class.inner_size);
+ size_t leaf_blocksize = RT_SLAB_BLOCK_SIZE(size_class.leaf_size);
+
+ tree->inner_slabs[i] = SlabContextCreate(ctx,
+ size_class.name,
+ inner_blocksize,
+ size_class.inner_size);
+ tree->leaf_slabs[i] = SlabContextCreate(ctx,
+ size_class.name,
+ leaf_blocksize,
+ size_class.leaf_size);
+ }
+#endif
+
+ tree->ctl->root = RT_INVALID_PTR_ALLOC;
+
+ MemoryContextSwitchTo(old_ctx);
+
+ return tree;
+}
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE *
+RT_ATTACH(dsa_area *dsa, RT_HANDLE handle)
+{
+ RT_RADIX_TREE *tree;
+ dsa_pointer control;
+
+ tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+
+ /* Find the control object in shard memory */
+ control = handle;
+
+ tree->dsa = dsa;
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, control);
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+ return tree;
+}
+
+RT_SCOPE void
+RT_DETACH(RT_RADIX_TREE *tree)
+{
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ pfree(tree);
+}
+
+RT_SCOPE RT_HANDLE
+RT_GET_HANDLE(RT_RADIX_TREE *tree)
+{
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ return tree->ctl->handle;
+}
+
+/*
+ * Recursively free all nodes allocated to the DSA area.
+ */
+static void
+RT_FREE_RECURSE(RT_RADIX_TREE *tree, RT_PTR_ALLOC ptr)
+{
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, ptr);
+
+ check_stack_depth();
+ CHECK_FOR_INTERRUPTS();
+
+ /* The leaf node doesn't have child pointers */
+ if (RT_NODE_IS_LEAF(node))
+ {
+ dsa_free(tree->dsa, ptr);
+ return;
+ }
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+ for (int i = 0; i < n3->base.n.count; i++)
+ RT_FREE_RECURSE(tree, n3->children[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+ for (int i = 0; i < n32->base.n.count; i++)
+ RT_FREE_RECURSE(tree, n32->children[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) node;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+ continue;
+
+ RT_FREE_RECURSE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+ }
+
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ RT_FREE_RECURSE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+ }
+
+ break;
+ }
+ }
+
+ /* Free the inner node */
+ dsa_free(tree->dsa, ptr);
+}
+#endif
+
+/*
+ * Free the given radix tree.
+ */
+RT_SCOPE void
+RT_FREE(RT_RADIX_TREE *tree)
+{
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+ /* Free all memory used for radix tree nodes */
+ if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ RT_FREE_RECURSE(tree, tree->ctl->root);
+
+ /*
+ * Vandalize the control block to help catch programming error where
+ * other backends access the memory formerly occupied by this radix tree.
+ */
+ tree->ctl->magic = 0;
+ dsa_free(tree->dsa, tree->ctl->handle);
+#else
+ pfree(tree->ctl);
+
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ MemoryContextDelete(tree->inner_slabs[i]);
+ MemoryContextDelete(tree->leaf_slabs[i]);
+ }
+#endif
+
+ pfree(tree);
+}
+
+/*
+ * Set key to value. If the entry already exists, we update its value to 'value'
+ * and return true. Returns false if entry doesn't yet exist.
+ */
+RT_SCOPE bool
+RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+ int shift;
+ bool updated;
+ RT_PTR_LOCAL parent;
+ RT_PTR_ALLOC stored_child;
+ RT_PTR_LOCAL child;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ RT_LOCK_EXCLUSIVE(tree);
+
+ /* Empty tree, create the root */
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ RT_NEW_ROOT(tree, key);
+
+ /* Extend the tree if necessary */
+ if (key > tree->ctl->max_val)
+ RT_EXTEND(tree, key);
+
+ stored_child = tree->ctl->root;
+ parent = RT_PTR_GET_LOCAL(tree, stored_child);
+ shift = parent->shift;
+
+ /* Descend the tree until we reach a leaf node */
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC new_child = RT_INVALID_PTR_ALLOC;
+
+ child = RT_PTR_GET_LOCAL(tree, stored_child);
+
+ if (RT_NODE_IS_LEAF(child))
+ break;
+
+ if (!RT_NODE_SEARCH_INNER(child, key, &new_child))
+ {
+ RT_SET_EXTEND(tree, key, value_p, parent, stored_child, child);
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ parent = child;
+ stored_child = new_child;
+ shift -= RT_NODE_SPAN;
+ }
+
+ updated = RT_NODE_INSERT_LEAF(tree, parent, stored_child, child, key, value_p);
+
+ /* Update the statistics */
+ if (!updated)
+ tree->ctl->num_keys++;
+
+ RT_UNLOCK(tree);
+ return updated;
+}
+
+/*
+ * Search the given key in the radix tree. Return true if there is the key,
+ * otherwise return false. On success, we set the value to *value_p so it must
+ * not be NULL.
+ */
+RT_SCOPE bool
+RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+ RT_PTR_LOCAL node;
+ int shift;
+ bool found;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+ Assert(value_p != NULL);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ node = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+ shift = node->shift;
+
+ /* Descend the tree until a leaf node */
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+ if (RT_NODE_IS_LEAF(node))
+ break;
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ node = RT_PTR_GET_LOCAL(tree, child);
+ shift -= RT_NODE_SPAN;
+ }
+
+ found = RT_NODE_SEARCH_LEAF(node, key, value_p);
+
+ RT_UNLOCK(tree);
+ return found;
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Delete the given key from the radix tree. Return true if the key is found (and
+ * deleted), otherwise do nothing and return false.
+ */
+RT_SCOPE bool
+RT_DELETE(RT_RADIX_TREE *tree, uint64 key)
+{
+ RT_PTR_LOCAL node;
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_ALLOC stack[RT_MAX_LEVEL] = {0};
+ int shift;
+ int level;
+ bool deleted;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ RT_LOCK_EXCLUSIVE(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ /*
+ * Descend the tree to search the key while building a stack of nodes we
+ * visited.
+ */
+ allocnode = tree->ctl->root;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift = node->shift;
+ level = -1;
+ while (shift > 0)
+ {
+ RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+ /* Push the current node to the stack */
+ stack[++level] = allocnode;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ allocnode = child;
+ shift -= RT_NODE_SPAN;
+ }
+
+ /* Delete the key from the leaf node if exists */
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ deleted = RT_NODE_DELETE_LEAF(node, key);
+
+ if (!deleted)
+ {
+ /* no key is found in the leaf node */
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ /* Found the key to delete. Update the statistics */
+ tree->ctl->num_keys--;
+
+ /*
+ * Return if the leaf node still has keys and we don't need to delete the
+ * node.
+ */
+ if (node->count > 0)
+ {
+ RT_UNLOCK(tree);
+ return true;
+ }
+
+ /* Free the empty leaf node */
+ RT_FREE_NODE(tree, allocnode);
+
+ /* Delete the key in inner nodes recursively */
+ while (level >= 0)
+ {
+ allocnode = stack[level--];
+
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ deleted = RT_NODE_DELETE_INNER(node, key);
+ Assert(deleted);
+
+ /* If the node didn't become empty, we stop deleting the key */
+ if (node->count > 0)
+ break;
+
+ /* The node became empty */
+ RT_FREE_NODE(tree, allocnode);
+ }
+
+ RT_UNLOCK(tree);
+ return true;
+}
+#endif
+
+static inline void
+RT_ITER_UPDATE_KEY(RT_ITER *iter, uint8 chunk, uint8 shift)
+{
+ iter->key &= ~(((uint64) RT_CHUNK_MASK) << shift);
+ iter->key |= (((uint64) chunk) << shift);
+}
+
+/*
+ * Advance the slot in the inner node. Return the child if exists, otherwise
+ * null.
+ */
+static inline RT_PTR_LOCAL
+RT_NODE_INNER_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Advance the slot in the leaf node. On success, return true and the value
+ * is set to value_p, otherwise return false.
+ */
+static inline bool
+RT_NODE_LEAF_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter,
+ RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Update each node_iter for inner nodes in the iterator node stack.
+ */
+static void
+RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
+{
+ int level = from;
+ RT_PTR_LOCAL node = from_node;
+
+ for (;;)
+ {
+ RT_NODE_ITER *node_iter = &(iter->stack[level--]);
+
+ node_iter->node = node;
+ node_iter->current_idx = -1;
+
+ /* We don't advance the leaf node iterator here */
+ if (RT_NODE_IS_LEAF(node))
+ return;
+
+ /* Advance to the next slot in the inner node */
+ node = RT_NODE_INNER_ITERATE_NEXT(iter, node_iter);
+
+ /* We must find the first children in the node */
+ Assert(node);
+ }
+}
+
+/*
+ * Create and return the iterator for the given radix tree.
+ *
+ * The radix tree is locked in shared mode during the iteration, so
+ * RT_END_ITERATE needs to be called when finished to release the lock.
+ */
+RT_SCOPE RT_ITER *
+RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
+{
+ MemoryContext old_ctx;
+ RT_ITER *iter;
+ RT_PTR_LOCAL root;
+ int top_level;
+
+ old_ctx = MemoryContextSwitchTo(tree->context);
+
+ iter = (RT_ITER *) palloc0(sizeof(RT_ITER));
+ iter->tree = tree;
+
+ RT_LOCK_SHARED(tree);
+
+ /* empty tree */
+ if (!iter->tree->ctl->root)
+ return iter;
+
+ root = RT_PTR_GET_LOCAL(tree, iter->tree->ctl->root);
+ top_level = root->shift / RT_NODE_SPAN;
+ iter->stack_len = top_level;
+
+ /*
+ * Descend to the left most leaf node from the root. The key is being
+ * constructed while descending to the leaf.
+ */
+ RT_UPDATE_ITER_STACK(iter, root, top_level);
+
+ MemoryContextSwitchTo(old_ctx);
+
+ return iter;
+}
+
+/*
+ * Return true with setting key_p and value_p if there is next key. Otherwise
+ * return false.
+ */
+RT_SCOPE bool
+RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p)
+{
+ /* Empty tree */
+ if (!iter->tree->ctl->root)
+ return false;
+
+ for (;;)
+ {
+ RT_PTR_LOCAL child = NULL;
+ RT_VALUE_TYPE value;
+ int level;
+ bool found;
+
+ /* Advance the leaf node iterator to get next key-value pair */
+ found = RT_NODE_LEAF_ITERATE_NEXT(iter, &(iter->stack[0]), &value);
+
+ if (found)
+ {
+ *key_p = iter->key;
+ *value_p = value;
+ return true;
+ }
+
+ /*
+ * We've visited all values in the leaf node, so advance inner node
+ * iterators from the level=1 until we find the next child node.
+ */
+ for (level = 1; level <= iter->stack_len; level++)
+ {
+ child = RT_NODE_INNER_ITERATE_NEXT(iter, &(iter->stack[level]));
+
+ if (child)
+ break;
+ }
+
+ /* the iteration finished */
+ if (!child)
+ return false;
+
+ /*
+ * Set the node to the node iterator and update the iterator stack
+ * from this node.
+ */
+ RT_UPDATE_ITER_STACK(iter, child, level - 1);
+
+ /* Node iterators are updated, so try again from the leaf */
+ }
+
+ return false;
+}
+
+/*
+ * Terminate the iteration and release the lock.
+ *
+ * This function needs to be called after finishing or when exiting an
+ * iteration.
+ */
+RT_SCOPE void
+RT_END_ITERATE(RT_ITER *iter)
+{
+#ifdef RT_SHMEM
+ Assert(LWLockHeldByMe(&iter->tree->ctl->lock));
+#endif
+
+ RT_UNLOCK(iter->tree);
+ pfree(iter);
+}
+
+/*
+ * Return the statistics of the amount of memory used by the radix tree.
+ */
+RT_SCOPE uint64
+RT_MEMORY_USAGE(RT_RADIX_TREE *tree)
+{
+ Size total = 0;
+
+ RT_LOCK_SHARED(tree);
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ total = dsa_get_total_size(tree->dsa);
+#else
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ total += MemoryContextMemAllocated(tree->inner_slabs[i], true);
+ total += MemoryContextMemAllocated(tree->leaf_slabs[i], true);
+ }
+#endif
+
+ RT_UNLOCK(tree);
+ return total;
+}
+
+/*
+ * Verify the radix tree node.
+ */
+static void
+RT_VERIFY_NODE(RT_PTR_LOCAL node)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(node->count >= 0);
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE_BASE_3 *n3 = (RT_NODE_BASE_3 *) node;
+
+ for (int i = 1; i < n3->n.count; i++)
+ Assert(n3->chunks[i - 1] < n3->chunks[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE_BASE_32 *n32 = (RT_NODE_BASE_32 *) node;
+
+ for (int i = 1; i < n32->n.count; i++)
+ Assert(n32->chunks[i - 1] < n32->chunks[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+ int cnt = 0;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ uint8 slot = n125->slot_idxs[i];
+ int idx = RT_BM_IDX(slot);
+ int bitnum = RT_BM_BIT(slot);
+
+ if (!RT_NODE_125_IS_CHUNK_USED(n125, i))
+ continue;
+
+ /* Check if the corresponding slot is used */
+ Assert(slot < node->fanout);
+ Assert((n125->isset[idx] & ((bitmapword) 1 << bitnum)) != 0);
+
+ cnt++;
+ }
+
+ Assert(n125->n.count == cnt);
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+ int cnt = 0;
+
+ for (int i = 0; i < RT_BM_IDX(RT_NODE_MAX_SLOTS); i++)
+ cnt += bmw_popcount(n256->isset[i]);
+
+ /* Check if the number of used chunk matches */
+ Assert(n256->base.n.count == cnt);
+
+ break;
+ }
+ }
+ }
+#endif
+}
+
+/***************** DEBUG FUNCTIONS *****************/
+#ifdef RT_DEBUG
+
+#define RT_UINT64_FORMAT_HEX "%" INT64_MODIFIER "X"
+
+RT_SCOPE void
+RT_STATS(RT_RADIX_TREE *tree)
+{
+ RT_LOCK_SHARED(tree);
+
+ fprintf(stderr, "max_val = " UINT64_FORMAT "\n", tree->ctl->max_val);
+ fprintf(stderr, "num_keys = " UINT64_FORMAT "\n", tree->ctl->num_keys);
+
+#ifdef RT_SHMEM
+ fprintf(stderr, "handle = " UINT64_FORMAT "\n", tree->ctl->handle);
+#endif
+
+ if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+
+ fprintf(stderr, "height = %d, n3 = %u, n15 = %u, n32 = %u, n125 = %u, n256 = %u\n",
+ root->shift / RT_NODE_SPAN,
+ tree->ctl->cnt[RT_CLASS_3],
+ tree->ctl->cnt[RT_CLASS_32_MIN],
+ tree->ctl->cnt[RT_CLASS_32_MAX],
+ tree->ctl->cnt[RT_CLASS_125],
+ tree->ctl->cnt[RT_CLASS_256]);
+ }
+
+ RT_UNLOCK(tree);
+}
+
+static void
+RT_DUMP_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, int level,
+ bool recurse, StringInfo buf)
+{
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+ StringInfoData spaces;
+
+ initStringInfo(&spaces);
+ appendStringInfoSpaces(&spaces, (level * 4) + 1);
+
+ appendStringInfo(buf, "%s%s[%s] kind %d, fanout %d, count %u, shift %u:\n",
+ spaces.data,
+ level == 0 ? "" : "-> ",
+ RT_NODE_IS_LEAF(node) ? "LEAF" : "INNR",
+ (node->kind == RT_NODE_KIND_3) ? 3 :
+ (node->kind == RT_NODE_KIND_32) ? 32 :
+ (node->kind == RT_NODE_KIND_125) ? 125 : 256,
+ node->fanout == 0 ? 256 : node->fanout,
+ node->count, node->shift);
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ for (int i = 0; i < node->count; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_3 *n3 = (RT_NODE_LEAF_3 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+ spaces.data, i, n3->base.chunks[i]);
+ }
+ else
+ {
+ RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X",
+ spaces.data, i, n3->base.chunks[i]);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, n3->children[i], level + 1,
+ recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ for (int i = 0; i < node->count; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_32 *n32 = (RT_NODE_LEAF_32 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+ spaces.data, i, n32->base.chunks[i]);
+ }
+ else
+ {
+ RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X",
+ spaces.data, i, n32->base.chunks[i]);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, n32->children[i], level + 1,
+ recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_BASE_125 *b125 = (RT_NODE_BASE_125 *) node;
+ char *sep = "";
+
+ appendStringInfo(buf, "%sslot_idxs: ", spaces.data);
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+ continue;
+
+ appendStringInfo(buf, "%s[%d]=%d ",
+ sep, i, b125->slot_idxs[i]);
+ sep = ",";
+ }
+
+ appendStringInfo(buf, "\n%sisset-bitmap: ", spaces.data);
+ for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+ appendStringInfo(buf, "%X ", ((uint8 *) b125->isset)[i]);
+ appendStringInfo(buf, "\n");
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+ continue;
+
+ if (RT_NODE_IS_LEAF(node))
+ appendStringInfo(buf, "%schunk 0x%X\n",
+ spaces.data, i);
+ else
+ {
+ RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) b125;
+
+ appendStringInfo(buf, "%schunk 0x%X",
+ spaces.data, i);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i),
+ level + 1, recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+ appendStringInfo(buf, "%sisset-bitmap: ", spaces.data);
+ for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+ appendStringInfo(buf, "%X ", ((uint8 *) n256->isset)[i]);
+ appendStringInfo(buf, "\n");
+ }
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ appendStringInfo(buf, "%schunk 0x%X\n",
+ spaces.data, i);
+ }
+ else
+ {
+ RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ appendStringInfo(buf, "%schunk 0x%X",
+ spaces.data, i);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i),
+ level + 1, recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ }
+}
+
+RT_SCOPE void
+RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key)
+{
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL node;
+ StringInfoData buf;
+ int shift;
+ int level = 0;
+
+ RT_STATS(tree);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "empty tree\n");
+ return;
+ }
+
+ if (key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "key " UINT64_FORMAT "(0x" RT_UINT64_FORMAT_HEX ") is larger than max val\n",
+ key, key);
+ return;
+ }
+
+ initStringInfo(&buf);
+ allocnode = tree->ctl->root;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift = node->shift;
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC child;
+
+ RT_DUMP_NODE(tree, allocnode, level, false, &buf);
+
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_VALUE_TYPE dummy;
+
+ /* We reached at a leaf node, find the corresponding slot */
+ RT_NODE_SEARCH_LEAF(node, key, &dummy);
+
+ break;
+ }
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ break;
+
+ allocnode = child;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift -= RT_NODE_SPAN;
+ level++;
+ }
+ RT_UNLOCK(tree);
+
+ fprintf(stderr, "%s", buf.data);
+}
+
+RT_SCOPE void
+RT_DUMP(RT_RADIX_TREE *tree)
+{
+ StringInfoData buf;
+
+ RT_STATS(tree);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "empty tree\n");
+ return;
+ }
+
+ initStringInfo(&buf);
+
+ RT_DUMP_NODE(tree, tree->ctl->root, 0, true, &buf);
+ RT_UNLOCK(tree);
+
+ fprintf(stderr, "%s",buf.data);
+}
+#endif
+
+#endif /* RT_DEFINE */
+
+
+/* undefine external parameters, so next radix tree can be defined */
+#undef RT_PREFIX
+#undef RT_SCOPE
+#undef RT_DECLARE
+#undef RT_DEFINE
+#undef RT_VALUE_TYPE
+
+/* locally declared macros */
+#undef RT_MAKE_PREFIX
+#undef RT_MAKE_NAME
+#undef RT_MAKE_NAME_
+#undef RT_NODE_SPAN
+#undef RT_NODE_MAX_SLOTS
+#undef RT_CHUNK_MASK
+#undef RT_MAX_SHIFT
+#undef RT_MAX_LEVEL
+#undef RT_GET_KEY_CHUNK
+#undef RT_BM_IDX
+#undef RT_BM_BIT
+#undef RT_LOCK_EXCLUSIVE
+#undef RT_LOCK_SHARED
+#undef RT_UNLOCK
+#undef RT_NODE_IS_LEAF
+#undef RT_NODE_MUST_GROW
+#undef RT_NODE_KIND_COUNT
+#undef RT_SIZE_CLASS_COUNT
+#undef RT_SLOT_IDX_LIMIT
+#undef RT_INVALID_SLOT_IDX
+#undef RT_SLAB_BLOCK_SIZE
+#undef RT_RADIX_TREE_MAGIC
+#undef RT_UINT64_FORMAT_HEX
+
+/* type declarations */
+#undef RT_RADIX_TREE
+#undef RT_RADIX_TREE_CONTROL
+#undef RT_PTR_LOCAL
+#undef RT_PTR_ALLOC
+#undef RT_INVALID_PTR_ALLOC
+#undef RT_HANDLE
+#undef RT_ITER
+#undef RT_NODE
+#undef RT_NODE_ITER
+#undef RT_NODE_KIND_3
+#undef RT_NODE_KIND_32
+#undef RT_NODE_KIND_125
+#undef RT_NODE_KIND_256
+#undef RT_NODE_BASE_3
+#undef RT_NODE_BASE_32
+#undef RT_NODE_BASE_125
+#undef RT_NODE_BASE_256
+#undef RT_NODE_INNER_3
+#undef RT_NODE_INNER_32
+#undef RT_NODE_INNER_125
+#undef RT_NODE_INNER_256
+#undef RT_NODE_LEAF_3
+#undef RT_NODE_LEAF_32
+#undef RT_NODE_LEAF_125
+#undef RT_NODE_LEAF_256
+#undef RT_SIZE_CLASS
+#undef RT_SIZE_CLASS_ELEM
+#undef RT_SIZE_CLASS_INFO
+#undef RT_CLASS_3
+#undef RT_CLASS_32_MIN
+#undef RT_CLASS_32_MAX
+#undef RT_CLASS_125
+#undef RT_CLASS_256
+
+/* function declarations */
+#undef RT_CREATE
+#undef RT_FREE
+#undef RT_ATTACH
+#undef RT_DETACH
+#undef RT_GET_HANDLE
+#undef RT_SEARCH
+#undef RT_SET
+#undef RT_BEGIN_ITERATE
+#undef RT_ITERATE_NEXT
+#undef RT_END_ITERATE
+#undef RT_USE_DELETE
+#undef RT_DELETE
+#undef RT_MEMORY_USAGE
+#undef RT_DUMP
+#undef RT_DUMP_NODE
+#undef RT_DUMP_SEARCH
+#undef RT_STATS
+
+/* internal helper functions */
+#undef RT_NEW_ROOT
+#undef RT_ALLOC_NODE
+#undef RT_INIT_NODE
+#undef RT_FREE_NODE
+#undef RT_FREE_RECURSE
+#undef RT_EXTEND
+#undef RT_SET_EXTEND
+#undef RT_SWITCH_NODE_KIND
+#undef RT_COPY_NODE
+#undef RT_REPLACE_NODE
+#undef RT_PTR_GET_LOCAL
+#undef RT_PTR_ALLOC_IS_VALID
+#undef RT_NODE_3_SEARCH_EQ
+#undef RT_NODE_32_SEARCH_EQ
+#undef RT_NODE_3_GET_INSERTPOS
+#undef RT_NODE_32_GET_INSERTPOS
+#undef RT_CHUNK_CHILDREN_ARRAY_SHIFT
+#undef RT_CHUNK_VALUES_ARRAY_SHIFT
+#undef RT_CHUNK_CHILDREN_ARRAY_DELETE
+#undef RT_CHUNK_VALUES_ARRAY_DELETE
+#undef RT_CHUNK_CHILDREN_ARRAY_COPY
+#undef RT_CHUNK_VALUES_ARRAY_COPY
+#undef RT_NODE_125_IS_CHUNK_USED
+#undef RT_NODE_INNER_125_GET_CHILD
+#undef RT_NODE_LEAF_125_GET_VALUE
+#undef RT_NODE_INNER_256_IS_CHUNK_USED
+#undef RT_NODE_LEAF_256_IS_CHUNK_USED
+#undef RT_NODE_INNER_256_GET_CHILD
+#undef RT_NODE_LEAF_256_GET_VALUE
+#undef RT_NODE_INNER_256_SET
+#undef RT_NODE_LEAF_256_SET
+#undef RT_NODE_INNER_256_DELETE
+#undef RT_NODE_LEAF_256_DELETE
+#undef RT_KEY_GET_SHIFT
+#undef RT_SHIFT_GET_MAX_VAL
+#undef RT_NODE_SEARCH_INNER
+#undef RT_NODE_SEARCH_LEAF
+#undef RT_NODE_UPDATE_INNER
+#undef RT_NODE_DELETE_INNER
+#undef RT_NODE_DELETE_LEAF
+#undef RT_NODE_INSERT_INNER
+#undef RT_NODE_INSERT_LEAF
+#undef RT_NODE_INNER_ITERATE_NEXT
+#undef RT_NODE_LEAF_ITERATE_NEXT
+#undef RT_UPDATE_ITER_STACK
+#undef RT_ITER_UPDATE_KEY
+#undef RT_VERIFY_NODE
+
+#undef RT_DEBUG
diff --git a/src/include/lib/radixtree_delete_impl.h b/src/include/lib/radixtree_delete_impl.h
new file mode 100644
index 0000000000..5f6dda1f12
--- /dev/null
+++ b/src/include/lib/radixtree_delete_impl.h
@@ -0,0 +1,122 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_delete_impl.h
+ * Common implementation for deletion in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ * TODO: Shrink nodes when deletion would allow them to fit in a smaller
+ * size class.
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_delete_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_DELETE(n3->base.chunks, n3->values,
+ n3->base.n.count, idx);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_DELETE(n3->base.chunks, n3->children,
+ n3->base.n.count, idx);
+#endif
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_DELETE(n32->base.chunks, n32->values,
+ n32->base.n.count, idx);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_DELETE(n32->base.chunks, n32->children,
+ n32->base.n.count, idx);
+#endif
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+ int idx;
+ int bitnum;
+
+ if (slotpos == RT_INVALID_SLOT_IDX)
+ return false;
+
+ idx = RT_BM_IDX(slotpos);
+ bitnum = RT_BM_BIT(slotpos);
+ n125->base.isset[idx] &= ~((bitmapword) 1 << bitnum);
+ n125->base.slot_idxs[chunk] = RT_INVALID_SLOT_IDX;
+
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_DELETE(n256, chunk);
+#else
+ RT_NODE_INNER_256_DELETE(n256, chunk);
+#endif
+ break;
+ }
+ }
+
+ /* update statistics */
+ node->count--;
+
+ return true;
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
new file mode 100644
index 0000000000..d56e58dcac
--- /dev/null
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -0,0 +1,328 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_insert_impl.h
+ * Common implementation for insertion in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_insert_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ const bool is_leaf = true;
+ bool chunk_exists = false;
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+ const bool is_leaf = false;
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ int idx = RT_NODE_3_SEARCH_EQ(&n3->base, chunk);
+
+ if (idx != -1)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n3->values[idx] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n3)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE32_TYPE *new32;
+ const uint8 new_kind = RT_NODE_KIND_32;
+ const RT_SIZE_CLASS new_class = RT_CLASS_32_MIN;
+
+ /* grow node from 3 to 32 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_COPY(n3->base.chunks, n3->values,
+ new32->base.chunks, new32->values);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_COPY(n3->base.chunks, n3->children,
+ new32->base.chunks, new32->children);
+#endif
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
+ int count = n3->base.n.count;
+
+ /* shift chunks and children */
+ if (insertpos < count)
+ {
+ Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_SHIFT(n3->base.chunks, n3->values,
+ count, insertpos);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_SHIFT(n3->base.chunks, n3->children,
+ count, insertpos);
+#endif
+ }
+
+ n3->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+ n3->values[insertpos] = *value_p;
+#else
+ n3->children[insertpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_32:
+ {
+ const RT_SIZE_CLASS_ELEM class32_max = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX];
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ int idx = RT_NODE_32_SEARCH_EQ(&n32->base, chunk);
+
+ if (idx != -1)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n32->values[idx] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n32)) &&
+ n32->base.n.fanout < class32_max.fanout)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ const RT_SIZE_CLASS_ELEM class32_min = RT_SIZE_CLASS_INFO[RT_CLASS_32_MIN];
+ const RT_SIZE_CLASS new_class = RT_CLASS_32_MAX;
+
+ Assert(n32->base.n.fanout == class32_min.fanout);
+
+ /* grow to the next size class of this kind */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ n32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ memcpy(newnode, node, class32_min.leaf_size);
+#else
+ memcpy(newnode, node, class32_min.inner_size);
+#endif
+ newnode->fanout = class32_max.fanout;
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+
+ if (unlikely(RT_NODE_MUST_GROW(n32)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE125_TYPE *new125;
+ const uint8 new_kind = RT_NODE_KIND_125;
+ const RT_SIZE_CLASS new_class = RT_CLASS_125;
+
+ Assert(n32->base.n.fanout == class32_max.fanout);
+
+ /* grow node from 32 to 125 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new125 = (RT_NODE125_TYPE *) newnode;
+
+ for (int i = 0; i < class32_max.fanout; i++)
+ {
+ new125->base.slot_idxs[n32->base.chunks[i]] = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ new125->values[i] = n32->values[i];
+#else
+ new125->children[i] = n32->children[i];
+#endif
+ }
+
+ /*
+ * Since we just copied a dense array, we can set the bits
+ * using a single store, provided the length of that array
+ * is at most the number of bits in a bitmapword.
+ */
+ Assert(class32_max.fanout <= sizeof(bitmapword) * BITS_PER_BYTE);
+ new125->base.isset[0] = (bitmapword) (((uint64) 1 << class32_max.fanout) - 1);
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
+ int count = n32->base.n.count;
+
+ if (insertpos < count)
+ {
+ Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_SHIFT(n32->base.chunks, n32->values,
+ count, insertpos);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_SHIFT(n32->base.chunks, n32->children,
+ count, insertpos);
+#endif
+ }
+
+ n32->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+ n32->values[insertpos] = *value_p;
+#else
+ n32->children[insertpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos;
+ int cnt = 0;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ slotpos = n125->base.slot_idxs[chunk];
+ if (slotpos != RT_INVALID_SLOT_IDX)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n125->values[slotpos] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n125)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE256_TYPE *new256;
+ const uint8 new_kind = RT_NODE_KIND_256;
+ const RT_SIZE_CLASS new_class = RT_CLASS_256;
+
+ /* grow node from 125 to 256 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new256 = (RT_NODE256_TYPE *) newnode;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS && cnt < n125->base.n.count; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+ continue;
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_SET(new256, i, RT_NODE_LEAF_125_GET_VALUE(n125, i));
+#else
+ RT_NODE_INNER_256_SET(new256, i, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+ cnt++;
+ }
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int idx;
+ bitmapword inverse;
+
+ /* get the first word with at least one bit not set */
+ for (idx = 0; idx < RT_BM_IDX(RT_SLOT_IDX_LIMIT); idx++)
+ {
+ if (n125->base.isset[idx] < ~((bitmapword) 0))
+ break;
+ }
+
+ /* To get the first unset bit in X, get the first set bit in ~X */
+ inverse = ~(n125->base.isset[idx]);
+ slotpos = idx * BITS_PER_BITMAPWORD;
+ slotpos += bmw_rightmost_one_pos(inverse);
+ Assert(slotpos < node->fanout);
+
+ /* mark the slot used */
+ n125->base.isset[idx] |= bmw_rightmost_one(inverse);
+ n125->base.slot_idxs[chunk] = slotpos;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ n125->values[slotpos] = *value_p;
+#else
+ n125->children[slotpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ chunk_exists = RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk);
+ Assert(chunk_exists || node->count < RT_NODE_MAX_SLOTS);
+ RT_NODE_LEAF_256_SET(n256, chunk, *value_p);
+#else
+ Assert(node->count < RT_NODE_MAX_SLOTS);
+ RT_NODE_INNER_256_SET(n256, chunk, child);
+#endif
+ break;
+ }
+ }
+
+ /* Update statistics */
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!chunk_exists)
+ node->count++;
+#else
+ node->count++;
+#endif
+
+ /*
+ * Done. Finally, verify the chunk and value is inserted or replaced
+ * properly in the node.
+ */
+ RT_VERIFY_NODE(node);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ return chunk_exists;
+#else
+ return;
+#endif
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_iter_impl.h b/src/include/lib/radixtree_iter_impl.h
new file mode 100644
index 0000000000..98c78eb237
--- /dev/null
+++ b/src/include/lib/radixtree_iter_impl.h
@@ -0,0 +1,153 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_iter_impl.h
+ * Common implementation for iteration in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_iter_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ bool found = false;
+ uint8 key_chunk;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_VALUE_TYPE value;
+
+ Assert(RT_NODE_IS_LEAF(node_iter->node));
+#else
+ RT_PTR_LOCAL child = NULL;
+
+ Assert(!RT_NODE_IS_LEAF(node_iter->node));
+#endif
+
+#ifdef RT_SHMEM
+ Assert(iter->tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ switch (node_iter->node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node_iter->node;
+
+ node_iter->current_idx++;
+ if (node_iter->current_idx >= n3->base.n.count)
+ break;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = n3->values[node_iter->current_idx];
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, n3->children[node_iter->current_idx]);
+#endif
+ key_chunk = n3->base.chunks[node_iter->current_idx];
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node_iter->node;
+
+ node_iter->current_idx++;
+ if (node_iter->current_idx >= n32->base.n.count)
+ break;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ value = n32->values[node_iter->current_idx];
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, n32->children[node_iter->current_idx]);
+#endif
+ key_chunk = n32->base.chunks[node_iter->current_idx];
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node_iter->node;
+ int i;
+
+ for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (RT_NODE_125_IS_CHUNK_USED((RT_NODE_BASE_125 *) n125, i))
+ break;
+ }
+
+ if (i >= RT_NODE_MAX_SLOTS)
+ break;
+
+ node_iter->current_idx = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = RT_NODE_LEAF_125_GET_VALUE(n125, i);
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+ key_chunk = i;
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node_iter->node;
+ int i;
+
+ for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ {
+#ifdef RT_NODE_LEVEL_LEAF
+ if (RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+#else
+ if (RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+#endif
+ break;
+ }
+
+ if (i >= RT_NODE_MAX_SLOTS)
+ break;
+
+ node_iter->current_idx = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = RT_NODE_LEAF_256_GET_VALUE(n256, i);
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+#endif
+ key_chunk = i;
+ found = true;
+ break;
+ }
+ }
+
+ if (found)
+ {
+ RT_ITER_UPDATE_KEY(iter, key_chunk, node_iter->node->shift);
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = value;
+#endif
+ }
+
+#ifdef RT_NODE_LEVEL_LEAF
+ return found;
+#else
+ return child;
+#endif
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_search_impl.h b/src/include/lib/radixtree_search_impl.h
new file mode 100644
index 0000000000..a8925c75d0
--- /dev/null
+++ b/src/include/lib/radixtree_search_impl.h
@@ -0,0 +1,138 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_search_impl.h
+ * Common implementation for search in leaf and inner nodes, plus
+ * update for inner nodes only.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_search_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(value_p != NULL);
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+#ifndef RT_ACTION_UPDATE
+ Assert(child_p != NULL);
+#endif
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+#ifdef RT_ACTION_UPDATE
+ Assert(idx >= 0);
+ n3->children[idx] = new_child;
+#else
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n3->values[idx];
+#else
+ *child_p = n3->children[idx];
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+#ifdef RT_ACTION_UPDATE
+ Assert(idx >= 0);
+ n32->children[idx] = new_child;
+#else
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n32->values[idx];
+#else
+ *child_p = n32->children[idx];
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+
+#ifdef RT_ACTION_UPDATE
+ Assert(slotpos != RT_INVALID_SLOT_IDX);
+ n125->children[slotpos] = new_child;
+#else
+ if (slotpos == RT_INVALID_SLOT_IDX)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_125_GET_VALUE(n125, chunk);
+#else
+ *child_p = RT_NODE_INNER_125_GET_CHILD(n125, chunk);
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_ACTION_UPDATE
+ RT_NODE_INNER_256_SET(n256, chunk, new_child);
+#else
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_256_GET_VALUE(n256, chunk);
+#else
+ *child_p = RT_NODE_INNER_256_GET_CHILD(n256, chunk);
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ }
+
+#ifdef RT_ACTION_UPDATE
+ return;
+#else
+ return true;
+#endif /* RT_ACTION_UPDATE */
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/utils/dsa.h b/src/include/utils/dsa.h
index 3ce4ee300a..2af215484f 100644
--- a/src/include/utils/dsa.h
+++ b/src/include/utils/dsa.h
@@ -121,6 +121,7 @@ extern dsa_handle dsa_get_handle(dsa_area *area);
extern dsa_pointer dsa_allocate_extended(dsa_area *area, size_t size, int flags);
extern void dsa_free(dsa_area *area, dsa_pointer dp);
extern void *dsa_get_address(dsa_area *area, dsa_pointer dp);
+extern size_t dsa_get_total_size(dsa_area *area);
extern void dsa_trim(dsa_area *area);
extern void dsa_dump(dsa_area *area);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index c629cbe383..9659eb85d7 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -28,6 +28,7 @@ SUBDIRS = \
test_pg_db_role_setting \
test_pg_dump \
test_predtest \
+ test_radixtree \
test_rbtree \
test_regex \
test_rls_hooks \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 1baa6b558d..232cbdac80 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -24,6 +24,7 @@ subdir('test_parser')
subdir('test_pg_db_role_setting')
subdir('test_pg_dump')
subdir('test_predtest')
+subdir('test_radixtree')
subdir('test_rbtree')
subdir('test_regex')
subdir('test_rls_hooks')
diff --git a/src/test/modules/test_radixtree/.gitignore b/src/test/modules/test_radixtree/.gitignore
new file mode 100644
index 0000000000..5dcb3ff972
--- /dev/null
+++ b/src/test/modules/test_radixtree/.gitignore
@@ -0,0 +1,4 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
diff --git a/src/test/modules/test_radixtree/Makefile b/src/test/modules/test_radixtree/Makefile
new file mode 100644
index 0000000000..da06b93da3
--- /dev/null
+++ b/src/test/modules/test_radixtree/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_radixtree/Makefile
+
+MODULE_big = test_radixtree
+OBJS = \
+ $(WIN32RES) \
+ test_radixtree.o
+PGFILEDESC = "test_radixtree - test code for src/backend/lib/radixtree.c"
+
+EXTENSION = test_radixtree
+DATA = test_radixtree--1.0.sql
+
+REGRESS = test_radixtree
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_radixtree
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_radixtree/README b/src/test/modules/test_radixtree/README
new file mode 100644
index 0000000000..a8b271869a
--- /dev/null
+++ b/src/test/modules/test_radixtree/README
@@ -0,0 +1,7 @@
+test_integerset contains unit tests for testing the integer set implementation
+in src/backend/lib/integerset.c.
+
+The tests verify the correctness of the implementation, but they can also be
+used as a micro-benchmark. If you set the 'intset_test_stats' flag in
+test_integerset.c, the tests will print extra information about execution time
+and memory usage.
diff --git a/src/test/modules/test_radixtree/expected/test_radixtree.out b/src/test/modules/test_radixtree/expected/test_radixtree.out
new file mode 100644
index 0000000000..ce645cb8b5
--- /dev/null
+++ b/src/test/modules/test_radixtree/expected/test_radixtree.out
@@ -0,0 +1,36 @@
+CREATE EXTENSION test_radixtree;
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
+NOTICE: testing basic operations with leaf node 4
+NOTICE: testing basic operations with inner node 4
+NOTICE: testing basic operations with leaf node 32
+NOTICE: testing basic operations with inner node 32
+NOTICE: testing basic operations with leaf node 125
+NOTICE: testing basic operations with inner node 125
+NOTICE: testing basic operations with leaf node 256
+NOTICE: testing basic operations with inner node 256
+NOTICE: testing radix tree node types with shift "0"
+NOTICE: testing radix tree node types with shift "8"
+NOTICE: testing radix tree node types with shift "16"
+NOTICE: testing radix tree node types with shift "24"
+NOTICE: testing radix tree node types with shift "32"
+NOTICE: testing radix tree node types with shift "40"
+NOTICE: testing radix tree node types with shift "48"
+NOTICE: testing radix tree node types with shift "56"
+NOTICE: testing radix tree with pattern "all ones"
+NOTICE: testing radix tree with pattern "alternating bits"
+NOTICE: testing radix tree with pattern "clusters of ten"
+NOTICE: testing radix tree with pattern "clusters of hundred"
+NOTICE: testing radix tree with pattern "one-every-64k"
+NOTICE: testing radix tree with pattern "sparse"
+NOTICE: testing radix tree with pattern "single values, distance > 2^32"
+NOTICE: testing radix tree with pattern "clusters, distance > 2^32"
+NOTICE: testing radix tree with pattern "clusters, distance > 2^60"
+ test_radixtree
+----------------
+
+(1 row)
+
diff --git a/src/test/modules/test_radixtree/meson.build b/src/test/modules/test_radixtree/meson.build
new file mode 100644
index 0000000000..6add06bbdb
--- /dev/null
+++ b/src/test/modules/test_radixtree/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_radixtree_sources = files(
+ 'test_radixtree.c',
+)
+
+if host_system == 'windows'
+ test_radixtree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_radixtree',
+ '--FILEDESC', 'test_radixtree - test code for src/include//lib/radixtree.h',])
+endif
+
+test_radixtree = shared_module('test_radixtree',
+ test_radixtree_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += test_radixtree
+
+install_data(
+ 'test_radixtree.control',
+ 'test_radixtree--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'test_radixtree',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'test_radixtree',
+ ],
+ },
+}
diff --git a/src/test/modules/test_radixtree/sql/test_radixtree.sql b/src/test/modules/test_radixtree/sql/test_radixtree.sql
new file mode 100644
index 0000000000..41ece5e9f5
--- /dev/null
+++ b/src/test/modules/test_radixtree/sql/test_radixtree.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_radixtree;
+
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
diff --git a/src/test/modules/test_radixtree/test_radixtree--1.0.sql b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
new file mode 100644
index 0000000000..074a5a7ea7
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_radixtree/test_radixtree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_radixtree" to load this file. \quit
+
+CREATE FUNCTION test_radixtree()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
new file mode 100644
index 0000000000..afe53382f3
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -0,0 +1,681 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_radixtree.c
+ * Test radixtree set data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_radixtree/test_radixtree.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "storage/lwlock.h"
+#include "utils/memutils.h"
+#include "utils/timestamp.h"
+
+#define UINT64_HEX_FORMAT "%" INT64_MODIFIER "X"
+
+/*
+ * The tests pass with uint32, but build with warnings because the string
+ * format expects uint64.
+ */
+typedef uint64 TestValueType;
+
+/*
+ * If you enable this, the "pattern" tests will print information about
+ * how long populating, probing, and iterating the test set takes, and
+ * how much memory the test set consumed. That can be used as
+ * micro-benchmark of various operations and input patterns (you might
+ * want to increase the number of values used in each of the test, if
+ * you do that, to reduce noise).
+ *
+ * The information is printed to the server's stderr, mostly because
+ * that's where MemoryContextStats() output goes.
+ */
+static const bool rt_test_stats = false;
+
+static int rt_node_kind_fanouts[] = {
+ 0,
+ 4, /* RT_NODE_KIND_4 */
+ 32, /* RT_NODE_KIND_32 */
+ 125, /* RT_NODE_KIND_125 */
+ 256 /* RT_NODE_KIND_256 */
+};
+/*
+ * A struct to define a pattern of integers, for use with the test_pattern()
+ * function.
+ */
+typedef struct
+{
+ char *test_name; /* short name of the test, for humans */
+ char *pattern_str; /* a bit pattern */
+ uint64 spacing; /* pattern repeats at this interval */
+ uint64 num_values; /* number of integers to set in total */
+} test_spec;
+
+/* Test patterns borrowed from test_integerset.c */
+static const test_spec test_specs[] = {
+ {
+ "all ones", "1111111111",
+ 10, 1000000
+ },
+ {
+ "alternating bits", "0101010101",
+ 10, 1000000
+ },
+ {
+ "clusters of ten", "1111111111",
+ 10000, 1000000
+ },
+ {
+ "clusters of hundred",
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111",
+ 10000, 1000000
+ },
+ {
+ "one-every-64k", "1",
+ 65536, 1000000
+ },
+ {
+ "sparse", "100000000000000000000000000000001",
+ 10000000, 1000000
+ },
+ {
+ "single values, distance > 2^32", "1",
+ UINT64CONST(10000000000), 100000
+ },
+ {
+ "clusters, distance > 2^32", "10101010",
+ UINT64CONST(10000000000), 1000000
+ },
+ {
+ "clusters, distance > 2^60", "10101010",
+ UINT64CONST(2000000000000000000),
+ 23 /* can't be much higher than this, or we
+ * overflow uint64 */
+ }
+};
+
+/* define the radix tree implementation to test */
+#define RT_PREFIX rt
+#define RT_SCOPE
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE TestValueType
+/* #define RT_SHMEM */
+#include "lib/radixtree.h"
+
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+ return tree->ctl->num_keys;
+}
+
+PG_MODULE_MAGIC;
+
+PG_FUNCTION_INFO_V1(test_radixtree);
+
+static void
+test_empty(void)
+{
+ rt_radix_tree *radixtree;
+ rt_iter *iter;
+ TestValueType dummy;
+ uint64 key;
+ TestValueType val;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
+ dsa = dsa_create(tranche_id);
+
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ if (rt_search(radixtree, 0, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_search(radixtree, 1, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_search(radixtree, PG_UINT64_MAX, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_delete(radixtree, 0))
+ elog(ERROR, "rt_delete on empty tree returned true");
+
+ if (rt_num_entries(radixtree) != 0)
+ elog(ERROR, "rt_num_entries on empty tree return non-zero");
+
+ iter = rt_begin_iterate(radixtree);
+
+ if (rt_iterate_next(iter, &key, &val))
+ elog(ERROR, "rt_itereate_next on empty tree returned true");
+
+ rt_end_iterate(iter);
+
+ rt_free(radixtree);
+
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+static void
+test_basic(int children, bool test_inner)
+{
+ rt_radix_tree *radixtree;
+ uint64 *keys;
+ int shift = test_inner ? 8 : 0;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing basic operations with %s node %d",
+ test_inner ? "inner" : "leaf", children);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ /* prepare keys in order like 1, 32, 2, 31, 2, ... */
+ keys = palloc(sizeof(uint64) * children);
+ for (int i = 0; i < children; i++)
+ {
+ if (i % 2 == 0)
+ keys[i] = (uint64) ((i / 2) + 1) << shift;
+ else
+ keys[i] = (uint64) (children - (i / 2)) << shift;
+ }
+
+ /* insert keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (rt_set(radixtree, keys[i], (TestValueType*) &keys[i]))
+ elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+ }
+
+ /* look up keys */
+ for (int i = 0; i < children; i++)
+ {
+ TestValueType value;
+
+ if (!rt_search(radixtree, keys[i], &value))
+ elog(ERROR, "could not find key 0x" UINT64_HEX_FORMAT, keys[i]);
+ if (value != (TestValueType) keys[i])
+ elog(ERROR, "rt_search returned 0x" UINT64_HEX_FORMAT ", expected " UINT64_HEX_FORMAT,
+ value, (TestValueType) keys[i]);
+ }
+
+ /* update keys */
+ for (int i = 0; i < children; i++)
+ {
+ TestValueType update = keys[i] + 1;
+ if (!rt_set(radixtree, keys[i], (TestValueType*) &update))
+ elog(ERROR, "could not update key 0x" UINT64_HEX_FORMAT, keys[i]);
+ }
+
+ /* repeat deleting and inserting keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (!rt_delete(radixtree, keys[i]))
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, keys[i]);
+ if (rt_set(radixtree, keys[i], (TestValueType*) &keys[i]))
+ elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+ }
+
+ pfree(keys);
+ rt_free(radixtree);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Check if keys from start to end with the shift exist in the tree.
+ */
+static void
+check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end,
+ int incr)
+{
+ for (int i = start; i < end; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ TestValueType val;
+
+ if (!rt_search(radixtree, key, &val))
+ elog(ERROR, "key 0x" UINT64_HEX_FORMAT " is not found on node-%d",
+ key, end);
+ if (val != (TestValueType) key)
+ elog(ERROR, "rt_search with key 0x" UINT64_HEX_FORMAT " returns 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+ key, val, key);
+ }
+}
+
+static void
+test_node_types_insert(rt_radix_tree *radixtree, uint8 shift, bool insert_asc)
+{
+ uint64 num_entries;
+ int ninserted = 0;
+ int start = insert_asc ? 0 : 256;
+ int incr = insert_asc ? 1 : -1;
+ int end = insert_asc ? 256 : 0;
+ int node_kind_idx = 1;
+
+ for (int i = start; i != end; i += incr)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_set(radixtree, key, (TestValueType*) &key);
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " is found", key);
+
+ /*
+ * After filling all slots in each node type, check if the values
+ * are stored properly.
+ */
+ if (ninserted == rt_node_kind_fanouts[node_kind_idx] - 1)
+ {
+ int check_start = insert_asc
+ ? rt_node_kind_fanouts[node_kind_idx - 1]
+ : rt_node_kind_fanouts[node_kind_idx];
+ int check_end = insert_asc
+ ? rt_node_kind_fanouts[node_kind_idx]
+ : rt_node_kind_fanouts[node_kind_idx - 1];
+
+ check_search_on_node(radixtree, shift, check_start, check_end, incr);
+ node_kind_idx++;
+ }
+
+ ninserted++;
+ }
+
+ num_entries = rt_num_entries(radixtree);
+
+ if (num_entries != 256)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+static void
+test_node_types_delete(rt_radix_tree *radixtree, uint8 shift)
+{
+ uint64 num_entries;
+
+ for (int i = 0; i < 256; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_delete(radixtree, key);
+
+ if (!found)
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, key);
+ }
+
+ num_entries = rt_num_entries(radixtree);
+
+ /* The tree must be empty */
+ if (num_entries != 0)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+/*
+ * Test for inserting and deleting key-value pairs to each node type at the given shift
+ * level.
+ */
+static void
+test_node_types(uint8 shift)
+{
+ rt_radix_tree *radixtree;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing radix tree node types with shift \"%d\"", shift);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ /*
+ * Insert and search entries for every node type at the 'shift' level,
+ * then delete all entries to make it empty, and insert and search entries
+ * again.
+ */
+ test_node_types_insert(radixtree, shift, true);
+ test_node_types_delete(radixtree, shift);
+ test_node_types_insert(radixtree, shift, false);
+
+ rt_free(radixtree);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Test with a repeating pattern, defined by the 'spec'.
+ */
+static void
+test_pattern(const test_spec * spec)
+{
+ rt_radix_tree *radixtree;
+ rt_iter *iter;
+ MemoryContext radixtree_ctx;
+ TimestampTz starttime;
+ TimestampTz endtime;
+ uint64 n;
+ uint64 last_int;
+ uint64 ndeleted;
+ uint64 nbefore;
+ uint64 nafter;
+ int patternlen;
+ uint64 *pattern_values;
+ uint64 pattern_num_values;
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing radix tree with pattern \"%s\"", spec->test_name);
+ if (rt_test_stats)
+ fprintf(stderr, "-----\ntesting radix tree with pattern \"%s\"\n", spec->test_name);
+
+ /* Pre-process the pattern, creating an array of integers from it. */
+ patternlen = strlen(spec->pattern_str);
+ pattern_values = palloc(patternlen * sizeof(uint64));
+ pattern_num_values = 0;
+ for (int i = 0; i < patternlen; i++)
+ {
+ if (spec->pattern_str[i] == '1')
+ pattern_values[pattern_num_values++] = i;
+ }
+
+ /*
+ * Allocate the radix tree.
+ *
+ * Allocate it in a separate memory context, so that we can print its
+ * memory usage easily.
+ */
+ radixtree_ctx = AllocSetContextCreate(CurrentMemoryContext,
+ "radixtree test",
+ ALLOCSET_SMALL_SIZES);
+ MemoryContextSetIdentifier(radixtree_ctx, spec->test_name);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(radixtree_ctx, dsa, tranche_id);
+#else
+ radixtree = rt_create(radixtree_ctx);
+#endif
+
+
+ /*
+ * Add values to the set.
+ */
+ starttime = GetCurrentTimestamp();
+
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ uint64 x = 0;
+
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ bool found;
+
+ x = last_int + pattern_values[i];
+
+ found = rt_set(radixtree, x, (TestValueType*) &x);
+
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " found", x);
+
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+
+ endtime = GetCurrentTimestamp();
+
+ if (rt_test_stats)
+ fprintf(stderr, "added " UINT64_FORMAT " values in %d ms\n",
+ spec->num_values, (int) (endtime - starttime) / 1000);
+
+ /*
+ * Print stats on the amount of memory used.
+ *
+ * We print the usage reported by rt_memory_usage(), as well as the stats
+ * from the memory context. They should be in the same ballpark, but it's
+ * hard to automate testing that, so if you're making changes to the
+ * implementation, just observe that manually.
+ */
+ if (rt_test_stats)
+ {
+ uint64 mem_usage;
+
+ /*
+ * Also print memory usage as reported by rt_memory_usage(). It
+ * should be in the same ballpark as the usage reported by
+ * MemoryContextStats().
+ */
+ mem_usage = rt_memory_usage(radixtree);
+ fprintf(stderr, "rt_memory_usage() reported " UINT64_FORMAT " (%0.2f bytes / integer)\n",
+ mem_usage, (double) mem_usage / spec->num_values);
+
+ MemoryContextStats(radixtree_ctx);
+ }
+
+ /* Check that rt_num_entries works */
+ n = rt_num_entries(radixtree);
+ if (n != spec->num_values)
+ elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT, n, spec->num_values);
+
+ /*
+ * Test random-access probes with rt_search()
+ */
+ starttime = GetCurrentTimestamp();
+
+ for (n = 0; n < 100000; n++)
+ {
+ bool found;
+ bool expected;
+ uint64 x;
+ TestValueType v;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit values that are actually in the set.
+ */
+ x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+ /* Do we expect this value to be present in the set? */
+ if (x >= last_int)
+ expected = false;
+ else
+ {
+ uint64 idx = x % spec->spacing;
+
+ if (idx >= patternlen)
+ expected = false;
+ else if (spec->pattern_str[idx] == '1')
+ expected = true;
+ else
+ expected = false;
+ }
+
+ /* Is it present according to rt_search() ? */
+ found = rt_search(radixtree, x, &v);
+
+ if (found != expected)
+ elog(ERROR, "mismatch at 0x" UINT64_HEX_FORMAT ": %d vs %d", x, found, expected);
+ if (found && (v != (TestValueType) x))
+ elog(ERROR, "found 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+ v, x);
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "probed " UINT64_FORMAT " values in %d ms\n",
+ n, (int) (endtime - starttime) / 1000);
+
+ /*
+ * Test iterator
+ */
+ starttime = GetCurrentTimestamp();
+
+ iter = rt_begin_iterate(radixtree);
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ uint64 expected = last_int + pattern_values[i];
+ uint64 x;
+ TestValueType val;
+
+ if (!rt_iterate_next(iter, &x, &val))
+ break;
+
+ if (x != expected)
+ elog(ERROR,
+ "iterate returned wrong key; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d",
+ x, expected, i);
+ if (val != (TestValueType) expected)
+ elog(ERROR,
+ "iterate returned wrong value; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d", x, expected, i);
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "iterated " UINT64_FORMAT " values in %d ms\n",
+ n, (int) (endtime - starttime) / 1000);
+
+ rt_end_iterate(iter);
+
+ if (n < spec->num_values)
+ elog(ERROR, "iterator stopped short after " UINT64_FORMAT " entries, expected " UINT64_FORMAT, n, spec->num_values);
+ if (n > spec->num_values)
+ elog(ERROR, "iterator returned " UINT64_FORMAT " entries, " UINT64_FORMAT " was expected", n, spec->num_values);
+
+ /*
+ * Test random-access probes with rt_delete()
+ */
+ starttime = GetCurrentTimestamp();
+
+ nbefore = rt_num_entries(radixtree);
+ ndeleted = 0;
+ for (n = 0; n < 1; n++)
+ {
+ bool found;
+ uint64 x;
+ TestValueType v;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit values that are actually in the set.
+ */
+ x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+ /* Is it present according to rt_search() ? */
+ found = rt_search(radixtree, x, &v);
+
+ if (!found)
+ continue;
+
+ /* If the key is found, delete it and check again */
+ if (!rt_delete(radixtree, x))
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, x);
+ if (rt_search(radixtree, x, &v))
+ elog(ERROR, "found deleted key 0x" UINT64_HEX_FORMAT, x);
+ if (rt_delete(radixtree, x))
+ elog(ERROR, "deleted already-deleted key 0x" UINT64_HEX_FORMAT, x);
+
+ ndeleted++;
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "deleted " UINT64_FORMAT " values in %d ms\n",
+ ndeleted, (int) (endtime - starttime) / 1000);
+
+ nafter = rt_num_entries(radixtree);
+
+ /* Check that rt_num_entries works */
+ if ((nbefore - ndeleted) != nafter)
+ elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT "after " UINT64_FORMAT " deletion",
+ nafter, (nbefore - ndeleted), ndeleted);
+
+ rt_free(radixtree);
+ MemoryContextDelete(radixtree_ctx);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+Datum
+test_radixtree(PG_FUNCTION_ARGS)
+{
+ test_empty();
+
+ for (int i = 1; i < lengthof(rt_node_kind_fanouts); i++)
+ {
+ test_basic(rt_node_kind_fanouts[i], false);
+ test_basic(rt_node_kind_fanouts[i], true);
+ }
+
+ for (int shift = 0; shift <= (64 - 8); shift += 8)
+ test_node_types(shift);
+
+ /* Test different test patterns, with lots of entries */
+ for (int i = 0; i < lengthof(test_specs); i++)
+ test_pattern(&test_specs[i]);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_radixtree/test_radixtree.control b/src/test/modules/test_radixtree/test_radixtree.control
new file mode 100644
index 0000000000..e53f2a3e0c
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.control
@@ -0,0 +1,4 @@
+comment = 'Test code for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/test_radixtree'
+relocatable = true
diff --git a/src/tools/pginclude/cpluspluscheck b/src/tools/pginclude/cpluspluscheck
index b0e9aa99a2..2f72d5ed4b 100755
--- a/src/tools/pginclude/cpluspluscheck
+++ b/src/tools/pginclude/cpluspluscheck
@@ -101,6 +101,12 @@ do
test "$f" = src/include/nodes/nodetags.h && continue
test "$f" = src/backend/nodes/nodetags.h && continue
+ # radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+ test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+ test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+ test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+ test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
# These files are not meant to be included standalone, because
# they contain lists that might have multiple use-cases.
test "$f" = src/include/access/rmgrlist.h && continue
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index 8dee1b5670..133313255c 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -96,6 +96,12 @@ do
test "$f" = src/include/nodes/nodetags.h && continue
test "$f" = src/backend/nodes/nodetags.h && continue
+ # radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+ test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+ test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+ test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+ test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
# These files are not meant to be included standalone, because
# they contain lists that might have multiple use-cases.
test "$f" = src/include/access/rmgrlist.h && continue
--
2.31.1
[application/octet-stream] v31-0006-Use-TIDStore-for-storing-dead-tuple-TID-during-l.patch (48.1K, ../../CAD21AoDBmD5q=eO+K=gyuVt53XvwpJ2dgxPwrtZ-eVOjVmtJjg@mail.gmail.com/10-v31-0006-Use-TIDStore-for-storing-dead-tuple-TID-during-l.patch)
download | inline diff:
From b3ac3b456aa1448f3e959674f16bed18630266be Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Tue, 7 Feb 2023 17:19:29 +0700
Subject: [PATCH v31 06/14] Use TIDStore for storing dead tuple TID during lazy
vacuum
Previously, we used an array of ItemPointerData to store dead tuple
TIDs, which was not space efficient and slow to lookup. Also, we had
the 1GB limit on its size.
Now we use TIDStore to store dead tuple TIDs. Since the TIDStore,
backed by the radix tree, incrementaly allocates the memory, we get
rid of the 1GB limit.
Since we are no longer able to exactly estimate the maximum number of
TIDs can be stored the pg_stat_progress_vacuum shows the progress
information based on the amount of memory in bytes. The column names
are also changed to max_dead_tuple_bytes and num_dead_tuple_bytes.
In addition, since the TIDStore use the radix tree internally, the
minimum amount of memory required by TIDStore is 1MB, the inital DSA
segment size. Due to that, we increase the minimum value of
maintenance_work_mem (also autovacuum_work_mem) from 1MB to 2MB.
XXX: needs to bump catalog version
---
doc/src/sgml/monitoring.sgml | 8 +-
src/backend/access/heap/vacuumlazy.c | 278 ++++++++-------------
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/vacuum.c | 78 +-----
src/backend/commands/vacuumparallel.c | 73 +++---
src/backend/postmaster/autovacuum.c | 6 +-
src/backend/storage/lmgr/lwlock.c | 2 +
src/backend/utils/misc/guc_tables.c | 2 +-
src/include/commands/progress.h | 4 +-
src/include/commands/vacuum.h | 25 +-
src/include/storage/lwlock.h | 1 +
src/test/regress/expected/cluster.out | 2 +-
src/test/regress/expected/create_index.out | 2 +-
src/test/regress/expected/rules.out | 4 +-
src/test/regress/sql/cluster.sql | 2 +-
src/test/regress/sql/create_index.sql | 2 +-
16 files changed, 177 insertions(+), 314 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 97d588b1d8..47b346d36c 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -7170,10 +7170,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>max_dead_tuples</structfield> <type>bigint</type>
+ <structfield>max_dead_tuple_bytes</structfield> <type>bigint</type>
</para>
<para>
- Number of dead tuples that we can store before needing to perform
+ Amount of dead tuple data that we can store before needing to perform
an index vacuum cycle, based on
<xref linkend="guc-maintenance-work-mem"/>.
</para></entry>
@@ -7181,10 +7181,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>num_dead_tuples</structfield> <type>bigint</type>
+ <structfield>num_dead_tuple_bytes</structfield> <type>bigint</type>
</para>
<para>
- Number of dead tuples collected since the last index vacuum cycle.
+ Amount of dead tuple data collected since the last index vacuum cycle.
</para></entry>
</row>
</tbody>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 8f14cf85f3..b4e40423a8 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3,18 +3,18 @@
* vacuumlazy.c
* Concurrent ("lazy") vacuuming.
*
- * The major space usage for vacuuming is storage for the array of dead TIDs
+ * The major space usage for vacuuming is TidStore, a storage for dead TIDs
* that are to be removed from indexes. We want to ensure we can vacuum even
* the very largest relations with finite memory space usage. To do that, we
- * set upper bounds on the number of TIDs we can keep track of at once.
+ * set upper bounds on the maximum memory that can be used for keeping track
+ * of dead TIDs at once.
*
* We are willing to use at most maintenance_work_mem (or perhaps
* autovacuum_work_mem) memory space to keep track of dead TIDs. We initially
- * allocate an array of TIDs of that size, with an upper limit that depends on
- * table size (this limit ensures we don't allocate a huge area uselessly for
- * vacuuming small tables). If the array threatens to overflow, we must call
- * lazy_vacuum to vacuum indexes (and to vacuum the pages that we've pruned).
- * This frees up the memory space dedicated to storing dead TIDs.
+ * create a TidStore with the maximum bytes that can be used by the TidStore.
+ * If the TidStore is full, we must call lazy_vacuum to vacuum indexes (and to
+ * vacuum the pages that we've pruned). This frees up the memory space dedicated
+ * to storing dead TIDs.
*
* In practice VACUUM will often complete its initial pass over the target
* heap relation without ever running out of space to store TIDs. This means
@@ -40,6 +40,7 @@
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
#include "access/multixact.h"
+#include "access/tidstore.h"
#include "access/transam.h"
#include "access/visibilitymap.h"
#include "access/xact.h"
@@ -188,7 +189,7 @@ typedef struct LVRelState
* lazy_vacuum_heap_rel, which marks the same LP_DEAD line pointers as
* LP_UNUSED during second heap pass.
*/
- VacDeadItems *dead_items; /* TIDs whose index tuples we'll delete */
+ TidStore *dead_items; /* TIDs whose index tuples we'll delete */
BlockNumber rel_pages; /* total number of pages */
BlockNumber scanned_pages; /* # pages examined (not skipped via VM) */
BlockNumber removed_pages; /* # pages removed by relation truncation */
@@ -220,11 +221,14 @@ typedef struct LVRelState
typedef struct LVPagePruneState
{
bool hastup; /* Page prevents rel truncation? */
- bool has_lpdead_items; /* includes existing LP_DEAD items */
+
+ /* collected offsets of LP_DEAD items including existing ones */
+ OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
+ int num_offsets;
/*
* State describes the proper VM bit states to set for the page following
- * pruning and freezing. all_visible implies !has_lpdead_items, but don't
+ * pruning and freezing. all_visible implies num_offsets == 0, but don't
* trust all_frozen result unless all_visible is also set to true.
*/
bool all_visible; /* Every item visible to all? */
@@ -259,8 +263,9 @@ static bool lazy_scan_noprune(LVRelState *vacrel, Buffer buf,
static void lazy_vacuum(LVRelState *vacrel);
static bool lazy_vacuum_all_indexes(LVRelState *vacrel);
static void lazy_vacuum_heap_rel(LVRelState *vacrel);
-static int lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
- Buffer buffer, int index, Buffer vmbuffer);
+static void lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
+ OffsetNumber *offsets, int num_offsets,
+ Buffer buffer, Buffer vmbuffer);
static bool lazy_check_wraparound_failsafe(LVRelState *vacrel);
static void lazy_cleanup_all_indexes(LVRelState *vacrel);
static IndexBulkDeleteResult *lazy_vacuum_one_index(Relation indrel,
@@ -487,11 +492,11 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
}
/*
- * Allocate dead_items array memory using dead_items_alloc. This handles
- * parallel VACUUM initialization as part of allocating shared memory
- * space used for dead_items. (But do a failsafe precheck first, to
- * ensure that parallel VACUUM won't be attempted at all when relfrozenxid
- * is already dangerously old.)
+ * Allocate dead_items memory using dead_items_alloc. This handles parallel
+ * VACUUM initialization as part of allocating shared memory space used for
+ * dead_items. (But do a failsafe precheck first, to ensure that parallel
+ * VACUUM won't be attempted at all when relfrozenxid is already dangerously
+ * old.)
*/
lazy_check_wraparound_failsafe(vacrel);
dead_items_alloc(vacrel, params->nworkers);
@@ -797,7 +802,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* have collected the TIDs whose index tuples need to be removed.
*
* Finally, invokes lazy_vacuum_heap_rel to vacuum heap pages, which
- * largely consists of marking LP_DEAD items (from collected TID array)
+ * largely consists of marking LP_DEAD items (from vacrel->dead_items)
* as LP_UNUSED. This has to happen in a second, final pass over the
* heap, to preserve a basic invariant that all index AMs rely on: no
* extant index tuple can ever be allowed to contain a TID that points to
@@ -825,21 +830,21 @@ lazy_scan_heap(LVRelState *vacrel)
blkno,
next_unskippable_block,
next_fsm_block_to_vacuum = 0;
- VacDeadItems *dead_items = vacrel->dead_items;
+ TidStore *dead_items = vacrel->dead_items;
Buffer vmbuffer = InvalidBuffer;
bool next_unskippable_allvis,
skipping_current_range;
const int initprog_index[] = {
PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_TOTAL_HEAP_BLKS,
- PROGRESS_VACUUM_MAX_DEAD_TUPLES
+ PROGRESS_VACUUM_MAX_DEAD_TUPLE_BYTES
};
int64 initprog_val[3];
/* Report that we're scanning the heap, advertising total # of blocks */
initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
initprog_val[1] = rel_pages;
- initprog_val[2] = dead_items->max_items;
+ initprog_val[2] = tidstore_max_memory(vacrel->dead_items);
pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
/* Set up an initial range of skippable blocks using the visibility map */
@@ -906,8 +911,7 @@ lazy_scan_heap(LVRelState *vacrel)
* dead_items TIDs, pause and do a cycle of vacuuming before we tackle
* this page.
*/
- Assert(dead_items->max_items >= MaxHeapTuplesPerPage);
- if (dead_items->max_items - dead_items->num_items < MaxHeapTuplesPerPage)
+ if (tidstore_is_full(vacrel->dead_items))
{
/*
* Before beginning index vacuuming, we release any pin we may
@@ -969,7 +973,7 @@ lazy_scan_heap(LVRelState *vacrel)
continue;
}
- /* Collect LP_DEAD items in dead_items array, count tuples */
+ /* Collect LP_DEAD items in dead_items, count tuples */
if (lazy_scan_noprune(vacrel, buf, blkno, page, &hastup,
&recordfreespace))
{
@@ -1011,14 +1015,14 @@ lazy_scan_heap(LVRelState *vacrel)
* Prune, freeze, and count tuples.
*
* Accumulates details of remaining LP_DEAD line pointers on page in
- * dead_items array. This includes LP_DEAD line pointers that we
- * pruned ourselves, as well as existing LP_DEAD line pointers that
- * were pruned some time earlier. Also considers freezing XIDs in the
- * tuple headers of remaining items with storage.
+ * dead_items. This includes LP_DEAD line pointers that we pruned
+ * ourselves, as well as existing LP_DEAD line pointers that were pruned
+ * some time earlier. Also considers freezing XIDs in the tuple headers
+ * of remaining items with storage.
*/
lazy_scan_prune(vacrel, buf, blkno, page, &prunestate);
- Assert(!prunestate.all_visible || !prunestate.has_lpdead_items);
+ Assert(!prunestate.all_visible || (prunestate.num_offsets == 0));
/* Remember the location of the last page with nonremovable tuples */
if (prunestate.hastup)
@@ -1034,14 +1038,12 @@ lazy_scan_heap(LVRelState *vacrel)
* performed here can be thought of as the one-pass equivalent of
* a call to lazy_vacuum().
*/
- if (prunestate.has_lpdead_items)
+ if (prunestate.num_offsets > 0)
{
Size freespace;
- lazy_vacuum_heap_page(vacrel, blkno, buf, 0, vmbuffer);
-
- /* Forget the LP_DEAD items that we just vacuumed */
- dead_items->num_items = 0;
+ lazy_vacuum_heap_page(vacrel, blkno, prunestate.deadoffsets,
+ prunestate.num_offsets, buf, vmbuffer);
/*
* Periodically perform FSM vacuuming to make newly-freed
@@ -1078,7 +1080,16 @@ lazy_scan_heap(LVRelState *vacrel)
* with prunestate-driven visibility map and FSM steps (just like
* the two-pass strategy).
*/
- Assert(dead_items->num_items == 0);
+ Assert(tidstore_num_tids(dead_items) == 0);
+ }
+ else if (prunestate.num_offsets > 0)
+ {
+ /* Save details of the LP_DEAD items from the page in dead_items */
+ tidstore_add_tids(dead_items, blkno, prunestate.deadoffsets,
+ prunestate.num_offsets);
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
+ tidstore_memory_usage(dead_items));
}
/*
@@ -1145,7 +1156,7 @@ lazy_scan_heap(LVRelState *vacrel)
* There should never be LP_DEAD items on a page with PD_ALL_VISIBLE
* set, however.
*/
- else if (prunestate.has_lpdead_items && PageIsAllVisible(page))
+ else if ((prunestate.num_offsets > 0) && PageIsAllVisible(page))
{
elog(WARNING, "page containing LP_DEAD items is marked as all-visible in relation \"%s\" page %u",
vacrel->relname, blkno);
@@ -1193,7 +1204,7 @@ lazy_scan_heap(LVRelState *vacrel)
* Final steps for block: drop cleanup lock, record free space in the
* FSM
*/
- if (prunestate.has_lpdead_items && vacrel->do_index_vacuuming)
+ if ((prunestate.num_offsets > 0) && vacrel->do_index_vacuuming)
{
/*
* Wait until lazy_vacuum_heap_rel() to save free space. This
@@ -1249,7 +1260,7 @@ lazy_scan_heap(LVRelState *vacrel)
* Do index vacuuming (call each index's ambulkdelete routine), then do
* related heap vacuuming
*/
- if (dead_items->num_items > 0)
+ if (tidstore_num_tids(dead_items) > 0)
lazy_vacuum(vacrel);
/*
@@ -1524,9 +1535,9 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* The approach we take now is to restart pruning when the race condition is
* detected. This allows heap_page_prune() to prune the tuples inserted by
* the now-aborted transaction. This is a little crude, but it guarantees
- * that any items that make it into the dead_items array are simple LP_DEAD
- * line pointers, and that every remaining item with tuple storage is
- * considered as a candidate for freezing.
+ * that any items that make it into the dead_items are simple LP_DEAD line
+ * pointers, and that every remaining item with tuple storage is considered
+ * as a candidate for freezing.
*/
static void
lazy_scan_prune(LVRelState *vacrel,
@@ -1543,13 +1554,11 @@ lazy_scan_prune(LVRelState *vacrel,
HTSV_Result res;
int tuples_deleted,
tuples_frozen,
- lpdead_items,
live_tuples,
recently_dead_tuples;
int nnewlpdead;
HeapPageFreeze pagefrz;
int64 fpi_before = pgWalUsage.wal_fpi;
- OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
@@ -1571,7 +1580,6 @@ retry:
pagefrz.NoFreezePageRelminMxid = vacrel->NewRelminMxid;
tuples_deleted = 0;
tuples_frozen = 0;
- lpdead_items = 0;
live_tuples = 0;
recently_dead_tuples = 0;
@@ -1580,9 +1588,9 @@ retry:
*
* We count tuples removed by the pruning step as tuples_deleted. Its
* final value can be thought of as the number of tuples that have been
- * deleted from the table. It should not be confused with lpdead_items;
- * lpdead_items's final value can be thought of as the number of tuples
- * that were deleted from indexes.
+ * deleted from the table. It should not be confused with
+ * prunestate->deadoffsets; prunestate->deadoffsets's final value can
+ * be thought of as the number of tuples that were deleted from indexes.
*/
tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest,
InvalidTransactionId, 0, &nnewlpdead,
@@ -1593,7 +1601,7 @@ retry:
* requiring freezing among remaining tuples with storage
*/
prunestate->hastup = false;
- prunestate->has_lpdead_items = false;
+ prunestate->num_offsets = 0;
prunestate->all_visible = true;
prunestate->all_frozen = true;
prunestate->visibility_cutoff_xid = InvalidTransactionId;
@@ -1638,7 +1646,7 @@ retry:
* (This is another case where it's useful to anticipate that any
* LP_DEAD items will become LP_UNUSED during the ongoing VACUUM.)
*/
- deadoffsets[lpdead_items++] = offnum;
+ prunestate->deadoffsets[prunestate->num_offsets++] = offnum;
continue;
}
@@ -1875,7 +1883,7 @@ retry:
*/
#ifdef USE_ASSERT_CHECKING
/* Note that all_frozen value does not matter when !all_visible */
- if (prunestate->all_visible && lpdead_items == 0)
+ if (prunestate->all_visible && prunestate->num_offsets == 0)
{
TransactionId cutoff;
bool all_frozen;
@@ -1888,28 +1896,9 @@ retry:
}
#endif
- /*
- * Now save details of the LP_DEAD items from the page in vacrel
- */
- if (lpdead_items > 0)
+ if (prunestate->num_offsets > 0)
{
- VacDeadItems *dead_items = vacrel->dead_items;
- ItemPointerData tmp;
-
vacrel->lpdead_item_pages++;
- prunestate->has_lpdead_items = true;
-
- ItemPointerSetBlockNumber(&tmp, blkno);
-
- for (int i = 0; i < lpdead_items; i++)
- {
- ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
- dead_items->items[dead_items->num_items++] = tmp;
- }
-
- Assert(dead_items->num_items <= dead_items->max_items);
- pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
- dead_items->num_items);
/*
* It was convenient to ignore LP_DEAD items in all_visible earlier on
@@ -1928,7 +1917,7 @@ retry:
/* Finally, add page-local counts to whole-VACUUM counts */
vacrel->tuples_deleted += tuples_deleted;
vacrel->tuples_frozen += tuples_frozen;
- vacrel->lpdead_items += lpdead_items;
+ vacrel->lpdead_items += prunestate->num_offsets;
vacrel->live_tuples += live_tuples;
vacrel->recently_dead_tuples += recently_dead_tuples;
}
@@ -1940,7 +1929,7 @@ retry:
* lazy_scan_prune, which requires a full cleanup lock. While pruning isn't
* performed here, it's quite possible that an earlier opportunistic pruning
* operation left LP_DEAD items behind. We'll at least collect any such items
- * in the dead_items array for removal from indexes.
+ * in the dead_items for removal from indexes.
*
* For aggressive VACUUM callers, we may return false to indicate that a full
* cleanup lock is required for processing by lazy_scan_prune. This is only
@@ -2099,7 +2088,7 @@ lazy_scan_noprune(LVRelState *vacrel,
vacrel->NewRelfrozenXid = NoFreezePageRelfrozenXid;
vacrel->NewRelminMxid = NoFreezePageRelminMxid;
- /* Save any LP_DEAD items found on the page in dead_items array */
+ /* Save any LP_DEAD items found on the page in dead_items */
if (vacrel->nindexes == 0)
{
/* Using one-pass strategy (since table has no indexes) */
@@ -2129,8 +2118,7 @@ lazy_scan_noprune(LVRelState *vacrel,
}
else
{
- VacDeadItems *dead_items = vacrel->dead_items;
- ItemPointerData tmp;
+ TidStore *dead_items = vacrel->dead_items;
/*
* Page has LP_DEAD items, and so any references/TIDs that remain in
@@ -2139,17 +2127,10 @@ lazy_scan_noprune(LVRelState *vacrel,
*/
vacrel->lpdead_item_pages++;
- ItemPointerSetBlockNumber(&tmp, blkno);
+ tidstore_add_tids(dead_items, blkno, deadoffsets, lpdead_items);
- for (int i = 0; i < lpdead_items; i++)
- {
- ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
- dead_items->items[dead_items->num_items++] = tmp;
- }
-
- Assert(dead_items->num_items <= dead_items->max_items);
- pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
- dead_items->num_items);
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
+ tidstore_memory_usage(dead_items));
vacrel->lpdead_items += lpdead_items;
@@ -2198,7 +2179,7 @@ lazy_vacuum(LVRelState *vacrel)
if (!vacrel->do_index_vacuuming)
{
Assert(!vacrel->do_index_cleanup);
- vacrel->dead_items->num_items = 0;
+ tidstore_reset(vacrel->dead_items);
return;
}
@@ -2227,7 +2208,7 @@ lazy_vacuum(LVRelState *vacrel)
BlockNumber threshold;
Assert(vacrel->num_index_scans == 0);
- Assert(vacrel->lpdead_items == vacrel->dead_items->num_items);
+ Assert(vacrel->lpdead_items == tidstore_num_tids(vacrel->dead_items));
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2254,8 +2235,8 @@ lazy_vacuum(LVRelState *vacrel)
* cases then this may need to be reconsidered.
*/
threshold = (double) vacrel->rel_pages * BYPASS_THRESHOLD_PAGES;
- bypass = (vacrel->lpdead_item_pages < threshold &&
- vacrel->lpdead_items < MAXDEADITEMS(32L * 1024L * 1024L));
+ bypass = (vacrel->lpdead_item_pages < threshold) &&
+ tidstore_memory_usage(vacrel->dead_items) < (32L * 1024L * 1024L);
}
if (bypass)
@@ -2300,7 +2281,7 @@ lazy_vacuum(LVRelState *vacrel)
* Forget the LP_DEAD items that we just vacuumed (or just decided to not
* vacuum)
*/
- vacrel->dead_items->num_items = 0;
+ tidstore_reset(vacrel->dead_items);
}
/*
@@ -2373,7 +2354,7 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
* place).
*/
Assert(vacrel->num_index_scans > 0 ||
- vacrel->dead_items->num_items == vacrel->lpdead_items);
+ tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items);
Assert(allindexes || vacrel->failsafe_active);
/*
@@ -2392,9 +2373,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
/*
* lazy_vacuum_heap_rel() -- second pass over the heap for two pass strategy
*
- * This routine marks LP_DEAD items in vacrel->dead_items array as LP_UNUSED.
- * Pages that never had lazy_scan_prune record LP_DEAD items are not visited
- * at all.
+ * This routine marks LP_DEAD items in vacrel->dead_items as LP_UNUSED. Pages
+ * that never had lazy_scan_prune record LP_DEAD items are not visited at all.
*
* We may also be able to truncate the line pointer array of the heap pages we
* visit. If there is a contiguous group of LP_UNUSED items at the end of the
@@ -2410,10 +2390,11 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
static void
lazy_vacuum_heap_rel(LVRelState *vacrel)
{
- int index = 0;
BlockNumber vacuumed_pages = 0;
Buffer vmbuffer = InvalidBuffer;
LVSavedErrInfo saved_err_info;
+ TidStoreIter *iter;
+ TidStoreIterResult *result;
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2428,7 +2409,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
VACUUM_ERRCB_PHASE_VACUUM_HEAP,
InvalidBlockNumber, InvalidOffsetNumber);
- while (index < vacrel->dead_items->num_items)
+ iter = tidstore_begin_iterate(vacrel->dead_items);
+ while ((result = tidstore_iterate_next(iter)) != NULL)
{
BlockNumber blkno;
Buffer buf;
@@ -2437,7 +2419,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
vacuum_delay_point();
- blkno = ItemPointerGetBlockNumber(&vacrel->dead_items->items[index]);
+ blkno = result->blkno;
vacrel->blkno = blkno;
/*
@@ -2451,7 +2433,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
buf = ReadBufferExtended(vacrel->rel, MAIN_FORKNUM, blkno, RBM_NORMAL,
vacrel->bstrategy);
LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
- index = lazy_vacuum_heap_page(vacrel, blkno, buf, index, vmbuffer);
+ lazy_vacuum_heap_page(vacrel, blkno, result->offsets, result->num_offsets,
+ buf, vmbuffer);
/* Now that we've vacuumed the page, record its available space */
page = BufferGetPage(buf);
@@ -2461,6 +2444,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
vacuumed_pages++;
}
+ tidstore_end_iterate(iter);
vacrel->blkno = InvalidBlockNumber;
if (BufferIsValid(vmbuffer))
@@ -2470,36 +2454,31 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
* We set all LP_DEAD items from the first heap pass to LP_UNUSED during
* the second heap pass. No more, no less.
*/
- Assert(index > 0);
Assert(vacrel->num_index_scans > 1 ||
- (index == vacrel->lpdead_items &&
+ (tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items &&
vacuumed_pages == vacrel->lpdead_item_pages));
ereport(DEBUG2,
- (errmsg("table \"%s\": removed %lld dead item identifiers in %u pages",
- vacrel->relname, (long long) index, vacuumed_pages)));
+ (errmsg("table \"%s\": removed " UINT64_FORMAT "dead item identifiers in %u pages",
+ vacrel->relname, tidstore_num_tids(vacrel->dead_items),
+ vacuumed_pages)));
/* Revert to the previous phase information for error traceback */
restore_vacuum_error_info(vacrel, &saved_err_info);
}
/*
- * lazy_vacuum_heap_page() -- free page's LP_DEAD items listed in the
- * vacrel->dead_items array.
+ * lazy_vacuum_heap_page() -- free page's LP_DEAD items.
*
* Caller must have an exclusive buffer lock on the buffer (though a full
* cleanup lock is also acceptable). vmbuffer must be valid and already have
* a pin on blkno's visibility map page.
- *
- * index is an offset into the vacrel->dead_items array for the first listed
- * LP_DEAD item on the page. The return value is the first index immediately
- * after all LP_DEAD items for the same page in the array.
*/
-static int
-lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
- int index, Buffer vmbuffer)
+static void
+lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
+ OffsetNumber *deadoffsets, int num_offsets, Buffer buffer,
+ Buffer vmbuffer)
{
- VacDeadItems *dead_items = vacrel->dead_items;
Page page = BufferGetPage(buffer);
OffsetNumber unused[MaxHeapTuplesPerPage];
int nunused = 0;
@@ -2518,16 +2497,11 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
START_CRIT_SECTION();
- for (; index < dead_items->num_items; index++)
+ for (int i = 0; i < num_offsets; i++)
{
- BlockNumber tblk;
- OffsetNumber toff;
ItemId itemid;
+ OffsetNumber toff = deadoffsets[i];
- tblk = ItemPointerGetBlockNumber(&dead_items->items[index]);
- if (tblk != blkno)
- break; /* past end of tuples for this block */
- toff = ItemPointerGetOffsetNumber(&dead_items->items[index]);
itemid = PageGetItemId(page, toff);
Assert(ItemIdIsDead(itemid) && !ItemIdHasStorage(itemid));
@@ -2597,7 +2571,6 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
/* Revert to the previous phase information for error traceback */
restore_vacuum_error_info(vacrel, &saved_err_info);
- return index;
}
/*
@@ -2687,8 +2660,8 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
* lazy_vacuum_one_index() -- vacuum index relation.
*
* Delete all the index tuples containing a TID collected in
- * vacrel->dead_items array. Also update running statistics.
- * Exact details depend on index AM's ambulkdelete routine.
+ * vacrel->dead_items. Also update running statistics. Exact
+ * details depend on index AM's ambulkdelete routine.
*
* reltuples is the number of heap tuples to be passed to the
* bulkdelete callback. It's always assumed to be estimated.
@@ -3094,48 +3067,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
}
/*
- * Returns the number of dead TIDs that VACUUM should allocate space to
- * store, given a heap rel of size vacrel->rel_pages, and given current
- * maintenance_work_mem setting (or current autovacuum_work_mem setting,
- * when applicable).
- *
- * See the comments at the head of this file for rationale.
- */
-static int
-dead_items_max_items(LVRelState *vacrel)
-{
- int64 max_items;
- int vac_work_mem = IsAutoVacuumWorkerProcess() &&
- autovacuum_work_mem != -1 ?
- autovacuum_work_mem : maintenance_work_mem;
-
- if (vacrel->nindexes > 0)
- {
- BlockNumber rel_pages = vacrel->rel_pages;
-
- max_items = MAXDEADITEMS(vac_work_mem * 1024L);
- max_items = Min(max_items, INT_MAX);
- max_items = Min(max_items, MAXDEADITEMS(MaxAllocSize));
-
- /* curious coding here to ensure the multiplication can't overflow */
- if ((BlockNumber) (max_items / MaxHeapTuplesPerPage) > rel_pages)
- max_items = rel_pages * MaxHeapTuplesPerPage;
-
- /* stay sane if small maintenance_work_mem */
- max_items = Max(max_items, MaxHeapTuplesPerPage);
- }
- else
- {
- /* One-pass case only stores a single heap page's TIDs at a time */
- max_items = MaxHeapTuplesPerPage;
- }
-
- return (int) max_items;
-}
-
-/*
- * Allocate dead_items (either using palloc, or in dynamic shared memory).
- * Sets dead_items in vacrel for caller.
+ * Allocate a (local or shared) TidStore for storing dead TIDs. Sets dead_items
+ * in vacrel for caller.
*
* Also handles parallel initialization as part of allocating dead_items in
* DSM when required.
@@ -3143,11 +3076,9 @@ dead_items_max_items(LVRelState *vacrel)
static void
dead_items_alloc(LVRelState *vacrel, int nworkers)
{
- VacDeadItems *dead_items;
- int max_items;
-
- max_items = dead_items_max_items(vacrel);
- Assert(max_items >= MaxHeapTuplesPerPage);
+ int vac_work_mem = IsAutoVacuumWorkerProcess() &&
+ autovacuum_work_mem != -1 ?
+ autovacuum_work_mem * 1024L : maintenance_work_mem * 1024L;
/*
* Initialize state for a parallel vacuum. As of now, only one worker can
@@ -3174,7 +3105,7 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
else
vacrel->pvs = parallel_vacuum_init(vacrel->rel, vacrel->indrels,
vacrel->nindexes, nworkers,
- max_items,
+ vac_work_mem, MaxHeapTuplesPerPage,
vacrel->verbose ? INFO : DEBUG2,
vacrel->bstrategy);
@@ -3187,11 +3118,8 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
}
/* Serial VACUUM case */
- dead_items = (VacDeadItems *) palloc(vac_max_items_to_alloc_size(max_items));
- dead_items->max_items = max_items;
- dead_items->num_items = 0;
-
- vacrel->dead_items = dead_items;
+ vacrel->dead_items = tidstore_create(vac_work_mem, MaxHeapTuplesPerPage,
+ NULL);
}
/*
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 34ca0e739f..149d41b41c 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1180,7 +1180,7 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuple_bytes, S.param7 AS dead_tuple_bytes
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 2e12baf8eb..785b825bbc 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -97,7 +97,6 @@ static bool vacuum_rel(Oid relid, RangeVar *relation, VacuumParams *params,
static double compute_parallel_delay(void);
static VacOptValue get_vacoptval_from_boolean(DefElem *def);
static bool vac_tid_reaped(ItemPointer itemptr, void *state);
-static int vac_cmp_itemptr(const void *left, const void *right);
/*
* Primary entry point for manual VACUUM and ANALYZE commands
@@ -2327,16 +2326,16 @@ get_vacoptval_from_boolean(DefElem *def)
*/
IndexBulkDeleteResult *
vac_bulkdel_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat,
- VacDeadItems *dead_items)
+ TidStore *dead_items)
{
/* Do bulk deletion */
istat = index_bulk_delete(ivinfo, istat, vac_tid_reaped,
(void *) dead_items);
ereport(ivinfo->message_level,
- (errmsg("scanned index \"%s\" to remove %d row versions",
+ (errmsg("scanned index \"%s\" to remove " UINT64_FORMAT " row versions",
RelationGetRelationName(ivinfo->index),
- dead_items->num_items)));
+ tidstore_num_tids(dead_items))));
return istat;
}
@@ -2367,82 +2366,15 @@ vac_cleanup_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat)
return istat;
}
-/*
- * Returns the total required space for VACUUM's dead_items array given a
- * max_items value.
- */
-Size
-vac_max_items_to_alloc_size(int max_items)
-{
- Assert(max_items <= MAXDEADITEMS(MaxAllocSize));
-
- return offsetof(VacDeadItems, items) + sizeof(ItemPointerData) * max_items;
-}
-
/*
* vac_tid_reaped() -- is a particular tid deletable?
*
* This has the right signature to be an IndexBulkDeleteCallback.
- *
- * Assumes dead_items array is sorted (in ascending TID order).
*/
static bool
vac_tid_reaped(ItemPointer itemptr, void *state)
{
- VacDeadItems *dead_items = (VacDeadItems *) state;
- int64 litem,
- ritem,
- item;
- ItemPointer res;
-
- litem = itemptr_encode(&dead_items->items[0]);
- ritem = itemptr_encode(&dead_items->items[dead_items->num_items - 1]);
- item = itemptr_encode(itemptr);
-
- /*
- * Doing a simple bound check before bsearch() is useful to avoid the
- * extra cost of bsearch(), especially if dead items on the heap are
- * concentrated in a certain range. Since this function is called for
- * every index tuple, it pays to be really fast.
- */
- if (item < litem || item > ritem)
- return false;
-
- res = (ItemPointer) bsearch(itemptr,
- dead_items->items,
- dead_items->num_items,
- sizeof(ItemPointerData),
- vac_cmp_itemptr);
-
- return (res != NULL);
-}
-
-/*
- * Comparator routines for use with qsort() and bsearch().
- */
-static int
-vac_cmp_itemptr(const void *left, const void *right)
-{
- BlockNumber lblk,
- rblk;
- OffsetNumber loff,
- roff;
-
- lblk = ItemPointerGetBlockNumber((ItemPointer) left);
- rblk = ItemPointerGetBlockNumber((ItemPointer) right);
-
- if (lblk < rblk)
- return -1;
- if (lblk > rblk)
- return 1;
-
- loff = ItemPointerGetOffsetNumber((ItemPointer) left);
- roff = ItemPointerGetOffsetNumber((ItemPointer) right);
-
- if (loff < roff)
- return -1;
- if (loff > roff)
- return 1;
+ TidStore *dead_items = (TidStore *) state;
- return 0;
+ return tidstore_lookup_tid(dead_items, itemptr);
}
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index bcd40c80a1..d653683693 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -9,12 +9,11 @@
* In a parallel vacuum, we perform both index bulk deletion and index cleanup
* with parallel worker processes. Individual indexes are processed by one
* vacuum process. ParalleVacuumState contains shared information as well as
- * the memory space for storing dead items allocated in the DSM segment. We
- * launch parallel worker processes at the start of parallel index
- * bulk-deletion and index cleanup and once all indexes are processed, the
- * parallel worker processes exit. Each time we process indexes in parallel,
- * the parallel context is re-initialized so that the same DSM can be used for
- * multiple passes of index bulk-deletion and index cleanup.
+ * the shared TidStore. We launch parallel worker processes at the start of
+ * parallel index bulk-deletion and index cleanup and once all indexes are
+ * processed, the parallel worker processes exit. Each time we process indexes
+ * in parallel, the parallel context is re-initialized so that the same DSM can
+ * be used for multiple passes of index bulk-deletion and index cleanup.
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -103,6 +102,9 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /* Handle of the shared TidStore */
+ tidstore_handle dead_items_handle;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -166,7 +168,8 @@ struct ParallelVacuumState
PVIndStats *indstats;
/* Shared dead items space among parallel vacuum workers */
- VacDeadItems *dead_items;
+ TidStore *dead_items;
+ dsa_area *dead_items_area;
/* Points to buffer usage area in DSM */
BufferUsage *buffer_usage;
@@ -222,20 +225,23 @@ static void parallel_vacuum_error_callback(void *arg);
*/
ParallelVacuumState *
parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
- int nrequested_workers, int max_items,
- int elevel, BufferAccessStrategy bstrategy)
+ int nrequested_workers, int vac_work_mem,
+ int max_offset, int elevel,
+ BufferAccessStrategy bstrategy)
{
ParallelVacuumState *pvs;
ParallelContext *pcxt;
PVShared *shared;
- VacDeadItems *dead_items;
+ TidStore *dead_items;
PVIndStats *indstats;
BufferUsage *buffer_usage;
WalUsage *wal_usage;
+ void *area_space;
+ dsa_area *dead_items_dsa;
bool *will_parallel_vacuum;
Size est_indstats_len;
Size est_shared_len;
- Size est_dead_items_len;
+ Size dsa_minsize = dsa_minimum_size();
int nindexes_mwm = 0;
int parallel_workers = 0;
int querylen;
@@ -283,9 +289,8 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_estimate_chunk(&pcxt->estimator, est_shared_len);
shm_toc_estimate_keys(&pcxt->estimator, 1);
- /* Estimate size for dead_items -- PARALLEL_VACUUM_KEY_DEAD_ITEMS */
- est_dead_items_len = vac_max_items_to_alloc_size(max_items);
- shm_toc_estimate_chunk(&pcxt->estimator, est_dead_items_len);
+ /* Estimate size for dead tuple DSA -- PARALLEL_VACUUM_KEY_DSA */
+ shm_toc_estimate_chunk(&pcxt->estimator, dsa_minsize);
shm_toc_estimate_keys(&pcxt->estimator, 1);
/*
@@ -351,6 +356,16 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_INDEX_STATS, indstats);
pvs->indstats = indstats;
+ /* Prepare DSA space for dead items */
+ area_space = shm_toc_allocate(pcxt->toc, dsa_minsize);
+ shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, area_space);
+ dead_items_dsa = dsa_create_in_place(area_space, dsa_minsize,
+ LWTRANCHE_PARALLEL_VACUUM_DSA,
+ pcxt->seg);
+ dead_items = tidstore_create(vac_work_mem, max_offset, dead_items_dsa);
+ pvs->dead_items = dead_items;
+ pvs->dead_items_area = dead_items_dsa;
+
/* Prepare shared information */
shared = (PVShared *) shm_toc_allocate(pcxt->toc, est_shared_len);
MemSet(shared, 0, est_shared_len);
@@ -360,6 +375,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
+ shared->dead_items_handle = tidstore_get_handle(dead_items);
pg_atomic_init_u32(&(shared->cost_balance), 0);
pg_atomic_init_u32(&(shared->active_nworkers), 0);
@@ -368,15 +384,6 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_SHARED, shared);
pvs->shared = shared;
- /* Prepare the dead_items space */
- dead_items = (VacDeadItems *) shm_toc_allocate(pcxt->toc,
- est_dead_items_len);
- dead_items->max_items = max_items;
- dead_items->num_items = 0;
- MemSet(dead_items->items, 0, sizeof(ItemPointerData) * max_items);
- shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, dead_items);
- pvs->dead_items = dead_items;
-
/*
* Allocate space for each worker's BufferUsage and WalUsage; no need to
* initialize
@@ -434,6 +441,9 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
istats[i] = NULL;
}
+ tidstore_destroy(pvs->dead_items);
+ dsa_detach(pvs->dead_items_area);
+
DestroyParallelContext(pvs->pcxt);
ExitParallelMode();
@@ -442,7 +452,7 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
}
/* Returns the dead items space */
-VacDeadItems *
+TidStore *
parallel_vacuum_get_dead_items(ParallelVacuumState *pvs)
{
return pvs->dead_items;
@@ -940,7 +950,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
Relation *indrels;
PVIndStats *indstats;
PVShared *shared;
- VacDeadItems *dead_items;
+ TidStore *dead_items;
+ void *area_space;
+ dsa_area *dead_items_area;
BufferUsage *buffer_usage;
WalUsage *wal_usage;
int nindexes;
@@ -984,10 +996,10 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
PARALLEL_VACUUM_KEY_INDEX_STATS,
false);
- /* Set dead_items space */
- dead_items = (VacDeadItems *) shm_toc_lookup(toc,
- PARALLEL_VACUUM_KEY_DEAD_ITEMS,
- false);
+ /* Set dead items */
+ area_space = shm_toc_lookup(toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, false);
+ dead_items_area = dsa_attach_in_place(area_space, seg);
+ dead_items = tidstore_attach(dead_items_area, shared->dead_items_handle);
/* Set cost-based vacuum delay */
VacuumCostActive = (VacuumCostDelay > 0);
@@ -1033,6 +1045,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
InstrEndParallelQuery(&buffer_usage[ParallelWorkerNumber],
&wal_usage[ParallelWorkerNumber]);
+ tidstore_detach(pvs.dead_items);
+ dsa_detach(dead_items_area);
+
/* Pop the error context stack */
error_context_stack = errcallback.previous;
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index c0e2e00a7e..60caeae739 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -3399,12 +3399,12 @@ check_autovacuum_work_mem(int *newval, void **extra, GucSource source)
return true;
/*
- * We clamp manually-set values to at least 1MB. Since
+ * We clamp manually-set values to at least 2MB. Since
* maintenance_work_mem is always set to at least this value, do the same
* here.
*/
- if (*newval < 1024)
- *newval = 1024;
+ if (*newval < 2048)
+ *newval = 2048;
return true;
}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 55b3a04097..c223a7dc94 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -192,6 +192,8 @@ static const char *const BuiltinTrancheNames[] = {
"LogicalRepLauncherDSA",
/* LWTRANCHE_LAUNCHER_HASH: */
"LogicalRepLauncherHash",
+ /* LWTRANCHE_PARALLEL_VACUUM_DSA: */
+ "ParallelVacuumDSA",
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 1c0583fe26..8a64614cd1 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2313,7 +2313,7 @@ struct config_int ConfigureNamesInt[] =
GUC_UNIT_KB
},
&maintenance_work_mem,
- 65536, 1024, MAX_KILOBYTES,
+ 65536, 2048, MAX_KILOBYTES,
NULL, NULL, NULL
},
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index e5add41352..b209d3cf84 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -23,8 +23,8 @@
#define PROGRESS_VACUUM_HEAP_BLKS_SCANNED 2
#define PROGRESS_VACUUM_HEAP_BLKS_VACUUMED 3
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
-#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
-#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_MAX_DEAD_TUPLE_BYTES 5
+#define PROGRESS_VACUUM_DEAD_TUPLE_BYTES 6
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index bdfd96cfec..cec2d1d356 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -17,6 +17,7 @@
#include "access/htup.h"
#include "access/genam.h"
#include "access/parallel.h"
+#include "access/tidstore.h"
#include "catalog/pg_class.h"
#include "catalog/pg_statistic.h"
#include "catalog/pg_type.h"
@@ -277,21 +278,6 @@ struct VacuumCutoffs
MultiXactId MultiXactCutoff;
};
-/*
- * VacDeadItems stores TIDs whose index tuples are deleted by index vacuuming.
- */
-typedef struct VacDeadItems
-{
- int max_items; /* # slots allocated in array */
- int num_items; /* current # of entries */
-
- /* Sorted array of TIDs to delete from indexes */
- ItemPointerData items[FLEXIBLE_ARRAY_MEMBER];
-} VacDeadItems;
-
-#define MAXDEADITEMS(avail_mem) \
- (((avail_mem) - offsetof(VacDeadItems, items)) / sizeof(ItemPointerData))
-
/* GUC parameters */
extern PGDLLIMPORT int default_statistics_target; /* PGDLLIMPORT for PostGIS */
extern PGDLLIMPORT int vacuum_freeze_min_age;
@@ -340,18 +326,17 @@ extern Relation vacuum_open_relation(Oid relid, RangeVar *relation,
LOCKMODE lmode);
extern IndexBulkDeleteResult *vac_bulkdel_one_index(IndexVacuumInfo *ivinfo,
IndexBulkDeleteResult *istat,
- VacDeadItems *dead_items);
+ TidStore *dead_items);
extern IndexBulkDeleteResult *vac_cleanup_one_index(IndexVacuumInfo *ivinfo,
IndexBulkDeleteResult *istat);
-extern Size vac_max_items_to_alloc_size(int max_items);
/* in commands/vacuumparallel.c */
extern ParallelVacuumState *parallel_vacuum_init(Relation rel, Relation *indrels,
int nindexes, int nrequested_workers,
- int max_items, int elevel,
- BufferAccessStrategy bstrategy);
+ int vac_work_mem, int max_offset,
+ int elevel, BufferAccessStrategy bstrategy);
extern void parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats);
-extern VacDeadItems *parallel_vacuum_get_dead_items(ParallelVacuumState *pvs);
+extern TidStore *parallel_vacuum_get_dead_items(ParallelVacuumState *pvs);
extern void parallel_vacuum_bulkdel_all_indexes(ParallelVacuumState *pvs,
long num_table_tuples,
int num_index_scans);
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index 07002fdfbe..537b34b30c 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -207,6 +207,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_PGSTATS_DATA,
LWTRANCHE_LAUNCHER_DSA,
LWTRANCHE_LAUNCHER_HASH,
+ LWTRANCHE_PARALLEL_VACUUM_DSA,
LWTRANCHE_FIRST_USER_DEFINED
} BuiltinTrancheIds;
diff --git a/src/test/regress/expected/cluster.out b/src/test/regress/expected/cluster.out
index 2eec483eaa..e04f50726f 100644
--- a/src/test/regress/expected/cluster.out
+++ b/src/test/regress/expected/cluster.out
@@ -526,7 +526,7 @@ create index cluster_sort on clstr_4 (hundred, thousand, tenthous);
-- ensure we don't use the index in CLUSTER nor the checking SELECTs
set enable_indexscan = off;
-- Use external sort:
-set maintenance_work_mem = '1MB';
+set maintenance_work_mem = '2MB';
cluster clstr_4 using cluster_sort;
select * from
(select hundred, lag(hundred) over () as lhundred,
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index acfd9d1f4f..d320ad87dd 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -1214,7 +1214,7 @@ DROP TABLE unlogged_hash_table;
-- CREATE INDEX hash_ovfl_index ON hash_ovfl_heap USING hash (x int4_ops);
-- Test hash index build tuplesorting. Force hash tuplesort using low
-- maintenance_work_mem setting and fillfactor:
-SET maintenance_work_mem = '1MB';
+SET maintenance_work_mem = '2MB';
CREATE INDEX hash_tuplesort_idx ON tenk1 USING hash (stringu1 name_ops) WITH (fillfactor = 10);
EXPLAIN (COSTS OFF)
SELECT count(*) FROM tenk1 WHERE stringu1 = 'TVAAAA';
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index e953d1f515..ef46c2994f 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2032,8 +2032,8 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param3 AS heap_blks_scanned,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
- s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param6 AS max_dead_tuple_bytes,
+ s.param7 AS dead_tuple_bytes
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_recovery_prefetch| SELECT stats_reset,
diff --git a/src/test/regress/sql/cluster.sql b/src/test/regress/sql/cluster.sql
index a4cfaae807..a4cb5b98a5 100644
--- a/src/test/regress/sql/cluster.sql
+++ b/src/test/regress/sql/cluster.sql
@@ -258,7 +258,7 @@ create index cluster_sort on clstr_4 (hundred, thousand, tenthous);
set enable_indexscan = off;
-- Use external sort:
-set maintenance_work_mem = '1MB';
+set maintenance_work_mem = '2MB';
cluster clstr_4 using cluster_sort;
select * from
(select hundred, lag(hundred) over () as lhundred,
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index d49ce9f300..d6e2471b00 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -367,7 +367,7 @@ DROP TABLE unlogged_hash_table;
-- Test hash index build tuplesorting. Force hash tuplesort using low
-- maintenance_work_mem setting and fillfactor:
-SET maintenance_work_mem = '1MB';
+SET maintenance_work_mem = '2MB';
CREATE INDEX hash_tuplesort_idx ON tenk1 USING hash (stringu1 name_ops) WITH (fillfactor = 10);
EXPLAIN (COSTS OFF)
SELECT count(*) FROM tenk1 WHERE stringu1 = 'TVAAAA';
--
2.31.1
[application/octet-stream] v31-0002-Move-some-bitmap-logic-out-of-bitmapset.c.patch (6.0K, ../../CAD21AoDBmD5q=eO+K=gyuVt53XvwpJ2dgxPwrtZ-eVOjVmtJjg@mail.gmail.com/11-v31-0002-Move-some-bitmap-logic-out-of-bitmapset.c.patch)
download | inline diff:
From 46ccfc2d0b588e090d1f46bc16f463789227aff4 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 6 Dec 2022 13:39:41 +0700
Subject: [PATCH v31 02/14] Move some bitmap logic out of bitmapset.c
Add pg_rightmost_one32/64 functions. This functionality was previously
private to bitmapset.c as the RIGHTMOST_ONE macro. It has practical
use in other contexts, so move to pg_bitutils.h.
Also move the logic for selecting appropriate pg_bitutils functions
based on word size to bitmapset.h for wider visibility and add
appropriate selection for bmw_rightmost_one().
Since the previous macro relied on casting to signedbitmapword,
and the new functions do not, remove that typedef.
Design input and review by Tom Lane
Discussion: https://www.postgresql.org/message-id/CAFBsxsFW2JjTo58jtDB%2B3sZhxMx3t-3evew8%3DAcr%2BGGhC%2BkFaA%40mail.gmail.com
---
src/backend/nodes/bitmapset.c | 34 +-------------------------------
src/include/nodes/bitmapset.h | 16 +++++++++++++--
src/include/port/pg_bitutils.h | 31 +++++++++++++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 -
4 files changed, 46 insertions(+), 36 deletions(-)
diff --git a/src/backend/nodes/bitmapset.c b/src/backend/nodes/bitmapset.c
index 7ba3cf635b..0b2962ed73 100644
--- a/src/backend/nodes/bitmapset.c
+++ b/src/backend/nodes/bitmapset.c
@@ -30,39 +30,7 @@
#define BITMAPSET_SIZE(nwords) \
(offsetof(Bitmapset, words) + (nwords) * sizeof(bitmapword))
-/*----------
- * This is a well-known cute trick for isolating the rightmost one-bit
- * in a word. It assumes two's complement arithmetic. Consider any
- * nonzero value, and focus attention on the rightmost one. The value is
- * then something like
- * xxxxxx10000
- * where x's are unspecified bits. The two's complement negative is formed
- * by inverting all the bits and adding one. Inversion gives
- * yyyyyy01111
- * where each y is the inverse of the corresponding x. Incrementing gives
- * yyyyyy10000
- * and then ANDing with the original value gives
- * 00000010000
- * This works for all cases except original value = zero, where of course
- * we get zero.
- *----------
- */
-#define RIGHTMOST_ONE(x) ((signedbitmapword) (x) & -((signedbitmapword) (x)))
-
-#define HAS_MULTIPLE_ONES(x) ((bitmapword) RIGHTMOST_ONE(x) != (x))
-
-/* Select appropriate bit-twiddling functions for bitmap word size */
-#if BITS_PER_BITMAPWORD == 32
-#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w)
-#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w)
-#define bmw_popcount(w) pg_popcount32(w)
-#elif BITS_PER_BITMAPWORD == 64
-#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w)
-#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w)
-#define bmw_popcount(w) pg_popcount64(w)
-#else
-#error "invalid BITS_PER_BITMAPWORD"
-#endif
+#define HAS_MULTIPLE_ONES(x) (bmw_rightmost_one(x) != (x))
static bool bms_is_empty_internal(const Bitmapset *a);
diff --git a/src/include/nodes/bitmapset.h b/src/include/nodes/bitmapset.h
index 14de6a9ff1..c7e1711147 100644
--- a/src/include/nodes/bitmapset.h
+++ b/src/include/nodes/bitmapset.h
@@ -36,13 +36,11 @@ struct List;
#define BITS_PER_BITMAPWORD 64
typedef uint64 bitmapword; /* must be an unsigned type */
-typedef int64 signedbitmapword; /* must be the matching signed type */
#else
#define BITS_PER_BITMAPWORD 32
typedef uint32 bitmapword; /* must be an unsigned type */
-typedef int32 signedbitmapword; /* must be the matching signed type */
#endif
@@ -73,6 +71,20 @@ typedef enum
BMS_MULTIPLE /* >1 member */
} BMS_Membership;
+/* Select appropriate bit-twiddling functions for bitmap word size */
+#if BITS_PER_BITMAPWORD == 32
+#define bmw_rightmost_one(w) pg_rightmost_one32(w)
+#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w)
+#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w)
+#define bmw_popcount(w) pg_popcount32(w)
+#elif BITS_PER_BITMAPWORD == 64
+#define bmw_rightmost_one(w) pg_rightmost_one64(w)
+#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w)
+#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w)
+#define bmw_popcount(w) pg_popcount64(w)
+#else
+#error "invalid BITS_PER_BITMAPWORD"
+#endif
/*
* function prototypes in nodes/bitmapset.c
diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h
index 158ef73a2b..bf7588e075 100644
--- a/src/include/port/pg_bitutils.h
+++ b/src/include/port/pg_bitutils.h
@@ -32,6 +32,37 @@ extern PGDLLIMPORT const uint8 pg_leftmost_one_pos[256];
extern PGDLLIMPORT const uint8 pg_rightmost_one_pos[256];
extern PGDLLIMPORT const uint8 pg_number_of_ones[256];
+/*----------
+ * This is a well-known cute trick for isolating the rightmost one-bit
+ * in a word. It assumes two's complement arithmetic. Consider any
+ * nonzero value, and focus attention on the rightmost one. The value is
+ * then something like
+ * xxxxxx10000
+ * where x's are unspecified bits. The two's complement negative is formed
+ * by inverting all the bits and adding one. Inversion gives
+ * yyyyyy01111
+ * where each y is the inverse of the corresponding x. Incrementing gives
+ * yyyyyy10000
+ * and then ANDing with the original value gives
+ * 00000010000
+ * This works for all cases except original value = zero, where of course
+ * we get zero.
+ *----------
+ */
+static inline uint32
+pg_rightmost_one32(uint32 word)
+{
+ int32 result = (int32) word & -((int32) word);
+ return (uint32) result;
+}
+
+static inline uint64
+pg_rightmost_one64(uint64 word)
+{
+ int64 result = (int64) word & -((int64) word);
+ return (uint64) result;
+}
+
/*
* pg_leftmost_one_pos32
* Returns the position of the most significant set bit in "word",
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 86a9303bf5..4a5e776703 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3675,7 +3675,6 @@ shmem_request_hook_type
shmem_startup_hook_type
sig_atomic_t
sigjmp_buf
-signedbitmapword
sigset_t
size_t
slist_head
--
2.31.1
[application/octet-stream] v31-0012-Revert-the-update-for-the-minimum-value-of-maint.patch (1.4K, ../../CAD21AoDBmD5q=eO+K=gyuVt53XvwpJ2dgxPwrtZ-eVOjVmtJjg@mail.gmail.com/12-v31-0012-Revert-the-update-for-the-minimum-value-of-maint.patch)
download | inline diff:
From 8080e74de8597b6e8567fbfce5dbd2771937287c Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 8 Mar 2023 15:09:22 +0900
Subject: [PATCH v31 12/14] Revert the update for the minimum value of
maintenance_work_mem.
---
src/backend/postmaster/autovacuum.c | 6 +++---
src/backend/utils/misc/guc_tables.c | 2 +-
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 60caeae739..c0e2e00a7e 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -3399,12 +3399,12 @@ check_autovacuum_work_mem(int *newval, void **extra, GucSource source)
return true;
/*
- * We clamp manually-set values to at least 2MB. Since
+ * We clamp manually-set values to at least 1MB. Since
* maintenance_work_mem is always set to at least this value, do the same
* here.
*/
- if (*newval < 2048)
- *newval = 2048;
+ if (*newval < 1024)
+ *newval = 1024;
return true;
}
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 8a64614cd1..1c0583fe26 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2313,7 +2313,7 @@ struct config_int ConfigureNamesInt[] =
GUC_UNIT_KB
},
&maintenance_work_mem,
- 65536, 2048, MAX_KILOBYTES,
+ 65536, 1024, MAX_KILOBYTES,
NULL, NULL, NULL
},
--
2.31.1
[application/octet-stream] v31-0001-Introduce-helper-SIMD-functions-for-small-byte-a.patch (2.9K, ../../CAD21AoDBmD5q=eO+K=gyuVt53XvwpJ2dgxPwrtZ-eVOjVmtJjg@mail.gmail.com/13-v31-0001-Introduce-helper-SIMD-functions-for-small-byte-a.patch)
download | inline diff:
From 2176fc0e5b4bee9e389f8a29637ef9ed29aec0da Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 24 Oct 2022 14:07:09 +0900
Subject: [PATCH v31 01/14] Introduce helper SIMD functions for small byte
arrays
vector8_min - helper for emulating ">=" semantics
vector8_highbit_mask - used to turn the result of a vector
comparison into a bitmask
Masahiko Sawada
Reviewed by Nathan Bossart, additional adjustments by me
Discussion: https://www.postgresql.org/message-id/CAD21AoDap240WDDdUDE0JMpCmuMMnGajrKrkCRxM7zn9Xk3JRA%40mail.gmail.com
---
src/include/port/simd.h | 47 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)
diff --git a/src/include/port/simd.h b/src/include/port/simd.h
index 1fa6c3bc6c..dfae14e463 100644
--- a/src/include/port/simd.h
+++ b/src/include/port/simd.h
@@ -79,6 +79,7 @@ static inline bool vector8_has_le(const Vector8 v, const uint8 c);
static inline bool vector8_is_highbit_set(const Vector8 v);
#ifndef USE_NO_SIMD
static inline bool vector32_is_highbit_set(const Vector32 v);
+static inline uint32 vector8_highbit_mask(const Vector8 v);
#endif
/* arithmetic operations */
@@ -96,6 +97,7 @@ static inline Vector8 vector8_ssub(const Vector8 v1, const Vector8 v2);
*/
#ifndef USE_NO_SIMD
static inline Vector8 vector8_eq(const Vector8 v1, const Vector8 v2);
+static inline Vector8 vector8_min(const Vector8 v1, const Vector8 v2);
static inline Vector32 vector32_eq(const Vector32 v1, const Vector32 v2);
#endif
@@ -299,6 +301,36 @@ vector32_is_highbit_set(const Vector32 v)
}
#endif /* ! USE_NO_SIMD */
+/*
+ * Return a bitmask formed from the high-bit of each element.
+ */
+#ifndef USE_NO_SIMD
+static inline uint32
+vector8_highbit_mask(const Vector8 v)
+{
+#ifdef USE_SSE2
+ return (uint32) _mm_movemask_epi8(v);
+#elif defined(USE_NEON)
+ /*
+ * Note: There is a faster way to do this, but it returns a uint64 and
+ * and if the caller wanted to extract the bit position using CTZ,
+ * it would have to divide that result by 4.
+ */
+ static const uint8 mask[16] = {
+ 1 << 0, 1 << 1, 1 << 2, 1 << 3,
+ 1 << 4, 1 << 5, 1 << 6, 1 << 7,
+ 1 << 0, 1 << 1, 1 << 2, 1 << 3,
+ 1 << 4, 1 << 5, 1 << 6, 1 << 7,
+ };
+
+ uint8x16_t masked = vandq_u8(vld1q_u8(mask), (uint8x16_t) vshrq_n_s8(v, 7));
+ uint8x16_t maskedhi = vextq_u8(masked, masked, 8);
+
+ return (uint32) vaddvq_u16((uint16x8_t) vzip1q_u8(masked, maskedhi));
+#endif
+}
+#endif /* ! USE_NO_SIMD */
+
/*
* Return the bitwise OR of the inputs
*/
@@ -372,4 +404,19 @@ vector32_eq(const Vector32 v1, const Vector32 v2)
}
#endif /* ! USE_NO_SIMD */
+/*
+ * Given two vectors, return a vector with the minimum element of each.
+ */
+#ifndef USE_NO_SIMD
+static inline Vector8
+vector8_min(const Vector8 v1, const Vector8 v2)
+{
+#ifdef USE_SSE2
+ return _mm_min_epu8(v1, v2);
+#elif defined(USE_NEON)
+ return vminq_u8(v1, v2);
+#endif
+}
+#endif /* ! USE_NO_SIMD */
+
#endif /* SIMD_H */
--
2.31.1
[application/octet-stream] v31-0010-Radix-tree-optionally-tracks-memory-usage-when-R.patch (8.0K, ../../CAD21AoDBmD5q=eO+K=gyuVt53XvwpJ2dgxPwrtZ-eVOjVmtJjg@mail.gmail.com/14-v31-0010-Radix-tree-optionally-tracks-memory-usage-when-R.patch)
download | inline diff:
From 7da5e7808ba51aed7ad22b9758b3200cbfcd7d19 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 8 Mar 2023 15:08:19 +0900
Subject: [PATCH v31 10/14] Radix tree optionally tracks memory usage, when
RT_MEASURE_MEMORY_USAGE.
---
contrib/bench_radix_tree/bench_radix_tree.c | 1 +
src/backend/utils/mmgr/dsa.c | 12 ---
src/include/lib/radixtree.h | 93 +++++++++++++++++--
src/include/utils/dsa.h | 1 -
.../modules/test_radixtree/test_radixtree.c | 1 +
5 files changed, 85 insertions(+), 23 deletions(-)
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
index 6e5149e2c4..8a0c754a2c 100644
--- a/contrib/bench_radix_tree/bench_radix_tree.c
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -34,6 +34,7 @@ PG_MODULE_MAGIC;
#define RT_DECLARE
#define RT_DEFINE
#define RT_USE_DELETE
+#define RT_MEASURE_MEMORY_USAGE
#define RT_VALUE_TYPE uint64
// WIP: compiles with warnings because rt_attach is defined but not used
// #define RT_SHMEM
diff --git a/src/backend/utils/mmgr/dsa.c b/src/backend/utils/mmgr/dsa.c
index 80555aefff..f5a62061a3 100644
--- a/src/backend/utils/mmgr/dsa.c
+++ b/src/backend/utils/mmgr/dsa.c
@@ -1024,18 +1024,6 @@ dsa_set_size_limit(dsa_area *area, size_t limit)
LWLockRelease(DSA_AREA_LOCK(area));
}
-size_t
-dsa_get_total_size(dsa_area *area)
-{
- size_t size;
-
- LWLockAcquire(DSA_AREA_LOCK(area), LW_SHARED);
- size = area->control->total_segment_size;
- LWLockRelease(DSA_AREA_LOCK(area));
-
- return size;
-}
-
/*
* Aggressively free all spare memory in the hope of returning DSM segments to
* the operating system.
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index 8bea606c62..f7812eb12a 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -84,7 +84,6 @@
* RT_BEGIN_ITERATE - Begin iterating through all key-value pairs
* RT_ITERATE_NEXT - Return next key-value pair, if any
* RT_END_ITERATE - End iteration
- * RT_MEMORY_USAGE - Get the memory usage
*
* Interface for Shared Memory
* ---------
@@ -97,6 +96,8 @@
* ---------
*
* RT_DELETE - Delete a key-value pair. Declared/define if RT_USE_DELETE is defined
+ * RT_MEMORY_USAGE - Get the memory usage. Declared/define if
+ * RT_MEASURE_MEMORY_USAGE is defined.
*
*
* Copyright (c) 2023, PostgreSQL Global Development Group
@@ -138,7 +139,9 @@
#ifdef RT_USE_DELETE
#define RT_DELETE RT_MAKE_NAME(delete)
#endif
+#ifdef RT_MEASURE_MEMORY_USAGE
#define RT_MEMORY_USAGE RT_MAKE_NAME(memory_usage)
+#endif
#ifdef RT_DEBUG
#define RT_DUMP RT_MAKE_NAME(dump)
#define RT_DUMP_NODE RT_MAKE_NAME(dump_node)
@@ -150,6 +153,9 @@
#define RT_NEW_ROOT RT_MAKE_NAME(new_root)
#define RT_ALLOC_NODE RT_MAKE_NAME(alloc_node)
#define RT_INIT_NODE RT_MAKE_NAME(init_node)
+#ifdef RT_MEASURE_MEMORY_USAGE
+#define RT_FANOUT_GET_NODE_SIZE RT_MAKE_NAME(fanout_get_node_size)
+#endif
#define RT_FREE_NODE RT_MAKE_NAME(free_node)
#define RT_FREE_RECURSE RT_MAKE_NAME(free_recurse)
#define RT_EXTEND_UP RT_MAKE_NAME(extend_up)
@@ -255,7 +261,9 @@ RT_SCOPE RT_ITER * RT_BEGIN_ITERATE(RT_RADIX_TREE *tree);
RT_SCOPE bool RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p);
RT_SCOPE void RT_END_ITERATE(RT_ITER *iter);
+#ifdef RT_MEASURE_MEMORY_USAGE
RT_SCOPE uint64 RT_MEMORY_USAGE(RT_RADIX_TREE *tree);
+#endif
#ifdef RT_DEBUG
RT_SCOPE void RT_DUMP(RT_RADIX_TREE *tree);
@@ -624,6 +632,10 @@ typedef struct RT_RADIX_TREE_CONTROL
uint64 max_val;
uint64 num_keys;
+#ifdef RT_MEASURE_MEMORY_USAGE
+ int64 mem_used;
+#endif
+
/* statistics */
#ifdef RT_DEBUG
int32 cnt[RT_SIZE_CLASS_COUNT];
@@ -1089,6 +1101,11 @@ RT_ALLOC_NODE(RT_RADIX_TREE *tree, RT_SIZE_CLASS size_class, bool is_leaf)
allocsize);
#endif
+#ifdef RT_MEASURE_MEMORY_USAGE
+ /* update memory usage */
+ tree->ctl->mem_used += allocsize;
+#endif
+
#ifdef RT_DEBUG
/* update the statistics */
tree->ctl->cnt[size_class]++;
@@ -1165,6 +1182,54 @@ RT_SWITCH_NODE_KIND(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, RT_PTR_LOCAL no
return newnode;
}
+#ifdef RT_MEASURE_MEMORY_USAGE
+/* Return the node size of the given fanout of the size class */
+static inline Size
+RT_FANOUT_GET_NODE_SIZE(int fanout, bool is_leaf)
+{
+ const Size fanout_inner_node_size[] = {
+ [3] = RT_SIZE_CLASS_INFO[RT_CLASS_3].inner_size,
+ [15] = RT_SIZE_CLASS_INFO[RT_CLASS_32_MIN].inner_size,
+ [32] = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX].inner_size,
+ [125] = RT_SIZE_CLASS_INFO[RT_CLASS_125].inner_size,
+ [256] = RT_SIZE_CLASS_INFO[RT_CLASS_256].inner_size,
+ };
+ const Size fanout_leaf_node_size[] = {
+ [3] = RT_SIZE_CLASS_INFO[RT_CLASS_3].leaf_size,
+ [15] = RT_SIZE_CLASS_INFO[RT_CLASS_32_MIN].leaf_size,
+ [32] = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX].leaf_size,
+ [125] = RT_SIZE_CLASS_INFO[RT_CLASS_125].leaf_size,
+ [256] = RT_SIZE_CLASS_INFO[RT_CLASS_256].leaf_size,
+ };
+ Size node_size;
+
+ node_size = is_leaf ?
+ fanout_leaf_node_size[fanout] : fanout_inner_node_size[fanout];
+
+#ifdef USE_ASSERT_CHECKING
+ {
+ Size assert_node_size = 0;
+
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ RT_SIZE_CLASS_ELEM size_class = RT_SIZE_CLASS_INFO[i];
+
+ if (size_class.fanout == fanout)
+ {
+ assert_node_size = is_leaf ?
+ size_class.leaf_size : size_class.inner_size;
+ break;
+ }
+ }
+
+ Assert(node_size == assert_node_size);
+ }
+#endif
+
+ return node_size;
+}
+#endif
+
/* Free the given node */
static void
RT_FREE_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode)
@@ -1197,11 +1262,22 @@ RT_FREE_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode)
}
#endif
+#ifdef RT_MEASURE_MEMORY_USAGE
+ /* update memory usage */
+ {
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+ tree->ctl->mem_used -= RT_FANOUT_GET_NODE_SIZE(node->fanout,
+ RT_NODE_IS_LEAF(node));
+ Assert(tree->ctl->mem_used >= 0);
+ }
+#endif
+
#ifdef RT_SHMEM
dsa_free(tree->dsa, allocnode);
#else
pfree(allocnode);
#endif
+
}
/* Update the parent's pointer when growing a node */
@@ -1989,27 +2065,23 @@ RT_END_ITERATE(RT_ITER *iter)
/*
* Return the statistics of the amount of memory used by the radix tree.
*/
+#ifdef RT_MEASURE_MEMORY_USAGE
RT_SCOPE uint64
RT_MEMORY_USAGE(RT_RADIX_TREE *tree)
{
Size total = 0;
- RT_LOCK_SHARED(tree);
-
#ifdef RT_SHMEM
Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
- total = dsa_get_total_size(tree->dsa);
-#else
- for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
- {
- total += MemoryContextMemAllocated(tree->inner_slabs[i], true);
- total += MemoryContextMemAllocated(tree->leaf_slabs[i], true);
- }
#endif
+ RT_LOCK_SHARED(tree);
+ total = tree->ctl->mem_used;
RT_UNLOCK(tree);
+
return total;
}
+#endif
/*
* Verify the radix tree node.
@@ -2476,6 +2548,7 @@ RT_DUMP(RT_RADIX_TREE *tree)
#undef RT_NEW_ROOT
#undef RT_ALLOC_NODE
#undef RT_INIT_NODE
+#undef RT_FANOUT_GET_NODE_SIZE
#undef RT_FREE_NODE
#undef RT_FREE_RECURSE
#undef RT_EXTEND_UP
diff --git a/src/include/utils/dsa.h b/src/include/utils/dsa.h
index 2af215484f..3ce4ee300a 100644
--- a/src/include/utils/dsa.h
+++ b/src/include/utils/dsa.h
@@ -121,7 +121,6 @@ extern dsa_handle dsa_get_handle(dsa_area *area);
extern dsa_pointer dsa_allocate_extended(dsa_area *area, size_t size, int flags);
extern void dsa_free(dsa_area *area, dsa_pointer dp);
extern void *dsa_get_address(dsa_area *area, dsa_pointer dp);
-extern size_t dsa_get_total_size(dsa_area *area);
extern void dsa_trim(dsa_area *area);
extern void dsa_dump(dsa_area *area);
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
index 5a169854d9..19d286d84b 100644
--- a/src/test/modules/test_radixtree/test_radixtree.c
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -114,6 +114,7 @@ static const test_spec test_specs[] = {
#define RT_DECLARE
#define RT_DEFINE
#define RT_USE_DELETE
+#define RT_MEASURE_MEMORY_USAGE
#define RT_VALUE_TYPE TestValueType
/* #define RT_SHMEM */
#include "lib/radixtree.h"
--
2.31.1
[application/octet-stream] v31-0004-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch (36.0K, ../../CAD21AoDBmD5q=eO+K=gyuVt53XvwpJ2dgxPwrtZ-eVOjVmtJjg@mail.gmail.com/15-v31-0004-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch)
download | inline diff:
From db646cb1da4a21182028096e036b0f86d61e8ce8 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 23 Dec 2022 23:50:39 +0900
Subject: [PATCH v31 04/14] Add TIDStore, to store sets of TIDs
(ItemPointerData) efficiently.
The TIDStore is designed to store large sets of TIDs efficiently, and
is backed by the radix tree. A TID is encoded into 64-bit key and
value and inserted to the radix tree.
The TIDStore is not used for anything yet, aside from the test code,
but the follow up patch integrates the TIDStore with lazy vacuum,
reducing lazy vacuum memory usage and lifting the 1GB limit on its
size, by storing the list of dead TIDs more efficiently.
This includes a unit test module, in src/test/modules/test_tidstore.
---
doc/src/sgml/monitoring.sgml | 4 +
src/backend/access/common/Makefile | 1 +
src/backend/access/common/meson.build | 1 +
src/backend/access/common/tidstore.c | 681 ++++++++++++++++++
src/backend/storage/lmgr/lwlock.c | 2 +
src/include/access/tidstore.h | 49 ++
src/include/storage/lwlock.h | 1 +
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_tidstore/Makefile | 23 +
.../test_tidstore/expected/test_tidstore.out | 13 +
src/test/modules/test_tidstore/meson.build | 35 +
.../test_tidstore/sql/test_tidstore.sql | 7 +
.../test_tidstore/test_tidstore--1.0.sql | 8 +
.../modules/test_tidstore/test_tidstore.c | 226 ++++++
.../test_tidstore/test_tidstore.control | 4 +
16 files changed, 1057 insertions(+)
create mode 100644 src/backend/access/common/tidstore.c
create mode 100644 src/include/access/tidstore.h
create mode 100644 src/test/modules/test_tidstore/Makefile
create mode 100644 src/test/modules/test_tidstore/expected/test_tidstore.out
create mode 100644 src/test/modules/test_tidstore/meson.build
create mode 100644 src/test/modules/test_tidstore/sql/test_tidstore.sql
create mode 100644 src/test/modules/test_tidstore/test_tidstore--1.0.sql
create mode 100644 src/test/modules/test_tidstore/test_tidstore.c
create mode 100644 src/test/modules/test_tidstore/test_tidstore.control
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 6249bb50d0..97d588b1d8 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2203,6 +2203,10 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<entry>Waiting to access a shared TID bitmap during a parallel bitmap
index scan.</entry>
</row>
+ <row>
+ <entry><literal>SharedTidStore</literal></entry>
+ <entry>Waiting to access a shared TID store.</entry>
+ </row>
<row>
<entry><literal>SharedTupleStore</literal></entry>
<entry>Waiting to access a shared tuple store during parallel
diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile
index b9aff0ccfd..67b8cc6108 100644
--- a/src/backend/access/common/Makefile
+++ b/src/backend/access/common/Makefile
@@ -27,6 +27,7 @@ OBJS = \
syncscan.o \
toast_compression.o \
toast_internals.o \
+ tidstore.o \
tupconvert.o \
tupdesc.o
diff --git a/src/backend/access/common/meson.build b/src/backend/access/common/meson.build
index f5ac17b498..fce19c09ce 100644
--- a/src/backend/access/common/meson.build
+++ b/src/backend/access/common/meson.build
@@ -15,6 +15,7 @@ backend_sources += files(
'syncscan.c',
'toast_compression.c',
'toast_internals.c',
+ 'tidstore.c',
'tupconvert.c',
'tupdesc.c',
)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
new file mode 100644
index 0000000000..8c05e60d92
--- /dev/null
+++ b/src/backend/access/common/tidstore.c
@@ -0,0 +1,681 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.c
+ * Tid (ItemPointerData) storage implementation.
+ *
+ * This module provides a in-memory data structure to store Tids (ItemPointer).
+ * Internally, a tid is encoded as a pair of 64-bit key and 64-bit value, and
+ * stored in the radix tree.
+ *
+ * A TidStore can be shared among parallel worker processes by passing DSA area
+ * to tidstore_create(). Other backends can attach to the shared TidStore by
+ * tidstore_attach().
+ *
+ * Regarding the concurrency, it basically relies on the concurrency support in
+ * the radix tree, but we acquires the lock on a TidStore in some cases, for
+ * example, when to reset the store and when to access the number tids in the
+ * store (num_tids).
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/access/common/tidstore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/tidstore.h"
+#include "miscadmin.h"
+#include "port/pg_bitutils.h"
+#include "storage/lwlock.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/*
+ * For encoding purposes, tids are represented as a pair of 64-bit key and
+ * 64-bit value. First, we construct 64-bit unsigned integer by combining
+ * the block number and the offset number. The number of bits used for the
+ * offset number is specified by max_offsets in tidstore_create(). We are
+ * frugal with the bits, because smaller keys could help keeping the radix
+ * tree shallow.
+ *
+ * For example, a tid of heap with 8kB blocks uses the lowest 9 bits for
+ * the offset number and uses the next 32 bits for the block number. That
+ * is, only 41 bits are used:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ *
+ * X = bits used for offset number
+ * Y = bits used for block number
+ * u = unused bit
+ * (high on the left, low on the right)
+ *
+ * 9 bits are enough for the offset number, because MaxHeapTuplesPerPage < 2^9
+ * on 8kB blocks.
+ *
+ * The 64-bit value is the bitmap representation of the lowest 6 bits
+ * (TIDSTORE_VALUE_NBITS) of the integer, and the rest 35 bits are used
+ * as the key:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ * |----| value
+ * |---------------------------------------------| key
+ *
+ * The maximum height of the radix tree is 5 in this case.
+ */
+#define TIDSTORE_VALUE_NBITS 6 /* log(64, 2) */
+#define TIDSTORE_OFFSET_MASK ((1 << TIDSTORE_VALUE_NBITS) - 1)
+
+/* A magic value used to identify our TidStores. */
+#define TIDSTORE_MAGIC 0x826f6a10
+
+#define RT_PREFIX local_rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE uint64
+#include "lib/radixtree.h"
+
+#define RT_PREFIX shared_rt
+#define RT_SHMEM
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE uint64
+#include "lib/radixtree.h"
+
+/* The control object for a TidStore */
+typedef struct TidStoreControl
+{
+ /* the number of tids in the store */
+ int64 num_tids;
+
+ /* These values are never changed after creation */
+ size_t max_bytes; /* the maximum bytes a TidStore can use */
+ int max_offset; /* the maximum offset number */
+ int offset_nbits; /* the number of bits required for an offset
+ * number */
+ int offset_key_nbits; /* the number of bits of an offset number
+ * used in a key */
+
+ /* The below fields are used only in shared case */
+
+ uint32 magic;
+ LWLock lock;
+
+ /* handles for TidStore and radix tree */
+ tidstore_handle handle;
+ shared_rt_handle tree_handle;
+} TidStoreControl;
+
+/* Per-backend state for a TidStore */
+struct TidStore
+{
+ /*
+ * Control object. This is allocated in DSA area 'area' in the shared
+ * case, otherwise in backend-local memory.
+ */
+ TidStoreControl *control;
+
+ /* Storage for Tids. Use either one depending on TidStoreIsShared() */
+ union
+ {
+ local_rt_radix_tree *local;
+ shared_rt_radix_tree *shared;
+ } tree;
+
+ /* DSA area for TidStore if used */
+ dsa_area *area;
+};
+#define TidStoreIsShared(ts) ((ts)->area != NULL)
+
+/* Iterator for TidStore */
+typedef struct TidStoreIter
+{
+ TidStore *ts;
+
+ /* iterator of radix tree. Use either one depending on TidStoreIsShared() */
+ union
+ {
+ shared_rt_iter *shared;
+ local_rt_iter *local;
+ } tree_iter;
+
+ /* we returned all tids? */
+ bool finished;
+
+ /* save for the next iteration */
+ uint64 next_key;
+ uint64 next_val;
+
+ /* output for the caller */
+ TidStoreIterResult result;
+} TidStoreIter;
+
+static void tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val);
+static inline BlockNumber key_get_blkno(TidStore *ts, uint64 key);
+static inline uint64 encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit);
+static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit);
+
+/*
+ * Create a TidStore. The returned object is allocated in backend-local memory.
+ * The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
+ */
+TidStore *
+tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
+{
+ TidStore *ts;
+
+ ts = palloc0(sizeof(TidStore));
+
+ /*
+ * Create the radix tree for the main storage.
+ *
+ * Memory consumption depends on the number of stored tids, but also on the
+ * distribution of them, how the radix tree stores, and the memory management
+ * that backed the radix tree. The maximum bytes that a TidStore can
+ * use is specified by the max_bytes in tidstore_create(). We want the total
+ * amount of memory consumption by a TidStore not to exceed the max_bytes.
+ *
+ * In local TidStore cases, the radix tree uses slab allocators for each kind
+ * of node class. The most memory consuming case while adding Tids associated
+ * with one page (i.e. during tidstore_add_tids()) is that we allocate a new
+ * slab block for a new radix tree node, which is approximately 70kB. Therefore,
+ * we deduct 70kB from the max_bytes.
+ *
+ * In shared cases, DSA allocates the memory segments big enough to follow
+ * a geometric series that approximately doubles the total DSA size (see
+ * make_new_segment() in dsa.c). We simulated the how DSA increases segment
+ * size and the simulation revealed, the 75% threshold for the maximum bytes
+ * perfectly works in case where the max_bytes is a power-of-2, and the 60%
+ * threshold works for other cases.
+ */
+ if (area != NULL)
+ {
+ dsa_pointer dp;
+ float ratio = ((max_bytes & (max_bytes - 1)) == 0) ? 0.75 : 0.6;
+
+ ts->tree.shared = shared_rt_create(CurrentMemoryContext, area,
+ LWTRANCHE_SHARED_TIDSTORE);
+
+ dp = dsa_allocate0(area, sizeof(TidStoreControl));
+ ts->control = (TidStoreControl *) dsa_get_address(area, dp);
+ ts->control->max_bytes = (uint64) (max_bytes * ratio);
+ ts->area = area;
+
+ ts->control->magic = TIDSTORE_MAGIC;
+ LWLockInitialize(&ts->control->lock, LWTRANCHE_SHARED_TIDSTORE);
+ ts->control->handle = dp;
+ ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+ }
+ else
+ {
+ ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+ ts->control = (TidStoreControl *) palloc0(sizeof(TidStoreControl));
+ ts->control->max_bytes = max_bytes - (70 * 1024);
+ }
+
+ ts->control->max_offset = max_offset;
+ ts->control->offset_nbits = pg_ceil_log2_32(max_offset);
+
+ if (ts->control->offset_nbits < TIDSTORE_VALUE_NBITS)
+ ts->control->offset_nbits = TIDSTORE_VALUE_NBITS;
+
+ ts->control->offset_key_nbits =
+ ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
+
+ return ts;
+}
+
+/*
+ * Attach to the shared TidStore using a handle. The returned object is
+ * allocated in backend-local memory using the CurrentMemoryContext.
+ */
+TidStore *
+tidstore_attach(dsa_area *area, tidstore_handle handle)
+{
+ TidStore *ts;
+ dsa_pointer control;
+
+ Assert(area != NULL);
+ Assert(DsaPointerIsValid(handle));
+
+ /* create per-backend state */
+ ts = palloc0(sizeof(TidStore));
+
+ /* Find the control object in shared memory */
+ control = handle;
+
+ /* Set up the TidStore */
+ ts->control = (TidStoreControl *) dsa_get_address(area, control);
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ ts->tree.shared = shared_rt_attach(area, ts->control->tree_handle);
+ ts->area = area;
+
+ return ts;
+}
+
+/*
+ * Detach from a TidStore. This detaches from radix tree and frees the
+ * backend-local resources. The radix tree will continue to exist until
+ * it is either explicitly destroyed, or the area that backs it is returned
+ * to the operating system.
+ */
+void
+tidstore_detach(TidStore *ts)
+{
+ Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+ shared_rt_detach(ts->tree.shared);
+ pfree(ts);
+}
+
+/*
+ * Destroy a TidStore, returning all memory.
+ *
+ * TODO: The caller must be certain that no other backend will attempt to
+ * access the TidStore before calling this function. Other backend must
+ * explicitly call tidstore_detach to free up backend-local memory associated
+ * with the TidStore. The backend that calls tidstore_destroy must not call
+ * tidstore_detach.
+ */
+void
+tidstore_destroy(TidStore *ts)
+{
+ if (TidStoreIsShared(ts))
+ {
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ /*
+ * Vandalize the control block to help catch programming error where
+ * other backends access the memory formerly occupied by this radix
+ * tree.
+ */
+ ts->control->magic = 0;
+ dsa_free(ts->area, ts->control->handle);
+ shared_rt_free(ts->tree.shared);
+ }
+ else
+ {
+ pfree(ts->control);
+ local_rt_free(ts->tree.local);
+ }
+
+ pfree(ts);
+}
+
+/*
+ * Forget all collected Tids. It's similar to tidstore_destroy but we don't free
+ * entire TidStore but recreate only the radix tree storage.
+ */
+void
+tidstore_reset(TidStore *ts)
+{
+ if (TidStoreIsShared(ts))
+ {
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+ /*
+ * Free the radix tree and return allocated DSA segments to
+ * the operating system.
+ */
+ shared_rt_free(ts->tree.shared);
+ dsa_trim(ts->area);
+
+ /* Recreate the radix tree */
+ ts->tree.shared = shared_rt_create(CurrentMemoryContext, ts->area,
+ LWTRANCHE_SHARED_TIDSTORE);
+
+ /* update the radix tree handle as we recreated it */
+ ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
+
+ LWLockRelease(&ts->control->lock);
+ }
+ else
+ {
+ local_rt_free(ts->tree.local);
+ ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
+ }
+}
+
+/* Add Tids on a block to TidStore */
+void
+tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets)
+{
+ uint64 *values;
+ uint64 key;
+ uint64 prev_key;
+ uint64 off_bitmap = 0;
+ int idx;
+ const uint64 key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
+ const int nkeys = UINT64CONST(1) << ts->control->offset_key_nbits;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ values = palloc(sizeof(uint64) * nkeys);
+ key = prev_key = key_base;
+
+ for (int i = 0; i < num_offsets; i++)
+ {
+ uint64 off_bit;
+
+ /* encode the tid to a key and partial offset */
+ key = encode_key_off(ts, blkno, offsets[i], &off_bit);
+
+ /* make sure we scanned the line pointer array in order */
+ Assert(key >= prev_key);
+
+ if (key > prev_key)
+ {
+ idx = prev_key - key_base;
+ Assert(idx >= 0 && idx < nkeys);
+
+ /* write out offset bitmap for this key */
+ values[idx] = off_bitmap;
+
+ /* zero out any gaps up to the current key */
+ for (int empty_idx = idx + 1; empty_idx < key - key_base; empty_idx++)
+ values[empty_idx] = 0;
+
+ /* reset for current key -- the current offset will be handled below */
+ off_bitmap = 0;
+ prev_key = key;
+ }
+
+ off_bitmap |= off_bit;
+ }
+
+ /* save the final index for later */
+ idx = key - key_base;
+ /* write out last offset bitmap */
+ values[idx] = off_bitmap;
+
+ if (TidStoreIsShared(ts))
+ LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+ /* insert the calculated key-values to the tree */
+ for (int i = 0; i <= idx; i++)
+ {
+ if (values[i])
+ {
+ key = key_base + i;
+
+ if (TidStoreIsShared(ts))
+ shared_rt_set(ts->tree.shared, key, &values[i]);
+ else
+ local_rt_set(ts->tree.local, key, &values[i]);
+ }
+ }
+
+ /* update statistics */
+ ts->control->num_tids += num_offsets;
+
+ if (TidStoreIsShared(ts))
+ LWLockRelease(&ts->control->lock);
+
+ pfree(values);
+}
+
+/* Return true if the given tid is present in the TidStore */
+bool
+tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
+{
+ uint64 key;
+ uint64 val = 0;
+ uint64 off_bit;
+ bool found;
+
+ key = tid_to_key_off(ts, tid, &off_bit);
+
+ if (TidStoreIsShared(ts))
+ found = shared_rt_search(ts->tree.shared, key, &val);
+ else
+ found = local_rt_search(ts->tree.local, key, &val);
+
+ if (!found)
+ return false;
+
+ return (val & off_bit) != 0;
+}
+
+/*
+ * Prepare to iterate through a TidStore. Since the radix tree is locked during the
+ * iteration, so tidstore_end_iterate() needs to called when finished.
+ *
+ * Concurrent updates during the iteration will be blocked when inserting a
+ * key-value to the radix tree.
+ */
+TidStoreIter *
+tidstore_begin_iterate(TidStore *ts)
+{
+ TidStoreIter *iter;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ iter = palloc0(sizeof(TidStoreIter));
+ iter->ts = ts;
+
+ iter->result.blkno = InvalidBlockNumber;
+ iter->result.offsets = palloc(sizeof(OffsetNumber) * ts->control->max_offset);
+
+ if (TidStoreIsShared(ts))
+ iter->tree_iter.shared = shared_rt_begin_iterate(ts->tree.shared);
+ else
+ iter->tree_iter.local = local_rt_begin_iterate(ts->tree.local);
+
+ /* If the TidStore is empty, there is no business */
+ if (tidstore_num_tids(ts) == 0)
+ iter->finished = true;
+
+ return iter;
+}
+
+static inline bool
+tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
+{
+ if (TidStoreIsShared(iter->ts))
+ return shared_rt_iterate_next(iter->tree_iter.shared, key, val);
+
+ return local_rt_iterate_next(iter->tree_iter.local, key, val);
+}
+
+/*
+ * Scan the TidStore and return a pointer to TidStoreIterResult that has tids
+ * in one block. We return the block numbers in ascending order and the offset
+ * numbers in each result is also sorted in ascending order.
+ */
+TidStoreIterResult *
+tidstore_iterate_next(TidStoreIter *iter)
+{
+ uint64 key;
+ uint64 val;
+ TidStoreIterResult *result = &(iter->result);
+
+ if (iter->finished)
+ return NULL;
+
+ if (BlockNumberIsValid(result->blkno))
+ {
+ /* Process the previously collected key-value */
+ result->num_offsets = 0;
+ tidstore_iter_extract_tids(iter, iter->next_key, iter->next_val);
+ }
+
+ while (tidstore_iter_kv(iter, &key, &val))
+ {
+ BlockNumber blkno;
+
+ blkno = key_get_blkno(iter->ts, key);
+
+ if (BlockNumberIsValid(result->blkno) && result->blkno != blkno)
+ {
+ /*
+ * We got a key-value pair for a different block. So return the
+ * collected tids, and remember the key-value for the next iteration.
+ */
+ iter->next_key = key;
+ iter->next_val = val;
+ return result;
+ }
+
+ /* Collect tids extracted from the key-value pair */
+ tidstore_iter_extract_tids(iter, key, val);
+ }
+
+ iter->finished = true;
+ return result;
+}
+
+/*
+ * Finish an iteration over TidStore. This needs to be called after finishing
+ * or when existing an iteration.
+ */
+void
+tidstore_end_iterate(TidStoreIter *iter)
+{
+ if (TidStoreIsShared(iter->ts))
+ shared_rt_end_iterate(iter->tree_iter.shared);
+ else
+ local_rt_end_iterate(iter->tree_iter.local);
+
+ pfree(iter->result.offsets);
+ pfree(iter);
+}
+
+/* Return the number of tids we collected so far */
+int64
+tidstore_num_tids(TidStore *ts)
+{
+ uint64 num_tids;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ if (!TidStoreIsShared(ts))
+ return ts->control->num_tids;
+
+ LWLockAcquire(&ts->control->lock, LW_SHARED);
+ num_tids = ts->control->num_tids;
+ LWLockRelease(&ts->control->lock);
+
+ return num_tids;
+}
+
+/* Return true if the current memory usage of TidStore exceeds the limit */
+bool
+tidstore_is_full(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ return (tidstore_memory_usage(ts) > ts->control->max_bytes);
+}
+
+/* Return the maximum memory TidStore can use */
+size_t
+tidstore_max_memory(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ return ts->control->max_bytes;
+}
+
+/* Return the memory usage of TidStore */
+size_t
+tidstore_memory_usage(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ /*
+ * In the shared case, TidStoreControl and radix_tree are backed by the
+ * same DSA area and rt_memory_usage() returns the value including both.
+ * So we don't need to add the size of TidStoreControl separately.
+ */
+ if (TidStoreIsShared(ts))
+ return sizeof(TidStore) + shared_rt_memory_usage(ts->tree.shared);
+
+ return sizeof(TidStore) + sizeof(TidStore) + local_rt_memory_usage(ts->tree.local);
+}
+
+/*
+ * Get a handle that can be used by other processes to attach to this TidStore
+ */
+tidstore_handle
+tidstore_get_handle(TidStore *ts)
+{
+ Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+ return ts->control->handle;
+}
+
+/* Extract tids from the given key-value pair */
+static void
+tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
+{
+ TidStoreIterResult *result = (&iter->result);
+
+ while (val)
+ {
+ uint64 tid_i;
+ OffsetNumber off;
+
+ tid_i = key << TIDSTORE_VALUE_NBITS;
+ tid_i |= pg_rightmost_one_pos64(val);
+
+ off = tid_i & ((UINT64CONST(1) << iter->ts->control->offset_nbits) - 1);
+
+ Assert(result->num_offsets < iter->ts->control->max_offset);
+ result->offsets[result->num_offsets++] = off;
+
+ /* unset the rightmost bit */
+ val &= ~pg_rightmost_one64(val);
+ }
+
+ result->blkno = key_get_blkno(iter->ts, key);
+}
+
+/* Get block number from the given key */
+static inline BlockNumber
+key_get_blkno(TidStore *ts, uint64 key)
+{
+ return (BlockNumber) (key >> ts->control->offset_key_nbits);
+}
+
+/* Encode a tid to key and offset */
+static inline uint64
+tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit)
+{
+ uint32 offset = ItemPointerGetOffsetNumber(tid);
+ BlockNumber block = ItemPointerGetBlockNumber(tid);
+
+ return encode_key_off(ts, block, offset, off_bit);
+}
+
+/* encode a block and offset to a key and partial offset */
+static inline uint64
+encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit)
+{
+ uint64 key;
+ uint64 tid_i;
+ uint32 off_lower;
+
+ off_lower = offset & TIDSTORE_OFFSET_MASK;
+ Assert(off_lower < (sizeof(uint64) * BITS_PER_BYTE));
+
+ *off_bit = UINT64CONST(1) << off_lower;
+ tid_i = offset | ((uint64) block << ts->control->offset_nbits);
+ key = tid_i >> TIDSTORE_VALUE_NBITS;
+
+ return key;
+}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index d2ec396045..55b3a04097 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -176,6 +176,8 @@ static const char *const BuiltinTrancheNames[] = {
"SharedTupleStore",
/* LWTRANCHE_SHARED_TIDBITMAP: */
"SharedTidBitmap",
+ /* LWTRANCHE_SHARED_TIDSTORE: */
+ "SharedTidStore",
/* LWTRANCHE_PARALLEL_APPEND: */
"ParallelAppend",
/* LWTRANCHE_PER_XACT_PREDICATE_LIST: */
diff --git a/src/include/access/tidstore.h b/src/include/access/tidstore.h
new file mode 100644
index 0000000000..a35a52124a
--- /dev/null
+++ b/src/include/access/tidstore.h
@@ -0,0 +1,49 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.h
+ * Tid storage.
+ *
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/tidstore.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef TIDSTORE_H
+#define TIDSTORE_H
+
+#include "storage/itemptr.h"
+#include "utils/dsa.h"
+
+typedef dsa_pointer tidstore_handle;
+
+typedef struct TidStore TidStore;
+typedef struct TidStoreIter TidStoreIter;
+
+typedef struct TidStoreIterResult
+{
+ BlockNumber blkno;
+ OffsetNumber *offsets;
+ int num_offsets;
+} TidStoreIterResult;
+
+extern TidStore *tidstore_create(size_t max_bytes, int max_offset, dsa_area *dsa);
+extern TidStore *tidstore_attach(dsa_area *dsa, dsa_pointer handle);
+extern void tidstore_detach(TidStore *ts);
+extern void tidstore_destroy(TidStore *ts);
+extern void tidstore_reset(TidStore *ts);
+extern void tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets);
+extern bool tidstore_lookup_tid(TidStore *ts, ItemPointer tid);
+extern TidStoreIter * tidstore_begin_iterate(TidStore *ts);
+extern TidStoreIterResult *tidstore_iterate_next(TidStoreIter *iter);
+extern void tidstore_end_iterate(TidStoreIter *iter);
+extern int64 tidstore_num_tids(TidStore *ts);
+extern bool tidstore_is_full(TidStore *ts);
+extern size_t tidstore_max_memory(TidStore *ts);
+extern size_t tidstore_memory_usage(TidStore *ts);
+extern tidstore_handle tidstore_get_handle(TidStore *ts);
+
+#endif /* TIDSTORE_H */
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index d2c7afb8f4..07002fdfbe 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -199,6 +199,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_PER_SESSION_RECORD_TYPMOD,
LWTRANCHE_SHARED_TUPLESTORE,
LWTRANCHE_SHARED_TIDBITMAP,
+ LWTRANCHE_SHARED_TIDSTORE,
LWTRANCHE_PARALLEL_APPEND,
LWTRANCHE_PER_XACT_PREDICATE_LIST,
LWTRANCHE_PGSTATS_DSA,
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 9659eb85d7..bddc16ada7 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
test_rls_hooks \
test_shm_mq \
test_slru \
+ test_tidstore \
unsafe_tests \
worker_spi
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 232cbdac80..c0d5645ad8 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -30,5 +30,6 @@ subdir('test_regex')
subdir('test_rls_hooks')
subdir('test_shm_mq')
subdir('test_slru')
+subdir('test_tidstore')
subdir('unsafe_tests')
subdir('worker_spi')
diff --git a/src/test/modules/test_tidstore/Makefile b/src/test/modules/test_tidstore/Makefile
new file mode 100644
index 0000000000..dab107d70c
--- /dev/null
+++ b/src/test/modules/test_tidstore/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_tidstore/Makefile
+
+MODULE_big = test_tidstore
+OBJS = \
+ $(WIN32RES) \
+ test_tidstore.o
+PGFILEDESC = "test_tidstore - test code for src/backend/access/common/tidstore.c"
+
+EXTENSION = test_tidstore
+DATA = test_tidstore--1.0.sql
+
+REGRESS = test_tidstore
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_tidstore
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_tidstore/expected/test_tidstore.out b/src/test/modules/test_tidstore/expected/test_tidstore.out
new file mode 100644
index 0000000000..7ff2f9af87
--- /dev/null
+++ b/src/test/modules/test_tidstore/expected/test_tidstore.out
@@ -0,0 +1,13 @@
+CREATE EXTENSION test_tidstore;
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
+NOTICE: testing empty tidstore
+NOTICE: testing basic operations
+ test_tidstore
+---------------
+
+(1 row)
+
diff --git a/src/test/modules/test_tidstore/meson.build b/src/test/modules/test_tidstore/meson.build
new file mode 100644
index 0000000000..31f2da7b61
--- /dev/null
+++ b/src/test/modules/test_tidstore/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_tidstore_sources = files(
+ 'test_tidstore.c',
+)
+
+if host_system == 'windows'
+ test_tidstore_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_tidstore',
+ '--FILEDESC', 'test_tidstore - test code for src/backend/access/common/tidstore.c',])
+endif
+
+test_tidstore = shared_module('test_tidstore',
+ test_tidstore_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += test_tidstore
+
+install_data(
+ 'test_tidstore.control',
+ 'test_tidstore--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'test_tidstore',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'test_tidstore',
+ ],
+ },
+}
diff --git a/src/test/modules/test_tidstore/sql/test_tidstore.sql b/src/test/modules/test_tidstore/sql/test_tidstore.sql
new file mode 100644
index 0000000000..03aea31815
--- /dev/null
+++ b/src/test/modules/test_tidstore/sql/test_tidstore.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_tidstore;
+
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
diff --git a/src/test/modules/test_tidstore/test_tidstore--1.0.sql b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
new file mode 100644
index 0000000000..47e9149900
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_tidstore/test_tidstore--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_tidstore" to load this file. \quit
+
+CREATE FUNCTION test_tidstore()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_tidstore/test_tidstore.c b/src/test/modules/test_tidstore/test_tidstore.c
new file mode 100644
index 0000000000..9a1217f833
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.c
@@ -0,0 +1,226 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_tidstore.c
+ * Test TidStore data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_tidstore/test_tidstore.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/tidstore.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "storage/lwlock.h"
+#include "utils/memutils.h"
+
+PG_MODULE_MAGIC;
+
+/* #define TEST_SHARED_TIDSTORE 1 */
+
+#define TEST_TIDSTORE_MAX_BYTES (2 * 1024 * 1024L) /* 2MB */
+
+PG_FUNCTION_INFO_V1(test_tidstore);
+
+static void
+check_tid(TidStore *ts, BlockNumber blkno, OffsetNumber off, bool expect)
+{
+ ItemPointerData tid;
+ bool found;
+
+ ItemPointerSet(&tid, blkno, off);
+
+ found = tidstore_lookup_tid(ts, &tid);
+
+ if (found != expect)
+ elog(ERROR, "lookup TID (%u, %u) returned %d, expected %d",
+ blkno, off, found, expect);
+}
+
+static void
+test_basic(int max_offset)
+{
+#define TEST_TIDSTORE_NUM_BLOCKS 5
+#define TEST_TIDSTORE_NUM_OFFSETS 5
+
+ TidStore *ts;
+ TidStoreIter *iter;
+ TidStoreIterResult *iter_result;
+ BlockNumber blks[TEST_TIDSTORE_NUM_BLOCKS] = {
+ 0, MaxBlockNumber, MaxBlockNumber - 1, 1, MaxBlockNumber / 2,
+ };
+ BlockNumber blks_sorted[TEST_TIDSTORE_NUM_BLOCKS] = {
+ 0, 1, MaxBlockNumber / 2, MaxBlockNumber - 1, MaxBlockNumber
+ };
+ OffsetNumber offs[TEST_TIDSTORE_NUM_OFFSETS];
+ int blk_idx;
+
+#ifdef TEST_SHARED_TIDSTORE
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_tidstore");
+ dsa = dsa_create(tranche_id);
+
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, dsa);
+#else
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
+#endif
+
+ /* prepare the offset array */
+ offs[0] = FirstOffsetNumber;
+ offs[1] = FirstOffsetNumber + 1;
+ offs[2] = max_offset / 2;
+ offs[3] = max_offset - 1;
+ offs[4] = max_offset;
+
+ /* add tids */
+ for (int i = 0; i < TEST_TIDSTORE_NUM_BLOCKS; i++)
+ tidstore_add_tids(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
+
+ /* lookup test */
+ for (OffsetNumber off = FirstOffsetNumber ; off < max_offset; off++)
+ {
+ bool expect = false;
+ for (int i = 0; i < TEST_TIDSTORE_NUM_OFFSETS; i++)
+ {
+ if (offs[i] == off)
+ {
+ expect = true;
+ break;
+ }
+ }
+
+ check_tid(ts, 0, off, expect);
+ check_tid(ts, 2, off, false);
+ check_tid(ts, MaxBlockNumber - 2, off, false);
+ check_tid(ts, MaxBlockNumber, off, expect);
+ }
+
+ /* test the number of tids */
+ if (tidstore_num_tids(ts) != (TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS))
+ elog(ERROR, "tidstore_num_tids returned " UINT64_FORMAT ", expected %d",
+ tidstore_num_tids(ts),
+ TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS);
+
+ /* iteration test */
+ iter = tidstore_begin_iterate(ts);
+ blk_idx = 0;
+ while ((iter_result = tidstore_iterate_next(iter)) != NULL)
+ {
+ /* check the returned block number */
+ if (blks_sorted[blk_idx] != iter_result->blkno)
+ elog(ERROR, "tidstore_iterate_next returned block number %u, expected %u",
+ iter_result->blkno, blks_sorted[blk_idx]);
+
+ /* check the returned offset numbers */
+ if (TEST_TIDSTORE_NUM_OFFSETS != iter_result->num_offsets)
+ elog(ERROR, "tidstore_iterate_next returned %u offsets, expected %u",
+ iter_result->num_offsets, TEST_TIDSTORE_NUM_OFFSETS);
+
+ for (int i = 0; i < iter_result->num_offsets; i++)
+ {
+ if (offs[i] != iter_result->offsets[i])
+ elog(ERROR, "tidstore_iterate_next returned offset number %u on block %u, expected %u",
+ iter_result->offsets[i], iter_result->blkno, offs[i]);
+ }
+
+ blk_idx++;
+ }
+
+ if (blk_idx != TEST_TIDSTORE_NUM_BLOCKS)
+ elog(ERROR, "tidstore_iterate_next returned %d blocks, expected %d",
+ blk_idx, TEST_TIDSTORE_NUM_BLOCKS);
+
+ /* remove all tids */
+ tidstore_reset(ts);
+
+ /* test the number of tids */
+ if (tidstore_num_tids(ts) != 0)
+ elog(ERROR, "tidstore_num_tids on empty store returned non-zero");
+
+ /* lookup test for empty store */
+ for (OffsetNumber off = FirstOffsetNumber ; off < MaxHeapTuplesPerPage;
+ off++)
+ {
+ check_tid(ts, 0, off, false);
+ check_tid(ts, 2, off, false);
+ check_tid(ts, MaxBlockNumber - 2, off, false);
+ check_tid(ts, MaxBlockNumber, off, false);
+ }
+
+ tidstore_destroy(ts);
+
+#ifdef TEST_SHARED_TIDSTORE
+ dsa_detach(dsa);
+#endif
+}
+
+static void
+test_empty(void)
+{
+ TidStore *ts;
+ TidStoreIter *iter;
+ ItemPointerData tid;
+
+#ifdef TEST_SHARED_TIDSTORE
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_tidstore");
+ dsa = dsa_create(tranche_id);
+
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, dsa);
+#else
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
+#endif
+
+ elog(NOTICE, "testing empty tidstore");
+
+ ItemPointerSet(&tid, 0, FirstOffsetNumber);
+ if (tidstore_lookup_tid(ts, &tid))
+ elog(ERROR, "tidstore_lookup_tid for (0,1) on empty store returned true");
+
+ ItemPointerSet(&tid, MaxBlockNumber, MaxOffsetNumber);
+ if (tidstore_lookup_tid(ts, &tid))
+ elog(ERROR, "tidstore_lookup_tid for (%u,%u) on empty store returned true",
+ MaxBlockNumber, MaxOffsetNumber);
+
+ if (tidstore_num_tids(ts) != 0)
+ elog(ERROR, "tidstore_num_entries on empty store returned non-zero");
+
+ if (tidstore_is_full(ts))
+ elog(ERROR, "tidstore_is_full on empty store returned true");
+
+ iter = tidstore_begin_iterate(ts);
+
+ if (tidstore_iterate_next(iter) != NULL)
+ elog(ERROR, "tidstore_iterate_next on empty store returned TIDs");
+
+ tidstore_end_iterate(iter);
+
+ tidstore_destroy(ts);
+
+#ifdef TEST_SHARED_TIDSTORE
+ dsa_detach(dsa);
+#endif
+}
+
+Datum
+test_tidstore(PG_FUNCTION_ARGS)
+{
+ test_empty();
+
+ elog(NOTICE, "testing basic operations");
+ test_basic(MaxHeapTuplesPerPage);
+ test_basic(10);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_tidstore/test_tidstore.control b/src/test/modules/test_tidstore/test_tidstore.control
new file mode 100644
index 0000000000..9b6bd4638f
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.control
@@ -0,0 +1,4 @@
+comment = 'Test code for tidstore'
+default_version = '1.0'
+module_pathname = '$libdir/test_tidstore'
+relocatable = true
--
2.31.1
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 15:26 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-04-17 15:20 ` Masahiko Sawada <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Masahiko Sawada @ 2023-04-17 15:20 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Sat, Mar 11, 2023 at 12:26 AM Masahiko Sawada <[email protected]> wrote:
>
> On Fri, Mar 10, 2023 at 11:30 PM Masahiko Sawada <[email protected]> wrote:
> >
> > On Fri, Mar 10, 2023 at 3:42 PM John Naylor
> > <[email protected]> wrote:
> > >
> > > On Thu, Mar 9, 2023 at 1:51 PM Masahiko Sawada <[email protected]> wrote:
> > >
> > > > I've attached the new version patches. I merged improvements and fixes
> > > > I did in the v29 patch.
> > >
> > > I haven't yet had a chance to look at those closely, since I've had to devote time to other commitments. I remember I wasn't particularly impressed that v29-0008 mixed my requested name-casing changes with a bunch of other random things. Separating those out would be an obvious way to make it easier for me to look at, whenever I can get back to this. I need to look at the iteration changes as well, in addition to testing memory measurement (thanks for the new results, they look encouraging).
> >
> > Okay, I'll separate them again.
>
> Attached new patch series. In addition to separate them again, I've
> fixed a conflict with HEAD.
>
I've attached updated version patches to make cfbot happy. Also, I've
splitted fixup patches further(from 0007 except for 0016 and 0018) to
make reviews easy. These patches have the prefix radix tree, tidstore,
and vacuum, indicating the part it changes. 0016 patch is to change
DSA so that we can specify both the initial and max segment size and
0017 makes use of it in vacuumparallel.c I'm still researching a
better solution for memory limitation but it's the best solution for
me for now.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] v32-0015-vacuum-Miscellaneous-updates.patch (4.9K, ../../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/2-v32-0015-vacuum-Miscellaneous-updates.patch)
download | inline diff:
From 16e55ffde1cb152dc94cf38a9f6c8442b78be284 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 18:07:04 +0900
Subject: [PATCH v32 15/18] vacuum: Miscellaneous updates
fix typos, comment updates, etc.
---
doc/src/sgml/monitoring.sgml | 2 +-
src/backend/access/heap/vacuumlazy.c | 17 ++++++++---------
src/backend/commands/vacuumparallel.c | 13 +++++++------
3 files changed, 16 insertions(+), 16 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 9b64614beb..67ab9fa2bc 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -7331,7 +7331,7 @@ FROM pg_stat_get_backend_idset() AS backendid;
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>num_dead_tuple_bytes</structfield> <type>bigint</type>
+ <structfield>dead_tuple_bytes</structfield> <type>bigint</type>
</para>
<para>
Amount of dead tuple data collected since the last index vacuum cycle.
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index be487aced6..228daad750 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -10,11 +10,10 @@
* of dead TIDs at once.
*
* We are willing to use at most maintenance_work_mem (or perhaps
- * autovacuum_work_mem) memory space to keep track of dead TIDs. We initially
- * create a TidStore with the maximum bytes that can be used by the TidStore.
- * If the TidStore is full, we must call lazy_vacuum to vacuum indexes (and to
- * vacuum the pages that we've pruned). This frees up the memory space dedicated
- * to storing dead TIDs.
+ * autovacuum_work_mem) memory space to keep track of dead TIDs. If the
+ * TidStore is full, we must call lazy_vacuum to vacuum indexes (and to vacuum
+ * the pages that we've pruned). This frees up the memory space dedicated to
+ * to store dead TIDs.
*
* In practice VACUUM will often complete its initial pass over the target
* heap relation without ever running out of space to store TIDs. This means
@@ -2392,7 +2391,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
Buffer vmbuffer = InvalidBuffer;
LVSavedErrInfo saved_err_info;
TidStoreIter *iter;
- TidStoreIterResult *result;
+ TidStoreIterResult *iter_result;
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2417,7 +2416,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
vacuum_delay_point();
- blkno = result->blkno;
+ blkno = iter_result->blkno;
vacrel->blkno = blkno;
/*
@@ -2431,8 +2430,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
buf = ReadBufferExtended(vacrel->rel, MAIN_FORKNUM, blkno, RBM_NORMAL,
vacrel->bstrategy);
LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
- lazy_vacuum_heap_page(vacrel, blkno, result->offsets, result->num_offsets,
- buf, vmbuffer);
+ lazy_vacuum_heap_page(vacrel, blkno, iter_result->offsets,
+ iter_result->num_offsets, buf, vmbuffer);
/* Now that we've vacuumed the page, record its available space */
page = BufferGetPage(buf);
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index be83ceb871..8385d375db 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -9,11 +9,12 @@
* In a parallel vacuum, we perform both index bulk deletion and index cleanup
* with parallel worker processes. Individual indexes are processed by one
* vacuum process. ParalleVacuumState contains shared information as well as
- * the shared TidStore. We launch parallel worker processes at the start of
- * parallel index bulk-deletion and index cleanup and once all indexes are
- * processed, the parallel worker processes exit. Each time we process indexes
- * in parallel, the parallel context is re-initialized so that the same DSM can
- * be used for multiple passes of index bulk-deletion and index cleanup.
+ * the memory space for storing dead items allocated in the DSA area. We
+ * launch parallel worker processes at the start of parallel index
+ * bulk-deletion and index cleanup and once all indexes are processed, the
+ * parallel worker processes exit. Each time we process indexes in parallel,
+ * the parallel context is re-initialized so that the same DSM can be used for
+ * multiple passes of index bulk-deletion and index cleanup.
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -299,7 +300,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_estimate_chunk(&pcxt->estimator, est_shared_len);
shm_toc_estimate_keys(&pcxt->estimator, 1);
- /* Estimate size for dead tuple DSA -- PARALLEL_VACUUM_KEY_DSA */
+ /* Initial size for dead tuple DSA -- PARALLEL_VACUUM_KEY_DEAD_ITEMS */
shm_toc_estimate_chunk(&pcxt->estimator, dsa_minsize);
shm_toc_estimate_keys(&pcxt->estimator, 1);
--
2.31.1
[application/octet-stream] v32-0016-Make-initial-and-maximum-DSA-segment-size-config.patch (9.6K, ../../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/3-v32-0016-Make-initial-and-maximum-DSA-segment-size-config.patch)
download | inline diff:
From bc7b41a404cc1c8050c400919836951f78456aef Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 21:59:12 +0900
Subject: [PATCH v32 16/18] Make initial and maximum DSA segment size
configurable
---
src/backend/utils/mmgr/dsa.c | 64 +++++++++++++++++-------------------
src/include/utils/dsa.h | 45 ++++++++++++++++++++++---
2 files changed, 71 insertions(+), 38 deletions(-)
diff --git a/src/backend/utils/mmgr/dsa.c b/src/backend/utils/mmgr/dsa.c
index 80555aefff..b6238bf4a3 100644
--- a/src/backend/utils/mmgr/dsa.c
+++ b/src/backend/utils/mmgr/dsa.c
@@ -60,14 +60,6 @@
#include "utils/freepage.h"
#include "utils/memutils.h"
-/*
- * The size of the initial DSM segment that backs a dsa_area created by
- * dsa_create. After creating some number of segments of this size we'll
- * double this size, and so on. Larger segments may be created if necessary
- * to satisfy large requests.
- */
-#define DSA_INITIAL_SEGMENT_SIZE ((size_t) (1 * 1024 * 1024))
-
/*
* How many segments to create before we double the segment size. If this is
* low, then there is likely to be a lot of wasted space in the largest
@@ -77,17 +69,6 @@
*/
#define DSA_NUM_SEGMENTS_AT_EACH_SIZE 2
-/*
- * The number of bits used to represent the offset part of a dsa_pointer.
- * This controls the maximum size of a segment, the maximum possible
- * allocation size and also the maximum number of segments per area.
- */
-#if SIZEOF_DSA_POINTER == 4
-#define DSA_OFFSET_WIDTH 27 /* 32 segments of size up to 128MB */
-#else
-#define DSA_OFFSET_WIDTH 40 /* 1024 segments of size up to 1TB */
-#endif
-
/*
* The maximum number of DSM segments that an area can own, determined by
* the number of bits remaining (but capped at 1024).
@@ -98,9 +79,6 @@
/* The bitmask for extracting the offset from a dsa_pointer. */
#define DSA_OFFSET_BITMASK (((dsa_pointer) 1 << DSA_OFFSET_WIDTH) - 1)
-/* The maximum size of a DSM segment. */
-#define DSA_MAX_SEGMENT_SIZE ((size_t) 1 << DSA_OFFSET_WIDTH)
-
/* Number of pages (see FPM_PAGE_SIZE) per regular superblock. */
#define DSA_PAGES_PER_SUPERBLOCK 16
@@ -319,6 +297,10 @@ typedef struct
dsa_segment_index segment_bins[DSA_NUM_SEGMENT_BINS];
/* The object pools for each size class. */
dsa_area_pool pools[DSA_NUM_SIZE_CLASSES];
+ /* initial allocation segment size */
+ size_t init_segment_size;
+ /* maximum allocation segment size */
+ size_t max_segment_size;
/* The total size of all active segments. */
size_t total_segment_size;
/* The maximum total size of backing storage we are allowed. */
@@ -413,7 +395,9 @@ static dsa_segment_map *make_new_segment(dsa_area *area, size_t requested_pages)
static dsa_area *create_internal(void *place, size_t size,
int tranche_id,
dsm_handle control_handle,
- dsm_segment *control_segment);
+ dsm_segment *control_segment,
+ size_t init_segment_size,
+ size_t max_segment_size);
static dsa_area *attach_internal(void *place, dsm_segment *segment,
dsa_handle handle);
static void check_for_freed_segments(dsa_area *area);
@@ -429,7 +413,8 @@ static void check_for_freed_segments_locked(dsa_area *area);
* we require the caller to provide one.
*/
dsa_area *
-dsa_create(int tranche_id)
+dsa_create_extended(int tranche_id, size_t init_segment_size,
+ size_t max_segment_size)
{
dsm_segment *segment;
dsa_area *area;
@@ -438,7 +423,7 @@ dsa_create(int tranche_id)
* Create the DSM segment that will hold the shared control object and the
* first segment of usable space.
*/
- segment = dsm_create(DSA_INITIAL_SEGMENT_SIZE, 0);
+ segment = dsm_create(init_segment_size, 0);
/*
* All segments backing this area are pinned, so that DSA can explicitly
@@ -450,9 +435,10 @@ dsa_create(int tranche_id)
/* Create a new DSA area with the control object in this segment. */
area = create_internal(dsm_segment_address(segment),
- DSA_INITIAL_SEGMENT_SIZE,
+ init_segment_size,
tranche_id,
- dsm_segment_handle(segment), segment);
+ dsm_segment_handle(segment), segment,
+ init_segment_size, max_segment_size);
/* Clean up when the control segment detaches. */
on_dsm_detach(segment, &dsa_on_dsm_detach_release_in_place,
@@ -478,13 +464,15 @@ dsa_create(int tranche_id)
* See dsa_create() for a note about the tranche arguments.
*/
dsa_area *
-dsa_create_in_place(void *place, size_t size,
- int tranche_id, dsm_segment *segment)
+dsa_create_in_place_extended(void *place, size_t size,
+ int tranche_id, dsm_segment *segment,
+ size_t init_segment_size, size_t max_segment_size)
{
dsa_area *area;
area = create_internal(place, size, tranche_id,
- DSM_HANDLE_INVALID, NULL);
+ DSM_HANDLE_INVALID, NULL,
+ init_segment_size, max_segment_size);
/*
* Clean up when the control segment detaches, if a containing DSM segment
@@ -1215,7 +1203,8 @@ static dsa_area *
create_internal(void *place, size_t size,
int tranche_id,
dsm_handle control_handle,
- dsm_segment *control_segment)
+ dsm_segment *control_segment,
+ size_t init_segment_size, size_t max_segment_size)
{
dsa_area_control *control;
dsa_area *area;
@@ -1225,6 +1214,11 @@ create_internal(void *place, size_t size,
size_t metadata_bytes;
int i;
+ /* Validate the initial and maximum block sizes */
+ Assert(init_segment_size >= 1024);
+ Assert(max_segment_size >= init_segment_size);
+ Assert(max_segment_size <= DSA_MAX_SEGMENT_SIZE);
+
/* Sanity check on the space we have to work in. */
if (size < dsa_minimum_size())
elog(ERROR, "dsa_area space must be at least %zu, but %zu provided",
@@ -1254,8 +1248,10 @@ create_internal(void *place, size_t size,
control->segment_header.prev = DSA_SEGMENT_INDEX_NONE;
control->segment_header.usable_pages = usable_pages;
control->segment_header.freed = false;
- control->segment_header.size = DSA_INITIAL_SEGMENT_SIZE;
+ control->segment_header.size = size;
control->handle = control_handle;
+ control->init_segment_size = init_segment_size;
+ control->max_segment_size = max_segment_size;
control->max_total_segment_size = (size_t) -1;
control->total_segment_size = size;
control->segment_handles[0] = control_handle;
@@ -2124,9 +2120,9 @@ make_new_segment(dsa_area *area, size_t requested_pages)
* move to huge pages in the future. Then we work back to the number of
* pages we can fit.
*/
- total_size = DSA_INITIAL_SEGMENT_SIZE *
+ total_size = area->control->init_segment_size *
((size_t) 1 << (new_index / DSA_NUM_SEGMENTS_AT_EACH_SIZE));
- total_size = Min(total_size, DSA_MAX_SEGMENT_SIZE);
+ total_size = Min(total_size, area->control->max_segment_size);
total_size = Min(total_size,
area->control->max_total_segment_size -
area->control->total_segment_size);
diff --git a/src/include/utils/dsa.h b/src/include/utils/dsa.h
index 2af215484f..90b7b0d93f 100644
--- a/src/include/utils/dsa.h
+++ b/src/include/utils/dsa.h
@@ -77,6 +77,28 @@ typedef pg_atomic_uint64 dsa_pointer_atomic;
/* A sentinel value for dsa_pointer used to indicate failure to allocate. */
#define InvalidDsaPointer ((dsa_pointer) 0)
+/*
+ * The default size of the initial DSM segment that backs a dsa_area created
+ * by dsa_create. After creating some number of segments of this size we'll
+ * double this size, and so on. Larger segments may be created if necessary
+ * to satisfy large requests.
+ */
+#define DSA_INITIAL_SEGMENT_SIZE ((size_t) (1 * 1024 * 1024))
+
+/*
+ * The number of bits used to represent the offset part of a dsa_pointer.
+ * This controls the maximum size of a segment, the maximum possible
+ * allocation size and also the maximum number of segments per area.
+ */
+#if SIZEOF_DSA_POINTER == 4
+#define DSA_OFFSET_WIDTH 27 /* 32 segments of size up to 128MB */
+#else
+#define DSA_OFFSET_WIDTH 40 /* 1024 segments of size up to 1TB */
+#endif
+
+/* The maximum size of a DSM segment. */
+#define DSA_MAX_SEGMENT_SIZE ((size_t) 1 << DSA_OFFSET_WIDTH)
+
/* Check if a dsa_pointer value is valid. */
#define DsaPointerIsValid(x) ((x) != InvalidDsaPointer)
@@ -88,6 +110,19 @@ typedef pg_atomic_uint64 dsa_pointer_atomic;
#define dsa_allocate0(area, size) \
dsa_allocate_extended(area, size, DSA_ALLOC_ZERO)
+/* Create dsa_area with default segment sizes */
+#define dsa_create(tranch_id) \
+ dsa_create_extended(tranch_id, DSA_INITIAL_SEGMENT_SIZE, \
+ DSA_MAX_SEGMENT_SIZE)
+
+/*
+ * Create dsa_area with default segment sizes in an existing share memory
+ * space.
+ */
+#define dsa_create_in_place(place, size, tranch_id, segment) \
+ dsa_create_in_place_extended(place, size, tranch_id, segment, \
+ DSA_INITIAL_SEGMENT_SIZE, DSA_MAX_SEGMENT_SIZE)
+
/*
* The type used for dsa_area handles. dsa_handle values can be shared with
* other processes, so that they can attach to them. This provides a way to
@@ -102,10 +137,12 @@ typedef dsm_handle dsa_handle;
/* Sentinel value to use for invalid dsa_handles. */
#define DSA_HANDLE_INVALID ((dsa_handle) DSM_HANDLE_INVALID)
-
-extern dsa_area *dsa_create(int tranche_id);
-extern dsa_area *dsa_create_in_place(void *place, size_t size,
- int tranche_id, dsm_segment *segment);
+extern dsa_area *dsa_create_extended(int tranche_id, size_t init_segment_size,
+ size_t max_segment_size);
+extern dsa_area *dsa_create_in_place_extended(void *place, size_t size,
+ int tranche_id, dsm_segment *segment,
+ size_t init_segment_size,
+ size_t max_segment_size);
extern dsa_area *dsa_attach(dsa_handle handle);
extern dsa_area *dsa_attach_in_place(void *place, dsm_segment *segment);
extern void dsa_release_in_place(void *place);
--
2.31.1
[application/octet-stream] v32-0014-tidstore-Miscellaneous-updates.patch (7.3K, ../../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/4-v32-0014-tidstore-Miscellaneous-updates.patch)
download | inline diff:
From f9be0044ee6e35dd44bceca59d733ba8cdf5373e Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 18:01:46 +0900
Subject: [PATCH v32 14/18] tidstore: Miscellaneous updates.
comment updates, fix typos, etc.
---
src/backend/access/common/tidstore.c | 78 +++++++++++--------
.../modules/test_tidstore/test_tidstore.c | 1 +
2 files changed, 47 insertions(+), 32 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 15b77b5bcb..9360520482 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -3,18 +3,19 @@
* tidstore.c
* Tid (ItemPointerData) storage implementation.
*
- * This module provides a in-memory data structure to store Tids (ItemPointer).
- * Internally, a tid is encoded as a pair of 64-bit key and 64-bit value, and
- * stored in the radix tree.
+ * TidStore is a in-memory data structure to store tids (ItemPointerData).
+ * Internally, a tid is encoded as a pair of 64-bit key and 64-bit value,
+ * and stored in the radix tree.
*
* TidStore can be shared among parallel worker processes by passing DSA area
* to TidStoreCreate(). Other backends can attach to the shared TidStore by
* TidStoreAttach().
*
- * Regarding the concurrency, it basically relies on the concurrency support in
- * the radix tree, but we acquires the lock on a TidStore in some cases, for
- * example, when to reset the store and when to access the number tids in the
- * store (num_tids).
+ * Regarding the concurrency support, we use a single LWLock for the TidStore.
+ * The TidStore is exclusively locked when inserting encoded tids to the
+ * radix tree or when resetting itself. When searching on the TidStore or
+ * doing the iteration, it is not locked but the underlying radix tree is
+ * locked in shared mode.
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -34,16 +35,18 @@
#include "utils/memutils.h"
/*
- * For encoding purposes, tids are represented as a pair of 64-bit key and
- * 64-bit value. First, we construct 64-bit unsigned integer by combining
- * the block number and the offset number. The number of bits used for the
- * offset number is specified by max_offsets in tidstore_create(). We are
- * frugal with the bits, because smaller keys could help keeping the radix
- * tree shallow.
+ * For encoding purposes, a tid is represented as a pair of 64-bit key and
+ * 64-bit value.
*
- * For example, a tid of heap with 8kB blocks uses the lowest 9 bits for
- * the offset number and uses the next 32 bits for the block number. That
- * is, only 41 bits are used:
+ * First, we construct a 64-bit unsigned integer by combining the block
+ * number and the offset number. The number of bits used for the offset number
+ * is specified by max_off in TidStoreCreate(). We are frugal with the bits,
+ * because smaller keys could help keeping the radix tree shallow.
+ *
+ * For example, a tid of heap on a 8kB block uses the lowest 9 bits for
+ * the offset number and uses the next 32 bits for the block number. 9 bits
+ * are enough for the offset number, because MaxHeapTuplesPerPage < 2^9
+ * on 8kB blocks. That is, only 41 bits are used:
*
* uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
*
@@ -52,25 +55,27 @@
* u = unused bit
* (high on the left, low on the right)
*
- * 9 bits are enough for the offset number, because MaxHeapTuplesPerPage < 2^9
- * on 8kB blocks.
- *
- * The 64-bit value is the bitmap representation of the lowest 6 bits
- * (TIDSTORE_VALUE_NBITS) of the integer, and the rest 35 bits are used
- * as the key:
+ * Then, 64-bit value is the bitmap representation of the lowest 6 bits
+ * (LOWER_OFFSET_NBITS) of the integer, and 64-bit key consists of the
+ * upper 3 bits of the offset number and the block number, 35 bits in
+ * total:
*
* uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
* |----| value
- * |---------------------------------------------| key
+ * |--------------------------------------| key
*
* The maximum height of the radix tree is 5 in this case.
+ *
+ * If the number of bits required for offset numbers fits in LOWER_OFFSET_NBITS,
+ * 64-bit value is the bitmap representation of the offset number, and the
+ * 64-bit key is the block number.
*/
typedef uint64 tidkey;
typedef uint64 offsetbm;
#define LOWER_OFFSET_NBITS 6 /* log(sizeof(offsetbm), 2) */
#define LOWER_OFFSET_MASK ((1 << LOWER_OFFSET_NBITS) - 1)
-/* A magic value used to identify our TidStores. */
+/* A magic value used to identify our TidStore. */
#define TIDSTORE_MAGIC 0x826f6a10
#define RT_PREFIX local_rt
@@ -152,8 +157,10 @@ typedef struct TidStoreIter
tidkey next_tidkey;
offsetbm next_off_bitmap;
- /* output for the caller */
- TidStoreIterResult result;
+ /*
+ * output for the caller. Must be last because variable-size.
+ */
+ TidStoreIterResult output;
} TidStoreIter;
static void iter_decode_key_off(TidStoreIter *iter, tidkey key, offsetbm off_bitmap);
@@ -205,7 +212,7 @@ TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
dp = dsa_allocate0(area, sizeof(TidStoreControl));
ts->control = (TidStoreControl *) dsa_get_address(area, dp);
- ts->control->max_bytes = (uint64) (max_bytes * ratio);
+ ts->control->max_bytes = (size_t) (max_bytes * ratio);
ts->area = area;
ts->control->magic = TIDSTORE_MAGIC;
@@ -353,7 +360,11 @@ TidStoreReset(TidStore *ts)
}
}
-/* Add Tids on a block to TidStore */
+/*
+ * Set the given tids on the blkno to TidStore.
+ *
+ * NB: the offset numbers in offsets must be sorted in ascending order.
+ */
void
TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
int num_offsets)
@@ -564,7 +575,7 @@ TidStoreEndIterate(TidStoreIter *iter)
int64
TidStoreNumTids(TidStore *ts)
{
- uint64 num_tids;
+ int64 num_tids;
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
@@ -624,11 +635,14 @@ TidStoreGetHandle(TidStore *ts)
return ts->control->handle;
}
-/* Extract tids from the given key-value pair */
+/*
+ * Decode the key and offset bitmap to tids and store them to the iteration
+ * result.
+ */
static void
iter_decode_key_off(TidStoreIter *iter, tidkey key, offsetbm off_bitmap)
{
- TidStoreIterResult *result = (&iter->result);
+ TidStoreIterResult *output = (&iter->output);
while (off_bitmap)
{
@@ -661,7 +675,7 @@ key_get_blkno(TidStore *ts, tidkey key)
static inline tidkey
encode_tid(TidStore *ts, ItemPointer tid, offsetbm *off_bit)
{
- uint32 offset = ItemPointerGetOffsetNumber(tid);
+ OffsetNumber offset = ItemPointerGetOffsetNumber(tid);
BlockNumber block = ItemPointerGetBlockNumber(tid);
return encode_blk_off(ts, block, offset, off_bit);
diff --git a/src/test/modules/test_tidstore/test_tidstore.c b/src/test/modules/test_tidstore/test_tidstore.c
index 12d3027624..8659e6780e 100644
--- a/src/test/modules/test_tidstore/test_tidstore.c
+++ b/src/test/modules/test_tidstore/test_tidstore.c
@@ -222,6 +222,7 @@ test_tidstore(PG_FUNCTION_ARGS)
elog(NOTICE, "testing basic operations");
test_basic(MaxHeapTuplesPerPage);
test_basic(10);
+ test_basic(MaxHeapTuplesPerPage * 2);
PG_RETURN_VOID();
}
--
2.31.1
[application/octet-stream] v32-0018-Revert-building-benchmark-module-for-CI.patch (694B, ../../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/5-v32-0018-Revert-building-benchmark-module-for-CI.patch)
download | inline diff:
From 9e42c43a7d081c06c02f0029e610c29d911732e3 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 14 Feb 2023 19:31:34 +0700
Subject: [PATCH v32 18/18] Revert building benchmark module for CI
---
contrib/meson.build | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/contrib/meson.build b/contrib/meson.build
index 421d469f8c..52253de793 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,7 +12,7 @@ subdir('amcheck')
subdir('auth_delay')
subdir('auto_explain')
subdir('basic_archive')
-subdir('bench_radix_tree')
+#subdir('bench_radix_tree')
subdir('bloom')
subdir('basebackup_to_shell')
subdir('bool_plperl')
--
2.31.1
[application/octet-stream] v32-0017-tidstore-vacuum-Specify-the-init-and-max-DSA-seg.patch (4.8K, ../../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/6-v32-0017-tidstore-vacuum-Specify-the-init-and-max-DSA-seg.patch)
download | inline diff:
From 11fda58f829c03d2a7c6476affc61862a078f741 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 22:27:04 +0900
Subject: [PATCH v32 17/18] tidstore, vacuum: Specify the init and max DSA
segment size based on m_w_m
---
src/backend/access/common/tidstore.c | 32 +++++----------------------
src/backend/commands/vacuumparallel.c | 21 ++++++++++++++----
2 files changed, 23 insertions(+), 30 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 9360520482..571d15c5c3 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -180,39 +180,15 @@ TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
ts = palloc0(sizeof(TidStore));
- /*
- * Create the radix tree for the main storage.
- *
- * Memory consumption depends on the number of stored tids, but also on the
- * distribution of them, how the radix tree stores, and the memory management
- * that backed the radix tree. The maximum bytes that a TidStore can
- * use is specified by the max_bytes in TidStoreCreate(). We want the total
- * amount of memory consumption by a TidStore not to exceed the max_bytes.
- *
- * In local TidStore cases, the radix tree uses slab allocators for each kind
- * of node class. The most memory consuming case while adding Tids associated
- * with one page (i.e. during TidStoreSetBlockOffsets()) is that we allocate a new
- * slab block for a new radix tree node, which is approximately 70kB. Therefore,
- * we deduct 70kB from the max_bytes.
- *
- * In shared cases, DSA allocates the memory segments big enough to follow
- * a geometric series that approximately doubles the total DSA size (see
- * make_new_segment() in dsa.c). We simulated the how DSA increases segment
- * size and the simulation revealed, the 75% threshold for the maximum bytes
- * perfectly works in case where the max_bytes is a power-of-2, and the 60%
- * threshold works for other cases.
- */
if (area != NULL)
{
dsa_pointer dp;
- float ratio = ((max_bytes & (max_bytes - 1)) == 0) ? 0.75 : 0.6;
ts->tree.shared = shared_rt_create(CurrentMemoryContext, area,
LWTRANCHE_SHARED_TIDSTORE);
dp = dsa_allocate0(area, sizeof(TidStoreControl));
ts->control = (TidStoreControl *) dsa_get_address(area, dp);
- ts->control->max_bytes = (size_t) (max_bytes * ratio);
ts->area = area;
ts->control->magic = TIDSTORE_MAGIC;
@@ -223,11 +199,15 @@ TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
else
{
ts->tree.local = local_rt_create(CurrentMemoryContext);
-
ts->control = (TidStoreControl *) palloc0(sizeof(TidStoreControl));
- ts->control->max_bytes = max_bytes - (70 * 1024);
}
+ /*
+ * max_bytes is forced to be at least 64kB, the current minimum valid value
+ * for the work_mem GUC.
+ */
+ ts->control->max_bytes = Max(64 * 1024L, max_bytes);
+
ts->control->max_off = max_off;
ts->control->max_off_nbits = pg_ceil_log2_32(max_off);
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 8385d375db..17699aa007 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -252,6 +252,8 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
Size est_indstats_len;
Size est_shared_len;
Size dsa_minsize = dsa_minimum_size();
+ Size init_segsize;
+ Size max_segsize;
int nindexes_mwm = 0;
int parallel_workers = 0;
int querylen;
@@ -367,12 +369,23 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_INDEX_STATS, indstats);
pvs->indstats = indstats;
- /* Prepare DSA space for dead items */
+ /*
+ * Prepare DSA space for dead items.
+ *
+ * Since total DSA size grows while following a geometric series by default,
+ * we specify both the initial DSA segment and maximum DSA segment sizes
+ * based on the memory available for parallel vacuum. Typically, the initial
+ * segment size is 1MB and the maximum segment size is vac_work_mem / 8, and
+ * heap scan stops after allocating 1.125 times more memory than vac_work_mem.
+ */
+ init_segsize = Min(vac_work_mem / 4, (1024 * 1024));
+ max_segsize = Max(vac_work_mem / 8, (8 * 1024 * 1024));
area_space = shm_toc_allocate(pcxt->toc, dsa_minsize);
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, area_space);
- dead_items_dsa = dsa_create_in_place(area_space, dsa_minsize,
- LWTRANCHE_PARALLEL_VACUUM_DSA,
- pcxt->seg);
+ dead_items_dsa = dsa_create_in_place_extended(area_space, dsa_minsize,
+ LWTRANCHE_PARALLEL_VACUUM_DSA,
+ pcxt->seg,
+ init_segsize, max_segsize);
dead_items = TidStoreCreate(vac_work_mem, max_offset, dead_items_dsa);
pvs->dead_items = dead_items;
pvs->dead_items_area = dead_items_dsa;
--
2.31.1
[application/octet-stream] v32-0010-radix-tree-fix-radix-tree-test-code.patch (6.8K, ../../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/7-v32-0010-radix-tree-fix-radix-tree-test-code.patch)
download | inline diff:
From 591bede6738ca9e5c7264db7ff1d3dd9ba29247f Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 17:35:14 +0900
Subject: [PATCH v32 10/18] radix tree: fix radix tree test code
fix tests for key insertion in ascending or descending order.
Also, we missed tests for MIN and MAX size classes.
---
.../expected/test_radixtree.out | 6 +-
.../modules/test_radixtree/test_radixtree.c | 103 ++++++++++++------
2 files changed, 71 insertions(+), 38 deletions(-)
diff --git a/src/test/modules/test_radixtree/expected/test_radixtree.out b/src/test/modules/test_radixtree/expected/test_radixtree.out
index ce645cb8b5..7ad1ce3605 100644
--- a/src/test/modules/test_radixtree/expected/test_radixtree.out
+++ b/src/test/modules/test_radixtree/expected/test_radixtree.out
@@ -4,8 +4,10 @@ CREATE EXTENSION test_radixtree;
-- an error if something fails.
--
SELECT test_radixtree();
-NOTICE: testing basic operations with leaf node 4
-NOTICE: testing basic operations with inner node 4
+NOTICE: testing basic operations with leaf node 3
+NOTICE: testing basic operations with inner node 3
+NOTICE: testing basic operations with leaf node 15
+NOTICE: testing basic operations with inner node 15
NOTICE: testing basic operations with leaf node 32
NOTICE: testing basic operations with inner node 32
NOTICE: testing basic operations with leaf node 125
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
index afe53382f3..5a169854d9 100644
--- a/src/test/modules/test_radixtree/test_radixtree.c
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -43,12 +43,15 @@ typedef uint64 TestValueType;
*/
static const bool rt_test_stats = false;
-static int rt_node_kind_fanouts[] = {
- 0,
- 4, /* RT_NODE_KIND_4 */
- 32, /* RT_NODE_KIND_32 */
- 125, /* RT_NODE_KIND_125 */
- 256 /* RT_NODE_KIND_256 */
+/*
+ * XXX: should we expose and use RT_SIZE_CLASS and RT_SIZE_CLASS_INFO?
+ */
+static int rt_node_class_fanouts[] = {
+ 3, /* RT_CLASS_3 */
+ 15, /* RT_CLASS_32_MIN */
+ 32, /* RT_CLASS_32_MAX */
+ 125, /* RT_CLASS_125 */
+ 256 /* RT_CLASS_256 */
};
/*
* A struct to define a pattern of integers, for use with the test_pattern()
@@ -260,10 +263,9 @@ test_basic(int children, bool test_inner)
* Check if keys from start to end with the shift exist in the tree.
*/
static void
-check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end,
- int incr)
+check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end)
{
- for (int i = start; i < end; i++)
+ for (int i = start; i <= end; i++)
{
uint64 key = ((uint64) i << shift);
TestValueType val;
@@ -277,22 +279,26 @@ check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end,
}
}
+/*
+ * Insert 256 key-value pairs, and check if keys are properly inserted on each
+ * node class.
+ */
+/* Test keys [0, 256) */
+#define NODE_TYPE_TEST_KEY_MIN 0
+#define NODE_TYPE_TEST_KEY_MAX 256
static void
-test_node_types_insert(rt_radix_tree *radixtree, uint8 shift, bool insert_asc)
+test_node_types_insert_asc(rt_radix_tree *radixtree, uint8 shift)
{
- uint64 num_entries;
- int ninserted = 0;
- int start = insert_asc ? 0 : 256;
- int incr = insert_asc ? 1 : -1;
- int end = insert_asc ? 256 : 0;
- int node_kind_idx = 1;
+ uint64 num_entries;
+ int node_class_idx = 0;
+ uint64 key_checked = 0;
- for (int i = start; i != end; i += incr)
+ for (int i = NODE_TYPE_TEST_KEY_MIN; i < NODE_TYPE_TEST_KEY_MAX; i++)
{
uint64 key = ((uint64) i << shift);
bool found;
- found = rt_set(radixtree, key, (TestValueType*) &key);
+ found = rt_set(radixtree, key, (TestValueType *) &key);
if (found)
elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " is found", key);
@@ -300,24 +306,49 @@ test_node_types_insert(rt_radix_tree *radixtree, uint8 shift, bool insert_asc)
* After filling all slots in each node type, check if the values
* are stored properly.
*/
- if (ninserted == rt_node_kind_fanouts[node_kind_idx] - 1)
+ if ((i + 1) == rt_node_class_fanouts[node_class_idx])
{
- int check_start = insert_asc
- ? rt_node_kind_fanouts[node_kind_idx - 1]
- : rt_node_kind_fanouts[node_kind_idx];
- int check_end = insert_asc
- ? rt_node_kind_fanouts[node_kind_idx]
- : rt_node_kind_fanouts[node_kind_idx - 1];
-
- check_search_on_node(radixtree, shift, check_start, check_end, incr);
- node_kind_idx++;
+ check_search_on_node(radixtree, shift, key_checked, i);
+ key_checked = i;
+ node_class_idx++;
}
-
- ninserted++;
}
num_entries = rt_num_entries(radixtree);
+ if (num_entries != 256)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+/*
+ * Similar to test_node_types_insert_asc(), but inserts keys in descending order.
+ */
+static void
+test_node_types_insert_desc(rt_radix_tree *radixtree, uint8 shift)
+{
+ uint64 num_entries;
+ int node_class_idx = 0;
+ uint64 key_checked = NODE_TYPE_TEST_KEY_MAX - 1;
+
+ for (int i = NODE_TYPE_TEST_KEY_MAX - 1; i >= NODE_TYPE_TEST_KEY_MIN; i--)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_set(radixtree, key, (TestValueType *) &key);
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " is found", key);
+ if ((i + 1) == rt_node_class_fanouts[node_class_idx])
+ {
+ check_search_on_node(radixtree, shift, i, key_checked);
+ key_checked = i;
+ node_class_idx++;
+ }
+ }
+
+ num_entries = rt_num_entries(radixtree);
if (num_entries != 256)
elog(ERROR,
"rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
@@ -329,7 +360,7 @@ test_node_types_delete(rt_radix_tree *radixtree, uint8 shift)
{
uint64 num_entries;
- for (int i = 0; i < 256; i++)
+ for (int i = NODE_TYPE_TEST_KEY_MIN; i < NODE_TYPE_TEST_KEY_MAX; i++)
{
uint64 key = ((uint64) i << shift);
bool found;
@@ -379,9 +410,9 @@ test_node_types(uint8 shift)
* then delete all entries to make it empty, and insert and search entries
* again.
*/
- test_node_types_insert(radixtree, shift, true);
+ test_node_types_insert_asc(radixtree, shift);
test_node_types_delete(radixtree, shift);
- test_node_types_insert(radixtree, shift, false);
+ test_node_types_insert_desc(radixtree, shift);
rt_free(radixtree);
#ifdef RT_SHMEM
@@ -664,10 +695,10 @@ test_radixtree(PG_FUNCTION_ARGS)
{
test_empty();
- for (int i = 1; i < lengthof(rt_node_kind_fanouts); i++)
+ for (int i = 0; i < lengthof(rt_node_class_fanouts); i++)
{
- test_basic(rt_node_kind_fanouts[i], false);
- test_basic(rt_node_kind_fanouts[i], true);
+ test_basic(rt_node_class_fanouts[i], false);
+ test_basic(rt_node_class_fanouts[i], true);
}
for (int shift = 0; shift <= (64 - 8); shift += 8)
--
2.31.1
[application/octet-stream] v32-0011-tidstore-vacuum-Use-camel-case-for-TidStore-APIs.patch (25.0K, ../../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/8-v32-0011-tidstore-vacuum-Use-camel-case-for-TidStore-APIs.patch)
download | inline diff:
From 6f4ff3584cbbf4db3ed7268ebc360df0ad328696 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 17:47:10 +0900
Subject: [PATCH v32 11/18] tidstore, vacuum: Use camel case for TidStore APIs
---
src/backend/access/common/tidstore.c | 64 +++++++++---------
src/backend/access/heap/vacuumlazy.c | 44 ++++++------
src/backend/commands/vacuum.c | 4 +-
src/backend/commands/vacuumparallel.c | 12 ++--
src/include/access/tidstore.h | 34 +++++-----
.../modules/test_tidstore/test_tidstore.c | 67 ++++++++++---------
6 files changed, 114 insertions(+), 111 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 8c05e60d92..283a326d13 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -7,9 +7,9 @@
* Internally, a tid is encoded as a pair of 64-bit key and 64-bit value, and
* stored in the radix tree.
*
- * A TidStore can be shared among parallel worker processes by passing DSA area
- * to tidstore_create(). Other backends can attach to the shared TidStore by
- * tidstore_attach().
+ * TidStore can be shared among parallel worker processes by passing DSA area
+ * to TidStoreCreate(). Other backends can attach to the shared TidStore by
+ * TidStoreAttach().
*
* Regarding the concurrency, it basically relies on the concurrency support in
* the radix tree, but we acquires the lock on a TidStore in some cases, for
@@ -106,7 +106,7 @@ typedef struct TidStoreControl
LWLock lock;
/* handles for TidStore and radix tree */
- tidstore_handle handle;
+ TidStoreHandle handle;
shared_rt_handle tree_handle;
} TidStoreControl;
@@ -164,7 +164,7 @@ static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_b
* The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
*/
TidStore *
-tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
+TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
{
TidStore *ts;
@@ -176,12 +176,12 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
* Memory consumption depends on the number of stored tids, but also on the
* distribution of them, how the radix tree stores, and the memory management
* that backed the radix tree. The maximum bytes that a TidStore can
- * use is specified by the max_bytes in tidstore_create(). We want the total
+ * use is specified by the max_bytes in TidStoreCreate(). We want the total
* amount of memory consumption by a TidStore not to exceed the max_bytes.
*
* In local TidStore cases, the radix tree uses slab allocators for each kind
* of node class. The most memory consuming case while adding Tids associated
- * with one page (i.e. during tidstore_add_tids()) is that we allocate a new
+ * with one page (i.e. during TidStoreSetBlockOffsets()) is that we allocate a new
* slab block for a new radix tree node, which is approximately 70kB. Therefore,
* we deduct 70kB from the max_bytes.
*
@@ -235,7 +235,7 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
* allocated in backend-local memory using the CurrentMemoryContext.
*/
TidStore *
-tidstore_attach(dsa_area *area, tidstore_handle handle)
+TidStoreAttach(dsa_area *area, TidStoreHandle handle)
{
TidStore *ts;
dsa_pointer control;
@@ -266,7 +266,7 @@ tidstore_attach(dsa_area *area, tidstore_handle handle)
* to the operating system.
*/
void
-tidstore_detach(TidStore *ts)
+TidStoreDetach(TidStore *ts)
{
Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
@@ -279,12 +279,12 @@ tidstore_detach(TidStore *ts)
*
* TODO: The caller must be certain that no other backend will attempt to
* access the TidStore before calling this function. Other backend must
- * explicitly call tidstore_detach to free up backend-local memory associated
- * with the TidStore. The backend that calls tidstore_destroy must not call
- * tidstore_detach.
+ * explicitly call TidStoreDetach() to free up backend-local memory associated
+ * with the TidStore. The backend that calls TidStoreDestroy() must not call
+ * TidStoreDetach().
*/
void
-tidstore_destroy(TidStore *ts)
+TidStoreDestroy(TidStore *ts)
{
if (TidStoreIsShared(ts))
{
@@ -309,11 +309,11 @@ tidstore_destroy(TidStore *ts)
}
/*
- * Forget all collected Tids. It's similar to tidstore_destroy but we don't free
+ * Forget all collected Tids. It's similar to TidStoreDestroy() but we don't free
* entire TidStore but recreate only the radix tree storage.
*/
void
-tidstore_reset(TidStore *ts)
+TidStoreReset(TidStore *ts)
{
if (TidStoreIsShared(ts))
{
@@ -352,8 +352,8 @@ tidstore_reset(TidStore *ts)
/* Add Tids on a block to TidStore */
void
-tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
- int num_offsets)
+TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets)
{
uint64 *values;
uint64 key;
@@ -431,7 +431,7 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
/* Return true if the given tid is present in the TidStore */
bool
-tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
+TidStoreIsMember(TidStore *ts, ItemPointer tid)
{
uint64 key;
uint64 val = 0;
@@ -452,14 +452,16 @@ tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
}
/*
- * Prepare to iterate through a TidStore. Since the radix tree is locked during the
- * iteration, so tidstore_end_iterate() needs to called when finished.
+ * Prepare to iterate through a TidStore. Since the radix tree is locked during
+ * the iteration, so TidStoreEndIterate() needs to be called when finished.
+ *
+ * The TidStoreIter struct is created in the caller's memory context.
*
* Concurrent updates during the iteration will be blocked when inserting a
* key-value to the radix tree.
*/
TidStoreIter *
-tidstore_begin_iterate(TidStore *ts)
+TidStoreBeginIterate(TidStore *ts)
{
TidStoreIter *iter;
@@ -477,7 +479,7 @@ tidstore_begin_iterate(TidStore *ts)
iter->tree_iter.local = local_rt_begin_iterate(ts->tree.local);
/* If the TidStore is empty, there is no business */
- if (tidstore_num_tids(ts) == 0)
+ if (TidStoreNumTids(ts) == 0)
iter->finished = true;
return iter;
@@ -498,7 +500,7 @@ tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
* numbers in each result is also sorted in ascending order.
*/
TidStoreIterResult *
-tidstore_iterate_next(TidStoreIter *iter)
+TidStoreIterateNext(TidStoreIter *iter)
{
uint64 key;
uint64 val;
@@ -544,7 +546,7 @@ tidstore_iterate_next(TidStoreIter *iter)
* or when existing an iteration.
*/
void
-tidstore_end_iterate(TidStoreIter *iter)
+TidStoreEndIterate(TidStoreIter *iter)
{
if (TidStoreIsShared(iter->ts))
shared_rt_end_iterate(iter->tree_iter.shared);
@@ -557,7 +559,7 @@ tidstore_end_iterate(TidStoreIter *iter)
/* Return the number of tids we collected so far */
int64
-tidstore_num_tids(TidStore *ts)
+TidStoreNumTids(TidStore *ts)
{
uint64 num_tids;
@@ -575,16 +577,16 @@ tidstore_num_tids(TidStore *ts)
/* Return true if the current memory usage of TidStore exceeds the limit */
bool
-tidstore_is_full(TidStore *ts)
+TidStoreIsFull(TidStore *ts)
{
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
- return (tidstore_memory_usage(ts) > ts->control->max_bytes);
+ return (TidStoreMemoryUsage(ts) > ts->control->max_bytes);
}
/* Return the maximum memory TidStore can use */
size_t
-tidstore_max_memory(TidStore *ts)
+TidStoreMaxMemory(TidStore *ts)
{
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
@@ -593,7 +595,7 @@ tidstore_max_memory(TidStore *ts)
/* Return the memory usage of TidStore */
size_t
-tidstore_memory_usage(TidStore *ts)
+TidStoreMemoryUsage(TidStore *ts)
{
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
@@ -611,8 +613,8 @@ tidstore_memory_usage(TidStore *ts)
/*
* Get a handle that can be used by other processes to attach to this TidStore
*/
-tidstore_handle
-tidstore_get_handle(TidStore *ts)
+TidStoreHandle
+TidStoreGetHandle(TidStore *ts)
{
Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 2c72088e69..be487aced6 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -842,7 +842,7 @@ lazy_scan_heap(LVRelState *vacrel)
/* Report that we're scanning the heap, advertising total # of blocks */
initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
initprog_val[1] = rel_pages;
- initprog_val[2] = tidstore_max_memory(vacrel->dead_items);
+ initprog_val[2] = TidStoreMaxMemory(vacrel->dead_items);
pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
/* Set up an initial range of skippable blocks using the visibility map */
@@ -909,7 +909,7 @@ lazy_scan_heap(LVRelState *vacrel)
* dead_items TIDs, pause and do a cycle of vacuuming before we tackle
* this page.
*/
- if (tidstore_is_full(vacrel->dead_items))
+ if (TidStoreIsFull(vacrel->dead_items))
{
/*
* Before beginning index vacuuming, we release any pin we may
@@ -1078,16 +1078,16 @@ lazy_scan_heap(LVRelState *vacrel)
* with prunestate-driven visibility map and FSM steps (just like
* the two-pass strategy).
*/
- Assert(tidstore_num_tids(dead_items) == 0);
+ Assert(TidStoreNumTids(dead_items) == 0);
}
else if (prunestate.num_offsets > 0)
{
/* Save details of the LP_DEAD items from the page in dead_items */
- tidstore_add_tids(dead_items, blkno, prunestate.deadoffsets,
- prunestate.num_offsets);
+ TidStoreSetBlockOffsets(dead_items, blkno, prunestate.deadoffsets,
+ prunestate.num_offsets);
pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
- tidstore_memory_usage(dead_items));
+ TidStoreMemoryUsage(dead_items));
}
/*
@@ -1258,7 +1258,7 @@ lazy_scan_heap(LVRelState *vacrel)
* Do index vacuuming (call each index's ambulkdelete routine), then do
* related heap vacuuming
*/
- if (tidstore_num_tids(dead_items) > 0)
+ if (TidStoreNumTids(dead_items) > 0)
lazy_vacuum(vacrel);
/*
@@ -2125,10 +2125,10 @@ lazy_scan_noprune(LVRelState *vacrel,
*/
vacrel->lpdead_item_pages++;
- tidstore_add_tids(dead_items, blkno, deadoffsets, lpdead_items);
+ TidStoreSetBlockOffsets(dead_items, blkno, deadoffsets, lpdead_items);
pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
- tidstore_memory_usage(dead_items));
+ TidStoreMemoryUsage(dead_items));
vacrel->lpdead_items += lpdead_items;
@@ -2177,7 +2177,7 @@ lazy_vacuum(LVRelState *vacrel)
if (!vacrel->do_index_vacuuming)
{
Assert(!vacrel->do_index_cleanup);
- tidstore_reset(vacrel->dead_items);
+ TidStoreReset(vacrel->dead_items);
return;
}
@@ -2206,7 +2206,7 @@ lazy_vacuum(LVRelState *vacrel)
BlockNumber threshold;
Assert(vacrel->num_index_scans == 0);
- Assert(vacrel->lpdead_items == tidstore_num_tids(vacrel->dead_items));
+ Assert(vacrel->lpdead_items == TidStoreNumTids(vacrel->dead_items));
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2234,7 +2234,7 @@ lazy_vacuum(LVRelState *vacrel)
*/
threshold = (double) vacrel->rel_pages * BYPASS_THRESHOLD_PAGES;
bypass = (vacrel->lpdead_item_pages < threshold) &&
- tidstore_memory_usage(vacrel->dead_items) < (32L * 1024L * 1024L);
+ TidStoreMemoryUsage(vacrel->dead_items) < (32L * 1024L * 1024L);
}
if (bypass)
@@ -2279,7 +2279,7 @@ lazy_vacuum(LVRelState *vacrel)
* Forget the LP_DEAD items that we just vacuumed (or just decided to not
* vacuum)
*/
- tidstore_reset(vacrel->dead_items);
+ TidStoreReset(vacrel->dead_items);
}
/*
@@ -2352,7 +2352,7 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
* place).
*/
Assert(vacrel->num_index_scans > 0 ||
- tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items);
+ TidStoreNumTids(vacrel->dead_items) == vacrel->lpdead_items);
Assert(allindexes || VacuumFailsafeActive);
/*
@@ -2407,8 +2407,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
VACUUM_ERRCB_PHASE_VACUUM_HEAP,
InvalidBlockNumber, InvalidOffsetNumber);
- iter = tidstore_begin_iterate(vacrel->dead_items);
- while ((result = tidstore_iterate_next(iter)) != NULL)
+ iter = TidStoreBeginIterate(vacrel->dead_items);
+ while ((iter_result = TidStoreIterateNext(iter)) != NULL)
{
BlockNumber blkno;
Buffer buf;
@@ -2442,7 +2442,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
vacuumed_pages++;
}
- tidstore_end_iterate(iter);
+ TidStoreEndIterate(iter);
vacrel->blkno = InvalidBlockNumber;
if (BufferIsValid(vmbuffer))
@@ -2453,12 +2453,12 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
* the second heap pass. No more, no less.
*/
Assert(vacrel->num_index_scans > 1 ||
- (tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items &&
+ (TidStoreNumTids(vacrel->dead_items) == vacrel->lpdead_items &&
vacuumed_pages == vacrel->lpdead_item_pages));
ereport(DEBUG2,
- (errmsg("table \"%s\": removed " UINT64_FORMAT "dead item identifiers in %u pages",
- vacrel->relname, tidstore_num_tids(vacrel->dead_items),
+ (errmsg("table \"%s\": removed " INT64_FORMAT "dead item identifiers in %u pages",
+ vacrel->relname, TidStoreNumTids(vacrel->dead_items),
vacuumed_pages)));
/* Revert to the previous phase information for error traceback */
@@ -3125,8 +3125,8 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
}
/* Serial VACUUM case */
- vacrel->dead_items = tidstore_create(vac_work_mem, MaxHeapTuplesPerPage,
- NULL);
+ vacrel->dead_items = TidStoreCreate(vac_work_mem, MaxHeapTuplesPerPage,
+ NULL);
}
/*
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index f3922b72dc..84f71fb14a 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -2486,7 +2486,7 @@ vac_bulkdel_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat,
ereport(ivinfo->message_level,
(errmsg("scanned index \"%s\" to remove " UINT64_FORMAT " row versions",
RelationGetRelationName(ivinfo->index),
- tidstore_num_tids(dead_items))));
+ TidStoreNumTids(dead_items))));
return istat;
}
@@ -2527,5 +2527,5 @@ vac_tid_reaped(ItemPointer itemptr, void *state)
{
TidStore *dead_items = (TidStore *) state;
- return tidstore_lookup_tid(dead_items, itemptr);
+ return TidStoreIsMember(dead_items, itemptr);
}
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index c363f45e32..be83ceb871 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -110,7 +110,7 @@ typedef struct PVShared
pg_atomic_uint32 idx;
/* Handle of the shared TidStore */
- tidstore_handle dead_items_handle;
+ TidStoreHandle dead_items_handle;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -372,7 +372,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
dead_items_dsa = dsa_create_in_place(area_space, dsa_minsize,
LWTRANCHE_PARALLEL_VACUUM_DSA,
pcxt->seg);
- dead_items = tidstore_create(vac_work_mem, max_offset, dead_items_dsa);
+ dead_items = TidStoreCreate(vac_work_mem, max_offset, dead_items_dsa);
pvs->dead_items = dead_items;
pvs->dead_items_area = dead_items_dsa;
@@ -385,7 +385,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
- shared->dead_items_handle = tidstore_get_handle(dead_items);
+ shared->dead_items_handle = TidStoreGetHandle(dead_items);
/* Use the same buffer size for all workers */
shared->ring_nbuffers = GetAccessStrategyBufferCount(bstrategy);
@@ -454,7 +454,7 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
istats[i] = NULL;
}
- tidstore_destroy(pvs->dead_items);
+ TidStoreDestroy(pvs->dead_items);
dsa_detach(pvs->dead_items_area);
DestroyParallelContext(pvs->pcxt);
@@ -1013,7 +1013,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
/* Set dead items */
area_space = shm_toc_lookup(toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, false);
dead_items_area = dsa_attach_in_place(area_space, seg);
- dead_items = tidstore_attach(dead_items_area, shared->dead_items_handle);
+ dead_items = TidStoreAttach(dead_items_area, shared->dead_items_handle);
/* Set cost-based vacuum delay */
VacuumUpdateCosts();
@@ -1061,7 +1061,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
InstrEndParallelQuery(&buffer_usage[ParallelWorkerNumber],
&wal_usage[ParallelWorkerNumber]);
- tidstore_detach(pvs.dead_items);
+ TidStoreDetach(dead_items);
dsa_detach(dead_items_area);
/* Pop the error context stack */
diff --git a/src/include/access/tidstore.h b/src/include/access/tidstore.h
index a35a52124a..f0a432d0da 100644
--- a/src/include/access/tidstore.h
+++ b/src/include/access/tidstore.h
@@ -17,7 +17,7 @@
#include "storage/itemptr.h"
#include "utils/dsa.h"
-typedef dsa_pointer tidstore_handle;
+typedef dsa_pointer TidStoreHandle;
typedef struct TidStore TidStore;
typedef struct TidStoreIter TidStoreIter;
@@ -29,21 +29,21 @@ typedef struct TidStoreIterResult
int num_offsets;
} TidStoreIterResult;
-extern TidStore *tidstore_create(size_t max_bytes, int max_offset, dsa_area *dsa);
-extern TidStore *tidstore_attach(dsa_area *dsa, dsa_pointer handle);
-extern void tidstore_detach(TidStore *ts);
-extern void tidstore_destroy(TidStore *ts);
-extern void tidstore_reset(TidStore *ts);
-extern void tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
- int num_offsets);
-extern bool tidstore_lookup_tid(TidStore *ts, ItemPointer tid);
-extern TidStoreIter * tidstore_begin_iterate(TidStore *ts);
-extern TidStoreIterResult *tidstore_iterate_next(TidStoreIter *iter);
-extern void tidstore_end_iterate(TidStoreIter *iter);
-extern int64 tidstore_num_tids(TidStore *ts);
-extern bool tidstore_is_full(TidStore *ts);
-extern size_t tidstore_max_memory(TidStore *ts);
-extern size_t tidstore_memory_usage(TidStore *ts);
-extern tidstore_handle tidstore_get_handle(TidStore *ts);
+extern TidStore *TidStoreCreate(size_t max_bytes, int max_off, dsa_area *dsa);
+extern TidStore *TidStoreAttach(dsa_area *dsa, dsa_pointer handle);
+extern void TidStoreDetach(TidStore *ts);
+extern void TidStoreDestroy(TidStore *ts);
+extern void TidStoreReset(TidStore *ts);
+extern void TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets);
+extern bool TidStoreIsMember(TidStore *ts, ItemPointer tid);
+extern TidStoreIter * TidStoreBeginIterate(TidStore *ts);
+extern TidStoreIterResult *TidStoreIterateNext(TidStoreIter *iter);
+extern void TidStoreEndIterate(TidStoreIter *iter);
+extern int64 TidStoreNumTids(TidStore *ts);
+extern bool TidStoreIsFull(TidStore *ts);
+extern size_t TidStoreMaxMemory(TidStore *ts);
+extern size_t TidStoreMemoryUsage(TidStore *ts);
+extern TidStoreHandle TidStoreGetHandle(TidStore *ts);
#endif /* TIDSTORE_H */
diff --git a/src/test/modules/test_tidstore/test_tidstore.c b/src/test/modules/test_tidstore/test_tidstore.c
index 9a1217f833..12d3027624 100644
--- a/src/test/modules/test_tidstore/test_tidstore.c
+++ b/src/test/modules/test_tidstore/test_tidstore.c
@@ -37,10 +37,10 @@ check_tid(TidStore *ts, BlockNumber blkno, OffsetNumber off, bool expect)
ItemPointerSet(&tid, blkno, off);
- found = tidstore_lookup_tid(ts, &tid);
+ found = TidStoreIsMember(ts, &tid);
if (found != expect)
- elog(ERROR, "lookup TID (%u, %u) returned %d, expected %d",
+ elog(ERROR, "TidStoreIsMember for TID (%u, %u) returned %d, expected %d",
blkno, off, found, expect);
}
@@ -69,9 +69,9 @@ test_basic(int max_offset)
LWLockRegisterTranche(tranche_id, "test_tidstore");
dsa = dsa_create(tranche_id);
- ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, dsa);
+ ts = TidStoreCreate(TEST_TIDSTORE_MAX_BYTES, max_offset, dsa);
#else
- ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
+ ts = TidStoreCreate(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
#endif
/* prepare the offset array */
@@ -83,7 +83,7 @@ test_basic(int max_offset)
/* add tids */
for (int i = 0; i < TEST_TIDSTORE_NUM_BLOCKS; i++)
- tidstore_add_tids(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
+ TidStoreSetBlockOffsets(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
/* lookup test */
for (OffsetNumber off = FirstOffsetNumber ; off < max_offset; off++)
@@ -105,30 +105,30 @@ test_basic(int max_offset)
}
/* test the number of tids */
- if (tidstore_num_tids(ts) != (TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS))
- elog(ERROR, "tidstore_num_tids returned " UINT64_FORMAT ", expected %d",
- tidstore_num_tids(ts),
+ if (TidStoreNumTids(ts) != (TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS))
+ elog(ERROR, "TidStoreNumTids returned " UINT64_FORMAT ", expected %d",
+ TidStoreNumTids(ts),
TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS);
/* iteration test */
- iter = tidstore_begin_iterate(ts);
+ iter = TidStoreBeginIterate(ts);
blk_idx = 0;
- while ((iter_result = tidstore_iterate_next(iter)) != NULL)
+ while ((iter_result = TidStoreIterateNext(iter)) != NULL)
{
/* check the returned block number */
if (blks_sorted[blk_idx] != iter_result->blkno)
- elog(ERROR, "tidstore_iterate_next returned block number %u, expected %u",
+ elog(ERROR, "TidStoreIterateNext returned block number %u, expected %u",
iter_result->blkno, blks_sorted[blk_idx]);
/* check the returned offset numbers */
if (TEST_TIDSTORE_NUM_OFFSETS != iter_result->num_offsets)
- elog(ERROR, "tidstore_iterate_next returned %u offsets, expected %u",
+ elog(ERROR, "TidStoreIterateNext %u offsets, expected %u",
iter_result->num_offsets, TEST_TIDSTORE_NUM_OFFSETS);
for (int i = 0; i < iter_result->num_offsets; i++)
{
if (offs[i] != iter_result->offsets[i])
- elog(ERROR, "tidstore_iterate_next returned offset number %u on block %u, expected %u",
+ elog(ERROR, "TidStoreIterateNext offset number %u on block %u, expected %u",
iter_result->offsets[i], iter_result->blkno, offs[i]);
}
@@ -136,15 +136,15 @@ test_basic(int max_offset)
}
if (blk_idx != TEST_TIDSTORE_NUM_BLOCKS)
- elog(ERROR, "tidstore_iterate_next returned %d blocks, expected %d",
+ elog(ERROR, "TidStoreIterateNext returned %d blocks, expected %d",
blk_idx, TEST_TIDSTORE_NUM_BLOCKS);
/* remove all tids */
- tidstore_reset(ts);
+ TidStoreReset(ts);
/* test the number of tids */
- if (tidstore_num_tids(ts) != 0)
- elog(ERROR, "tidstore_num_tids on empty store returned non-zero");
+ if (TidStoreNumTids(ts) != 0)
+ elog(ERROR, "TidStoreNumTids on empty store returned non-zero");
/* lookup test for empty store */
for (OffsetNumber off = FirstOffsetNumber ; off < MaxHeapTuplesPerPage;
@@ -156,7 +156,7 @@ test_basic(int max_offset)
check_tid(ts, MaxBlockNumber, off, false);
}
- tidstore_destroy(ts);
+ TidStoreDestroy(ts);
#ifdef TEST_SHARED_TIDSTORE
dsa_detach(dsa);
@@ -177,36 +177,37 @@ test_empty(void)
LWLockRegisterTranche(tranche_id, "test_tidstore");
dsa = dsa_create(tranche_id);
- ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, dsa);
+ ts = TidStoreCreate(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, dsa);
#else
- ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
+ ts = TidStoreCreate(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
#endif
elog(NOTICE, "testing empty tidstore");
ItemPointerSet(&tid, 0, FirstOffsetNumber);
- if (tidstore_lookup_tid(ts, &tid))
- elog(ERROR, "tidstore_lookup_tid for (0,1) on empty store returned true");
+ if (TidStoreIsMember(ts, &tid))
+ elog(ERROR, "TidStoreIsMember for TID (%u,%u) on empty store returned true",
+ 0, FirstOffsetNumber);
ItemPointerSet(&tid, MaxBlockNumber, MaxOffsetNumber);
- if (tidstore_lookup_tid(ts, &tid))
- elog(ERROR, "tidstore_lookup_tid for (%u,%u) on empty store returned true",
+ if (TidStoreIsMember(ts, &tid))
+ elog(ERROR, "TidStoreIsMember for TID (%u,%u) on empty store returned true",
MaxBlockNumber, MaxOffsetNumber);
- if (tidstore_num_tids(ts) != 0)
- elog(ERROR, "tidstore_num_entries on empty store returned non-zero");
+ if (TidStoreNumTids(ts) != 0)
+ elog(ERROR, "TidStoreNumTids on empty store returned non-zero");
- if (tidstore_is_full(ts))
- elog(ERROR, "tidstore_is_full on empty store returned true");
+ if (TidStoreIsFull(ts))
+ elog(ERROR, "TidStoreIsFull on empty store returned true");
- iter = tidstore_begin_iterate(ts);
+ iter = TidStoreBeginIterate(ts);
- if (tidstore_iterate_next(iter) != NULL)
- elog(ERROR, "tidstore_iterate_next on empty store returned TIDs");
+ if (TidStoreIterateNext(iter) != NULL)
+ elog(ERROR, "TidStoreIterateNext on empty store returned TIDs");
- tidstore_end_iterate(iter);
+ TidStoreEndIterate(iter);
- tidstore_destroy(ts);
+ TidStoreDestroy(ts);
#ifdef TEST_SHARED_TIDSTORE
dsa_detach(dsa);
--
2.31.1
[application/octet-stream] v32-0012-tidstore-Use-concept-of-off_upper-and-off_lower.patch (12.6K, ../../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/9-v32-0012-tidstore-Use-concept-of-off_upper-and-off_lower.patch)
download | inline diff:
From 3f38c7722deb260e5cc4ac003ab37cfe959b1954 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 17:54:49 +0900
Subject: [PATCH v32 12/18] tidstore: Use concept of off_upper and off_lower.
The key is block number + the upper of offset number, whereas the
value is the bitmap representation of the lower offset number.
Updated function and variable names accordingly.
---
src/backend/access/common/tidstore.c | 191 ++++++++++++++-------------
1 file changed, 99 insertions(+), 92 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 283a326d13..d9fe3d5f15 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -65,8 +65,10 @@
*
* The maximum height of the radix tree is 5 in this case.
*/
-#define TIDSTORE_VALUE_NBITS 6 /* log(64, 2) */
-#define TIDSTORE_OFFSET_MASK ((1 << TIDSTORE_VALUE_NBITS) - 1)
+typedef uint64 tidkey;
+typedef uint64 offsetbm;
+#define LOWER_OFFSET_NBITS 6 /* log(sizeof(offsetbm), 2) */
+#define LOWER_OFFSET_MASK ((1 << LOWER_OFFSET_NBITS) - 1)
/* A magic value used to identify our TidStores. */
#define TIDSTORE_MAGIC 0x826f6a10
@@ -75,7 +77,7 @@
#define RT_SCOPE static
#define RT_DECLARE
#define RT_DEFINE
-#define RT_VALUE_TYPE uint64
+#define RT_VALUE_TYPE tidkey
#include "lib/radixtree.h"
#define RT_PREFIX shared_rt
@@ -83,7 +85,7 @@
#define RT_SCOPE static
#define RT_DECLARE
#define RT_DEFINE
-#define RT_VALUE_TYPE uint64
+#define RT_VALUE_TYPE tidkey
#include "lib/radixtree.h"
/* The control object for a TidStore */
@@ -94,10 +96,10 @@ typedef struct TidStoreControl
/* These values are never changed after creation */
size_t max_bytes; /* the maximum bytes a TidStore can use */
- int max_offset; /* the maximum offset number */
- int offset_nbits; /* the number of bits required for an offset
- * number */
- int offset_key_nbits; /* the number of bits of an offset number
+ int max_off; /* the maximum offset number */
+ int max_off_nbits; /* the number of bits required for offset
+ * numbers */
+ int upper_off_nbits; /* the number of bits of offset numbers
* used in a key */
/* The below fields are used only in shared case */
@@ -147,17 +149,18 @@ typedef struct TidStoreIter
bool finished;
/* save for the next iteration */
- uint64 next_key;
- uint64 next_val;
+ tidkey next_tidkey;
+ offsetbm next_off_bitmap;
/* output for the caller */
TidStoreIterResult result;
} TidStoreIter;
-static void tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val);
-static inline BlockNumber key_get_blkno(TidStore *ts, uint64 key);
-static inline uint64 encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit);
-static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit);
+static void iter_decode_key_off(TidStoreIter *iter, tidkey key, offsetbm off_bitmap);
+static inline BlockNumber key_get_blkno(TidStore *ts, tidkey key);
+static inline tidkey encode_blk_off(TidStore *ts, BlockNumber block,
+ OffsetNumber offset, offsetbm *off_bit);
+static inline tidkey encode_tid(TidStore *ts, ItemPointer tid, offsetbm *off_bit);
/*
* Create a TidStore. The returned object is allocated in backend-local memory.
@@ -218,14 +221,14 @@ TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
ts->control->max_bytes = max_bytes - (70 * 1024);
}
- ts->control->max_offset = max_offset;
- ts->control->offset_nbits = pg_ceil_log2_32(max_offset);
+ ts->control->max_off = max_off;
+ ts->control->max_off_nbits = pg_ceil_log2_32(max_off);
- if (ts->control->offset_nbits < TIDSTORE_VALUE_NBITS)
- ts->control->offset_nbits = TIDSTORE_VALUE_NBITS;
+ if (ts->control->max_off_nbits < LOWER_OFFSET_NBITS)
+ ts->control->max_off_nbits = LOWER_OFFSET_NBITS;
- ts->control->offset_key_nbits =
- ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
+ ts->control->upper_off_nbits =
+ ts->control->max_off_nbits - LOWER_OFFSET_NBITS;
return ts;
}
@@ -355,25 +358,25 @@ void
TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
int num_offsets)
{
- uint64 *values;
- uint64 key;
- uint64 prev_key;
- uint64 off_bitmap = 0;
+ offsetbm *bitmaps;
+ tidkey key;
+ tidkey prev_key;
+ offsetbm off_bitmap = 0;
int idx;
- const uint64 key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
- const int nkeys = UINT64CONST(1) << ts->control->offset_key_nbits;
+ const tidkey key_base = ((uint64) blkno) << ts->control->upper_off_nbits;
+ const int nkeys = UINT64CONST(1) << ts->control->upper_off_nbits;
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
- values = palloc(sizeof(uint64) * nkeys);
+ bitmaps = palloc(sizeof(offsetbm) * nkeys);
key = prev_key = key_base;
for (int i = 0; i < num_offsets; i++)
{
- uint64 off_bit;
+ offsetbm off_bit;
/* encode the tid to a key and partial offset */
- key = encode_key_off(ts, blkno, offsets[i], &off_bit);
+ key = encode_blk_off(ts, blkno, offsets[i], &off_bit);
/* make sure we scanned the line pointer array in order */
Assert(key >= prev_key);
@@ -384,11 +387,11 @@ TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
Assert(idx >= 0 && idx < nkeys);
/* write out offset bitmap for this key */
- values[idx] = off_bitmap;
+ bitmaps[idx] = off_bitmap;
/* zero out any gaps up to the current key */
for (int empty_idx = idx + 1; empty_idx < key - key_base; empty_idx++)
- values[empty_idx] = 0;
+ bitmaps[empty_idx] = 0;
/* reset for current key -- the current offset will be handled below */
off_bitmap = 0;
@@ -401,7 +404,7 @@ TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
/* save the final index for later */
idx = key - key_base;
/* write out last offset bitmap */
- values[idx] = off_bitmap;
+ bitmaps[idx] = off_bitmap;
if (TidStoreIsShared(ts))
LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
@@ -409,14 +412,14 @@ TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
/* insert the calculated key-values to the tree */
for (int i = 0; i <= idx; i++)
{
- if (values[i])
+ if (bitmaps[i])
{
key = key_base + i;
if (TidStoreIsShared(ts))
- shared_rt_set(ts->tree.shared, key, &values[i]);
+ shared_rt_set(ts->tree.shared, key, &bitmaps[i]);
else
- local_rt_set(ts->tree.local, key, &values[i]);
+ local_rt_set(ts->tree.local, key, &bitmaps[i]);
}
}
@@ -426,29 +429,29 @@ TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
if (TidStoreIsShared(ts))
LWLockRelease(&ts->control->lock);
- pfree(values);
+ pfree(bitmaps);
}
/* Return true if the given tid is present in the TidStore */
bool
TidStoreIsMember(TidStore *ts, ItemPointer tid)
{
- uint64 key;
- uint64 val = 0;
- uint64 off_bit;
+ tidkey key;
+ offsetbm off_bitmap = 0;
+ offsetbm off_bit;
bool found;
- key = tid_to_key_off(ts, tid, &off_bit);
+ key = encode_tid(ts, tid, &off_bit);
if (TidStoreIsShared(ts))
- found = shared_rt_search(ts->tree.shared, key, &val);
+ found = shared_rt_search(ts->tree.shared, key, &off_bitmap);
else
- found = local_rt_search(ts->tree.local, key, &val);
+ found = local_rt_search(ts->tree.local, key, &off_bitmap);
if (!found)
return false;
- return (val & off_bit) != 0;
+ return (off_bitmap & off_bit) != 0;
}
/*
@@ -486,12 +489,12 @@ TidStoreBeginIterate(TidStore *ts)
}
static inline bool
-tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
+tidstore_iter(TidStoreIter *iter, tidkey *key, offsetbm *off_bitmap)
{
if (TidStoreIsShared(iter->ts))
- return shared_rt_iterate_next(iter->tree_iter.shared, key, val);
+ return shared_rt_iterate_next(iter->tree_iter.shared, key, off_bitmap);
- return local_rt_iterate_next(iter->tree_iter.local, key, val);
+ return local_rt_iterate_next(iter->tree_iter.local, key, off_bitmap);
}
/*
@@ -502,43 +505,46 @@ tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
TidStoreIterResult *
TidStoreIterateNext(TidStoreIter *iter)
{
- uint64 key;
- uint64 val;
- TidStoreIterResult *result = &(iter->result);
+ tidkey key;
+ offsetbm off_bitmap = 0;
+ TidStoreIterResult *output = &(iter->output);
if (iter->finished)
return NULL;
- if (BlockNumberIsValid(result->blkno))
- {
- /* Process the previously collected key-value */
- result->num_offsets = 0;
- tidstore_iter_extract_tids(iter, iter->next_key, iter->next_val);
- }
+ /* Initialize the outputs */
+ output->blkno = InvalidBlockNumber;
+ output->num_offsets = 0;
- while (tidstore_iter_kv(iter, &key, &val))
- {
- BlockNumber blkno;
+ /*
+ * Decode the key and offset bitmap that are collected in the previous
+ * time, if exists.
+ */
+ if (iter->next_off_bitmap > 0)
+ iter_decode_key_off(iter, iter->next_tidkey, iter->next_off_bitmap);
- blkno = key_get_blkno(iter->ts, key);
+ while (tidstore_iter(iter, &key, &off_bitmap))
+ {
+ BlockNumber blkno = key_get_blkno(iter->ts, key);
- if (BlockNumberIsValid(result->blkno) && result->blkno != blkno)
+ if (BlockNumberIsValid(output->blkno) && output->blkno != blkno)
{
/*
- * We got a key-value pair for a different block. So return the
- * collected tids, and remember the key-value for the next iteration.
+ * We got tids for a different block. We return the collected
+ * tids so far, and remember the key-value for the next
+ * iteration.
*/
- iter->next_key = key;
- iter->next_val = val;
- return result;
+ iter->next_tidkey = key;
+ iter->next_off_bitmap = off_bitmap;
+ return output;
}
- /* Collect tids extracted from the key-value pair */
- tidstore_iter_extract_tids(iter, key, val);
+ /* Collect tids decoded from the key and offset bitmap */
+ iter_decode_key_off(iter, key, off_bitmap);
}
iter->finished = true;
- return result;
+ return output;
}
/*
@@ -623,61 +629,62 @@ TidStoreGetHandle(TidStore *ts)
/* Extract tids from the given key-value pair */
static void
-tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
+iter_decode_key_off(TidStoreIter *iter, tidkey key, offsetbm off_bitmap)
{
TidStoreIterResult *result = (&iter->result);
- while (val)
+ while (off_bitmap)
{
- uint64 tid_i;
+ uint64 compressed_tid;
OffsetNumber off;
- tid_i = key << TIDSTORE_VALUE_NBITS;
- tid_i |= pg_rightmost_one_pos64(val);
+ compressed_tid = key << LOWER_OFFSET_NBITS;
+ compressed_tid |= pg_rightmost_one_pos64(off_bitmap);
- off = tid_i & ((UINT64CONST(1) << iter->ts->control->offset_nbits) - 1);
+ off = compressed_tid & ((UINT64CONST(1) << iter->ts->control->max_off_nbits) - 1);
- Assert(result->num_offsets < iter->ts->control->max_offset);
- result->offsets[result->num_offsets++] = off;
+ Assert(output->num_offsets < iter->ts->control->max_off);
+ output->offsets[output->num_offsets++] = off;
/* unset the rightmost bit */
- val &= ~pg_rightmost_one64(val);
+ off_bitmap &= ~pg_rightmost_one64(off_bitmap);
}
- result->blkno = key_get_blkno(iter->ts, key);
+ output->blkno = key_get_blkno(iter->ts, key);
}
/* Get block number from the given key */
static inline BlockNumber
-key_get_blkno(TidStore *ts, uint64 key)
+key_get_blkno(TidStore *ts, tidkey key)
{
- return (BlockNumber) (key >> ts->control->offset_key_nbits);
+ return (BlockNumber) (key >> ts->control->upper_off_nbits);
}
-/* Encode a tid to key and offset */
-static inline uint64
-tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit)
+/* Encode a tid to key and partial offset */
+static inline tidkey
+encode_tid(TidStore *ts, ItemPointer tid, offsetbm *off_bit)
{
uint32 offset = ItemPointerGetOffsetNumber(tid);
BlockNumber block = ItemPointerGetBlockNumber(tid);
- return encode_key_off(ts, block, offset, off_bit);
+ return encode_blk_off(ts, block, offset, off_bit);
}
/* encode a block and offset to a key and partial offset */
-static inline uint64
-encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit)
+static inline tidkey
+encode_blk_off(TidStore *ts, BlockNumber block, OffsetNumber offset,
+ offsetbm *off_bit)
{
- uint64 key;
- uint64 tid_i;
+ tidkey key;
+ uint64 compressed_tid;
uint32 off_lower;
- off_lower = offset & TIDSTORE_OFFSET_MASK;
- Assert(off_lower < (sizeof(uint64) * BITS_PER_BYTE));
+ off_lower = offset & LOWER_OFFSET_MASK;
+ Assert(off_lower < (sizeof(offsetbm) * BITS_PER_BYTE));
*off_bit = UINT64CONST(1) << off_lower;
- tid_i = offset | ((uint64) block << ts->control->offset_nbits);
- key = tid_i >> TIDSTORE_VALUE_NBITS;
+ compressed_tid = offset | ((uint64) block << ts->control->max_off_nbits);
+ key = compressed_tid >> LOWER_OFFSET_NBITS;
return key;
}
--
2.31.1
[application/octet-stream] v32-0009-radix-tree-Review-tree-iteration-code.patch (13.9K, ../../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/10-v32-0009-radix-tree-Review-tree-iteration-code.patch)
download | inline diff:
From 989dd2cb442c1c2a6182bb5f7785c52f4d5cdb5e Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 17:33:21 +0900
Subject: [PATCH v32 09/18] radix tree: Review tree iteration code
Cleanup the routines and improve comments and variable names.
---
src/include/lib/radixtree.h | 152 ++++++++++++++------------
src/include/lib/radixtree_iter_impl.h | 85 +++++++-------
2 files changed, 118 insertions(+), 119 deletions(-)
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index 088d1dfd9d..8bea606c62 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -83,7 +83,7 @@
* RT_SET - Set a key-value pair
* RT_BEGIN_ITERATE - Begin iterating through all key-value pairs
* RT_ITERATE_NEXT - Return next key-value pair, if any
- * RT_END_ITER - End iteration
+ * RT_END_ITERATE - End iteration
* RT_MEMORY_USAGE - Get the memory usage
*
* Interface for Shared Memory
@@ -191,7 +191,7 @@
#define RT_NODE_INSERT_LEAF RT_MAKE_NAME(node_insert_leaf)
#define RT_NODE_INNER_ITERATE_NEXT RT_MAKE_NAME(node_inner_iterate_next)
#define RT_NODE_LEAF_ITERATE_NEXT RT_MAKE_NAME(node_leaf_iterate_next)
-#define RT_UPDATE_ITER_STACK RT_MAKE_NAME(update_iter_stack)
+#define RT_ITER_SET_NODE_FROM RT_MAKE_NAME(iter_set_node_from)
#define RT_ITER_UPDATE_KEY RT_MAKE_NAME(iter_update_key)
#define RT_VERIFY_NODE RT_MAKE_NAME(verify_node)
@@ -650,36 +650,40 @@ typedef struct RT_RADIX_TREE
* Iteration support.
*
* Iterating the radix tree returns each pair of key and value in the ascending
- * order of the key. To support this, the we iterate nodes of each level.
+ * order of the key.
*
- * RT_NODE_ITER struct is used to track the iteration within a node.
+ * RT_NODE_ITER is the struct for iteration of one radix tree node.
*
* RT_ITER is the struct for iteration of the radix tree, and uses RT_NODE_ITER
- * in order to track the iteration of each level. During iteration, we also
- * construct the key whenever updating the node iteration information, e.g., when
- * advancing the current index within the node or when moving to the next node
- * at the same level.
- *
- * XXX: Currently we allow only one process to do iteration. Therefore, rt_node_iter
- * has the local pointers to nodes, rather than RT_PTR_ALLOC.
- * We need either a safeguard to disallow other processes to begin the iteration
- * while one process is doing or to allow multiple processes to do the iteration.
+ * for each level to track the iteration within the node.
*/
typedef struct RT_NODE_ITER
{
- RT_PTR_LOCAL node; /* current node being iterated */
- int current_idx; /* current position. -1 for initial value */
+ /*
+ * Local pointer to the node we are iterating over.
+ *
+ * Since the radix tree doesn't support the shared iteration among multiple
+ * processes, we use RT_PTR_LOCAL rather than RT_PTR_ALLOC.
+ */
+ RT_PTR_LOCAL node;
+
+ /*
+ * The next index of the chunk array in RT_NODE_KIND_3 and
+ * RT_NODE_KIND_32 nodes, or the next chunk in RT_NODE_KIND_125 and
+ * RT_NODE_KIND_256 nodes. 0 for the initial value.
+ */
+ int idx;
} RT_NODE_ITER;
typedef struct RT_ITER
{
RT_RADIX_TREE *tree;
- /* Track the iteration on nodes of each level */
- RT_NODE_ITER stack[RT_MAX_LEVEL];
- int stack_len;
+ /* Track the nodes for each level. level = 0 is for a leaf node */
+ RT_NODE_ITER node_iters[RT_MAX_LEVEL];
+ int top_level;
- /* The key is constructed during iteration */
+ /* The key constructed during the iteration */
uint64 key;
} RT_ITER;
@@ -1804,16 +1808,9 @@ RT_DELETE(RT_RADIX_TREE *tree, uint64 key)
}
#endif
-static inline void
-RT_ITER_UPDATE_KEY(RT_ITER *iter, uint8 chunk, uint8 shift)
-{
- iter->key &= ~(((uint64) RT_CHUNK_MASK) << shift);
- iter->key |= (((uint64) chunk) << shift);
-}
-
/*
- * Advance the slot in the inner node. Return the child if exists, otherwise
- * null.
+ * Scan the inner node and return the next child node if exist, otherwise
+ * return NULL.
*/
static inline RT_PTR_LOCAL
RT_NODE_INNER_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter)
@@ -1824,8 +1821,8 @@ RT_NODE_INNER_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter)
}
/*
- * Advance the slot in the leaf node. On success, return true and the value
- * is set to value_p, otherwise return false.
+ * Scan the leaf node, and return true and the next value is set to value_p
+ * if exists. Otherwise return false.
*/
static inline bool
RT_NODE_LEAF_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter,
@@ -1837,29 +1834,50 @@ RT_NODE_LEAF_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter,
}
/*
- * Update each node_iter for inner nodes in the iterator node stack.
+ * While descending the radix tree from the 'from' node to the bottom, we
+ * set the next node to iterate for each level.
*/
static void
-RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
+RT_ITER_SET_NODE_FROM(RT_ITER *iter, RT_PTR_LOCAL from)
{
- int level = from;
- RT_PTR_LOCAL node = from_node;
+ int level = from->shift / RT_NODE_SPAN;
+ RT_PTR_LOCAL node = from;
for (;;)
{
- RT_NODE_ITER *node_iter = &(iter->stack[level--]);
+ RT_NODE_ITER *node_iter = &(iter->node_iters[level--]);
+
+#ifdef USE_ASSERT_CHECKING
+ if (node_iter->node)
+ {
+ /* We must have finished the iteration on the previous node */
+ if (RT_NODE_IS_LEAF(node_iter->node))
+ {
+ uint64 dummy;
+ Assert(!RT_NODE_LEAF_ITERATE_NEXT(iter, node_iter, &dummy));
+ }
+ else
+ Assert(!RT_NODE_INNER_ITERATE_NEXT(iter, node_iter));
+ }
+#endif
+ /* Set the node to the node iterator of this level */
node_iter->node = node;
- node_iter->current_idx = -1;
+ node_iter->idx = 0;
- /* We don't advance the leaf node iterator here */
if (RT_NODE_IS_LEAF(node))
- return;
+ {
+ /* We will visit the leaf node when RT_ITERATE_NEXT() */
+ break;
+ }
- /* Advance to the next slot in the inner node */
+ /*
+ * Get the first child node from the node, which corresponds to the
+ * lowest chunk within the node.
+ */
node = RT_NODE_INNER_ITERATE_NEXT(iter, node_iter);
- /* We must find the first children in the node */
+ /* The first child must be found */
Assert(node);
}
}
@@ -1873,14 +1891,11 @@ RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
RT_SCOPE RT_ITER *
RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
{
- MemoryContext old_ctx;
RT_ITER *iter;
RT_PTR_LOCAL root;
- int top_level;
- old_ctx = MemoryContextSwitchTo(tree->context);
-
- iter = (RT_ITER *) palloc0(sizeof(RT_ITER));
+ iter = (RT_ITER *) MemoryContextAllocZero(tree->context,
+ sizeof(RT_ITER));
iter->tree = tree;
RT_LOCK_SHARED(tree);
@@ -1890,16 +1905,13 @@ RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
return iter;
root = RT_PTR_GET_LOCAL(tree, iter->tree->ctl->root);
- top_level = root->shift / RT_NODE_SPAN;
- iter->stack_len = top_level;
+ iter->top_level = root->shift / RT_NODE_SPAN;
/*
- * Descend to the left most leaf node from the root. The key is being
- * constructed while descending to the leaf.
+ * Set the next node to iterate for each level from the level of the
+ * root node.
*/
- RT_UPDATE_ITER_STACK(iter, root, top_level);
-
- MemoryContextSwitchTo(old_ctx);
+ RT_ITER_SET_NODE_FROM(iter, root);
return iter;
}
@@ -1911,6 +1923,8 @@ RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
RT_SCOPE bool
RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p)
{
+ Assert(value_p != NULL);
+
/* Empty tree */
if (!iter->tree->ctl->root)
return false;
@@ -1918,43 +1932,38 @@ RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p)
for (;;)
{
RT_PTR_LOCAL child = NULL;
- RT_VALUE_TYPE value;
- int level;
- bool found;
-
- /* Advance the leaf node iterator to get next key-value pair */
- found = RT_NODE_LEAF_ITERATE_NEXT(iter, &(iter->stack[0]), &value);
- if (found)
+ /* Get the next chunk of the leaf node */
+ if (RT_NODE_LEAF_ITERATE_NEXT(iter, &(iter->node_iters[0]), value_p))
{
*key_p = iter->key;
- *value_p = value;
return true;
}
/*
- * We've visited all values in the leaf node, so advance inner node
- * iterators from the level=1 until we find the next child node.
+ * We've visited all values in the leaf node, so advance all inner node
+ * iterators by visiting inner nodes from the level = 1 until we find the
+ * next inner node that has a child node.
*/
- for (level = 1; level <= iter->stack_len; level++)
+ for (int level = 1; level <= iter->top_level; level++)
{
- child = RT_NODE_INNER_ITERATE_NEXT(iter, &(iter->stack[level]));
+ child = RT_NODE_INNER_ITERATE_NEXT(iter, &(iter->node_iters[level]));
if (child)
break;
}
- /* the iteration finished */
+ /* We've visited all nodes, so the iteration finished */
if (!child)
- return false;
+ break;
/*
- * Set the node to the node iterator and update the iterator stack
- * from this node.
+ * Found the new child node. We update the next node to iterate for each
+ * level from the level of this child node.
*/
- RT_UPDATE_ITER_STACK(iter, child, level - 1);
+ RT_ITER_SET_NODE_FROM(iter, child);
- /* Node iterators are updated, so try again from the leaf */
+ /* Find key-value from the leaf node again */
}
return false;
@@ -2508,8 +2517,7 @@ RT_DUMP(RT_RADIX_TREE *tree)
#undef RT_NODE_INSERT_LEAF
#undef RT_NODE_INNER_ITERATE_NEXT
#undef RT_NODE_LEAF_ITERATE_NEXT
-#undef RT_UPDATE_ITER_STACK
-#undef RT_ITER_UPDATE_KEY
+#undef RT_RT_ITER_SET_NODE_FROM
#undef RT_VERIFY_NODE
#undef RT_DEBUG
diff --git a/src/include/lib/radixtree_iter_impl.h b/src/include/lib/radixtree_iter_impl.h
index 98c78eb237..5c1034768e 100644
--- a/src/include/lib/radixtree_iter_impl.h
+++ b/src/include/lib/radixtree_iter_impl.h
@@ -27,12 +27,10 @@
#error node level must be either inner or leaf
#endif
- bool found = false;
- uint8 key_chunk;
+ uint8 key_chunk = 0;
#ifdef RT_NODE_LEVEL_LEAF
- RT_VALUE_TYPE value;
-
+ Assert(value_p != NULL);
Assert(RT_NODE_IS_LEAF(node_iter->node));
#else
RT_PTR_LOCAL child = NULL;
@@ -50,99 +48,92 @@
{
RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node_iter->node;
- node_iter->current_idx++;
- if (node_iter->current_idx >= n3->base.n.count)
- break;
+ if (node_iter->idx >= n3->base.n.count)
+ return false;
+
#ifdef RT_NODE_LEVEL_LEAF
- value = n3->values[node_iter->current_idx];
+ *value_p = n3->values[node_iter->idx];
#else
- child = RT_PTR_GET_LOCAL(iter->tree, n3->children[node_iter->current_idx]);
+ child = RT_PTR_GET_LOCAL(iter->tree, n3->children[node_iter->idx]);
#endif
- key_chunk = n3->base.chunks[node_iter->current_idx];
- found = true;
+ key_chunk = n3->base.chunks[node_iter->idx];
+ node_iter->idx++;
break;
}
case RT_NODE_KIND_32:
{
RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node_iter->node;
- node_iter->current_idx++;
- if (node_iter->current_idx >= n32->base.n.count)
- break;
+ if (node_iter->idx >= n32->base.n.count)
+ return false;
#ifdef RT_NODE_LEVEL_LEAF
- value = n32->values[node_iter->current_idx];
+ *value_p = n32->values[node_iter->idx];
#else
- child = RT_PTR_GET_LOCAL(iter->tree, n32->children[node_iter->current_idx]);
+ child = RT_PTR_GET_LOCAL(iter->tree, n32->children[node_iter->idx]);
#endif
- key_chunk = n32->base.chunks[node_iter->current_idx];
- found = true;
+ key_chunk = n32->base.chunks[node_iter->idx];
+ node_iter->idx++;
break;
}
case RT_NODE_KIND_125:
{
RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node_iter->node;
- int i;
+ int chunk;
- for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ for (chunk = node_iter->idx; chunk < RT_NODE_MAX_SLOTS; chunk++)
{
- if (RT_NODE_125_IS_CHUNK_USED((RT_NODE_BASE_125 *) n125, i))
+ if (RT_NODE_125_IS_CHUNK_USED((RT_NODE_BASE_125 *) n125, chunk))
break;
}
- if (i >= RT_NODE_MAX_SLOTS)
- break;
+ if (chunk >= RT_NODE_MAX_SLOTS)
+ return false;
- node_iter->current_idx = i;
#ifdef RT_NODE_LEVEL_LEAF
- value = RT_NODE_LEAF_125_GET_VALUE(n125, i);
+ *value_p = RT_NODE_LEAF_125_GET_VALUE(n125, chunk);
#else
- child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_125_GET_CHILD(n125, chunk));
#endif
- key_chunk = i;
- found = true;
+ key_chunk = chunk;
+ node_iter->idx = chunk + 1;
break;
}
case RT_NODE_KIND_256:
{
RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node_iter->node;
- int i;
+ int chunk;
- for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ for (chunk = node_iter->idx; chunk < RT_NODE_MAX_SLOTS; chunk++)
{
#ifdef RT_NODE_LEVEL_LEAF
- if (RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+ if (RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
#else
- if (RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ if (RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
#endif
break;
}
- if (i >= RT_NODE_MAX_SLOTS)
- break;
+ if (chunk >= RT_NODE_MAX_SLOTS)
+ return false;
- node_iter->current_idx = i;
#ifdef RT_NODE_LEVEL_LEAF
- value = RT_NODE_LEAF_256_GET_VALUE(n256, i);
+ *value_p = RT_NODE_LEAF_256_GET_VALUE(n256, chunk);
#else
- child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_256_GET_CHILD(n256, chunk));
#endif
- key_chunk = i;
- found = true;
+ key_chunk = chunk;
+ node_iter->idx = chunk + 1;
break;
}
}
- if (found)
- {
- RT_ITER_UPDATE_KEY(iter, key_chunk, node_iter->node->shift);
-#ifdef RT_NODE_LEVEL_LEAF
- *value_p = value;
-#endif
- }
+ /* Update the part of the key */
+ iter->key &= ~(((uint64) RT_CHUNK_MASK) << node_iter->node->shift);
+ iter->key |= (((uint64) key_chunk) << node_iter->node->shift);
#ifdef RT_NODE_LEVEL_LEAF
- return found;
+ return true;
#else
return child;
#endif
--
2.31.1
[application/octet-stream] v32-0013-tidstore-Embed-output-offsets-in-TidStoreIterRes.patch (1.9K, ../../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/11-v32-0013-tidstore-Embed-output-offsets-in-TidStoreIterRes.patch)
download | inline diff:
From 453dc7fd8078ba202569417d4ed65ce6e7f4a850 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 18:00:44 +0900
Subject: [PATCH v32 13/18] tidstore: Embed output offsets in
TidStoreIterResult.
---
src/backend/access/common/tidstore.c | 7 ++-----
src/include/access/tidstore.h | 3 ++-
2 files changed, 4 insertions(+), 6 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index d9fe3d5f15..15b77b5bcb 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -470,12 +470,10 @@ TidStoreBeginIterate(TidStore *ts)
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
- iter = palloc0(sizeof(TidStoreIter));
+ iter = palloc0(sizeof(TidStoreIter) +
+ sizeof(OffsetNumber) * ts->control->max_off);
iter->ts = ts;
- iter->result.blkno = InvalidBlockNumber;
- iter->result.offsets = palloc(sizeof(OffsetNumber) * ts->control->max_offset);
-
if (TidStoreIsShared(ts))
iter->tree_iter.shared = shared_rt_begin_iterate(ts->tree.shared);
else
@@ -559,7 +557,6 @@ TidStoreEndIterate(TidStoreIter *iter)
else
local_rt_end_iterate(iter->tree_iter.local);
- pfree(iter->result.offsets);
pfree(iter);
}
diff --git a/src/include/access/tidstore.h b/src/include/access/tidstore.h
index f0a432d0da..66f0fdd482 100644
--- a/src/include/access/tidstore.h
+++ b/src/include/access/tidstore.h
@@ -22,11 +22,12 @@ typedef dsa_pointer TidStoreHandle;
typedef struct TidStore TidStore;
typedef struct TidStoreIter TidStoreIter;
+/* Result struct for TidStoreIterateNext */
typedef struct TidStoreIterResult
{
BlockNumber blkno;
- OffsetNumber *offsets;
int num_offsets;
+ OffsetNumber offsets[FLEXIBLE_ARRAY_MEMBER];
} TidStoreIterResult;
extern TidStore *TidStoreCreate(size_t max_bytes, int max_off, dsa_area *dsa);
--
2.31.1
[application/octet-stream] v32-0008-radix-tree-remove-resolved-TODO.patch (706B, ../../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/12-v32-0008-radix-tree-remove-resolved-TODO.patch)
download | inline diff:
From 84bad553eecc97bbc3d7ccacc90723ae22b7888f Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 17:29:32 +0900
Subject: [PATCH v32 08/18] radix tree: remove resolved TODO
---
src/include/lib/radixtree.h | 1 -
1 file changed, 1 deletion(-)
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index c277d5a484..088d1dfd9d 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -612,7 +612,6 @@ static const RT_SIZE_CLASS_ELEM RT_SIZE_CLASS_INFO[] = {
#endif
/* Contains the actual tree and ancillary info */
-// WIP: this name is a bit strange
typedef struct RT_RADIX_TREE_CONTROL
{
#ifdef RT_SHMEM
--
2.31.1
[application/octet-stream] v32-0007-radix-tree-rename-RT_EXTEND-and-RT_SET_EXTEND-to.patch (2.6K, ../../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/13-v32-0007-radix-tree-rename-RT_EXTEND-and-RT_SET_EXTEND-to.patch)
download | inline diff:
From e25dc39fd502ae5c6c1c44a798a24dc5c6a1c7b0 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 17:26:52 +0900
Subject: [PATCH v32 07/18] radix tree: rename RT_EXTEND and RT_SET_EXTEND to
RT_EXTEND_UP/DOWN
---
src/include/lib/radixtree.h | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index e546bd705c..c277d5a484 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -152,8 +152,8 @@
#define RT_INIT_NODE RT_MAKE_NAME(init_node)
#define RT_FREE_NODE RT_MAKE_NAME(free_node)
#define RT_FREE_RECURSE RT_MAKE_NAME(free_recurse)
-#define RT_EXTEND RT_MAKE_NAME(extend)
-#define RT_SET_EXTEND RT_MAKE_NAME(set_extend)
+#define RT_EXTEND_UP RT_MAKE_NAME(extend_up)
+#define RT_EXTEND_DOWN RT_MAKE_NAME(extend_down)
#define RT_SWITCH_NODE_KIND RT_MAKE_NAME(grow_node_kind)
#define RT_COPY_NODE RT_MAKE_NAME(copy_node)
#define RT_REPLACE_NODE RT_MAKE_NAME(replace_node)
@@ -1243,7 +1243,7 @@ RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
* it can store the key.
*/
static pg_noinline void
-RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
+RT_EXTEND_UP(RT_RADIX_TREE *tree, uint64 key)
{
int target_shift;
RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
@@ -1282,7 +1282,7 @@ RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
* Insert inner and leaf nodes from 'node' to bottom.
*/
static pg_noinline void
-RT_SET_EXTEND(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
+RT_EXTEND_DOWN(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node)
{
int shift = node->shift;
@@ -1613,7 +1613,7 @@ RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
/* Extend the tree if necessary */
if (key > tree->ctl->max_val)
- RT_EXTEND(tree, key);
+ RT_EXTEND_UP(tree, key);
stored_child = tree->ctl->root;
parent = RT_PTR_GET_LOCAL(tree, stored_child);
@@ -1631,7 +1631,7 @@ RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
if (!RT_NODE_SEARCH_INNER(child, key, &new_child))
{
- RT_SET_EXTEND(tree, key, value_p, parent, stored_child, child);
+ RT_EXTEND_DOWN(tree, key, value_p, parent, stored_child, child);
RT_UNLOCK(tree);
return false;
}
@@ -2470,8 +2470,8 @@ RT_DUMP(RT_RADIX_TREE *tree)
#undef RT_INIT_NODE
#undef RT_FREE_NODE
#undef RT_FREE_RECURSE
-#undef RT_EXTEND
-#undef RT_SET_EXTEND
+#undef RT_EXTEND_UP
+#undef RT_EXTEND_DOWN
#undef RT_SWITCH_NODE_KIND
#undef RT_COPY_NODE
#undef RT_REPLACE_NODE
--
2.31.1
[application/octet-stream] v32-0006-Use-TIDStore-for-storing-dead-tuple-TID-during-l.patch (48.3K, ../../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/14-v32-0006-Use-TIDStore-for-storing-dead-tuple-TID-during-l.patch)
download | inline diff:
From 1f6c4aa27d734b8c81369541481b0d3abd0d5dec Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 17:22:03 +0900
Subject: [PATCH v32 06/18] Use TIDStore for storing dead tuple TID during lazy
vacuum
Previously, we used an array of ItemPointerData to store dead tuple
TIDs, which was not space efficient and slow to lookup. Also, we had
the 1GB limit on its size.
Now we use TIDStore to store dead tuple TIDs. Since the TIDStore,
backed by the radix tree, incrementaly allocates the memory, we get
rid of the 1GB limit.
Since we are no longer able to exactly estimate the maximum number of
TIDs can be stored the pg_stat_progress_vacuum shows the progress
information based on the amount of memory in bytes. The column names
are also changed to max_dead_tuple_bytes and num_dead_tuple_bytes.
In addition, since the TIDStore use the radix tree internally, the
minimum amount of memory required by TIDStore is 1MB, the inital DSA
segment size. Due to that, we increase the minimum value of
maintenance_work_mem (also autovacuum_work_mem) from 1MB to 2MB.
XXX: needs to bump catalog version
---
doc/src/sgml/monitoring.sgml | 8 +-
src/backend/access/heap/vacuumlazy.c | 278 ++++++++-------------
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/vacuum.c | 78 +-----
src/backend/commands/vacuumparallel.c | 73 +++---
src/backend/postmaster/autovacuum.c | 6 +-
src/backend/storage/lmgr/lwlock.c | 2 +
src/backend/utils/misc/guc_tables.c | 2 +-
src/include/commands/progress.h | 4 +-
src/include/commands/vacuum.h | 25 +-
src/include/storage/lwlock.h | 1 +
src/test/regress/expected/cluster.out | 2 +-
src/test/regress/expected/create_index.out | 2 +-
src/test/regress/expected/rules.out | 4 +-
src/test/regress/sql/cluster.sql | 2 +-
src/test/regress/sql/create_index.sql | 2 +-
16 files changed, 177 insertions(+), 314 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index be4448fe6e..9b64614beb 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -7320,10 +7320,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>max_dead_tuples</structfield> <type>bigint</type>
+ <structfield>max_dead_tuple_bytes</structfield> <type>bigint</type>
</para>
<para>
- Number of dead tuples that we can store before needing to perform
+ Amount of dead tuple data that we can store before needing to perform
an index vacuum cycle, based on
<xref linkend="guc-maintenance-work-mem"/>.
</para></entry>
@@ -7331,10 +7331,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>num_dead_tuples</structfield> <type>bigint</type>
+ <structfield>num_dead_tuple_bytes</structfield> <type>bigint</type>
</para>
<para>
- Number of dead tuples collected since the last index vacuum cycle.
+ Amount of dead tuple data collected since the last index vacuum cycle.
</para></entry>
</row>
</tbody>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 0a9ebd22bd..2c72088e69 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3,18 +3,18 @@
* vacuumlazy.c
* Concurrent ("lazy") vacuuming.
*
- * The major space usage for vacuuming is storage for the array of dead TIDs
+ * The major space usage for vacuuming is TidStore, a storage for dead TIDs
* that are to be removed from indexes. We want to ensure we can vacuum even
* the very largest relations with finite memory space usage. To do that, we
- * set upper bounds on the number of TIDs we can keep track of at once.
+ * set upper bounds on the maximum memory that can be used for keeping track
+ * of dead TIDs at once.
*
* We are willing to use at most maintenance_work_mem (or perhaps
* autovacuum_work_mem) memory space to keep track of dead TIDs. We initially
- * allocate an array of TIDs of that size, with an upper limit that depends on
- * table size (this limit ensures we don't allocate a huge area uselessly for
- * vacuuming small tables). If the array threatens to overflow, we must call
- * lazy_vacuum to vacuum indexes (and to vacuum the pages that we've pruned).
- * This frees up the memory space dedicated to storing dead TIDs.
+ * create a TidStore with the maximum bytes that can be used by the TidStore.
+ * If the TidStore is full, we must call lazy_vacuum to vacuum indexes (and to
+ * vacuum the pages that we've pruned). This frees up the memory space dedicated
+ * to storing dead TIDs.
*
* In practice VACUUM will often complete its initial pass over the target
* heap relation without ever running out of space to store TIDs. This means
@@ -40,6 +40,7 @@
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
#include "access/multixact.h"
+#include "access/tidstore.h"
#include "access/transam.h"
#include "access/visibilitymap.h"
#include "access/xact.h"
@@ -186,7 +187,7 @@ typedef struct LVRelState
* lazy_vacuum_heap_rel, which marks the same LP_DEAD line pointers as
* LP_UNUSED during second heap pass.
*/
- VacDeadItems *dead_items; /* TIDs whose index tuples we'll delete */
+ TidStore *dead_items; /* TIDs whose index tuples we'll delete */
BlockNumber rel_pages; /* total number of pages */
BlockNumber scanned_pages; /* # pages examined (not skipped via VM) */
BlockNumber removed_pages; /* # pages removed by relation truncation */
@@ -218,11 +219,14 @@ typedef struct LVRelState
typedef struct LVPagePruneState
{
bool hastup; /* Page prevents rel truncation? */
- bool has_lpdead_items; /* includes existing LP_DEAD items */
+
+ /* collected offsets of LP_DEAD items including existing ones */
+ OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
+ int num_offsets;
/*
* State describes the proper VM bit states to set for the page following
- * pruning and freezing. all_visible implies !has_lpdead_items, but don't
+ * pruning and freezing. all_visible implies num_offsets == 0, but don't
* trust all_frozen result unless all_visible is also set to true.
*/
bool all_visible; /* Every item visible to all? */
@@ -257,8 +261,9 @@ static bool lazy_scan_noprune(LVRelState *vacrel, Buffer buf,
static void lazy_vacuum(LVRelState *vacrel);
static bool lazy_vacuum_all_indexes(LVRelState *vacrel);
static void lazy_vacuum_heap_rel(LVRelState *vacrel);
-static int lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
- Buffer buffer, int index, Buffer vmbuffer);
+static void lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
+ OffsetNumber *offsets, int num_offsets,
+ Buffer buffer, Buffer vmbuffer);
static bool lazy_check_wraparound_failsafe(LVRelState *vacrel);
static void lazy_cleanup_all_indexes(LVRelState *vacrel);
static IndexBulkDeleteResult *lazy_vacuum_one_index(Relation indrel,
@@ -485,11 +490,11 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
}
/*
- * Allocate dead_items array memory using dead_items_alloc. This handles
- * parallel VACUUM initialization as part of allocating shared memory
- * space used for dead_items. (But do a failsafe precheck first, to
- * ensure that parallel VACUUM won't be attempted at all when relfrozenxid
- * is already dangerously old.)
+ * Allocate dead_items memory using dead_items_alloc. This handles parallel
+ * VACUUM initialization as part of allocating shared memory space used for
+ * dead_items. (But do a failsafe precheck first, to ensure that parallel
+ * VACUUM won't be attempted at all when relfrozenxid is already dangerously
+ * old.)
*/
lazy_check_wraparound_failsafe(vacrel);
dead_items_alloc(vacrel, params->nworkers);
@@ -795,7 +800,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* have collected the TIDs whose index tuples need to be removed.
*
* Finally, invokes lazy_vacuum_heap_rel to vacuum heap pages, which
- * largely consists of marking LP_DEAD items (from collected TID array)
+ * largely consists of marking LP_DEAD items (from vacrel->dead_items)
* as LP_UNUSED. This has to happen in a second, final pass over the
* heap, to preserve a basic invariant that all index AMs rely on: no
* extant index tuple can ever be allowed to contain a TID that points to
@@ -823,21 +828,21 @@ lazy_scan_heap(LVRelState *vacrel)
blkno,
next_unskippable_block,
next_fsm_block_to_vacuum = 0;
- VacDeadItems *dead_items = vacrel->dead_items;
+ TidStore *dead_items = vacrel->dead_items;
Buffer vmbuffer = InvalidBuffer;
bool next_unskippable_allvis,
skipping_current_range;
const int initprog_index[] = {
PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_TOTAL_HEAP_BLKS,
- PROGRESS_VACUUM_MAX_DEAD_TUPLES
+ PROGRESS_VACUUM_MAX_DEAD_TUPLE_BYTES
};
int64 initprog_val[3];
/* Report that we're scanning the heap, advertising total # of blocks */
initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
initprog_val[1] = rel_pages;
- initprog_val[2] = dead_items->max_items;
+ initprog_val[2] = tidstore_max_memory(vacrel->dead_items);
pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
/* Set up an initial range of skippable blocks using the visibility map */
@@ -904,8 +909,7 @@ lazy_scan_heap(LVRelState *vacrel)
* dead_items TIDs, pause and do a cycle of vacuuming before we tackle
* this page.
*/
- Assert(dead_items->max_items >= MaxHeapTuplesPerPage);
- if (dead_items->max_items - dead_items->num_items < MaxHeapTuplesPerPage)
+ if (tidstore_is_full(vacrel->dead_items))
{
/*
* Before beginning index vacuuming, we release any pin we may
@@ -967,7 +971,7 @@ lazy_scan_heap(LVRelState *vacrel)
continue;
}
- /* Collect LP_DEAD items in dead_items array, count tuples */
+ /* Collect LP_DEAD items in dead_items, count tuples */
if (lazy_scan_noprune(vacrel, buf, blkno, page, &hastup,
&recordfreespace))
{
@@ -1009,14 +1013,14 @@ lazy_scan_heap(LVRelState *vacrel)
* Prune, freeze, and count tuples.
*
* Accumulates details of remaining LP_DEAD line pointers on page in
- * dead_items array. This includes LP_DEAD line pointers that we
- * pruned ourselves, as well as existing LP_DEAD line pointers that
- * were pruned some time earlier. Also considers freezing XIDs in the
- * tuple headers of remaining items with storage.
+ * dead_items. This includes LP_DEAD line pointers that we pruned
+ * ourselves, as well as existing LP_DEAD line pointers that were pruned
+ * some time earlier. Also considers freezing XIDs in the tuple headers
+ * of remaining items with storage.
*/
lazy_scan_prune(vacrel, buf, blkno, page, &prunestate);
- Assert(!prunestate.all_visible || !prunestate.has_lpdead_items);
+ Assert(!prunestate.all_visible || (prunestate.num_offsets == 0));
/* Remember the location of the last page with nonremovable tuples */
if (prunestate.hastup)
@@ -1032,14 +1036,12 @@ lazy_scan_heap(LVRelState *vacrel)
* performed here can be thought of as the one-pass equivalent of
* a call to lazy_vacuum().
*/
- if (prunestate.has_lpdead_items)
+ if (prunestate.num_offsets > 0)
{
Size freespace;
- lazy_vacuum_heap_page(vacrel, blkno, buf, 0, vmbuffer);
-
- /* Forget the LP_DEAD items that we just vacuumed */
- dead_items->num_items = 0;
+ lazy_vacuum_heap_page(vacrel, blkno, prunestate.deadoffsets,
+ prunestate.num_offsets, buf, vmbuffer);
/*
* Periodically perform FSM vacuuming to make newly-freed
@@ -1076,7 +1078,16 @@ lazy_scan_heap(LVRelState *vacrel)
* with prunestate-driven visibility map and FSM steps (just like
* the two-pass strategy).
*/
- Assert(dead_items->num_items == 0);
+ Assert(tidstore_num_tids(dead_items) == 0);
+ }
+ else if (prunestate.num_offsets > 0)
+ {
+ /* Save details of the LP_DEAD items from the page in dead_items */
+ tidstore_add_tids(dead_items, blkno, prunestate.deadoffsets,
+ prunestate.num_offsets);
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
+ tidstore_memory_usage(dead_items));
}
/*
@@ -1143,7 +1154,7 @@ lazy_scan_heap(LVRelState *vacrel)
* There should never be LP_DEAD items on a page with PD_ALL_VISIBLE
* set, however.
*/
- else if (prunestate.has_lpdead_items && PageIsAllVisible(page))
+ else if ((prunestate.num_offsets > 0) && PageIsAllVisible(page))
{
elog(WARNING, "page containing LP_DEAD items is marked as all-visible in relation \"%s\" page %u",
vacrel->relname, blkno);
@@ -1191,7 +1202,7 @@ lazy_scan_heap(LVRelState *vacrel)
* Final steps for block: drop cleanup lock, record free space in the
* FSM
*/
- if (prunestate.has_lpdead_items && vacrel->do_index_vacuuming)
+ if ((prunestate.num_offsets > 0) && vacrel->do_index_vacuuming)
{
/*
* Wait until lazy_vacuum_heap_rel() to save free space. This
@@ -1247,7 +1258,7 @@ lazy_scan_heap(LVRelState *vacrel)
* Do index vacuuming (call each index's ambulkdelete routine), then do
* related heap vacuuming
*/
- if (dead_items->num_items > 0)
+ if (tidstore_num_tids(dead_items) > 0)
lazy_vacuum(vacrel);
/*
@@ -1522,9 +1533,9 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* The approach we take now is to restart pruning when the race condition is
* detected. This allows heap_page_prune() to prune the tuples inserted by
* the now-aborted transaction. This is a little crude, but it guarantees
- * that any items that make it into the dead_items array are simple LP_DEAD
- * line pointers, and that every remaining item with tuple storage is
- * considered as a candidate for freezing.
+ * that any items that make it into the dead_items are simple LP_DEAD line
+ * pointers, and that every remaining item with tuple storage is considered
+ * as a candidate for freezing.
*/
static void
lazy_scan_prune(LVRelState *vacrel,
@@ -1541,13 +1552,11 @@ lazy_scan_prune(LVRelState *vacrel,
HTSV_Result res;
int tuples_deleted,
tuples_frozen,
- lpdead_items,
live_tuples,
recently_dead_tuples;
int nnewlpdead;
HeapPageFreeze pagefrz;
int64 fpi_before = pgWalUsage.wal_fpi;
- OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
@@ -1569,7 +1578,6 @@ retry:
pagefrz.NoFreezePageRelminMxid = vacrel->NewRelminMxid;
tuples_deleted = 0;
tuples_frozen = 0;
- lpdead_items = 0;
live_tuples = 0;
recently_dead_tuples = 0;
@@ -1578,9 +1586,9 @@ retry:
*
* We count tuples removed by the pruning step as tuples_deleted. Its
* final value can be thought of as the number of tuples that have been
- * deleted from the table. It should not be confused with lpdead_items;
- * lpdead_items's final value can be thought of as the number of tuples
- * that were deleted from indexes.
+ * deleted from the table. It should not be confused with
+ * prunestate->deadoffsets; prunestate->deadoffsets's final value can
+ * be thought of as the number of tuples that were deleted from indexes.
*/
tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest,
InvalidTransactionId, 0, &nnewlpdead,
@@ -1591,7 +1599,7 @@ retry:
* requiring freezing among remaining tuples with storage
*/
prunestate->hastup = false;
- prunestate->has_lpdead_items = false;
+ prunestate->num_offsets = 0;
prunestate->all_visible = true;
prunestate->all_frozen = true;
prunestate->visibility_cutoff_xid = InvalidTransactionId;
@@ -1636,7 +1644,7 @@ retry:
* (This is another case where it's useful to anticipate that any
* LP_DEAD items will become LP_UNUSED during the ongoing VACUUM.)
*/
- deadoffsets[lpdead_items++] = offnum;
+ prunestate->deadoffsets[prunestate->num_offsets++] = offnum;
continue;
}
@@ -1873,7 +1881,7 @@ retry:
*/
#ifdef USE_ASSERT_CHECKING
/* Note that all_frozen value does not matter when !all_visible */
- if (prunestate->all_visible && lpdead_items == 0)
+ if (prunestate->all_visible && prunestate->num_offsets == 0)
{
TransactionId cutoff;
bool all_frozen;
@@ -1886,28 +1894,9 @@ retry:
}
#endif
- /*
- * Now save details of the LP_DEAD items from the page in vacrel
- */
- if (lpdead_items > 0)
+ if (prunestate->num_offsets > 0)
{
- VacDeadItems *dead_items = vacrel->dead_items;
- ItemPointerData tmp;
-
vacrel->lpdead_item_pages++;
- prunestate->has_lpdead_items = true;
-
- ItemPointerSetBlockNumber(&tmp, blkno);
-
- for (int i = 0; i < lpdead_items; i++)
- {
- ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
- dead_items->items[dead_items->num_items++] = tmp;
- }
-
- Assert(dead_items->num_items <= dead_items->max_items);
- pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
- dead_items->num_items);
/*
* It was convenient to ignore LP_DEAD items in all_visible earlier on
@@ -1926,7 +1915,7 @@ retry:
/* Finally, add page-local counts to whole-VACUUM counts */
vacrel->tuples_deleted += tuples_deleted;
vacrel->tuples_frozen += tuples_frozen;
- vacrel->lpdead_items += lpdead_items;
+ vacrel->lpdead_items += prunestate->num_offsets;
vacrel->live_tuples += live_tuples;
vacrel->recently_dead_tuples += recently_dead_tuples;
}
@@ -1938,7 +1927,7 @@ retry:
* lazy_scan_prune, which requires a full cleanup lock. While pruning isn't
* performed here, it's quite possible that an earlier opportunistic pruning
* operation left LP_DEAD items behind. We'll at least collect any such items
- * in the dead_items array for removal from indexes.
+ * in the dead_items for removal from indexes.
*
* For aggressive VACUUM callers, we may return false to indicate that a full
* cleanup lock is required for processing by lazy_scan_prune. This is only
@@ -2097,7 +2086,7 @@ lazy_scan_noprune(LVRelState *vacrel,
vacrel->NewRelfrozenXid = NoFreezePageRelfrozenXid;
vacrel->NewRelminMxid = NoFreezePageRelminMxid;
- /* Save any LP_DEAD items found on the page in dead_items array */
+ /* Save any LP_DEAD items found on the page in dead_items */
if (vacrel->nindexes == 0)
{
/* Using one-pass strategy (since table has no indexes) */
@@ -2127,8 +2116,7 @@ lazy_scan_noprune(LVRelState *vacrel,
}
else
{
- VacDeadItems *dead_items = vacrel->dead_items;
- ItemPointerData tmp;
+ TidStore *dead_items = vacrel->dead_items;
/*
* Page has LP_DEAD items, and so any references/TIDs that remain in
@@ -2137,17 +2125,10 @@ lazy_scan_noprune(LVRelState *vacrel,
*/
vacrel->lpdead_item_pages++;
- ItemPointerSetBlockNumber(&tmp, blkno);
+ tidstore_add_tids(dead_items, blkno, deadoffsets, lpdead_items);
- for (int i = 0; i < lpdead_items; i++)
- {
- ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
- dead_items->items[dead_items->num_items++] = tmp;
- }
-
- Assert(dead_items->num_items <= dead_items->max_items);
- pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
- dead_items->num_items);
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
+ tidstore_memory_usage(dead_items));
vacrel->lpdead_items += lpdead_items;
@@ -2196,7 +2177,7 @@ lazy_vacuum(LVRelState *vacrel)
if (!vacrel->do_index_vacuuming)
{
Assert(!vacrel->do_index_cleanup);
- vacrel->dead_items->num_items = 0;
+ tidstore_reset(vacrel->dead_items);
return;
}
@@ -2225,7 +2206,7 @@ lazy_vacuum(LVRelState *vacrel)
BlockNumber threshold;
Assert(vacrel->num_index_scans == 0);
- Assert(vacrel->lpdead_items == vacrel->dead_items->num_items);
+ Assert(vacrel->lpdead_items == tidstore_num_tids(vacrel->dead_items));
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2252,8 +2233,8 @@ lazy_vacuum(LVRelState *vacrel)
* cases then this may need to be reconsidered.
*/
threshold = (double) vacrel->rel_pages * BYPASS_THRESHOLD_PAGES;
- bypass = (vacrel->lpdead_item_pages < threshold &&
- vacrel->lpdead_items < MAXDEADITEMS(32L * 1024L * 1024L));
+ bypass = (vacrel->lpdead_item_pages < threshold) &&
+ tidstore_memory_usage(vacrel->dead_items) < (32L * 1024L * 1024L);
}
if (bypass)
@@ -2298,7 +2279,7 @@ lazy_vacuum(LVRelState *vacrel)
* Forget the LP_DEAD items that we just vacuumed (or just decided to not
* vacuum)
*/
- vacrel->dead_items->num_items = 0;
+ tidstore_reset(vacrel->dead_items);
}
/*
@@ -2371,7 +2352,7 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
* place).
*/
Assert(vacrel->num_index_scans > 0 ||
- vacrel->dead_items->num_items == vacrel->lpdead_items);
+ tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items);
Assert(allindexes || VacuumFailsafeActive);
/*
@@ -2390,9 +2371,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
/*
* lazy_vacuum_heap_rel() -- second pass over the heap for two pass strategy
*
- * This routine marks LP_DEAD items in vacrel->dead_items array as LP_UNUSED.
- * Pages that never had lazy_scan_prune record LP_DEAD items are not visited
- * at all.
+ * This routine marks LP_DEAD items in vacrel->dead_items as LP_UNUSED. Pages
+ * that never had lazy_scan_prune record LP_DEAD items are not visited at all.
*
* We may also be able to truncate the line pointer array of the heap pages we
* visit. If there is a contiguous group of LP_UNUSED items at the end of the
@@ -2408,10 +2388,11 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
static void
lazy_vacuum_heap_rel(LVRelState *vacrel)
{
- int index = 0;
BlockNumber vacuumed_pages = 0;
Buffer vmbuffer = InvalidBuffer;
LVSavedErrInfo saved_err_info;
+ TidStoreIter *iter;
+ TidStoreIterResult *result;
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2426,7 +2407,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
VACUUM_ERRCB_PHASE_VACUUM_HEAP,
InvalidBlockNumber, InvalidOffsetNumber);
- while (index < vacrel->dead_items->num_items)
+ iter = tidstore_begin_iterate(vacrel->dead_items);
+ while ((result = tidstore_iterate_next(iter)) != NULL)
{
BlockNumber blkno;
Buffer buf;
@@ -2435,7 +2417,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
vacuum_delay_point();
- blkno = ItemPointerGetBlockNumber(&vacrel->dead_items->items[index]);
+ blkno = result->blkno;
vacrel->blkno = blkno;
/*
@@ -2449,7 +2431,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
buf = ReadBufferExtended(vacrel->rel, MAIN_FORKNUM, blkno, RBM_NORMAL,
vacrel->bstrategy);
LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
- index = lazy_vacuum_heap_page(vacrel, blkno, buf, index, vmbuffer);
+ lazy_vacuum_heap_page(vacrel, blkno, result->offsets, result->num_offsets,
+ buf, vmbuffer);
/* Now that we've vacuumed the page, record its available space */
page = BufferGetPage(buf);
@@ -2459,6 +2442,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
vacuumed_pages++;
}
+ tidstore_end_iterate(iter);
vacrel->blkno = InvalidBlockNumber;
if (BufferIsValid(vmbuffer))
@@ -2468,36 +2452,31 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
* We set all LP_DEAD items from the first heap pass to LP_UNUSED during
* the second heap pass. No more, no less.
*/
- Assert(index > 0);
Assert(vacrel->num_index_scans > 1 ||
- (index == vacrel->lpdead_items &&
+ (tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items &&
vacuumed_pages == vacrel->lpdead_item_pages));
ereport(DEBUG2,
- (errmsg("table \"%s\": removed %lld dead item identifiers in %u pages",
- vacrel->relname, (long long) index, vacuumed_pages)));
+ (errmsg("table \"%s\": removed " UINT64_FORMAT "dead item identifiers in %u pages",
+ vacrel->relname, tidstore_num_tids(vacrel->dead_items),
+ vacuumed_pages)));
/* Revert to the previous phase information for error traceback */
restore_vacuum_error_info(vacrel, &saved_err_info);
}
/*
- * lazy_vacuum_heap_page() -- free page's LP_DEAD items listed in the
- * vacrel->dead_items array.
+ * lazy_vacuum_heap_page() -- free page's LP_DEAD items.
*
* Caller must have an exclusive buffer lock on the buffer (though a full
* cleanup lock is also acceptable). vmbuffer must be valid and already have
* a pin on blkno's visibility map page.
- *
- * index is an offset into the vacrel->dead_items array for the first listed
- * LP_DEAD item on the page. The return value is the first index immediately
- * after all LP_DEAD items for the same page in the array.
*/
-static int
-lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
- int index, Buffer vmbuffer)
+static void
+lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
+ OffsetNumber *deadoffsets, int num_offsets, Buffer buffer,
+ Buffer vmbuffer)
{
- VacDeadItems *dead_items = vacrel->dead_items;
Page page = BufferGetPage(buffer);
OffsetNumber unused[MaxHeapTuplesPerPage];
int nunused = 0;
@@ -2516,16 +2495,11 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
START_CRIT_SECTION();
- for (; index < dead_items->num_items; index++)
+ for (int i = 0; i < num_offsets; i++)
{
- BlockNumber tblk;
- OffsetNumber toff;
ItemId itemid;
+ OffsetNumber toff = deadoffsets[i];
- tblk = ItemPointerGetBlockNumber(&dead_items->items[index]);
- if (tblk != blkno)
- break; /* past end of tuples for this block */
- toff = ItemPointerGetOffsetNumber(&dead_items->items[index]);
itemid = PageGetItemId(page, toff);
Assert(ItemIdIsDead(itemid) && !ItemIdHasStorage(itemid));
@@ -2595,7 +2569,6 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
/* Revert to the previous phase information for error traceback */
restore_vacuum_error_info(vacrel, &saved_err_info);
- return index;
}
/*
@@ -2692,8 +2665,8 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
* lazy_vacuum_one_index() -- vacuum index relation.
*
* Delete all the index tuples containing a TID collected in
- * vacrel->dead_items array. Also update running statistics.
- * Exact details depend on index AM's ambulkdelete routine.
+ * vacrel->dead_items. Also update running statistics. Exact
+ * details depend on index AM's ambulkdelete routine.
*
* reltuples is the number of heap tuples to be passed to the
* bulkdelete callback. It's always assumed to be estimated.
@@ -3101,48 +3074,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
}
/*
- * Returns the number of dead TIDs that VACUUM should allocate space to
- * store, given a heap rel of size vacrel->rel_pages, and given current
- * maintenance_work_mem setting (or current autovacuum_work_mem setting,
- * when applicable).
- *
- * See the comments at the head of this file for rationale.
- */
-static int
-dead_items_max_items(LVRelState *vacrel)
-{
- int64 max_items;
- int vac_work_mem = IsAutoVacuumWorkerProcess() &&
- autovacuum_work_mem != -1 ?
- autovacuum_work_mem : maintenance_work_mem;
-
- if (vacrel->nindexes > 0)
- {
- BlockNumber rel_pages = vacrel->rel_pages;
-
- max_items = MAXDEADITEMS(vac_work_mem * 1024L);
- max_items = Min(max_items, INT_MAX);
- max_items = Min(max_items, MAXDEADITEMS(MaxAllocSize));
-
- /* curious coding here to ensure the multiplication can't overflow */
- if ((BlockNumber) (max_items / MaxHeapTuplesPerPage) > rel_pages)
- max_items = rel_pages * MaxHeapTuplesPerPage;
-
- /* stay sane if small maintenance_work_mem */
- max_items = Max(max_items, MaxHeapTuplesPerPage);
- }
- else
- {
- /* One-pass case only stores a single heap page's TIDs at a time */
- max_items = MaxHeapTuplesPerPage;
- }
-
- return (int) max_items;
-}
-
-/*
- * Allocate dead_items (either using palloc, or in dynamic shared memory).
- * Sets dead_items in vacrel for caller.
+ * Allocate a (local or shared) TidStore for storing dead TIDs. Sets dead_items
+ * in vacrel for caller.
*
* Also handles parallel initialization as part of allocating dead_items in
* DSM when required.
@@ -3150,11 +3083,9 @@ dead_items_max_items(LVRelState *vacrel)
static void
dead_items_alloc(LVRelState *vacrel, int nworkers)
{
- VacDeadItems *dead_items;
- int max_items;
-
- max_items = dead_items_max_items(vacrel);
- Assert(max_items >= MaxHeapTuplesPerPage);
+ int vac_work_mem = IsAutoVacuumWorkerProcess() &&
+ autovacuum_work_mem != -1 ?
+ autovacuum_work_mem * 1024L : maintenance_work_mem * 1024L;
/*
* Initialize state for a parallel vacuum. As of now, only one worker can
@@ -3181,7 +3112,7 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
else
vacrel->pvs = parallel_vacuum_init(vacrel->rel, vacrel->indrels,
vacrel->nindexes, nworkers,
- max_items,
+ vac_work_mem, MaxHeapTuplesPerPage,
vacrel->verbose ? INFO : DEBUG2,
vacrel->bstrategy);
@@ -3194,11 +3125,8 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
}
/* Serial VACUUM case */
- dead_items = (VacDeadItems *) palloc(vac_max_items_to_alloc_size(max_items));
- dead_items->max_items = max_items;
- dead_items->num_items = 0;
-
- vacrel->dead_items = dead_items;
+ vacrel->dead_items = tidstore_create(vac_work_mem, MaxHeapTuplesPerPage,
+ NULL);
}
/*
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 2129c916aa..134df925ce 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1190,7 +1190,7 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuple_bytes, S.param7 AS dead_tuple_bytes
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index a843f9ad92..f3922b72dc 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -119,7 +119,6 @@ static bool vacuum_rel(Oid relid, RangeVar *relation, VacuumParams *params,
static double compute_parallel_delay(void);
static VacOptValue get_vacoptval_from_boolean(DefElem *def);
static bool vac_tid_reaped(ItemPointer itemptr, void *state);
-static int vac_cmp_itemptr(const void *left, const void *right);
/*
* GUC check function to ensure GUC value specified is within the allowable
@@ -2478,16 +2477,16 @@ get_vacoptval_from_boolean(DefElem *def)
*/
IndexBulkDeleteResult *
vac_bulkdel_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat,
- VacDeadItems *dead_items)
+ TidStore *dead_items)
{
/* Do bulk deletion */
istat = index_bulk_delete(ivinfo, istat, vac_tid_reaped,
(void *) dead_items);
ereport(ivinfo->message_level,
- (errmsg("scanned index \"%s\" to remove %d row versions",
+ (errmsg("scanned index \"%s\" to remove " UINT64_FORMAT " row versions",
RelationGetRelationName(ivinfo->index),
- dead_items->num_items)));
+ tidstore_num_tids(dead_items))));
return istat;
}
@@ -2518,82 +2517,15 @@ vac_cleanup_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat)
return istat;
}
-/*
- * Returns the total required space for VACUUM's dead_items array given a
- * max_items value.
- */
-Size
-vac_max_items_to_alloc_size(int max_items)
-{
- Assert(max_items <= MAXDEADITEMS(MaxAllocSize));
-
- return offsetof(VacDeadItems, items) + sizeof(ItemPointerData) * max_items;
-}
-
/*
* vac_tid_reaped() -- is a particular tid deletable?
*
* This has the right signature to be an IndexBulkDeleteCallback.
- *
- * Assumes dead_items array is sorted (in ascending TID order).
*/
static bool
vac_tid_reaped(ItemPointer itemptr, void *state)
{
- VacDeadItems *dead_items = (VacDeadItems *) state;
- int64 litem,
- ritem,
- item;
- ItemPointer res;
-
- litem = itemptr_encode(&dead_items->items[0]);
- ritem = itemptr_encode(&dead_items->items[dead_items->num_items - 1]);
- item = itemptr_encode(itemptr);
-
- /*
- * Doing a simple bound check before bsearch() is useful to avoid the
- * extra cost of bsearch(), especially if dead items on the heap are
- * concentrated in a certain range. Since this function is called for
- * every index tuple, it pays to be really fast.
- */
- if (item < litem || item > ritem)
- return false;
-
- res = (ItemPointer) bsearch(itemptr,
- dead_items->items,
- dead_items->num_items,
- sizeof(ItemPointerData),
- vac_cmp_itemptr);
-
- return (res != NULL);
-}
-
-/*
- * Comparator routines for use with qsort() and bsearch().
- */
-static int
-vac_cmp_itemptr(const void *left, const void *right)
-{
- BlockNumber lblk,
- rblk;
- OffsetNumber loff,
- roff;
-
- lblk = ItemPointerGetBlockNumber((ItemPointer) left);
- rblk = ItemPointerGetBlockNumber((ItemPointer) right);
-
- if (lblk < rblk)
- return -1;
- if (lblk > rblk)
- return 1;
-
- loff = ItemPointerGetOffsetNumber((ItemPointer) left);
- roff = ItemPointerGetOffsetNumber((ItemPointer) right);
-
- if (loff < roff)
- return -1;
- if (loff > roff)
- return 1;
+ TidStore *dead_items = (TidStore *) state;
- return 0;
+ return tidstore_lookup_tid(dead_items, itemptr);
}
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 87ea5c5242..c363f45e32 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -9,12 +9,11 @@
* In a parallel vacuum, we perform both index bulk deletion and index cleanup
* with parallel worker processes. Individual indexes are processed by one
* vacuum process. ParalleVacuumState contains shared information as well as
- * the memory space for storing dead items allocated in the DSM segment. We
- * launch parallel worker processes at the start of parallel index
- * bulk-deletion and index cleanup and once all indexes are processed, the
- * parallel worker processes exit. Each time we process indexes in parallel,
- * the parallel context is re-initialized so that the same DSM can be used for
- * multiple passes of index bulk-deletion and index cleanup.
+ * the shared TidStore. We launch parallel worker processes at the start of
+ * parallel index bulk-deletion and index cleanup and once all indexes are
+ * processed, the parallel worker processes exit. Each time we process indexes
+ * in parallel, the parallel context is re-initialized so that the same DSM can
+ * be used for multiple passes of index bulk-deletion and index cleanup.
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -109,6 +108,9 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /* Handle of the shared TidStore */
+ tidstore_handle dead_items_handle;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -175,7 +177,8 @@ struct ParallelVacuumState
PVIndStats *indstats;
/* Shared dead items space among parallel vacuum workers */
- VacDeadItems *dead_items;
+ TidStore *dead_items;
+ dsa_area *dead_items_area;
/* Points to buffer usage area in DSM */
BufferUsage *buffer_usage;
@@ -231,20 +234,23 @@ static void parallel_vacuum_error_callback(void *arg);
*/
ParallelVacuumState *
parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
- int nrequested_workers, int max_items,
- int elevel, BufferAccessStrategy bstrategy)
+ int nrequested_workers, int vac_work_mem,
+ int max_offset, int elevel,
+ BufferAccessStrategy bstrategy)
{
ParallelVacuumState *pvs;
ParallelContext *pcxt;
PVShared *shared;
- VacDeadItems *dead_items;
+ TidStore *dead_items;
PVIndStats *indstats;
BufferUsage *buffer_usage;
WalUsage *wal_usage;
+ void *area_space;
+ dsa_area *dead_items_dsa;
bool *will_parallel_vacuum;
Size est_indstats_len;
Size est_shared_len;
- Size est_dead_items_len;
+ Size dsa_minsize = dsa_minimum_size();
int nindexes_mwm = 0;
int parallel_workers = 0;
int querylen;
@@ -293,9 +299,8 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_estimate_chunk(&pcxt->estimator, est_shared_len);
shm_toc_estimate_keys(&pcxt->estimator, 1);
- /* Estimate size for dead_items -- PARALLEL_VACUUM_KEY_DEAD_ITEMS */
- est_dead_items_len = vac_max_items_to_alloc_size(max_items);
- shm_toc_estimate_chunk(&pcxt->estimator, est_dead_items_len);
+ /* Estimate size for dead tuple DSA -- PARALLEL_VACUUM_KEY_DSA */
+ shm_toc_estimate_chunk(&pcxt->estimator, dsa_minsize);
shm_toc_estimate_keys(&pcxt->estimator, 1);
/*
@@ -361,6 +366,16 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_INDEX_STATS, indstats);
pvs->indstats = indstats;
+ /* Prepare DSA space for dead items */
+ area_space = shm_toc_allocate(pcxt->toc, dsa_minsize);
+ shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, area_space);
+ dead_items_dsa = dsa_create_in_place(area_space, dsa_minsize,
+ LWTRANCHE_PARALLEL_VACUUM_DSA,
+ pcxt->seg);
+ dead_items = tidstore_create(vac_work_mem, max_offset, dead_items_dsa);
+ pvs->dead_items = dead_items;
+ pvs->dead_items_area = dead_items_dsa;
+
/* Prepare shared information */
shared = (PVShared *) shm_toc_allocate(pcxt->toc, est_shared_len);
MemSet(shared, 0, est_shared_len);
@@ -370,6 +385,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
+ shared->dead_items_handle = tidstore_get_handle(dead_items);
/* Use the same buffer size for all workers */
shared->ring_nbuffers = GetAccessStrategyBufferCount(bstrategy);
@@ -381,15 +397,6 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_SHARED, shared);
pvs->shared = shared;
- /* Prepare the dead_items space */
- dead_items = (VacDeadItems *) shm_toc_allocate(pcxt->toc,
- est_dead_items_len);
- dead_items->max_items = max_items;
- dead_items->num_items = 0;
- MemSet(dead_items->items, 0, sizeof(ItemPointerData) * max_items);
- shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, dead_items);
- pvs->dead_items = dead_items;
-
/*
* Allocate space for each worker's BufferUsage and WalUsage; no need to
* initialize
@@ -447,6 +454,9 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
istats[i] = NULL;
}
+ tidstore_destroy(pvs->dead_items);
+ dsa_detach(pvs->dead_items_area);
+
DestroyParallelContext(pvs->pcxt);
ExitParallelMode();
@@ -455,7 +465,7 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
}
/* Returns the dead items space */
-VacDeadItems *
+TidStore *
parallel_vacuum_get_dead_items(ParallelVacuumState *pvs)
{
return pvs->dead_items;
@@ -954,7 +964,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
Relation *indrels;
PVIndStats *indstats;
PVShared *shared;
- VacDeadItems *dead_items;
+ TidStore *dead_items;
+ void *area_space;
+ dsa_area *dead_items_area;
BufferUsage *buffer_usage;
WalUsage *wal_usage;
int nindexes;
@@ -998,10 +1010,10 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
PARALLEL_VACUUM_KEY_INDEX_STATS,
false);
- /* Set dead_items space */
- dead_items = (VacDeadItems *) shm_toc_lookup(toc,
- PARALLEL_VACUUM_KEY_DEAD_ITEMS,
- false);
+ /* Set dead items */
+ area_space = shm_toc_lookup(toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, false);
+ dead_items_area = dsa_attach_in_place(area_space, seg);
+ dead_items = tidstore_attach(dead_items_area, shared->dead_items_handle);
/* Set cost-based vacuum delay */
VacuumUpdateCosts();
@@ -1049,6 +1061,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
InstrEndParallelQuery(&buffer_usage[ParallelWorkerNumber],
&wal_usage[ParallelWorkerNumber]);
+ tidstore_detach(pvs.dead_items);
+ dsa_detach(dead_items_area);
+
/* Pop the error context stack */
error_context_stack = errcallback.previous;
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 53c8f8d79c..74915bee9b 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -3474,12 +3474,12 @@ check_autovacuum_work_mem(int *newval, void **extra, GucSource source)
return true;
/*
- * We clamp manually-set values to at least 1MB. Since
+ * We clamp manually-set values to at least 2MB. Since
* maintenance_work_mem is always set to at least this value, do the same
* here.
*/
- if (*newval < 1024)
- *newval = 1024;
+ if (*newval < 2048)
+ *newval = 2048;
return true;
}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 55b3a04097..c223a7dc94 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -192,6 +192,8 @@ static const char *const BuiltinTrancheNames[] = {
"LogicalRepLauncherDSA",
/* LWTRANCHE_LAUNCHER_HASH: */
"LogicalRepLauncherHash",
+ /* LWTRANCHE_PARALLEL_VACUUM_DSA: */
+ "ParallelVacuumDSA",
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index cab3ddbe11..0bbdf04980 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2353,7 +2353,7 @@ struct config_int ConfigureNamesInt[] =
GUC_UNIT_KB
},
&maintenance_work_mem,
- 65536, 1024, MAX_KILOBYTES,
+ 65536, 2048, MAX_KILOBYTES,
NULL, NULL, NULL
},
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index e5add41352..b209d3cf84 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -23,8 +23,8 @@
#define PROGRESS_VACUUM_HEAP_BLKS_SCANNED 2
#define PROGRESS_VACUUM_HEAP_BLKS_VACUUMED 3
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
-#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
-#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_MAX_DEAD_TUPLE_BYTES 5
+#define PROGRESS_VACUUM_DEAD_TUPLE_BYTES 6
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 17e9b4f68e..b48c6ebf2d 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -17,6 +17,7 @@
#include "access/htup.h"
#include "access/genam.h"
#include "access/parallel.h"
+#include "access/tidstore.h"
#include "catalog/pg_class.h"
#include "catalog/pg_statistic.h"
#include "catalog/pg_type.h"
@@ -280,21 +281,6 @@ struct VacuumCutoffs
MultiXactId MultiXactCutoff;
};
-/*
- * VacDeadItems stores TIDs whose index tuples are deleted by index vacuuming.
- */
-typedef struct VacDeadItems
-{
- int max_items; /* # slots allocated in array */
- int num_items; /* current # of entries */
-
- /* Sorted array of TIDs to delete from indexes */
- ItemPointerData items[FLEXIBLE_ARRAY_MEMBER];
-} VacDeadItems;
-
-#define MAXDEADITEMS(avail_mem) \
- (((avail_mem) - offsetof(VacDeadItems, items)) / sizeof(ItemPointerData))
-
/* GUC parameters */
extern PGDLLIMPORT int default_statistics_target; /* PGDLLIMPORT for PostGIS */
extern PGDLLIMPORT int vacuum_freeze_min_age;
@@ -347,10 +333,9 @@ extern Relation vacuum_open_relation(Oid relid, RangeVar *relation,
LOCKMODE lmode);
extern IndexBulkDeleteResult *vac_bulkdel_one_index(IndexVacuumInfo *ivinfo,
IndexBulkDeleteResult *istat,
- VacDeadItems *dead_items);
+ TidStore *dead_items);
extern IndexBulkDeleteResult *vac_cleanup_one_index(IndexVacuumInfo *ivinfo,
IndexBulkDeleteResult *istat);
-extern Size vac_max_items_to_alloc_size(int max_items);
/* In postmaster/autovacuum.c */
extern void AutoVacuumUpdateCostLimit(void);
@@ -359,10 +344,10 @@ extern void VacuumUpdateCosts(void);
/* in commands/vacuumparallel.c */
extern ParallelVacuumState *parallel_vacuum_init(Relation rel, Relation *indrels,
int nindexes, int nrequested_workers,
- int max_items, int elevel,
- BufferAccessStrategy bstrategy);
+ int vac_work_mem, int max_offset,
+ int elevel, BufferAccessStrategy bstrategy);
extern void parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats);
-extern VacDeadItems *parallel_vacuum_get_dead_items(ParallelVacuumState *pvs);
+extern TidStore *parallel_vacuum_get_dead_items(ParallelVacuumState *pvs);
extern void parallel_vacuum_bulkdel_all_indexes(ParallelVacuumState *pvs,
long num_table_tuples,
int num_index_scans);
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index 07002fdfbe..537b34b30c 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -207,6 +207,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_PGSTATS_DATA,
LWTRANCHE_LAUNCHER_DSA,
LWTRANCHE_LAUNCHER_HASH,
+ LWTRANCHE_PARALLEL_VACUUM_DSA,
LWTRANCHE_FIRST_USER_DEFINED
} BuiltinTrancheIds;
diff --git a/src/test/regress/expected/cluster.out b/src/test/regress/expected/cluster.out
index 2eec483eaa..e04f50726f 100644
--- a/src/test/regress/expected/cluster.out
+++ b/src/test/regress/expected/cluster.out
@@ -526,7 +526,7 @@ create index cluster_sort on clstr_4 (hundred, thousand, tenthous);
-- ensure we don't use the index in CLUSTER nor the checking SELECTs
set enable_indexscan = off;
-- Use external sort:
-set maintenance_work_mem = '1MB';
+set maintenance_work_mem = '2MB';
cluster clstr_4 using cluster_sort;
select * from
(select hundred, lag(hundred) over () as lhundred,
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index acfd9d1f4f..d320ad87dd 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -1214,7 +1214,7 @@ DROP TABLE unlogged_hash_table;
-- CREATE INDEX hash_ovfl_index ON hash_ovfl_heap USING hash (x int4_ops);
-- Test hash index build tuplesorting. Force hash tuplesort using low
-- maintenance_work_mem setting and fillfactor:
-SET maintenance_work_mem = '1MB';
+SET maintenance_work_mem = '2MB';
CREATE INDEX hash_tuplesort_idx ON tenk1 USING hash (stringu1 name_ops) WITH (fillfactor = 10);
EXPLAIN (COSTS OFF)
SELECT count(*) FROM tenk1 WHERE stringu1 = 'TVAAAA';
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 919d947ec0..66d671a641 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2041,8 +2041,8 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param3 AS heap_blks_scanned,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
- s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param6 AS max_dead_tuple_bytes,
+ s.param7 AS dead_tuple_bytes
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_recovery_prefetch| SELECT stats_reset,
diff --git a/src/test/regress/sql/cluster.sql b/src/test/regress/sql/cluster.sql
index a4cfaae807..a4cb5b98a5 100644
--- a/src/test/regress/sql/cluster.sql
+++ b/src/test/regress/sql/cluster.sql
@@ -258,7 +258,7 @@ create index cluster_sort on clstr_4 (hundred, thousand, tenthous);
set enable_indexscan = off;
-- Use external sort:
-set maintenance_work_mem = '1MB';
+set maintenance_work_mem = '2MB';
cluster clstr_4 using cluster_sort;
select * from
(select hundred, lag(hundred) over () as lhundred,
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index d49ce9f300..d6e2471b00 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -367,7 +367,7 @@ DROP TABLE unlogged_hash_table;
-- Test hash index build tuplesorting. Force hash tuplesort using low
-- maintenance_work_mem setting and fillfactor:
-SET maintenance_work_mem = '1MB';
+SET maintenance_work_mem = '2MB';
CREATE INDEX hash_tuplesort_idx ON tenk1 USING hash (stringu1 name_ops) WITH (fillfactor = 10);
EXPLAIN (COSTS OFF)
SELECT count(*) FROM tenk1 WHERE stringu1 = 'TVAAAA';
--
2.31.1
[application/octet-stream] v32-0005-Tool-for-measuring-radix-tree-and-tidstore-perfo.patch (24.7K, ../../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/15-v32-0005-Tool-for-measuring-radix-tree-and-tidstore-perfo.patch)
download | inline diff:
From cff1ffa9af592765cf9073291fb1665b09b61d8a Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 16 Sep 2022 11:57:03 +0900
Subject: [PATCH v32 05/18] Tool for measuring radix tree and tidstore
performance
Includes Meson support, but commented out to avoid warnings
XXX: Not for commit
---
contrib/bench_radix_tree/Makefile | 21 +
.../bench_radix_tree--1.0.sql | 88 +++
contrib/bench_radix_tree/bench_radix_tree.c | 747 ++++++++++++++++++
.../bench_radix_tree/bench_radix_tree.control | 6 +
contrib/bench_radix_tree/expected/bench.out | 13 +
contrib/bench_radix_tree/meson.build | 33 +
contrib/bench_radix_tree/sql/bench.sql | 16 +
contrib/meson.build | 1 +
8 files changed, 925 insertions(+)
create mode 100644 contrib/bench_radix_tree/Makefile
create mode 100644 contrib/bench_radix_tree/bench_radix_tree--1.0.sql
create mode 100644 contrib/bench_radix_tree/bench_radix_tree.c
create mode 100644 contrib/bench_radix_tree/bench_radix_tree.control
create mode 100644 contrib/bench_radix_tree/expected/bench.out
create mode 100644 contrib/bench_radix_tree/meson.build
create mode 100644 contrib/bench_radix_tree/sql/bench.sql
diff --git a/contrib/bench_radix_tree/Makefile b/contrib/bench_radix_tree/Makefile
new file mode 100644
index 0000000000..952bb0ceae
--- /dev/null
+++ b/contrib/bench_radix_tree/Makefile
@@ -0,0 +1,21 @@
+# contrib/bench_radix_tree/Makefile
+
+MODULE_big = bench_radix_tree
+OBJS = \
+ bench_radix_tree.o
+
+EXTENSION = bench_radix_tree
+DATA = bench_radix_tree--1.0.sql
+
+REGRESS = bench_fixed_height
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/bench_radix_tree
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
new file mode 100644
index 0000000000..ad66265e23
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
@@ -0,0 +1,88 @@
+/* contrib/bench_radix_tree/bench_radix_tree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION bench_radix_tree" to load this file. \quit
+
+create function bench_shuffle_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_seq_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_load_random_int(
+cnt int8,
+OUT mem_allocated int8,
+OUT load_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_search_random_nodes(
+cnt int8,
+filter_str text DEFAULT NULL,
+OUT mem_allocated int8,
+OUT load_ms int8,
+OUT search_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C VOLATILE PARALLEL UNSAFE;
+
+create function bench_fixed_height_search(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT rt_search_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_node128_load(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_sparseload_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_tidstore_load(
+minblk int4,
+maxblk int4,
+OUT mem_allocated int8,
+OUT load_ms int8,
+OUT iter_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
new file mode 100644
index 0000000000..6e5149e2c4
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -0,0 +1,747 @@
+/*-------------------------------------------------------------------------
+ *
+ * bench_radix_tree.c
+ *
+ * Copyright (c) 2016-2022, PostgreSQL Global Development Group
+ *
+ * contrib/bench_radix_tree/bench_radix_tree.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/tidstore.h"
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "funcapi.h"
+#include "lib/radixtree.h"
+#include <math.h>
+#include "miscadmin.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/timestamp.h"
+
+PG_MODULE_MAGIC;
+
+/* run benchmark also for binary-search case? */
+/* #define MEASURE_BINARY_SEARCH 1 */
+
+#define TIDS_PER_BLOCK_FOR_LOAD 30
+#define TIDS_PER_BLOCK_FOR_LOOKUP 50
+
+//#define RT_DEBUG
+#define RT_PREFIX rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE uint64
+// WIP: compiles with warnings because rt_attach is defined but not used
+// #define RT_SHMEM
+#include "lib/radixtree.h"
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+ return tree->ctl->num_keys;
+}
+
+
+PG_FUNCTION_INFO_V1(bench_seq_search);
+PG_FUNCTION_INFO_V1(bench_shuffle_search);
+PG_FUNCTION_INFO_V1(bench_load_random_int);
+PG_FUNCTION_INFO_V1(bench_fixed_height_search);
+PG_FUNCTION_INFO_V1(bench_search_random_nodes);
+PG_FUNCTION_INFO_V1(bench_node128_load);
+PG_FUNCTION_INFO_V1(bench_tidstore_load);
+
+static uint64
+tid_to_key_off(ItemPointer tid, uint32 *off)
+{
+ uint64 upper;
+ uint32 shift = pg_ceil_log2_32(MaxHeapTuplesPerPage);
+ int64 tid_i;
+
+ Assert(ItemPointerGetOffsetNumber(tid) < MaxHeapTuplesPerPage);
+
+ tid_i = ItemPointerGetOffsetNumber(tid);
+ tid_i |= (uint64) ItemPointerGetBlockNumber(tid) << shift;
+
+ /* log(sizeof(uint64) * BITS_PER_BYTE, 2) = log(64, 2) = 6 */
+ *off = tid_i & ((1 << 6) - 1);
+ upper = tid_i >> 6;
+ Assert(*off < (sizeof(uint64) * BITS_PER_BYTE));
+
+ Assert(*off < 64);
+
+ return upper;
+}
+
+static int
+shuffle_randrange(pg_prng_state *state, int lower, int upper)
+{
+ return (int) floor(pg_prng_double(state) * ((upper - lower) + 0.999999)) + lower;
+}
+
+/* Naive Fisher-Yates implementation*/
+static void
+shuffle_itemptrs(ItemPointer itemptr, uint64 nitems)
+{
+ /* reproducability */
+ pg_prng_state state;
+
+ pg_prng_seed(&state, 0);
+
+ for (int i = 0; i < nitems - 1; i++)
+ {
+ int j = shuffle_randrange(&state, i, nitems - 1);
+ ItemPointerData t = itemptr[j];
+
+ itemptr[j] = itemptr[i];
+ itemptr[i] = t;
+ }
+}
+
+static ItemPointer
+generate_tids(BlockNumber minblk, BlockNumber maxblk, int ntids_per_blk, uint64 *ntids_p,
+ bool random_block)
+{
+ ItemPointer tids;
+ uint64 maxitems;
+ uint64 ntids = 0;
+ pg_prng_state state;
+
+ maxitems = (maxblk - minblk + 1) * ntids_per_blk;
+ tids = MemoryContextAllocHuge(TopTransactionContext,
+ sizeof(ItemPointerData) * maxitems);
+
+ if (random_block)
+ pg_prng_seed(&state, 0x9E3779B185EBCA87);
+
+ for (BlockNumber blk = minblk; blk < maxblk; blk++)
+ {
+ if (random_block && !pg_prng_bool(&state))
+ continue;
+
+ for (OffsetNumber off = FirstOffsetNumber;
+ off <= ntids_per_blk; off++)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ ItemPointerSetBlockNumber(&(tids[ntids]), blk);
+ ItemPointerSetOffsetNumber(&(tids[ntids]), off);
+
+ ntids++;
+ }
+ }
+
+ *ntids_p = ntids;
+ return tids;
+}
+
+#ifdef MEASURE_BINARY_SEARCH
+static int
+vac_cmp_itemptr(const void *left, const void *right)
+{
+ BlockNumber lblk,
+ rblk;
+ OffsetNumber loff,
+ roff;
+
+ lblk = ItemPointerGetBlockNumber((ItemPointer) left);
+ rblk = ItemPointerGetBlockNumber((ItemPointer) right);
+
+ if (lblk < rblk)
+ return -1;
+ if (lblk > rblk)
+ return 1;
+
+ loff = ItemPointerGetOffsetNumber((ItemPointer) left);
+ roff = ItemPointerGetOffsetNumber((ItemPointer) right);
+
+ if (loff < roff)
+ return -1;
+ if (loff > roff)
+ return 1;
+
+ return 0;
+}
+#endif
+
+Datum
+bench_tidstore_load(PG_FUNCTION_ARGS)
+{
+ BlockNumber minblk = PG_GETARG_INT32(0);
+ BlockNumber maxblk = PG_GETARG_INT32(1);
+ TidStore *ts;
+ TidStoreIter *iter;
+ TidStoreIterResult *result;
+ OffsetNumber *offs;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_ms;
+ int64 iter_ms;
+ TupleDesc tupdesc;
+ Datum values[3];
+ bool nulls[3] = {false};
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ offs = palloc(sizeof(OffsetNumber) * TIDS_PER_BLOCK_FOR_LOAD);
+ for (int i = 0; i < TIDS_PER_BLOCK_FOR_LOAD; i++)
+ offs[i] = i + 1; /* FirstOffsetNumber is 1 */
+
+ ts = tidstore_create(1 * 1024L * 1024L * 1024L, MaxHeapTuplesPerPage, NULL);
+
+ /* load tids */
+ start_time = GetCurrentTimestamp();
+ for (BlockNumber blkno = minblk; blkno < maxblk; blkno++)
+ tidstore_add_tids(ts, blkno, offs, TIDS_PER_BLOCK_FOR_LOAD);
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_ms = secs * 1000 + usecs / 1000;
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* iterate through tids */
+ iter = tidstore_begin_iterate(ts);
+ start_time = GetCurrentTimestamp();
+ while ((result = tidstore_iterate_next(iter)) != NULL)
+ ;
+ tidstore_end_iterate(iter);
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ iter_ms = secs * 1000 + usecs / 1000;
+
+ values[0] = Int64GetDatum(tidstore_memory_usage(ts));
+ values[1] = Int64GetDatum(load_ms);
+ values[2] = Int64GetDatum(iter_ms);
+
+ tidstore_destroy(ts);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+static Datum
+bench_search(FunctionCallInfo fcinfo, bool shuffle)
+{
+ BlockNumber minblk = PG_GETARG_INT32(0);
+ BlockNumber maxblk = PG_GETARG_INT32(1);
+ bool random_block = PG_GETARG_BOOL(2);
+ rt_radix_tree *rt = NULL;
+ uint64 ntids;
+ uint64 key;
+ uint64 last_key = PG_UINT64_MAX;
+ uint64 val = 0;
+ ItemPointer tids;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_load_ms,
+ rt_search_ms;
+ Datum values[7];
+ bool nulls[7];
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ tids = generate_tids(minblk, maxblk, TIDS_PER_BLOCK_FOR_LOAD, &ntids, random_block);
+
+ /* measure the load time of the radix tree */
+ rt = rt_create(CurrentMemoryContext);
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ uint32 off;
+
+ CHECK_FOR_INTERRUPTS();
+
+ key = tid_to_key_off(tid, &off);
+
+ if (last_key != PG_UINT64_MAX && last_key != key)
+ {
+ rt_set(rt, last_key, &val);
+ val = 0;
+ }
+
+ last_key = key;
+ val |= (uint64) 1 << off;
+ }
+ if (last_key != PG_UINT64_MAX)
+ rt_set(rt, last_key, &val);
+
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_load_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+ if (shuffle)
+ shuffle_itemptrs(tids, ntids);
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* meaure the serach time of the radix tree */
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ uint32 off;
+ volatile bool ret; /* prevent calling rt_search from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ key = tid_to_key_off(tid, &off);
+
+ ret = rt_search(rt, key, &val);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_search_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int64GetDatum(rt_num_entries(rt));
+ values[1] = Int64GetDatum(rt_memory_usage(rt));
+ values[2] = Int64GetDatum(sizeof(ItemPointerData) * ntids);
+ values[3] = Int64GetDatum(rt_load_ms);
+ nulls[4] = true; /* ar_load_ms */
+ values[5] = Int64GetDatum(rt_search_ms);
+ nulls[6] = true; /* ar_search_ms */
+
+#ifdef MEASURE_BINARY_SEARCH
+ {
+ ItemPointer itemptrs = NULL;
+
+ int64 ar_load_ms,
+ ar_search_ms;
+
+ /* measure the load time of the array */
+ itemptrs = MemoryContextAllocHuge(CurrentMemoryContext,
+ sizeof(ItemPointerData) * ntids);
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointerSetBlockNumber(&(itemptrs[i]),
+ ItemPointerGetBlockNumber(&(tids[i])));
+ ItemPointerSetOffsetNumber(&(itemptrs[i]),
+ ItemPointerGetOffsetNumber(&(tids[i])));
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ ar_load_ms = secs * 1000 + usecs / 1000;
+
+ /* next, measure the serach time of the array */
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ volatile bool ret; /* prevent calling bsearch from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ ret = bsearch((void *) tid,
+ (void *) itemptrs,
+ ntids,
+ sizeof(ItemPointerData),
+ vac_cmp_itemptr);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ ar_search_ms = secs * 1000 + usecs / 1000;
+
+ /* set the result */
+ nulls[4] = false;
+ values[4] = Int64GetDatum(ar_load_ms);
+ nulls[6] = false;
+ values[6] = Int64GetDatum(ar_search_ms);
+ }
+#endif
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_seq_search(PG_FUNCTION_ARGS)
+{
+ return bench_search(fcinfo, false);
+}
+
+Datum
+bench_shuffle_search(PG_FUNCTION_ARGS)
+{
+ return bench_search(fcinfo, true);
+}
+
+Datum
+bench_load_random_int(PG_FUNCTION_ARGS)
+{
+ uint64 cnt = (uint64) PG_GETARG_INT64(0);
+ rt_radix_tree *rt;
+ pg_prng_state state;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_time_ms;
+ Datum values[2];
+ bool nulls[2];
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ pg_prng_seed(&state, 0);
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 key = pg_prng_uint64(&state);
+
+ rt_set(rt, key, &key);
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_time_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int64GetDatum(rt_memory_usage(rt));
+ values[1] = Int64GetDatum(load_time_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+/* copy of splitmix64() */
+static uint64
+hash64(uint64 x)
+{
+ x ^= x >> 30;
+ x *= UINT64CONST(0xbf58476d1ce4e5b9);
+ x ^= x >> 27;
+ x *= UINT64CONST(0x94d049bb133111eb);
+ x ^= x >> 31;
+ return x;
+}
+
+/* attempts to have a relatively even population of node kinds */
+Datum
+bench_search_random_nodes(PG_FUNCTION_ARGS)
+{
+ uint64 cnt = (uint64) PG_GETARG_INT64(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_time_ms;
+ int64 search_time_ms;
+ Datum values[3] = {0};
+ bool nulls[3] = {0};
+ /* from trial and error */
+ uint64 filter = (((uint64) 0x7F<<32) | (0x07<<24) | (0xFF<<16) | 0xFF);
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (!PG_ARGISNULL(1))
+ {
+ char *filter_str = text_to_cstring(PG_GETARG_TEXT_P(1));
+
+ if (sscanf(filter_str, "0x%lX", &filter) == 0)
+ elog(ERROR, "invalid filter string %s", filter_str);
+ }
+ elog(NOTICE, "bench with filter 0x%lX", filter);
+
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 hash = hash64(i);
+ uint64 key = hash & filter;
+
+ rt_set(rt, key, &key);
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_time_ms = secs * 1000 + usecs / 1000;
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 hash = hash64(i);
+ uint64 key = hash & filter;
+ uint64 val;
+ volatile bool ret; /* prevent calling rt_search from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ ret = rt_search(rt, key, &val);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ search_time_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ values[0] = Int64GetDatum(rt_memory_usage(rt));
+ values[1] = Int64GetDatum(load_time_ms);
+ values[2] = Int64GetDatum(search_time_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_fixed_height_search(PG_FUNCTION_ARGS)
+{
+ int fanout = PG_GETARG_INT32(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_load_ms,
+ rt_search_ms;
+ Datum values[5];
+ bool nulls[5];
+
+ /* test boundary between vector and iteration */
+ const int n_keys = 5 * 16 * 16 * 16 * 16;
+ uint64 r,
+ h,
+ i,
+ j,
+ k;
+ uint64 key_id;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+
+ key_id = 0;
+
+ /*
+ * lower nodes have limited fanout, the top is only limited by
+ * bits-per-byte
+ */
+ for (r = 1;; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ for (i = 1; i <= fanout; i++)
+ {
+ for (j = 1; j <= fanout; j++)
+ {
+ for (k = 1; k <= fanout; k++)
+ {
+ uint64 key;
+
+ key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+ CHECK_FOR_INTERRUPTS();
+
+ key_id++;
+ if (key_id > n_keys)
+ goto finish_set;
+
+ rt_set(rt, key, &key_id);
+ }
+ }
+ }
+ }
+ }
+finish_set:
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_load_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ /* meaure the search time of the radix tree */
+ start_time = GetCurrentTimestamp();
+
+ key_id = 0;
+ for (r = 1;; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ for (i = 1; i <= fanout; i++)
+ {
+ for (j = 1; j <= fanout; j++)
+ {
+ for (k = 1; k <= fanout; k++)
+ {
+ uint64 key,
+ val;
+
+ key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+ CHECK_FOR_INTERRUPTS();
+
+ key_id++;
+ if (key_id > n_keys)
+ goto finish_search;
+
+ rt_search(rt, key, &val);
+ }
+ }
+ }
+ }
+ }
+finish_search:
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_search_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int32GetDatum(fanout);
+ values[1] = Int64GetDatum(rt_num_entries(rt));
+ values[2] = Int64GetDatum(rt_memory_usage(rt));
+ values[3] = Int64GetDatum(rt_load_ms);
+ values[4] = Int64GetDatum(rt_search_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_node128_load(PG_FUNCTION_ARGS)
+{
+ int fanout = PG_GETARG_INT32(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_sparseload_ms;
+ Datum values[5];
+ bool nulls[5];
+
+ uint64 r,
+ h;
+ uint64 key_id;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ rt = rt_create(CurrentMemoryContext);
+
+ key_id = 0;
+
+ for (r = 1; r <= fanout; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (h);
+
+ key_id++;
+ rt_set(rt, key, &key_id);
+ }
+ }
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ /* measure sparse deletion and re-loading */
+ start_time = GetCurrentTimestamp();
+
+ for (int t = 0; t<10000; t++)
+ {
+ /* delete one key in each leaf */
+ for (r = 1; r <= fanout; r++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (fanout);
+
+ rt_delete(rt, key);
+ }
+
+ /* add them all back */
+ key_id = 0;
+ for (r = 1; r <= fanout; r++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (fanout);
+
+ key_id++;
+ rt_set(rt, key, &key_id);
+ }
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_sparseload_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int32GetDatum(fanout);
+ values[1] = Int64GetDatum(rt_num_entries(rt));
+ values[2] = Int64GetDatum(rt_memory_usage(rt));
+ values[3] = Int64GetDatum(rt_sparseload_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+/* to silence warnings about unused iter functions */
+static void pg_attribute_unused()
+stub_iter()
+{
+ rt_radix_tree *rt;
+ rt_iter *iter;
+ uint64 key = 1;
+ uint64 value = 1;
+
+ rt = rt_create(CurrentMemoryContext);
+
+ iter = rt_begin_iterate(rt);
+ rt_iterate_next(iter, &key, &value);
+ rt_end_iterate(iter);
+}
\ No newline at end of file
diff --git a/contrib/bench_radix_tree/bench_radix_tree.control b/contrib/bench_radix_tree/bench_radix_tree.control
new file mode 100644
index 0000000000..1d988e6c9a
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.control
@@ -0,0 +1,6 @@
+# bench_radix_tree extension
+comment = 'benchmark suits for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/bench_radix_tree'
+relocatable = true
+trusted = true
diff --git a/contrib/bench_radix_tree/expected/bench.out b/contrib/bench_radix_tree/expected/bench.out
new file mode 100644
index 0000000000..60c303892e
--- /dev/null
+++ b/contrib/bench_radix_tree/expected/bench.out
@@ -0,0 +1,13 @@
+create extension bench_radix_tree;
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/bench_radix_tree/meson.build b/contrib/bench_radix_tree/meson.build
new file mode 100644
index 0000000000..332c1ae7df
--- /dev/null
+++ b/contrib/bench_radix_tree/meson.build
@@ -0,0 +1,33 @@
+bench_radix_tree_sources = files(
+ 'bench_radix_tree.c',
+)
+
+if host_system == 'windows'
+ bench_radix_tree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'bench_radix_tree',
+ '--FILEDESC', 'bench_radix_tree - performance test code for radix tree',])
+endif
+
+bench_radix_tree = shared_module('bench_radix_tree',
+ bench_radix_tree_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += bench_radix_tree
+
+install_data(
+ 'bench_radix_tree.control',
+ 'bench_radix_tree--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'bench_radix_tree',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'bench_radix_tree',
+ ],
+ },
+}
diff --git a/contrib/bench_radix_tree/sql/bench.sql b/contrib/bench_radix_tree/sql/bench.sql
new file mode 100644
index 0000000000..a46018c9d4
--- /dev/null
+++ b/contrib/bench_radix_tree/sql/bench.sql
@@ -0,0 +1,16 @@
+create extension bench_radix_tree;
+
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/meson.build b/contrib/meson.build
index bd4a57c43c..421d469f8c 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,6 +12,7 @@ subdir('amcheck')
subdir('auth_delay')
subdir('auto_explain')
subdir('basic_archive')
+subdir('bench_radix_tree')
subdir('bloom')
subdir('basebackup_to_shell')
subdir('bool_plperl')
--
2.31.1
[application/octet-stream] v32-0004-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch (36.0K, ../../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/16-v32-0004-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch)
download | inline diff:
From a804e3ebba8733d65497d5e9c3a47b32f175ea1e Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 23 Dec 2022 23:50:39 +0900
Subject: [PATCH v32 04/18] Add TIDStore, to store sets of TIDs
(ItemPointerData) efficiently.
The TIDStore is designed to store large sets of TIDs efficiently, and
is backed by the radix tree. A TID is encoded into 64-bit key and
value and inserted to the radix tree.
The TIDStore is not used for anything yet, aside from the test code,
but the follow up patch integrates the TIDStore with lazy vacuum,
reducing lazy vacuum memory usage and lifting the 1GB limit on its
size, by storing the list of dead TIDs more efficiently.
This includes a unit test module, in src/test/modules/test_tidstore.
---
doc/src/sgml/monitoring.sgml | 4 +
src/backend/access/common/Makefile | 1 +
src/backend/access/common/meson.build | 1 +
src/backend/access/common/tidstore.c | 681 ++++++++++++++++++
src/backend/storage/lmgr/lwlock.c | 2 +
src/include/access/tidstore.h | 49 ++
src/include/storage/lwlock.h | 1 +
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_tidstore/Makefile | 23 +
.../test_tidstore/expected/test_tidstore.out | 13 +
src/test/modules/test_tidstore/meson.build | 35 +
.../test_tidstore/sql/test_tidstore.sql | 7 +
.../test_tidstore/test_tidstore--1.0.sql | 8 +
.../modules/test_tidstore/test_tidstore.c | 226 ++++++
.../test_tidstore/test_tidstore.control | 4 +
16 files changed, 1057 insertions(+)
create mode 100644 src/backend/access/common/tidstore.c
create mode 100644 src/include/access/tidstore.h
create mode 100644 src/test/modules/test_tidstore/Makefile
create mode 100644 src/test/modules/test_tidstore/expected/test_tidstore.out
create mode 100644 src/test/modules/test_tidstore/meson.build
create mode 100644 src/test/modules/test_tidstore/sql/test_tidstore.sql
create mode 100644 src/test/modules/test_tidstore/test_tidstore--1.0.sql
create mode 100644 src/test/modules/test_tidstore/test_tidstore.c
create mode 100644 src/test/modules/test_tidstore/test_tidstore.control
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 2903b67170..be4448fe6e 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2211,6 +2211,10 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<entry>Waiting to access a shared TID bitmap during a parallel bitmap
index scan.</entry>
</row>
+ <row>
+ <entry><literal>SharedTidStore</literal></entry>
+ <entry>Waiting to access a shared TID store.</entry>
+ </row>
<row>
<entry><literal>SharedTupleStore</literal></entry>
<entry>Waiting to access a shared tuple store during parallel
diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile
index b9aff0ccfd..67b8cc6108 100644
--- a/src/backend/access/common/Makefile
+++ b/src/backend/access/common/Makefile
@@ -27,6 +27,7 @@ OBJS = \
syncscan.o \
toast_compression.o \
toast_internals.o \
+ tidstore.o \
tupconvert.o \
tupdesc.o
diff --git a/src/backend/access/common/meson.build b/src/backend/access/common/meson.build
index f5ac17b498..fce19c09ce 100644
--- a/src/backend/access/common/meson.build
+++ b/src/backend/access/common/meson.build
@@ -15,6 +15,7 @@ backend_sources += files(
'syncscan.c',
'toast_compression.c',
'toast_internals.c',
+ 'tidstore.c',
'tupconvert.c',
'tupdesc.c',
)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
new file mode 100644
index 0000000000..8c05e60d92
--- /dev/null
+++ b/src/backend/access/common/tidstore.c
@@ -0,0 +1,681 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.c
+ * Tid (ItemPointerData) storage implementation.
+ *
+ * This module provides a in-memory data structure to store Tids (ItemPointer).
+ * Internally, a tid is encoded as a pair of 64-bit key and 64-bit value, and
+ * stored in the radix tree.
+ *
+ * A TidStore can be shared among parallel worker processes by passing DSA area
+ * to tidstore_create(). Other backends can attach to the shared TidStore by
+ * tidstore_attach().
+ *
+ * Regarding the concurrency, it basically relies on the concurrency support in
+ * the radix tree, but we acquires the lock on a TidStore in some cases, for
+ * example, when to reset the store and when to access the number tids in the
+ * store (num_tids).
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/access/common/tidstore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/tidstore.h"
+#include "miscadmin.h"
+#include "port/pg_bitutils.h"
+#include "storage/lwlock.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/*
+ * For encoding purposes, tids are represented as a pair of 64-bit key and
+ * 64-bit value. First, we construct 64-bit unsigned integer by combining
+ * the block number and the offset number. The number of bits used for the
+ * offset number is specified by max_offsets in tidstore_create(). We are
+ * frugal with the bits, because smaller keys could help keeping the radix
+ * tree shallow.
+ *
+ * For example, a tid of heap with 8kB blocks uses the lowest 9 bits for
+ * the offset number and uses the next 32 bits for the block number. That
+ * is, only 41 bits are used:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ *
+ * X = bits used for offset number
+ * Y = bits used for block number
+ * u = unused bit
+ * (high on the left, low on the right)
+ *
+ * 9 bits are enough for the offset number, because MaxHeapTuplesPerPage < 2^9
+ * on 8kB blocks.
+ *
+ * The 64-bit value is the bitmap representation of the lowest 6 bits
+ * (TIDSTORE_VALUE_NBITS) of the integer, and the rest 35 bits are used
+ * as the key:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ * |----| value
+ * |---------------------------------------------| key
+ *
+ * The maximum height of the radix tree is 5 in this case.
+ */
+#define TIDSTORE_VALUE_NBITS 6 /* log(64, 2) */
+#define TIDSTORE_OFFSET_MASK ((1 << TIDSTORE_VALUE_NBITS) - 1)
+
+/* A magic value used to identify our TidStores. */
+#define TIDSTORE_MAGIC 0x826f6a10
+
+#define RT_PREFIX local_rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE uint64
+#include "lib/radixtree.h"
+
+#define RT_PREFIX shared_rt
+#define RT_SHMEM
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE uint64
+#include "lib/radixtree.h"
+
+/* The control object for a TidStore */
+typedef struct TidStoreControl
+{
+ /* the number of tids in the store */
+ int64 num_tids;
+
+ /* These values are never changed after creation */
+ size_t max_bytes; /* the maximum bytes a TidStore can use */
+ int max_offset; /* the maximum offset number */
+ int offset_nbits; /* the number of bits required for an offset
+ * number */
+ int offset_key_nbits; /* the number of bits of an offset number
+ * used in a key */
+
+ /* The below fields are used only in shared case */
+
+ uint32 magic;
+ LWLock lock;
+
+ /* handles for TidStore and radix tree */
+ tidstore_handle handle;
+ shared_rt_handle tree_handle;
+} TidStoreControl;
+
+/* Per-backend state for a TidStore */
+struct TidStore
+{
+ /*
+ * Control object. This is allocated in DSA area 'area' in the shared
+ * case, otherwise in backend-local memory.
+ */
+ TidStoreControl *control;
+
+ /* Storage for Tids. Use either one depending on TidStoreIsShared() */
+ union
+ {
+ local_rt_radix_tree *local;
+ shared_rt_radix_tree *shared;
+ } tree;
+
+ /* DSA area for TidStore if used */
+ dsa_area *area;
+};
+#define TidStoreIsShared(ts) ((ts)->area != NULL)
+
+/* Iterator for TidStore */
+typedef struct TidStoreIter
+{
+ TidStore *ts;
+
+ /* iterator of radix tree. Use either one depending on TidStoreIsShared() */
+ union
+ {
+ shared_rt_iter *shared;
+ local_rt_iter *local;
+ } tree_iter;
+
+ /* we returned all tids? */
+ bool finished;
+
+ /* save for the next iteration */
+ uint64 next_key;
+ uint64 next_val;
+
+ /* output for the caller */
+ TidStoreIterResult result;
+} TidStoreIter;
+
+static void tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val);
+static inline BlockNumber key_get_blkno(TidStore *ts, uint64 key);
+static inline uint64 encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit);
+static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit);
+
+/*
+ * Create a TidStore. The returned object is allocated in backend-local memory.
+ * The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
+ */
+TidStore *
+tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
+{
+ TidStore *ts;
+
+ ts = palloc0(sizeof(TidStore));
+
+ /*
+ * Create the radix tree for the main storage.
+ *
+ * Memory consumption depends on the number of stored tids, but also on the
+ * distribution of them, how the radix tree stores, and the memory management
+ * that backed the radix tree. The maximum bytes that a TidStore can
+ * use is specified by the max_bytes in tidstore_create(). We want the total
+ * amount of memory consumption by a TidStore not to exceed the max_bytes.
+ *
+ * In local TidStore cases, the radix tree uses slab allocators for each kind
+ * of node class. The most memory consuming case while adding Tids associated
+ * with one page (i.e. during tidstore_add_tids()) is that we allocate a new
+ * slab block for a new radix tree node, which is approximately 70kB. Therefore,
+ * we deduct 70kB from the max_bytes.
+ *
+ * In shared cases, DSA allocates the memory segments big enough to follow
+ * a geometric series that approximately doubles the total DSA size (see
+ * make_new_segment() in dsa.c). We simulated the how DSA increases segment
+ * size and the simulation revealed, the 75% threshold for the maximum bytes
+ * perfectly works in case where the max_bytes is a power-of-2, and the 60%
+ * threshold works for other cases.
+ */
+ if (area != NULL)
+ {
+ dsa_pointer dp;
+ float ratio = ((max_bytes & (max_bytes - 1)) == 0) ? 0.75 : 0.6;
+
+ ts->tree.shared = shared_rt_create(CurrentMemoryContext, area,
+ LWTRANCHE_SHARED_TIDSTORE);
+
+ dp = dsa_allocate0(area, sizeof(TidStoreControl));
+ ts->control = (TidStoreControl *) dsa_get_address(area, dp);
+ ts->control->max_bytes = (uint64) (max_bytes * ratio);
+ ts->area = area;
+
+ ts->control->magic = TIDSTORE_MAGIC;
+ LWLockInitialize(&ts->control->lock, LWTRANCHE_SHARED_TIDSTORE);
+ ts->control->handle = dp;
+ ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+ }
+ else
+ {
+ ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+ ts->control = (TidStoreControl *) palloc0(sizeof(TidStoreControl));
+ ts->control->max_bytes = max_bytes - (70 * 1024);
+ }
+
+ ts->control->max_offset = max_offset;
+ ts->control->offset_nbits = pg_ceil_log2_32(max_offset);
+
+ if (ts->control->offset_nbits < TIDSTORE_VALUE_NBITS)
+ ts->control->offset_nbits = TIDSTORE_VALUE_NBITS;
+
+ ts->control->offset_key_nbits =
+ ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
+
+ return ts;
+}
+
+/*
+ * Attach to the shared TidStore using a handle. The returned object is
+ * allocated in backend-local memory using the CurrentMemoryContext.
+ */
+TidStore *
+tidstore_attach(dsa_area *area, tidstore_handle handle)
+{
+ TidStore *ts;
+ dsa_pointer control;
+
+ Assert(area != NULL);
+ Assert(DsaPointerIsValid(handle));
+
+ /* create per-backend state */
+ ts = palloc0(sizeof(TidStore));
+
+ /* Find the control object in shared memory */
+ control = handle;
+
+ /* Set up the TidStore */
+ ts->control = (TidStoreControl *) dsa_get_address(area, control);
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ ts->tree.shared = shared_rt_attach(area, ts->control->tree_handle);
+ ts->area = area;
+
+ return ts;
+}
+
+/*
+ * Detach from a TidStore. This detaches from radix tree and frees the
+ * backend-local resources. The radix tree will continue to exist until
+ * it is either explicitly destroyed, or the area that backs it is returned
+ * to the operating system.
+ */
+void
+tidstore_detach(TidStore *ts)
+{
+ Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+ shared_rt_detach(ts->tree.shared);
+ pfree(ts);
+}
+
+/*
+ * Destroy a TidStore, returning all memory.
+ *
+ * TODO: The caller must be certain that no other backend will attempt to
+ * access the TidStore before calling this function. Other backend must
+ * explicitly call tidstore_detach to free up backend-local memory associated
+ * with the TidStore. The backend that calls tidstore_destroy must not call
+ * tidstore_detach.
+ */
+void
+tidstore_destroy(TidStore *ts)
+{
+ if (TidStoreIsShared(ts))
+ {
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ /*
+ * Vandalize the control block to help catch programming error where
+ * other backends access the memory formerly occupied by this radix
+ * tree.
+ */
+ ts->control->magic = 0;
+ dsa_free(ts->area, ts->control->handle);
+ shared_rt_free(ts->tree.shared);
+ }
+ else
+ {
+ pfree(ts->control);
+ local_rt_free(ts->tree.local);
+ }
+
+ pfree(ts);
+}
+
+/*
+ * Forget all collected Tids. It's similar to tidstore_destroy but we don't free
+ * entire TidStore but recreate only the radix tree storage.
+ */
+void
+tidstore_reset(TidStore *ts)
+{
+ if (TidStoreIsShared(ts))
+ {
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+ /*
+ * Free the radix tree and return allocated DSA segments to
+ * the operating system.
+ */
+ shared_rt_free(ts->tree.shared);
+ dsa_trim(ts->area);
+
+ /* Recreate the radix tree */
+ ts->tree.shared = shared_rt_create(CurrentMemoryContext, ts->area,
+ LWTRANCHE_SHARED_TIDSTORE);
+
+ /* update the radix tree handle as we recreated it */
+ ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
+
+ LWLockRelease(&ts->control->lock);
+ }
+ else
+ {
+ local_rt_free(ts->tree.local);
+ ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
+ }
+}
+
+/* Add Tids on a block to TidStore */
+void
+tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets)
+{
+ uint64 *values;
+ uint64 key;
+ uint64 prev_key;
+ uint64 off_bitmap = 0;
+ int idx;
+ const uint64 key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
+ const int nkeys = UINT64CONST(1) << ts->control->offset_key_nbits;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ values = palloc(sizeof(uint64) * nkeys);
+ key = prev_key = key_base;
+
+ for (int i = 0; i < num_offsets; i++)
+ {
+ uint64 off_bit;
+
+ /* encode the tid to a key and partial offset */
+ key = encode_key_off(ts, blkno, offsets[i], &off_bit);
+
+ /* make sure we scanned the line pointer array in order */
+ Assert(key >= prev_key);
+
+ if (key > prev_key)
+ {
+ idx = prev_key - key_base;
+ Assert(idx >= 0 && idx < nkeys);
+
+ /* write out offset bitmap for this key */
+ values[idx] = off_bitmap;
+
+ /* zero out any gaps up to the current key */
+ for (int empty_idx = idx + 1; empty_idx < key - key_base; empty_idx++)
+ values[empty_idx] = 0;
+
+ /* reset for current key -- the current offset will be handled below */
+ off_bitmap = 0;
+ prev_key = key;
+ }
+
+ off_bitmap |= off_bit;
+ }
+
+ /* save the final index for later */
+ idx = key - key_base;
+ /* write out last offset bitmap */
+ values[idx] = off_bitmap;
+
+ if (TidStoreIsShared(ts))
+ LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+ /* insert the calculated key-values to the tree */
+ for (int i = 0; i <= idx; i++)
+ {
+ if (values[i])
+ {
+ key = key_base + i;
+
+ if (TidStoreIsShared(ts))
+ shared_rt_set(ts->tree.shared, key, &values[i]);
+ else
+ local_rt_set(ts->tree.local, key, &values[i]);
+ }
+ }
+
+ /* update statistics */
+ ts->control->num_tids += num_offsets;
+
+ if (TidStoreIsShared(ts))
+ LWLockRelease(&ts->control->lock);
+
+ pfree(values);
+}
+
+/* Return true if the given tid is present in the TidStore */
+bool
+tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
+{
+ uint64 key;
+ uint64 val = 0;
+ uint64 off_bit;
+ bool found;
+
+ key = tid_to_key_off(ts, tid, &off_bit);
+
+ if (TidStoreIsShared(ts))
+ found = shared_rt_search(ts->tree.shared, key, &val);
+ else
+ found = local_rt_search(ts->tree.local, key, &val);
+
+ if (!found)
+ return false;
+
+ return (val & off_bit) != 0;
+}
+
+/*
+ * Prepare to iterate through a TidStore. Since the radix tree is locked during the
+ * iteration, so tidstore_end_iterate() needs to called when finished.
+ *
+ * Concurrent updates during the iteration will be blocked when inserting a
+ * key-value to the radix tree.
+ */
+TidStoreIter *
+tidstore_begin_iterate(TidStore *ts)
+{
+ TidStoreIter *iter;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ iter = palloc0(sizeof(TidStoreIter));
+ iter->ts = ts;
+
+ iter->result.blkno = InvalidBlockNumber;
+ iter->result.offsets = palloc(sizeof(OffsetNumber) * ts->control->max_offset);
+
+ if (TidStoreIsShared(ts))
+ iter->tree_iter.shared = shared_rt_begin_iterate(ts->tree.shared);
+ else
+ iter->tree_iter.local = local_rt_begin_iterate(ts->tree.local);
+
+ /* If the TidStore is empty, there is no business */
+ if (tidstore_num_tids(ts) == 0)
+ iter->finished = true;
+
+ return iter;
+}
+
+static inline bool
+tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
+{
+ if (TidStoreIsShared(iter->ts))
+ return shared_rt_iterate_next(iter->tree_iter.shared, key, val);
+
+ return local_rt_iterate_next(iter->tree_iter.local, key, val);
+}
+
+/*
+ * Scan the TidStore and return a pointer to TidStoreIterResult that has tids
+ * in one block. We return the block numbers in ascending order and the offset
+ * numbers in each result is also sorted in ascending order.
+ */
+TidStoreIterResult *
+tidstore_iterate_next(TidStoreIter *iter)
+{
+ uint64 key;
+ uint64 val;
+ TidStoreIterResult *result = &(iter->result);
+
+ if (iter->finished)
+ return NULL;
+
+ if (BlockNumberIsValid(result->blkno))
+ {
+ /* Process the previously collected key-value */
+ result->num_offsets = 0;
+ tidstore_iter_extract_tids(iter, iter->next_key, iter->next_val);
+ }
+
+ while (tidstore_iter_kv(iter, &key, &val))
+ {
+ BlockNumber blkno;
+
+ blkno = key_get_blkno(iter->ts, key);
+
+ if (BlockNumberIsValid(result->blkno) && result->blkno != blkno)
+ {
+ /*
+ * We got a key-value pair for a different block. So return the
+ * collected tids, and remember the key-value for the next iteration.
+ */
+ iter->next_key = key;
+ iter->next_val = val;
+ return result;
+ }
+
+ /* Collect tids extracted from the key-value pair */
+ tidstore_iter_extract_tids(iter, key, val);
+ }
+
+ iter->finished = true;
+ return result;
+}
+
+/*
+ * Finish an iteration over TidStore. This needs to be called after finishing
+ * or when existing an iteration.
+ */
+void
+tidstore_end_iterate(TidStoreIter *iter)
+{
+ if (TidStoreIsShared(iter->ts))
+ shared_rt_end_iterate(iter->tree_iter.shared);
+ else
+ local_rt_end_iterate(iter->tree_iter.local);
+
+ pfree(iter->result.offsets);
+ pfree(iter);
+}
+
+/* Return the number of tids we collected so far */
+int64
+tidstore_num_tids(TidStore *ts)
+{
+ uint64 num_tids;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ if (!TidStoreIsShared(ts))
+ return ts->control->num_tids;
+
+ LWLockAcquire(&ts->control->lock, LW_SHARED);
+ num_tids = ts->control->num_tids;
+ LWLockRelease(&ts->control->lock);
+
+ return num_tids;
+}
+
+/* Return true if the current memory usage of TidStore exceeds the limit */
+bool
+tidstore_is_full(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ return (tidstore_memory_usage(ts) > ts->control->max_bytes);
+}
+
+/* Return the maximum memory TidStore can use */
+size_t
+tidstore_max_memory(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ return ts->control->max_bytes;
+}
+
+/* Return the memory usage of TidStore */
+size_t
+tidstore_memory_usage(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ /*
+ * In the shared case, TidStoreControl and radix_tree are backed by the
+ * same DSA area and rt_memory_usage() returns the value including both.
+ * So we don't need to add the size of TidStoreControl separately.
+ */
+ if (TidStoreIsShared(ts))
+ return sizeof(TidStore) + shared_rt_memory_usage(ts->tree.shared);
+
+ return sizeof(TidStore) + sizeof(TidStore) + local_rt_memory_usage(ts->tree.local);
+}
+
+/*
+ * Get a handle that can be used by other processes to attach to this TidStore
+ */
+tidstore_handle
+tidstore_get_handle(TidStore *ts)
+{
+ Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+ return ts->control->handle;
+}
+
+/* Extract tids from the given key-value pair */
+static void
+tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
+{
+ TidStoreIterResult *result = (&iter->result);
+
+ while (val)
+ {
+ uint64 tid_i;
+ OffsetNumber off;
+
+ tid_i = key << TIDSTORE_VALUE_NBITS;
+ tid_i |= pg_rightmost_one_pos64(val);
+
+ off = tid_i & ((UINT64CONST(1) << iter->ts->control->offset_nbits) - 1);
+
+ Assert(result->num_offsets < iter->ts->control->max_offset);
+ result->offsets[result->num_offsets++] = off;
+
+ /* unset the rightmost bit */
+ val &= ~pg_rightmost_one64(val);
+ }
+
+ result->blkno = key_get_blkno(iter->ts, key);
+}
+
+/* Get block number from the given key */
+static inline BlockNumber
+key_get_blkno(TidStore *ts, uint64 key)
+{
+ return (BlockNumber) (key >> ts->control->offset_key_nbits);
+}
+
+/* Encode a tid to key and offset */
+static inline uint64
+tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit)
+{
+ uint32 offset = ItemPointerGetOffsetNumber(tid);
+ BlockNumber block = ItemPointerGetBlockNumber(tid);
+
+ return encode_key_off(ts, block, offset, off_bit);
+}
+
+/* encode a block and offset to a key and partial offset */
+static inline uint64
+encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit)
+{
+ uint64 key;
+ uint64 tid_i;
+ uint32 off_lower;
+
+ off_lower = offset & TIDSTORE_OFFSET_MASK;
+ Assert(off_lower < (sizeof(uint64) * BITS_PER_BYTE));
+
+ *off_bit = UINT64CONST(1) << off_lower;
+ tid_i = offset | ((uint64) block << ts->control->offset_nbits);
+ key = tid_i >> TIDSTORE_VALUE_NBITS;
+
+ return key;
+}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index d2ec396045..55b3a04097 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -176,6 +176,8 @@ static const char *const BuiltinTrancheNames[] = {
"SharedTupleStore",
/* LWTRANCHE_SHARED_TIDBITMAP: */
"SharedTidBitmap",
+ /* LWTRANCHE_SHARED_TIDSTORE: */
+ "SharedTidStore",
/* LWTRANCHE_PARALLEL_APPEND: */
"ParallelAppend",
/* LWTRANCHE_PER_XACT_PREDICATE_LIST: */
diff --git a/src/include/access/tidstore.h b/src/include/access/tidstore.h
new file mode 100644
index 0000000000..a35a52124a
--- /dev/null
+++ b/src/include/access/tidstore.h
@@ -0,0 +1,49 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.h
+ * Tid storage.
+ *
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/tidstore.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef TIDSTORE_H
+#define TIDSTORE_H
+
+#include "storage/itemptr.h"
+#include "utils/dsa.h"
+
+typedef dsa_pointer tidstore_handle;
+
+typedef struct TidStore TidStore;
+typedef struct TidStoreIter TidStoreIter;
+
+typedef struct TidStoreIterResult
+{
+ BlockNumber blkno;
+ OffsetNumber *offsets;
+ int num_offsets;
+} TidStoreIterResult;
+
+extern TidStore *tidstore_create(size_t max_bytes, int max_offset, dsa_area *dsa);
+extern TidStore *tidstore_attach(dsa_area *dsa, dsa_pointer handle);
+extern void tidstore_detach(TidStore *ts);
+extern void tidstore_destroy(TidStore *ts);
+extern void tidstore_reset(TidStore *ts);
+extern void tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets);
+extern bool tidstore_lookup_tid(TidStore *ts, ItemPointer tid);
+extern TidStoreIter * tidstore_begin_iterate(TidStore *ts);
+extern TidStoreIterResult *tidstore_iterate_next(TidStoreIter *iter);
+extern void tidstore_end_iterate(TidStoreIter *iter);
+extern int64 tidstore_num_tids(TidStore *ts);
+extern bool tidstore_is_full(TidStore *ts);
+extern size_t tidstore_max_memory(TidStore *ts);
+extern size_t tidstore_memory_usage(TidStore *ts);
+extern tidstore_handle tidstore_get_handle(TidStore *ts);
+
+#endif /* TIDSTORE_H */
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index d2c7afb8f4..07002fdfbe 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -199,6 +199,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_PER_SESSION_RECORD_TYPMOD,
LWTRANCHE_SHARED_TUPLESTORE,
LWTRANCHE_SHARED_TIDBITMAP,
+ LWTRANCHE_SHARED_TIDSTORE,
LWTRANCHE_PARALLEL_APPEND,
LWTRANCHE_PER_XACT_PREDICATE_LIST,
LWTRANCHE_PGSTATS_DSA,
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 89f42bf9e3..a6ec135430 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
test_rls_hooks \
test_shm_mq \
test_slru \
+ test_tidstore \
unsafe_tests \
worker_spi
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index beaf4080fb..f126ea9f2e 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -31,5 +31,6 @@ subdir('test_regex')
subdir('test_rls_hooks')
subdir('test_shm_mq')
subdir('test_slru')
+subdir('test_tidstore')
subdir('unsafe_tests')
subdir('worker_spi')
diff --git a/src/test/modules/test_tidstore/Makefile b/src/test/modules/test_tidstore/Makefile
new file mode 100644
index 0000000000..dab107d70c
--- /dev/null
+++ b/src/test/modules/test_tidstore/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_tidstore/Makefile
+
+MODULE_big = test_tidstore
+OBJS = \
+ $(WIN32RES) \
+ test_tidstore.o
+PGFILEDESC = "test_tidstore - test code for src/backend/access/common/tidstore.c"
+
+EXTENSION = test_tidstore
+DATA = test_tidstore--1.0.sql
+
+REGRESS = test_tidstore
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_tidstore
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_tidstore/expected/test_tidstore.out b/src/test/modules/test_tidstore/expected/test_tidstore.out
new file mode 100644
index 0000000000..7ff2f9af87
--- /dev/null
+++ b/src/test/modules/test_tidstore/expected/test_tidstore.out
@@ -0,0 +1,13 @@
+CREATE EXTENSION test_tidstore;
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
+NOTICE: testing empty tidstore
+NOTICE: testing basic operations
+ test_tidstore
+---------------
+
+(1 row)
+
diff --git a/src/test/modules/test_tidstore/meson.build b/src/test/modules/test_tidstore/meson.build
new file mode 100644
index 0000000000..31f2da7b61
--- /dev/null
+++ b/src/test/modules/test_tidstore/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_tidstore_sources = files(
+ 'test_tidstore.c',
+)
+
+if host_system == 'windows'
+ test_tidstore_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_tidstore',
+ '--FILEDESC', 'test_tidstore - test code for src/backend/access/common/tidstore.c',])
+endif
+
+test_tidstore = shared_module('test_tidstore',
+ test_tidstore_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += test_tidstore
+
+install_data(
+ 'test_tidstore.control',
+ 'test_tidstore--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'test_tidstore',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'test_tidstore',
+ ],
+ },
+}
diff --git a/src/test/modules/test_tidstore/sql/test_tidstore.sql b/src/test/modules/test_tidstore/sql/test_tidstore.sql
new file mode 100644
index 0000000000..03aea31815
--- /dev/null
+++ b/src/test/modules/test_tidstore/sql/test_tidstore.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_tidstore;
+
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
diff --git a/src/test/modules/test_tidstore/test_tidstore--1.0.sql b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
new file mode 100644
index 0000000000..47e9149900
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_tidstore/test_tidstore--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_tidstore" to load this file. \quit
+
+CREATE FUNCTION test_tidstore()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_tidstore/test_tidstore.c b/src/test/modules/test_tidstore/test_tidstore.c
new file mode 100644
index 0000000000..9a1217f833
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.c
@@ -0,0 +1,226 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_tidstore.c
+ * Test TidStore data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_tidstore/test_tidstore.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/tidstore.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "storage/lwlock.h"
+#include "utils/memutils.h"
+
+PG_MODULE_MAGIC;
+
+/* #define TEST_SHARED_TIDSTORE 1 */
+
+#define TEST_TIDSTORE_MAX_BYTES (2 * 1024 * 1024L) /* 2MB */
+
+PG_FUNCTION_INFO_V1(test_tidstore);
+
+static void
+check_tid(TidStore *ts, BlockNumber blkno, OffsetNumber off, bool expect)
+{
+ ItemPointerData tid;
+ bool found;
+
+ ItemPointerSet(&tid, blkno, off);
+
+ found = tidstore_lookup_tid(ts, &tid);
+
+ if (found != expect)
+ elog(ERROR, "lookup TID (%u, %u) returned %d, expected %d",
+ blkno, off, found, expect);
+}
+
+static void
+test_basic(int max_offset)
+{
+#define TEST_TIDSTORE_NUM_BLOCKS 5
+#define TEST_TIDSTORE_NUM_OFFSETS 5
+
+ TidStore *ts;
+ TidStoreIter *iter;
+ TidStoreIterResult *iter_result;
+ BlockNumber blks[TEST_TIDSTORE_NUM_BLOCKS] = {
+ 0, MaxBlockNumber, MaxBlockNumber - 1, 1, MaxBlockNumber / 2,
+ };
+ BlockNumber blks_sorted[TEST_TIDSTORE_NUM_BLOCKS] = {
+ 0, 1, MaxBlockNumber / 2, MaxBlockNumber - 1, MaxBlockNumber
+ };
+ OffsetNumber offs[TEST_TIDSTORE_NUM_OFFSETS];
+ int blk_idx;
+
+#ifdef TEST_SHARED_TIDSTORE
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_tidstore");
+ dsa = dsa_create(tranche_id);
+
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, dsa);
+#else
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
+#endif
+
+ /* prepare the offset array */
+ offs[0] = FirstOffsetNumber;
+ offs[1] = FirstOffsetNumber + 1;
+ offs[2] = max_offset / 2;
+ offs[3] = max_offset - 1;
+ offs[4] = max_offset;
+
+ /* add tids */
+ for (int i = 0; i < TEST_TIDSTORE_NUM_BLOCKS; i++)
+ tidstore_add_tids(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
+
+ /* lookup test */
+ for (OffsetNumber off = FirstOffsetNumber ; off < max_offset; off++)
+ {
+ bool expect = false;
+ for (int i = 0; i < TEST_TIDSTORE_NUM_OFFSETS; i++)
+ {
+ if (offs[i] == off)
+ {
+ expect = true;
+ break;
+ }
+ }
+
+ check_tid(ts, 0, off, expect);
+ check_tid(ts, 2, off, false);
+ check_tid(ts, MaxBlockNumber - 2, off, false);
+ check_tid(ts, MaxBlockNumber, off, expect);
+ }
+
+ /* test the number of tids */
+ if (tidstore_num_tids(ts) != (TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS))
+ elog(ERROR, "tidstore_num_tids returned " UINT64_FORMAT ", expected %d",
+ tidstore_num_tids(ts),
+ TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS);
+
+ /* iteration test */
+ iter = tidstore_begin_iterate(ts);
+ blk_idx = 0;
+ while ((iter_result = tidstore_iterate_next(iter)) != NULL)
+ {
+ /* check the returned block number */
+ if (blks_sorted[blk_idx] != iter_result->blkno)
+ elog(ERROR, "tidstore_iterate_next returned block number %u, expected %u",
+ iter_result->blkno, blks_sorted[blk_idx]);
+
+ /* check the returned offset numbers */
+ if (TEST_TIDSTORE_NUM_OFFSETS != iter_result->num_offsets)
+ elog(ERROR, "tidstore_iterate_next returned %u offsets, expected %u",
+ iter_result->num_offsets, TEST_TIDSTORE_NUM_OFFSETS);
+
+ for (int i = 0; i < iter_result->num_offsets; i++)
+ {
+ if (offs[i] != iter_result->offsets[i])
+ elog(ERROR, "tidstore_iterate_next returned offset number %u on block %u, expected %u",
+ iter_result->offsets[i], iter_result->blkno, offs[i]);
+ }
+
+ blk_idx++;
+ }
+
+ if (blk_idx != TEST_TIDSTORE_NUM_BLOCKS)
+ elog(ERROR, "tidstore_iterate_next returned %d blocks, expected %d",
+ blk_idx, TEST_TIDSTORE_NUM_BLOCKS);
+
+ /* remove all tids */
+ tidstore_reset(ts);
+
+ /* test the number of tids */
+ if (tidstore_num_tids(ts) != 0)
+ elog(ERROR, "tidstore_num_tids on empty store returned non-zero");
+
+ /* lookup test for empty store */
+ for (OffsetNumber off = FirstOffsetNumber ; off < MaxHeapTuplesPerPage;
+ off++)
+ {
+ check_tid(ts, 0, off, false);
+ check_tid(ts, 2, off, false);
+ check_tid(ts, MaxBlockNumber - 2, off, false);
+ check_tid(ts, MaxBlockNumber, off, false);
+ }
+
+ tidstore_destroy(ts);
+
+#ifdef TEST_SHARED_TIDSTORE
+ dsa_detach(dsa);
+#endif
+}
+
+static void
+test_empty(void)
+{
+ TidStore *ts;
+ TidStoreIter *iter;
+ ItemPointerData tid;
+
+#ifdef TEST_SHARED_TIDSTORE
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_tidstore");
+ dsa = dsa_create(tranche_id);
+
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, dsa);
+#else
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
+#endif
+
+ elog(NOTICE, "testing empty tidstore");
+
+ ItemPointerSet(&tid, 0, FirstOffsetNumber);
+ if (tidstore_lookup_tid(ts, &tid))
+ elog(ERROR, "tidstore_lookup_tid for (0,1) on empty store returned true");
+
+ ItemPointerSet(&tid, MaxBlockNumber, MaxOffsetNumber);
+ if (tidstore_lookup_tid(ts, &tid))
+ elog(ERROR, "tidstore_lookup_tid for (%u,%u) on empty store returned true",
+ MaxBlockNumber, MaxOffsetNumber);
+
+ if (tidstore_num_tids(ts) != 0)
+ elog(ERROR, "tidstore_num_entries on empty store returned non-zero");
+
+ if (tidstore_is_full(ts))
+ elog(ERROR, "tidstore_is_full on empty store returned true");
+
+ iter = tidstore_begin_iterate(ts);
+
+ if (tidstore_iterate_next(iter) != NULL)
+ elog(ERROR, "tidstore_iterate_next on empty store returned TIDs");
+
+ tidstore_end_iterate(iter);
+
+ tidstore_destroy(ts);
+
+#ifdef TEST_SHARED_TIDSTORE
+ dsa_detach(dsa);
+#endif
+}
+
+Datum
+test_tidstore(PG_FUNCTION_ARGS)
+{
+ test_empty();
+
+ elog(NOTICE, "testing basic operations");
+ test_basic(MaxHeapTuplesPerPage);
+ test_basic(10);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_tidstore/test_tidstore.control b/src/test/modules/test_tidstore/test_tidstore.control
new file mode 100644
index 0000000000..9b6bd4638f
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.control
@@ -0,0 +1,4 @@
+comment = 'Test code for tidstore'
+default_version = '1.0'
+module_pathname = '$libdir/test_tidstore'
+relocatable = true
--
2.31.1
[application/octet-stream] v32-0002-Move-some-bitmap-logic-out-of-bitmapset.c.patch (6.0K, ../../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/17-v32-0002-Move-some-bitmap-logic-out-of-bitmapset.c.patch)
download | inline diff:
From 7fe0c744e052286a8c44716494fe4d644b0e8451 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 6 Dec 2022 13:39:41 +0700
Subject: [PATCH v32 02/18] Move some bitmap logic out of bitmapset.c
Add pg_rightmost_one32/64 functions. This functionality was previously
private to bitmapset.c as the RIGHTMOST_ONE macro. It has practical
use in other contexts, so move to pg_bitutils.h.
Also move the logic for selecting appropriate pg_bitutils functions
based on word size to bitmapset.h for wider visibility and add
appropriate selection for bmw_rightmost_one().
Since the previous macro relied on casting to signedbitmapword,
and the new functions do not, remove that typedef.
Design input and review by Tom Lane
Discussion: https://www.postgresql.org/message-id/CAFBsxsFW2JjTo58jtDB%2B3sZhxMx3t-3evew8%3DAcr%2BGGhC%2BkFaA%40mail.gmail.com
---
src/backend/nodes/bitmapset.c | 34 +-------------------------------
src/include/nodes/bitmapset.h | 16 +++++++++++++--
src/include/port/pg_bitutils.h | 31 +++++++++++++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 -
4 files changed, 46 insertions(+), 36 deletions(-)
diff --git a/src/backend/nodes/bitmapset.c b/src/backend/nodes/bitmapset.c
index 7ba3cf635b..0b2962ed73 100644
--- a/src/backend/nodes/bitmapset.c
+++ b/src/backend/nodes/bitmapset.c
@@ -30,39 +30,7 @@
#define BITMAPSET_SIZE(nwords) \
(offsetof(Bitmapset, words) + (nwords) * sizeof(bitmapword))
-/*----------
- * This is a well-known cute trick for isolating the rightmost one-bit
- * in a word. It assumes two's complement arithmetic. Consider any
- * nonzero value, and focus attention on the rightmost one. The value is
- * then something like
- * xxxxxx10000
- * where x's are unspecified bits. The two's complement negative is formed
- * by inverting all the bits and adding one. Inversion gives
- * yyyyyy01111
- * where each y is the inverse of the corresponding x. Incrementing gives
- * yyyyyy10000
- * and then ANDing with the original value gives
- * 00000010000
- * This works for all cases except original value = zero, where of course
- * we get zero.
- *----------
- */
-#define RIGHTMOST_ONE(x) ((signedbitmapword) (x) & -((signedbitmapword) (x)))
-
-#define HAS_MULTIPLE_ONES(x) ((bitmapword) RIGHTMOST_ONE(x) != (x))
-
-/* Select appropriate bit-twiddling functions for bitmap word size */
-#if BITS_PER_BITMAPWORD == 32
-#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w)
-#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w)
-#define bmw_popcount(w) pg_popcount32(w)
-#elif BITS_PER_BITMAPWORD == 64
-#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w)
-#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w)
-#define bmw_popcount(w) pg_popcount64(w)
-#else
-#error "invalid BITS_PER_BITMAPWORD"
-#endif
+#define HAS_MULTIPLE_ONES(x) (bmw_rightmost_one(x) != (x))
static bool bms_is_empty_internal(const Bitmapset *a);
diff --git a/src/include/nodes/bitmapset.h b/src/include/nodes/bitmapset.h
index 14de6a9ff1..c7e1711147 100644
--- a/src/include/nodes/bitmapset.h
+++ b/src/include/nodes/bitmapset.h
@@ -36,13 +36,11 @@ struct List;
#define BITS_PER_BITMAPWORD 64
typedef uint64 bitmapword; /* must be an unsigned type */
-typedef int64 signedbitmapword; /* must be the matching signed type */
#else
#define BITS_PER_BITMAPWORD 32
typedef uint32 bitmapword; /* must be an unsigned type */
-typedef int32 signedbitmapword; /* must be the matching signed type */
#endif
@@ -73,6 +71,20 @@ typedef enum
BMS_MULTIPLE /* >1 member */
} BMS_Membership;
+/* Select appropriate bit-twiddling functions for bitmap word size */
+#if BITS_PER_BITMAPWORD == 32
+#define bmw_rightmost_one(w) pg_rightmost_one32(w)
+#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w)
+#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w)
+#define bmw_popcount(w) pg_popcount32(w)
+#elif BITS_PER_BITMAPWORD == 64
+#define bmw_rightmost_one(w) pg_rightmost_one64(w)
+#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w)
+#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w)
+#define bmw_popcount(w) pg_popcount64(w)
+#else
+#error "invalid BITS_PER_BITMAPWORD"
+#endif
/*
* function prototypes in nodes/bitmapset.c
diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h
index 158ef73a2b..bf7588e075 100644
--- a/src/include/port/pg_bitutils.h
+++ b/src/include/port/pg_bitutils.h
@@ -32,6 +32,37 @@ extern PGDLLIMPORT const uint8 pg_leftmost_one_pos[256];
extern PGDLLIMPORT const uint8 pg_rightmost_one_pos[256];
extern PGDLLIMPORT const uint8 pg_number_of_ones[256];
+/*----------
+ * This is a well-known cute trick for isolating the rightmost one-bit
+ * in a word. It assumes two's complement arithmetic. Consider any
+ * nonzero value, and focus attention on the rightmost one. The value is
+ * then something like
+ * xxxxxx10000
+ * where x's are unspecified bits. The two's complement negative is formed
+ * by inverting all the bits and adding one. Inversion gives
+ * yyyyyy01111
+ * where each y is the inverse of the corresponding x. Incrementing gives
+ * yyyyyy10000
+ * and then ANDing with the original value gives
+ * 00000010000
+ * This works for all cases except original value = zero, where of course
+ * we get zero.
+ *----------
+ */
+static inline uint32
+pg_rightmost_one32(uint32 word)
+{
+ int32 result = (int32) word & -((int32) word);
+ return (uint32) result;
+}
+
+static inline uint64
+pg_rightmost_one64(uint64 word)
+{
+ int64 result = (int64) word & -((int64) word);
+ return (uint64) result;
+}
+
/*
* pg_leftmost_one_pos32
* Returns the position of the most significant set bit in "word",
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b4058b88c3..fd3d83c781 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3684,7 +3684,6 @@ shmem_request_hook_type
shmem_startup_hook_type
sig_atomic_t
sigjmp_buf
-signedbitmapword
sigset_t
size_t
slist_head
--
2.31.1
[application/octet-stream] v32-0001-Introduce-helper-SIMD-functions-for-small-byte-a.patch (2.9K, ../../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/18-v32-0001-Introduce-helper-SIMD-functions-for-small-byte-a.patch)
download | inline diff:
From 51fe658fcecefb2b8c0d826c7d7d6070eb9e878c Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 24 Oct 2022 14:07:09 +0900
Subject: [PATCH v32 01/18] Introduce helper SIMD functions for small byte
arrays
vector8_min - helper for emulating ">=" semantics
vector8_highbit_mask - used to turn the result of a vector
comparison into a bitmask
Masahiko Sawada
Reviewed by Nathan Bossart, additional adjustments by me
Discussion: https://www.postgresql.org/message-id/CAD21AoDap240WDDdUDE0JMpCmuMMnGajrKrkCRxM7zn9Xk3JRA%40mail.gmail.com
---
src/include/port/simd.h | 47 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)
diff --git a/src/include/port/simd.h b/src/include/port/simd.h
index 1fa6c3bc6c..dfae14e463 100644
--- a/src/include/port/simd.h
+++ b/src/include/port/simd.h
@@ -79,6 +79,7 @@ static inline bool vector8_has_le(const Vector8 v, const uint8 c);
static inline bool vector8_is_highbit_set(const Vector8 v);
#ifndef USE_NO_SIMD
static inline bool vector32_is_highbit_set(const Vector32 v);
+static inline uint32 vector8_highbit_mask(const Vector8 v);
#endif
/* arithmetic operations */
@@ -96,6 +97,7 @@ static inline Vector8 vector8_ssub(const Vector8 v1, const Vector8 v2);
*/
#ifndef USE_NO_SIMD
static inline Vector8 vector8_eq(const Vector8 v1, const Vector8 v2);
+static inline Vector8 vector8_min(const Vector8 v1, const Vector8 v2);
static inline Vector32 vector32_eq(const Vector32 v1, const Vector32 v2);
#endif
@@ -299,6 +301,36 @@ vector32_is_highbit_set(const Vector32 v)
}
#endif /* ! USE_NO_SIMD */
+/*
+ * Return a bitmask formed from the high-bit of each element.
+ */
+#ifndef USE_NO_SIMD
+static inline uint32
+vector8_highbit_mask(const Vector8 v)
+{
+#ifdef USE_SSE2
+ return (uint32) _mm_movemask_epi8(v);
+#elif defined(USE_NEON)
+ /*
+ * Note: There is a faster way to do this, but it returns a uint64 and
+ * and if the caller wanted to extract the bit position using CTZ,
+ * it would have to divide that result by 4.
+ */
+ static const uint8 mask[16] = {
+ 1 << 0, 1 << 1, 1 << 2, 1 << 3,
+ 1 << 4, 1 << 5, 1 << 6, 1 << 7,
+ 1 << 0, 1 << 1, 1 << 2, 1 << 3,
+ 1 << 4, 1 << 5, 1 << 6, 1 << 7,
+ };
+
+ uint8x16_t masked = vandq_u8(vld1q_u8(mask), (uint8x16_t) vshrq_n_s8(v, 7));
+ uint8x16_t maskedhi = vextq_u8(masked, masked, 8);
+
+ return (uint32) vaddvq_u16((uint16x8_t) vzip1q_u8(masked, maskedhi));
+#endif
+}
+#endif /* ! USE_NO_SIMD */
+
/*
* Return the bitwise OR of the inputs
*/
@@ -372,4 +404,19 @@ vector32_eq(const Vector32 v1, const Vector32 v2)
}
#endif /* ! USE_NO_SIMD */
+/*
+ * Given two vectors, return a vector with the minimum element of each.
+ */
+#ifndef USE_NO_SIMD
+static inline Vector8
+vector8_min(const Vector8 v1, const Vector8 v2)
+{
+#ifdef USE_SSE2
+ return _mm_min_epu8(v1, v2);
+#elif defined(USE_NEON)
+ return vminq_u8(v1, v2);
+#endif
+}
+#endif /* ! USE_NO_SIMD */
+
#endif /* SIMD_H */
--
2.31.1
[application/octet-stream] v32-0003-Add-radixtree-template.patch (117.0K, ../../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/19-v32-0003-Add-radixtree-template.patch)
download | inline diff:
From b88b152cac7c31b49416c4e59e93b3b5f0813759 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 14 Sep 2022 12:38:51 +0000
Subject: [PATCH v32 03/18] Add radixtree template
WIP: commit message based on template comments
---
src/backend/utils/mmgr/dsa.c | 12 +
src/include/lib/radixtree.h | 2516 +++++++++++++++++
src/include/lib/radixtree_delete_impl.h | 122 +
src/include/lib/radixtree_insert_impl.h | 328 +++
src/include/lib/radixtree_iter_impl.h | 153 +
src/include/lib/radixtree_search_impl.h | 138 +
src/include/utils/dsa.h | 1 +
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_radixtree/.gitignore | 4 +
src/test/modules/test_radixtree/Makefile | 23 +
src/test/modules/test_radixtree/README | 7 +
.../expected/test_radixtree.out | 36 +
src/test/modules/test_radixtree/meson.build | 35 +
.../test_radixtree/sql/test_radixtree.sql | 7 +
.../test_radixtree/test_radixtree--1.0.sql | 8 +
.../modules/test_radixtree/test_radixtree.c | 681 +++++
.../test_radixtree/test_radixtree.control | 4 +
src/tools/pginclude/cpluspluscheck | 6 +
src/tools/pginclude/headerscheck | 6 +
20 files changed, 4089 insertions(+)
create mode 100644 src/include/lib/radixtree.h
create mode 100644 src/include/lib/radixtree_delete_impl.h
create mode 100644 src/include/lib/radixtree_insert_impl.h
create mode 100644 src/include/lib/radixtree_iter_impl.h
create mode 100644 src/include/lib/radixtree_search_impl.h
create mode 100644 src/test/modules/test_radixtree/.gitignore
create mode 100644 src/test/modules/test_radixtree/Makefile
create mode 100644 src/test/modules/test_radixtree/README
create mode 100644 src/test/modules/test_radixtree/expected/test_radixtree.out
create mode 100644 src/test/modules/test_radixtree/meson.build
create mode 100644 src/test/modules/test_radixtree/sql/test_radixtree.sql
create mode 100644 src/test/modules/test_radixtree/test_radixtree--1.0.sql
create mode 100644 src/test/modules/test_radixtree/test_radixtree.c
create mode 100644 src/test/modules/test_radixtree/test_radixtree.control
diff --git a/src/backend/utils/mmgr/dsa.c b/src/backend/utils/mmgr/dsa.c
index f5a62061a3..80555aefff 100644
--- a/src/backend/utils/mmgr/dsa.c
+++ b/src/backend/utils/mmgr/dsa.c
@@ -1024,6 +1024,18 @@ dsa_set_size_limit(dsa_area *area, size_t limit)
LWLockRelease(DSA_AREA_LOCK(area));
}
+size_t
+dsa_get_total_size(dsa_area *area)
+{
+ size_t size;
+
+ LWLockAcquire(DSA_AREA_LOCK(area), LW_SHARED);
+ size = area->control->total_segment_size;
+ LWLockRelease(DSA_AREA_LOCK(area));
+
+ return size;
+}
+
/*
* Aggressively free all spare memory in the hope of returning DSM segments to
* the operating system.
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
new file mode 100644
index 0000000000..e546bd705c
--- /dev/null
+++ b/src/include/lib/radixtree.h
@@ -0,0 +1,2516 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree.h
+ * Template for adaptive radix tree.
+ *
+ * This module employs the idea from the paper "The Adaptive Radix Tree: ARTful
+ * Indexing for Main-Memory Databases" by Viktor Leis, Alfons Kemper, and Thomas
+ * Neumann, 2013. The radix tree uses adaptive node sizes, a small number of node
+ * types, each with a different numbers of elements. Depending on the number of
+ * children, the appropriate node type is used.
+ *
+ * WIP: notes about traditional radix tree trading off span vs height...
+ *
+ * There are two kinds of nodes, inner nodes and leaves. Inner nodes
+ * map partial keys to child pointers.
+ *
+ * The ART paper mentions three ways to implement leaves:
+ *
+ * "- Single-value leaves: The values are stored using an addi-
+ * tional leaf node type which stores one value.
+ * - Multi-value leaves: The values are stored in one of four
+ * different leaf node types, which mirror the structure of
+ * inner nodes, but contain values instead of pointers.
+ * - Combined pointer/value slots: If values fit into point-
+ * ers, no separate node types are necessary. Instead, each
+ * pointer storage location in an inner node can either
+ * store a pointer or a value."
+ *
+ * We chose "multi-value leaves" to avoid the additional pointer traversal
+ * required by "single-value leaves"
+ *
+ * For simplicity, the key is assumed to be 64-bit unsigned integer. The
+ * tree doesn't need to contain paths where the highest bytes of all keys
+ * are zero. That way, the tree's height adapts to the distribution of keys.
+ *
+ * TODO: In the future it might be worthwhile to offer configurability of
+ * leaf implementation for different use cases. Single-values leaves would
+ * give more flexibility in key type, including variable-length keys.
+ *
+ * There are some optimizations not yet implemented, particularly path
+ * compression and lazy path expansion.
+ *
+ * To handle concurrency, we use a single reader-writer lock for the radix
+ * tree. The radix tree is exclusively locked during write operations such
+ * as RT_SET() and RT_DELETE(), and shared locked during read operations
+ * such as RT_SEARCH(). An iteration also holds the shared lock on the radix
+ * tree until it is completed.
+ *
+ * TODO: The current locking mechanism is not optimized for high concurrency
+ * with mixed read-write workloads. In the future it might be worthwhile
+ * to replace it with the Optimistic Lock Coupling or ROWEX mentioned in
+ * the paper "The ART of Practical Synchronization" by the same authors as
+ * the ART paper, 2016.
+ *
+ * WIP: the radix tree nodes don't shrink.
+ *
+ * To generate a radix tree and associated functions for a use case several
+ * macros have to be #define'ed before this file is included. Including
+ * the file #undef's all those, so a new radix tree can be generated
+ * afterwards.
+ * The relevant parameters are:
+ * - RT_PREFIX - prefix for all symbol names generated. A prefix of 'foo'
+ * will result in radix tree type 'foo_radix_tree' and functions like
+ * 'foo_create'/'foo_free' and so forth.
+ * - RT_DECLARE - if defined function prototypes and type declarations are
+ * generated
+ * - RT_DEFINE - if defined function definitions are generated
+ * - RT_SCOPE - in which scope (e.g. extern, static inline) do function
+ * declarations reside
+ * - RT_VALUE_TYPE - the type of the value.
+ *
+ * Optional parameters:
+ * - RT_SHMEM - if defined, the radix tree is created in the DSA area
+ * so that multiple processes can access it simultaneously.
+ * - RT_DEBUG - if defined add stats tracking and debugging functions
+ *
+ * Interface
+ * ---------
+ *
+ * RT_CREATE - Create a new, empty radix tree
+ * RT_FREE - Free the radix tree
+ * RT_SEARCH - Search a key-value pair
+ * RT_SET - Set a key-value pair
+ * RT_BEGIN_ITERATE - Begin iterating through all key-value pairs
+ * RT_ITERATE_NEXT - Return next key-value pair, if any
+ * RT_END_ITER - End iteration
+ * RT_MEMORY_USAGE - Get the memory usage
+ *
+ * Interface for Shared Memory
+ * ---------
+ *
+ * RT_ATTACH - Attach to the radix tree
+ * RT_DETACH - Detach from the radix tree
+ * RT_GET_HANDLE - Return the handle of the radix tree
+ *
+ * Optional Interface
+ * ---------
+ *
+ * RT_DELETE - Delete a key-value pair. Declared/define if RT_USE_DELETE is defined
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/include/lib/radixtree.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/stringinfo.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "port/pg_bitutils.h"
+#include "port/simd.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/* helpers */
+#define RT_MAKE_PREFIX(a) CppConcat(a,_)
+#define RT_MAKE_NAME(name) RT_MAKE_NAME_(RT_MAKE_PREFIX(RT_PREFIX),name)
+#define RT_MAKE_NAME_(a,b) CppConcat(a,b)
+
+/* function declarations */
+#define RT_CREATE RT_MAKE_NAME(create)
+#define RT_FREE RT_MAKE_NAME(free)
+#define RT_SEARCH RT_MAKE_NAME(search)
+#ifdef RT_SHMEM
+#define RT_ATTACH RT_MAKE_NAME(attach)
+#define RT_DETACH RT_MAKE_NAME(detach)
+#define RT_GET_HANDLE RT_MAKE_NAME(get_handle)
+#endif
+#define RT_SET RT_MAKE_NAME(set)
+#define RT_BEGIN_ITERATE RT_MAKE_NAME(begin_iterate)
+#define RT_ITERATE_NEXT RT_MAKE_NAME(iterate_next)
+#define RT_END_ITERATE RT_MAKE_NAME(end_iterate)
+#ifdef RT_USE_DELETE
+#define RT_DELETE RT_MAKE_NAME(delete)
+#endif
+#define RT_MEMORY_USAGE RT_MAKE_NAME(memory_usage)
+#ifdef RT_DEBUG
+#define RT_DUMP RT_MAKE_NAME(dump)
+#define RT_DUMP_NODE RT_MAKE_NAME(dump_node)
+#define RT_DUMP_SEARCH RT_MAKE_NAME(dump_search)
+#define RT_STATS RT_MAKE_NAME(stats)
+#endif
+
+/* internal helper functions (no externally visible prototypes) */
+#define RT_NEW_ROOT RT_MAKE_NAME(new_root)
+#define RT_ALLOC_NODE RT_MAKE_NAME(alloc_node)
+#define RT_INIT_NODE RT_MAKE_NAME(init_node)
+#define RT_FREE_NODE RT_MAKE_NAME(free_node)
+#define RT_FREE_RECURSE RT_MAKE_NAME(free_recurse)
+#define RT_EXTEND RT_MAKE_NAME(extend)
+#define RT_SET_EXTEND RT_MAKE_NAME(set_extend)
+#define RT_SWITCH_NODE_KIND RT_MAKE_NAME(grow_node_kind)
+#define RT_COPY_NODE RT_MAKE_NAME(copy_node)
+#define RT_REPLACE_NODE RT_MAKE_NAME(replace_node)
+#define RT_PTR_GET_LOCAL RT_MAKE_NAME(ptr_get_local)
+#define RT_PTR_ALLOC_IS_VALID RT_MAKE_NAME(ptr_stored_is_valid)
+#define RT_NODE_3_SEARCH_EQ RT_MAKE_NAME(node_3_search_eq)
+#define RT_NODE_32_SEARCH_EQ RT_MAKE_NAME(node_32_search_eq)
+#define RT_NODE_3_GET_INSERTPOS RT_MAKE_NAME(node_3_get_insertpos)
+#define RT_NODE_32_GET_INSERTPOS RT_MAKE_NAME(node_32_get_insertpos)
+#define RT_CHUNK_CHILDREN_ARRAY_SHIFT RT_MAKE_NAME(chunk_children_array_shift)
+#define RT_CHUNK_VALUES_ARRAY_SHIFT RT_MAKE_NAME(chunk_values_array_shift)
+#define RT_CHUNK_CHILDREN_ARRAY_DELETE RT_MAKE_NAME(chunk_children_array_delete)
+#define RT_CHUNK_VALUES_ARRAY_DELETE RT_MAKE_NAME(chunk_values_array_delete)
+#define RT_CHUNK_CHILDREN_ARRAY_COPY RT_MAKE_NAME(chunk_children_array_copy)
+#define RT_CHUNK_VALUES_ARRAY_COPY RT_MAKE_NAME(chunk_values_array_copy)
+#define RT_NODE_125_IS_CHUNK_USED RT_MAKE_NAME(node_125_is_chunk_used)
+#define RT_NODE_INNER_125_GET_CHILD RT_MAKE_NAME(node_inner_125_get_child)
+#define RT_NODE_LEAF_125_GET_VALUE RT_MAKE_NAME(node_leaf_125_get_value)
+#define RT_NODE_INNER_256_IS_CHUNK_USED RT_MAKE_NAME(node_inner_256_is_chunk_used)
+#define RT_NODE_LEAF_256_IS_CHUNK_USED RT_MAKE_NAME(node_leaf_256_is_chunk_used)
+#define RT_NODE_INNER_256_GET_CHILD RT_MAKE_NAME(node_inner_256_get_child)
+#define RT_NODE_LEAF_256_GET_VALUE RT_MAKE_NAME(node_leaf_256_get_value)
+#define RT_NODE_INNER_256_SET RT_MAKE_NAME(node_inner_256_set)
+#define RT_NODE_LEAF_256_SET RT_MAKE_NAME(node_leaf_256_set)
+#define RT_NODE_INNER_256_DELETE RT_MAKE_NAME(node_inner_256_delete)
+#define RT_NODE_LEAF_256_DELETE RT_MAKE_NAME(node_leaf_256_delete)
+#define RT_KEY_GET_SHIFT RT_MAKE_NAME(key_get_shift)
+#define RT_SHIFT_GET_MAX_VAL RT_MAKE_NAME(shift_get_max_val)
+#define RT_NODE_SEARCH_INNER RT_MAKE_NAME(node_search_inner)
+#define RT_NODE_SEARCH_LEAF RT_MAKE_NAME(node_search_leaf)
+#define RT_NODE_UPDATE_INNER RT_MAKE_NAME(node_update_inner)
+#define RT_NODE_DELETE_INNER RT_MAKE_NAME(node_delete_inner)
+#define RT_NODE_DELETE_LEAF RT_MAKE_NAME(node_delete_leaf)
+#define RT_NODE_INSERT_INNER RT_MAKE_NAME(node_insert_inner)
+#define RT_NODE_INSERT_LEAF RT_MAKE_NAME(node_insert_leaf)
+#define RT_NODE_INNER_ITERATE_NEXT RT_MAKE_NAME(node_inner_iterate_next)
+#define RT_NODE_LEAF_ITERATE_NEXT RT_MAKE_NAME(node_leaf_iterate_next)
+#define RT_UPDATE_ITER_STACK RT_MAKE_NAME(update_iter_stack)
+#define RT_ITER_UPDATE_KEY RT_MAKE_NAME(iter_update_key)
+#define RT_VERIFY_NODE RT_MAKE_NAME(verify_node)
+
+/* type declarations */
+#define RT_RADIX_TREE RT_MAKE_NAME(radix_tree)
+#define RT_RADIX_TREE_CONTROL RT_MAKE_NAME(radix_tree_control)
+#define RT_ITER RT_MAKE_NAME(iter)
+#ifdef RT_SHMEM
+#define RT_HANDLE RT_MAKE_NAME(handle)
+#endif
+#define RT_NODE RT_MAKE_NAME(node)
+#define RT_NODE_ITER RT_MAKE_NAME(node_iter)
+#define RT_NODE_BASE_3 RT_MAKE_NAME(node_base_3)
+#define RT_NODE_BASE_32 RT_MAKE_NAME(node_base_32)
+#define RT_NODE_BASE_125 RT_MAKE_NAME(node_base_125)
+#define RT_NODE_BASE_256 RT_MAKE_NAME(node_base_256)
+#define RT_NODE_INNER_3 RT_MAKE_NAME(node_inner_3)
+#define RT_NODE_INNER_32 RT_MAKE_NAME(node_inner_32)
+#define RT_NODE_INNER_125 RT_MAKE_NAME(node_inner_125)
+#define RT_NODE_INNER_256 RT_MAKE_NAME(node_inner_256)
+#define RT_NODE_LEAF_3 RT_MAKE_NAME(node_leaf_3)
+#define RT_NODE_LEAF_32 RT_MAKE_NAME(node_leaf_32)
+#define RT_NODE_LEAF_125 RT_MAKE_NAME(node_leaf_125)
+#define RT_NODE_LEAF_256 RT_MAKE_NAME(node_leaf_256)
+#define RT_SIZE_CLASS RT_MAKE_NAME(size_class)
+#define RT_SIZE_CLASS_ELEM RT_MAKE_NAME(size_class_elem)
+#define RT_SIZE_CLASS_INFO RT_MAKE_NAME(size_class_info)
+#define RT_CLASS_3 RT_MAKE_NAME(class_3)
+#define RT_CLASS_32_MIN RT_MAKE_NAME(class_32_min)
+#define RT_CLASS_32_MAX RT_MAKE_NAME(class_32_max)
+#define RT_CLASS_125 RT_MAKE_NAME(class_125)
+#define RT_CLASS_256 RT_MAKE_NAME(class_256)
+
+/* generate forward declarations necessary to use the radix tree */
+#ifdef RT_DECLARE
+
+typedef struct RT_RADIX_TREE RT_RADIX_TREE;
+typedef struct RT_ITER RT_ITER;
+
+#ifdef RT_SHMEM
+typedef dsa_pointer RT_HANDLE;
+#endif
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id);
+RT_SCOPE RT_RADIX_TREE * RT_ATTACH(dsa_area *dsa, dsa_pointer dp);
+RT_SCOPE void RT_DETACH(RT_RADIX_TREE *tree);
+RT_SCOPE RT_HANDLE RT_GET_HANDLE(RT_RADIX_TREE *tree);
+#else
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx);
+#endif
+RT_SCOPE void RT_FREE(RT_RADIX_TREE *tree);
+
+RT_SCOPE bool RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p);
+RT_SCOPE bool RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p);
+#ifdef RT_USE_DELETE
+RT_SCOPE bool RT_DELETE(RT_RADIX_TREE *tree, uint64 key);
+#endif
+
+RT_SCOPE RT_ITER * RT_BEGIN_ITERATE(RT_RADIX_TREE *tree);
+RT_SCOPE bool RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p);
+RT_SCOPE void RT_END_ITERATE(RT_ITER *iter);
+
+RT_SCOPE uint64 RT_MEMORY_USAGE(RT_RADIX_TREE *tree);
+
+#ifdef RT_DEBUG
+RT_SCOPE void RT_DUMP(RT_RADIX_TREE *tree);
+RT_SCOPE void RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key);
+RT_SCOPE void RT_STATS(RT_RADIX_TREE *tree);
+#endif
+
+#endif /* RT_DECLARE */
+
+
+/* generate implementation of the radix tree */
+#ifdef RT_DEFINE
+
+/* The number of bits encoded in one tree level */
+#define RT_NODE_SPAN BITS_PER_BYTE
+
+/* The number of maximum slots in the node */
+#define RT_NODE_MAX_SLOTS (1 << RT_NODE_SPAN)
+
+/* Mask for extracting a chunk from the key */
+#define RT_CHUNK_MASK ((1 << RT_NODE_SPAN) - 1)
+
+/* Maximum shift the radix tree uses */
+#define RT_MAX_SHIFT RT_KEY_GET_SHIFT(UINT64_MAX)
+
+/* Tree level the radix tree uses */
+#define RT_MAX_LEVEL ((sizeof(uint64) * BITS_PER_BYTE) / RT_NODE_SPAN)
+
+/*
+ * Number of bits necessary for isset array in the slot-index node.
+ * Since bitmapword can be 64 bits, the only values that make sense
+ * here are 64 and 128.
+ */
+#define RT_SLOT_IDX_LIMIT (RT_NODE_MAX_SLOTS / 2)
+
+/* Invalid index used in node-125 */
+#define RT_INVALID_SLOT_IDX 0xFF
+
+/* Get a chunk from the key */
+#define RT_GET_KEY_CHUNK(key, shift) ((uint8) (((key) >> (shift)) & RT_CHUNK_MASK))
+
+/* For accessing bitmaps */
+#define RT_BM_IDX(x) ((x) / BITS_PER_BITMAPWORD)
+#define RT_BM_BIT(x) ((x) % BITS_PER_BITMAPWORD)
+
+/*
+ * Node kinds
+ *
+ * The different node kinds are what make the tree "adaptive".
+ *
+ * Each node kind is associated with a different datatype and different
+ * search/set/delete/iterate algorithms adapted for its size. The largest
+ * kind, node256 is basically the same as a traditional radix tree,
+ * and would be most wasteful of memory when sparsely populated. The
+ * smaller nodes expend some additional CPU time to enable a smaller
+ * memory footprint.
+ *
+ * XXX There are 4 node kinds, and this should never be increased,
+ * for several reasons:
+ * 1. With 5 or more kinds, gcc tends to use a jump table for switch
+ * statements.
+ * 2. The 4 kinds can be represented with 2 bits, so we have the option
+ * in the future to tag the node pointer with the kind, even on
+ * platforms with 32-bit pointers. This might speed up node traversal
+ * in trees with highly random node kinds.
+ * 3. We can have multiple size classes per node kind.
+ */
+#define RT_NODE_KIND_3 0x00
+#define RT_NODE_KIND_32 0x01
+#define RT_NODE_KIND_125 0x02
+#define RT_NODE_KIND_256 0x03
+#define RT_NODE_KIND_COUNT 4
+
+/*
+ * Calculate the slab blocksize so that we can allocate at least 32 chunks
+ * from the block.
+ */
+#define RT_SLAB_BLOCK_SIZE(size) \
+ Max((SLAB_DEFAULT_BLOCK_SIZE / (size)) * (size), (size) * 32)
+
+/* Common type for all nodes types */
+typedef struct RT_NODE
+{
+ /*
+ * Number of children. We use uint16 to be able to indicate 256 children
+ * at the fanout of 8.
+ */
+ uint16 count;
+
+ /*
+ * Max capacity for the current size class. Storing this in the
+ * node enables multiple size classes per node kind.
+ * Technically, kinds with a single size class don't need this, so we could
+ * keep this in the individual base types, but the code is simpler this way.
+ * Note: node256 is unique in that it cannot possibly have more than a
+ * single size class, so for that kind we store zero, and uint8 is
+ * sufficient for other kinds.
+ */
+ uint8 fanout;
+
+ /*
+ * Shift indicates which part of the key space is represented by this
+ * node. That is, the key is shifted by 'shift' and the lowest
+ * RT_NODE_SPAN bits are then represented in chunk.
+ */
+ uint8 shift;
+
+ /* Node kind, one per search/set algorithm */
+ uint8 kind;
+} RT_NODE;
+
+
+#define RT_PTR_LOCAL RT_NODE *
+
+#ifdef RT_SHMEM
+#define RT_PTR_ALLOC dsa_pointer
+#else
+#define RT_PTR_ALLOC RT_PTR_LOCAL
+#endif
+
+
+#ifdef RT_SHMEM
+#define RT_INVALID_PTR_ALLOC InvalidDsaPointer
+#else
+#define RT_INVALID_PTR_ALLOC NULL
+#endif
+
+#ifdef RT_SHMEM
+#define RT_LOCK_EXCLUSIVE(tree) LWLockAcquire(&tree->ctl->lock, LW_EXCLUSIVE)
+#define RT_LOCK_SHARED(tree) LWLockAcquire(&tree->ctl->lock, LW_SHARED)
+#define RT_UNLOCK(tree) LWLockRelease(&tree->ctl->lock);
+#else
+#define RT_LOCK_EXCLUSIVE(tree) ((void) 0)
+#define RT_LOCK_SHARED(tree) ((void) 0)
+#define RT_UNLOCK(tree) ((void) 0)
+#endif
+
+/*
+ * Inner nodes and leaf nodes have analogous structure. To distinguish
+ * them at runtime, we take advantage of the fact that the key chunk
+ * is accessed by shifting: Inner tree nodes (shift > 0), store the
+ * pointer to its child node in the slot. In leaf nodes (shift == 0),
+ * the slot contains the value corresponding to the key.
+ */
+#define RT_NODE_IS_LEAF(n) (((RT_PTR_LOCAL) (n))->shift == 0)
+
+#define RT_NODE_MUST_GROW(node) \
+ ((node)->base.n.count == (node)->base.n.fanout)
+
+/*
+ * Base type of each node kinds for leaf and inner nodes.
+ * The base types must be a be able to accommodate the largest size
+ * class for variable-sized node kinds.
+ */
+typedef struct RT_NODE_BASE_3
+{
+ RT_NODE n;
+
+ /* 3 children, for key chunks */
+ uint8 chunks[3];
+} RT_NODE_BASE_3;
+
+typedef struct RT_NODE_BASE_32
+{
+ RT_NODE n;
+
+ /* 32 children, for key chunks */
+ uint8 chunks[32];
+} RT_NODE_BASE_32;
+
+/*
+ * node-125 uses slot_idx array, an array of RT_NODE_MAX_SLOTS length
+ * to store indexes into a second array that contains the values (or
+ * child pointers).
+ */
+typedef struct RT_NODE_BASE_125
+{
+ RT_NODE n;
+
+ /* The index of slots for each fanout */
+ uint8 slot_idxs[RT_NODE_MAX_SLOTS];
+
+ /* bitmap to track which slots are in use */
+ bitmapword isset[RT_BM_IDX(RT_SLOT_IDX_LIMIT)];
+} RT_NODE_BASE_125;
+
+typedef struct RT_NODE_BASE_256
+{
+ RT_NODE n;
+} RT_NODE_BASE_256;
+
+/*
+ * Inner and leaf nodes.
+ *
+ * These are separate because the value type might be different than
+ * something fitting into a pointer-width type.
+ */
+typedef struct RT_NODE_INNER_3
+{
+ RT_NODE_BASE_3 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_3;
+
+typedef struct RT_NODE_LEAF_3
+{
+ RT_NODE_BASE_3 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_3;
+
+typedef struct RT_NODE_INNER_32
+{
+ RT_NODE_BASE_32 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_32;
+
+typedef struct RT_NODE_LEAF_32
+{
+ RT_NODE_BASE_32 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_32;
+
+typedef struct RT_NODE_INNER_125
+{
+ RT_NODE_BASE_125 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_125;
+
+typedef struct RT_NODE_LEAF_125
+{
+ RT_NODE_BASE_125 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_125;
+
+/*
+ * node-256 is the largest node type. This node has an array
+ * for directly storing values (or child pointers in inner nodes).
+ * Unlike other node kinds, it's array size is by definition
+ * fixed.
+ */
+typedef struct RT_NODE_INNER_256
+{
+ RT_NODE_BASE_256 base;
+
+ /* Slots for 256 children */
+ RT_PTR_ALLOC children[RT_NODE_MAX_SLOTS];
+} RT_NODE_INNER_256;
+
+typedef struct RT_NODE_LEAF_256
+{
+ RT_NODE_BASE_256 base;
+
+ /*
+ * Unlike with inner256, zero is a valid value here, so we use a
+ * bitmap to track which slots are in use.
+ */
+ bitmapword isset[RT_BM_IDX(RT_NODE_MAX_SLOTS)];
+
+ /* Slots for 256 values */
+ RT_VALUE_TYPE values[RT_NODE_MAX_SLOTS];
+} RT_NODE_LEAF_256;
+
+/*
+ * Node size classes
+ *
+ * Nodes of different kinds necessarily belong to different size classes.
+ * The main innovation in our implementation compared to the ART paper
+ * is decoupling the notion of size class from kind.
+ *
+ * The size classes within a given node kind have the same underlying
+ * type, but a variable number of children/values. This is possible
+ * because the base type contains small fixed data structures that
+ * work the same way regardless of how full the node is. We store the
+ * node's allocated capacity in the "fanout" member of RT_NODE, to allow
+ * runtime introspection.
+ *
+ * Growing from one node kind to another requires special code for each
+ * case, but growing from one size class to another within the same kind
+ * is basically just allocate + memcpy.
+ *
+ * The size classes have been chosen so that inner nodes on platforms
+ * with 64-bit pointers (and leaf nodes when using a 64-bit key) are
+ * equal to or slightly smaller than some DSA size class.
+ */
+typedef enum RT_SIZE_CLASS
+{
+ RT_CLASS_3 = 0,
+ RT_CLASS_32_MIN,
+ RT_CLASS_32_MAX,
+ RT_CLASS_125,
+ RT_CLASS_256
+} RT_SIZE_CLASS;
+
+/* Information for each size class */
+typedef struct RT_SIZE_CLASS_ELEM
+{
+ const char *name;
+ int fanout;
+
+ /* slab chunk size */
+ Size inner_size;
+ Size leaf_size;
+} RT_SIZE_CLASS_ELEM;
+
+static const RT_SIZE_CLASS_ELEM RT_SIZE_CLASS_INFO[] = {
+ [RT_CLASS_3] = {
+ .name = "radix tree node 3",
+ .fanout = 3,
+ .inner_size = sizeof(RT_NODE_INNER_3) + 3 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_3) + 3 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_32_MIN] = {
+ .name = "radix tree node 15",
+ .fanout = 15,
+ .inner_size = sizeof(RT_NODE_INNER_32) + 15 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_32) + 15 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_32_MAX] = {
+ .name = "radix tree node 32",
+ .fanout = 32,
+ .inner_size = sizeof(RT_NODE_INNER_32) + 32 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_32) + 32 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_125] = {
+ .name = "radix tree node 125",
+ .fanout = 125,
+ .inner_size = sizeof(RT_NODE_INNER_125) + 125 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_125) + 125 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_256] = {
+ .name = "radix tree node 256",
+ .fanout = 256,
+ .inner_size = sizeof(RT_NODE_INNER_256),
+ .leaf_size = sizeof(RT_NODE_LEAF_256),
+ },
+};
+
+#define RT_SIZE_CLASS_COUNT lengthof(RT_SIZE_CLASS_INFO)
+
+#ifdef RT_SHMEM
+/* A magic value used to identify our radix tree */
+#define RT_RADIX_TREE_MAGIC 0x54A48167
+#endif
+
+/* Contains the actual tree and ancillary info */
+// WIP: this name is a bit strange
+typedef struct RT_RADIX_TREE_CONTROL
+{
+#ifdef RT_SHMEM
+ RT_HANDLE handle;
+ uint32 magic;
+ LWLock lock;
+#endif
+
+ RT_PTR_ALLOC root;
+ uint64 max_val;
+ uint64 num_keys;
+
+ /* statistics */
+#ifdef RT_DEBUG
+ int32 cnt[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE_CONTROL;
+
+/* Entry point for allocating and accessing the tree */
+typedef struct RT_RADIX_TREE
+{
+ MemoryContext context;
+
+ /* pointing to either local memory or DSA */
+ RT_RADIX_TREE_CONTROL *ctl;
+
+#ifdef RT_SHMEM
+ dsa_area *dsa;
+#else
+ MemoryContextData *inner_slabs[RT_SIZE_CLASS_COUNT];
+ MemoryContextData *leaf_slabs[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE;
+
+/*
+ * Iteration support.
+ *
+ * Iterating the radix tree returns each pair of key and value in the ascending
+ * order of the key. To support this, the we iterate nodes of each level.
+ *
+ * RT_NODE_ITER struct is used to track the iteration within a node.
+ *
+ * RT_ITER is the struct for iteration of the radix tree, and uses RT_NODE_ITER
+ * in order to track the iteration of each level. During iteration, we also
+ * construct the key whenever updating the node iteration information, e.g., when
+ * advancing the current index within the node or when moving to the next node
+ * at the same level.
+ *
+ * XXX: Currently we allow only one process to do iteration. Therefore, rt_node_iter
+ * has the local pointers to nodes, rather than RT_PTR_ALLOC.
+ * We need either a safeguard to disallow other processes to begin the iteration
+ * while one process is doing or to allow multiple processes to do the iteration.
+ */
+typedef struct RT_NODE_ITER
+{
+ RT_PTR_LOCAL node; /* current node being iterated */
+ int current_idx; /* current position. -1 for initial value */
+} RT_NODE_ITER;
+
+typedef struct RT_ITER
+{
+ RT_RADIX_TREE *tree;
+
+ /* Track the iteration on nodes of each level */
+ RT_NODE_ITER stack[RT_MAX_LEVEL];
+ int stack_len;
+
+ /* The key is constructed during iteration */
+ uint64 key;
+} RT_ITER;
+
+
+static void RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_PTR_ALLOC child);
+static bool RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_VALUE_TYPE *value_p);
+
+/* verification (available only with assertion) */
+static void RT_VERIFY_NODE(RT_PTR_LOCAL node);
+
+/* Get the local address of an allocated node */
+static inline RT_PTR_LOCAL
+RT_PTR_GET_LOCAL(RT_RADIX_TREE *tree, RT_PTR_ALLOC node)
+{
+#ifdef RT_SHMEM
+ return dsa_get_address(tree->dsa, (dsa_pointer) node);
+#else
+ return node;
+#endif
+}
+
+static inline bool
+RT_PTR_ALLOC_IS_VALID(RT_PTR_ALLOC ptr)
+{
+#ifdef RT_SHMEM
+ return DsaPointerIsValid(ptr);
+#else
+ return PointerIsValid(ptr);
+#endif
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_3_SEARCH_EQ(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+ int idx = -1;
+
+ for (int i = 0; i < node->n.count; i++)
+ {
+ if (node->chunks[i] == chunk)
+ {
+ idx = i;
+ break;
+ }
+ }
+
+ return idx;
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_3_GET_INSERTPOS(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+ int idx;
+
+ for (idx = 0; idx < node->n.count; idx++)
+ {
+ if (node->chunks[idx] >= chunk)
+ break;
+ }
+
+ return idx;
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_32_SEARCH_EQ(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+ int count = node->n.count;
+#ifndef USE_NO_SIMD
+ Vector8 spread_chunk;
+ Vector8 haystack1;
+ Vector8 haystack2;
+ Vector8 cmp1;
+ Vector8 cmp2;
+ uint32 bitfield;
+ int index_simd = -1;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+ int index = -1;
+
+ for (int i = 0; i < count; i++)
+ {
+ if (node->chunks[i] == chunk)
+ {
+ index = i;
+ break;
+ }
+ }
+#endif
+
+#ifndef USE_NO_SIMD
+ /* replicate the search key */
+ spread_chunk = vector8_broadcast(chunk);
+
+ /* compare to all 32 keys stored in the node */
+ vector8_load(&haystack1, &node->chunks[0]);
+ vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+ cmp1 = vector8_eq(spread_chunk, haystack1);
+ cmp2 = vector8_eq(spread_chunk, haystack2);
+
+ /* convert comparison to a bitfield */
+ bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+
+ /* mask off invalid entries */
+ bitfield &= ((UINT64CONST(1) << count) - 1);
+
+ /* convert bitfield to index by counting trailing zeros */
+ if (bitfield)
+ index_simd = pg_rightmost_one_pos32(bitfield);
+
+ Assert(index_simd == index);
+ return index_simd;
+#else
+ return index;
+#endif
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_32_GET_INSERTPOS(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+ int count = node->n.count;
+#ifndef USE_NO_SIMD
+ Vector8 spread_chunk;
+ Vector8 haystack1;
+ Vector8 haystack2;
+ Vector8 cmp1;
+ Vector8 cmp2;
+ Vector8 min1;
+ Vector8 min2;
+ uint32 bitfield;
+ int index_simd;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+ int index;
+
+ for (index = 0; index < count; index++)
+ {
+ /*
+ * This is coded with '>=' to match what we can do with SIMD,
+ * with an assert to keep us honest.
+ */
+ if (node->chunks[index] >= chunk)
+ {
+ Assert(node->chunks[index] != chunk);
+ break;
+ }
+ }
+#endif
+
+#ifndef USE_NO_SIMD
+ /*
+ * This is a bit more complicated than RT_NODE_32_SEARCH_EQ(), because
+ * no unsigned uint8 comparison instruction exists, at least for SSE2. So
+ * we need to play some trickery using vector8_min() to effectively get
+ * >=. There'll never be any equal elements in current uses, but that's
+ * what we get here...
+ */
+ spread_chunk = vector8_broadcast(chunk);
+ vector8_load(&haystack1, &node->chunks[0]);
+ vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+ min1 = vector8_min(spread_chunk, haystack1);
+ min2 = vector8_min(spread_chunk, haystack2);
+ cmp1 = vector8_eq(spread_chunk, min1);
+ cmp2 = vector8_eq(spread_chunk, min2);
+ bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+ bitfield &= ((UINT64CONST(1) << count) - 1);
+
+ if (bitfield)
+ index_simd = pg_rightmost_one_pos32(bitfield);
+ else
+ index_simd = count;
+
+ Assert(index_simd == index);
+ return index_simd;
+#else
+ return index;
+#endif
+}
+
+
+/*
+ * Functions to manipulate both chunks array and children/values array.
+ * These are used for node-3 and node-32.
+ */
+
+/* Shift the elements right at 'idx' by one */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_SHIFT(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+ memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+ memmove(&(children[idx + 1]), &(children[idx]), sizeof(RT_PTR_ALLOC) * (count - idx));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_SHIFT(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+ memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+ memmove(&(values[idx + 1]), &(values[idx]), sizeof(RT_VALUE_TYPE) * (count - idx));
+}
+
+/* Delete the element at 'idx' */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_DELETE(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+ memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+ memmove(&(children[idx]), &(children[idx + 1]), sizeof(RT_PTR_ALLOC) * (count - idx - 1));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_DELETE(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+ memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+ memmove(&(values[idx]), &(values[idx + 1]), sizeof(RT_VALUE_TYPE) * (count - idx - 1));
+}
+
+/* Copy both chunks and children/values arrays */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_COPY(uint8 *src_chunks, RT_PTR_ALLOC *src_children,
+ uint8 *dst_chunks, RT_PTR_ALLOC *dst_children)
+{
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const Size chunk_size = sizeof(uint8) * fanout;
+ const Size children_size = sizeof(RT_PTR_ALLOC) * fanout;
+
+ memcpy(dst_chunks, src_chunks, chunk_size);
+ memcpy(dst_children, src_children, children_size);
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_COPY(uint8 *src_chunks, RT_VALUE_TYPE *src_values,
+ uint8 *dst_chunks, RT_VALUE_TYPE *dst_values)
+{
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const Size chunk_size = sizeof(uint8) * fanout;
+ const Size values_size = sizeof(RT_VALUE_TYPE) * fanout;
+
+ memcpy(dst_chunks, src_chunks, chunk_size);
+ memcpy(dst_values, src_values, values_size);
+}
+
+/* Functions to manipulate inner and leaf node-125 */
+
+/* Does the given chunk in the node has the value? */
+static inline bool
+RT_NODE_125_IS_CHUNK_USED(RT_NODE_BASE_125 *node, uint8 chunk)
+{
+ return node->slot_idxs[chunk] != RT_INVALID_SLOT_IDX;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_125_GET_CHILD(RT_NODE_INNER_125 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ return node->children[node->base.slot_idxs[chunk]];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_125_GET_VALUE(RT_NODE_LEAF_125 *node, uint8 chunk)
+{
+ Assert(RT_NODE_IS_LEAF(node));
+ Assert(((RT_NODE_BASE_125 *) node)->slot_idxs[chunk] != RT_INVALID_SLOT_IDX);
+ return node->values[node->base.slot_idxs[chunk]];
+}
+
+/* Functions to manipulate inner and leaf node-256 */
+
+/* Return true if the slot corresponding to the given chunk is in use */
+static inline bool
+RT_NODE_INNER_256_IS_CHUNK_USED(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ return node->children[chunk] != RT_INVALID_PTR_ALLOC;
+}
+
+static inline bool
+RT_NODE_LEAF_256_IS_CHUNK_USED(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ return (node->isset[idx] & ((bitmapword) 1 << bitnum)) != 0;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_256_GET_CHILD(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ Assert(RT_NODE_INNER_256_IS_CHUNK_USED(node, chunk));
+ return node->children[chunk];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_256_GET_VALUE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ Assert(RT_NODE_IS_LEAF(node));
+ Assert(RT_NODE_LEAF_256_IS_CHUNK_USED(node, chunk));
+ return node->values[chunk];
+}
+
+/* Set the child in the node-256 */
+static inline void
+RT_NODE_INNER_256_SET(RT_NODE_INNER_256 *node, uint8 chunk, RT_PTR_ALLOC child)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ node->children[chunk] = child;
+}
+
+/* Set the value in the node-256 */
+static inline void
+RT_NODE_LEAF_256_SET(RT_NODE_LEAF_256 *node, uint8 chunk, RT_VALUE_TYPE value)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ node->isset[idx] |= ((bitmapword) 1 << bitnum);
+ node->values[chunk] = value;
+}
+
+/* Set the slot at the given chunk position */
+static inline void
+RT_NODE_INNER_256_DELETE(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ node->children[chunk] = RT_INVALID_PTR_ALLOC;
+}
+
+static inline void
+RT_NODE_LEAF_256_DELETE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ node->isset[idx] &= ~((bitmapword) 1 << bitnum);
+}
+
+/*
+ * Return the largest shift that will allowing storing the given key.
+ */
+static inline int
+RT_KEY_GET_SHIFT(uint64 key)
+{
+ if (key == 0)
+ return 0;
+ else
+ return (pg_leftmost_one_pos64(key) / RT_NODE_SPAN) * RT_NODE_SPAN;
+}
+
+/*
+ * Return the max value that can be stored in the tree with the given shift.
+ */
+static uint64
+RT_SHIFT_GET_MAX_VAL(int shift)
+{
+ if (shift == RT_MAX_SHIFT)
+ return UINT64_MAX;
+
+ return (UINT64CONST(1) << (shift + RT_NODE_SPAN)) - 1;
+}
+
+/*
+ * Allocate a new node with the given node kind.
+ */
+static RT_PTR_ALLOC
+RT_ALLOC_NODE(RT_RADIX_TREE *tree, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+ RT_PTR_ALLOC allocnode;
+ size_t allocsize;
+
+ if (is_leaf)
+ allocsize = RT_SIZE_CLASS_INFO[size_class].leaf_size;
+ else
+ allocsize = RT_SIZE_CLASS_INFO[size_class].inner_size;
+
+#ifdef RT_SHMEM
+ allocnode = dsa_allocate(tree->dsa, allocsize);
+#else
+ if (is_leaf)
+ allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->leaf_slabs[size_class],
+ allocsize);
+ else
+ allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->inner_slabs[size_class],
+ allocsize);
+#endif
+
+#ifdef RT_DEBUG
+ /* update the statistics */
+ tree->ctl->cnt[size_class]++;
+#endif
+
+ return allocnode;
+}
+
+/* Initialize the node contents */
+static inline void
+RT_INIT_NODE(RT_PTR_LOCAL node, uint8 kind, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+ if (is_leaf)
+ MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].leaf_size);
+ else
+ MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].inner_size);
+
+ node->kind = kind;
+
+ if (kind == RT_NODE_KIND_256)
+ /* See comment for the RT_NODE type */
+ Assert(node->fanout == 0);
+ else
+ node->fanout = RT_SIZE_CLASS_INFO[size_class].fanout;
+
+ /* Initialize slot_idxs to invalid values */
+ if (kind == RT_NODE_KIND_125)
+ {
+ RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+
+ memset(n125->slot_idxs, RT_INVALID_SLOT_IDX, sizeof(n125->slot_idxs));
+ }
+}
+
+/*
+ * Create a new node as the root. Subordinate nodes will be created during
+ * the insertion.
+ */
+static pg_noinline void
+RT_NEW_ROOT(RT_RADIX_TREE *tree, uint64 key)
+{
+ int shift = RT_KEY_GET_SHIFT(key);
+ bool is_leaf = shift == 0;
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(newnode, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ newnode->shift = shift;
+ tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(shift);
+ tree->ctl->root = allocnode;
+}
+
+static inline void
+RT_COPY_NODE(RT_PTR_LOCAL newnode, RT_PTR_LOCAL oldnode)
+{
+ newnode->shift = oldnode->shift;
+ newnode->count = oldnode->count;
+}
+
+/*
+ * Given a new allocated node and an old node, initalize the new
+ * node with the necessary fields and return its local pointer.
+ */
+static inline RT_PTR_LOCAL
+RT_SWITCH_NODE_KIND(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, RT_PTR_LOCAL node,
+ uint8 new_kind, uint8 new_class, bool is_leaf)
+{
+ RT_PTR_LOCAL newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(newnode, new_kind, new_class, is_leaf);
+ RT_COPY_NODE(newnode, node);
+
+ return newnode;
+}
+
+/* Free the given node */
+static void
+RT_FREE_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode)
+{
+ /* If we're deleting the root node, make the tree empty */
+ if (tree->ctl->root == allocnode)
+ {
+ tree->ctl->root = RT_INVALID_PTR_ALLOC;
+ tree->ctl->max_val = 0;
+ }
+
+#ifdef RT_DEBUG
+ {
+ int i;
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+ /* update the statistics */
+ for (i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ if (node->fanout == RT_SIZE_CLASS_INFO[i].fanout)
+ break;
+ }
+
+ /* fanout of node256 is intentionally 0 */
+ if (i == RT_SIZE_CLASS_COUNT)
+ i = RT_CLASS_256;
+
+ tree->ctl->cnt[i]--;
+ Assert(tree->ctl->cnt[i] >= 0);
+ }
+#endif
+
+#ifdef RT_SHMEM
+ dsa_free(tree->dsa, allocnode);
+#else
+ pfree(allocnode);
+#endif
+}
+
+/* Update the parent's pointer when growing a node */
+static inline void
+RT_NODE_UPDATE_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC new_child)
+{
+#define RT_ACTION_UPDATE
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+#undef RT_ACTION_UPDATE
+}
+
+/*
+ * Replace old_child with new_child, and free the old one.
+ */
+static inline void
+RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
+ RT_PTR_ALLOC stored_old_child, RT_PTR_LOCAL old_child,
+ RT_PTR_ALLOC new_child, uint64 key)
+{
+#ifdef USE_ASSERT_CHECKING
+ RT_PTR_LOCAL new = RT_PTR_GET_LOCAL(tree, new_child);
+
+ Assert(old_child->shift == new->shift);
+ Assert(old_child->count == new->count);
+#endif
+
+ if (parent == old_child)
+ {
+ /* Replace the root node with the new larger node */
+ tree->ctl->root = new_child;
+ }
+ else
+ RT_NODE_UPDATE_INNER(parent, key, new_child);
+
+ RT_FREE_NODE(tree, stored_old_child);
+}
+
+/*
+ * The radix tree doesn't have sufficient height. Extend the radix tree so
+ * it can store the key.
+ */
+static pg_noinline void
+RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
+{
+ int target_shift;
+ RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+ int shift = root->shift + RT_NODE_SPAN;
+
+ target_shift = RT_KEY_GET_SHIFT(key);
+
+ /* Grow tree from 'shift' to 'target_shift' */
+ while (shift <= target_shift)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL node;
+ RT_NODE_INNER_3 *n3;
+
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, true);
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(node, RT_NODE_KIND_3, RT_CLASS_3, true);
+ node->shift = shift;
+ node->count = 1;
+
+ n3 = (RT_NODE_INNER_3 *) node;
+ n3->base.chunks[0] = 0;
+ n3->children[0] = tree->ctl->root;
+
+ /* Update the root */
+ tree->ctl->root = allocnode;
+
+ shift += RT_NODE_SPAN;
+ }
+
+ tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(target_shift);
+}
+
+/*
+ * The radix tree doesn't have inner and leaf nodes for given key-value pair.
+ * Insert inner and leaf nodes from 'node' to bottom.
+ */
+static pg_noinline void
+RT_SET_EXTEND(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
+ RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node)
+{
+ int shift = node->shift;
+
+ Assert(RT_PTR_GET_LOCAL(tree, stored_node) == node);
+
+ while (shift >= RT_NODE_SPAN)
+ {
+ RT_PTR_ALLOC allocchild;
+ RT_PTR_LOCAL newchild;
+ int newshift = shift - RT_NODE_SPAN;
+ bool is_leaf = newshift == 0;
+
+ allocchild = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ newchild = RT_PTR_GET_LOCAL(tree, allocchild);
+ RT_INIT_NODE(newchild, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ newchild->shift = newshift;
+ RT_NODE_INSERT_INNER(tree, parent, stored_node, node, key, allocchild);
+
+ parent = node;
+ node = newchild;
+ stored_node = allocchild;
+ shift -= RT_NODE_SPAN;
+ }
+
+ RT_NODE_INSERT_LEAF(tree, parent, stored_node, node, key, value_p);
+ tree->ctl->num_keys++;
+}
+
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the child
+ * pointer is set to child_p.
+ */
+static inline bool
+RT_NODE_SEARCH_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC *child_p)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the pointer
+ * to the value is set to value_p.
+ */
+static inline bool
+RT_NODE_SEARCH_LEAF(RT_PTR_LOCAL node, uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_INNER(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_LEAF(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+#endif
+
+/*
+ * Insert "child" into "node".
+ *
+ * "parent" is the parent of "node", so the grandparent of the child.
+ * If the node we're inserting into needs to grow, we update the parent's
+ * child pointer with the pointer to the new larger node.
+ */
+static void
+RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_PTR_ALLOC child)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/* Like RT_NODE_INSERT_INNER, but for leaf nodes */
+static bool
+RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Create the radix tree in the given memory context and return it.
+ */
+RT_SCOPE RT_RADIX_TREE *
+#ifdef RT_SHMEM
+RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id)
+#else
+RT_CREATE(MemoryContext ctx)
+#endif
+{
+ RT_RADIX_TREE *tree;
+ MemoryContext old_ctx;
+#ifdef RT_SHMEM
+ dsa_pointer dp;
+#endif
+
+ old_ctx = MemoryContextSwitchTo(ctx);
+
+ tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+ tree->context = ctx;
+
+#ifdef RT_SHMEM
+ tree->dsa = dsa;
+ dp = dsa_allocate0(dsa, sizeof(RT_RADIX_TREE_CONTROL));
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, dp);
+ tree->ctl->handle = dp;
+ tree->ctl->magic = RT_RADIX_TREE_MAGIC;
+ LWLockInitialize(&tree->ctl->lock, tranche_id);
+#else
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) palloc0(sizeof(RT_RADIX_TREE_CONTROL));
+
+ /* Create a slab context for each size class */
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ RT_SIZE_CLASS_ELEM size_class = RT_SIZE_CLASS_INFO[i];
+ size_t inner_blocksize = RT_SLAB_BLOCK_SIZE(size_class.inner_size);
+ size_t leaf_blocksize = RT_SLAB_BLOCK_SIZE(size_class.leaf_size);
+
+ tree->inner_slabs[i] = SlabContextCreate(ctx,
+ size_class.name,
+ inner_blocksize,
+ size_class.inner_size);
+ tree->leaf_slabs[i] = SlabContextCreate(ctx,
+ size_class.name,
+ leaf_blocksize,
+ size_class.leaf_size);
+ }
+#endif
+
+ tree->ctl->root = RT_INVALID_PTR_ALLOC;
+
+ MemoryContextSwitchTo(old_ctx);
+
+ return tree;
+}
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE *
+RT_ATTACH(dsa_area *dsa, RT_HANDLE handle)
+{
+ RT_RADIX_TREE *tree;
+ dsa_pointer control;
+
+ tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+
+ /* Find the control object in shard memory */
+ control = handle;
+
+ tree->dsa = dsa;
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, control);
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+ return tree;
+}
+
+RT_SCOPE void
+RT_DETACH(RT_RADIX_TREE *tree)
+{
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ pfree(tree);
+}
+
+RT_SCOPE RT_HANDLE
+RT_GET_HANDLE(RT_RADIX_TREE *tree)
+{
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ return tree->ctl->handle;
+}
+
+/*
+ * Recursively free all nodes allocated to the DSA area.
+ */
+static void
+RT_FREE_RECURSE(RT_RADIX_TREE *tree, RT_PTR_ALLOC ptr)
+{
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, ptr);
+
+ check_stack_depth();
+ CHECK_FOR_INTERRUPTS();
+
+ /* The leaf node doesn't have child pointers */
+ if (RT_NODE_IS_LEAF(node))
+ {
+ dsa_free(tree->dsa, ptr);
+ return;
+ }
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+ for (int i = 0; i < n3->base.n.count; i++)
+ RT_FREE_RECURSE(tree, n3->children[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+ for (int i = 0; i < n32->base.n.count; i++)
+ RT_FREE_RECURSE(tree, n32->children[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) node;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+ continue;
+
+ RT_FREE_RECURSE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+ }
+
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ RT_FREE_RECURSE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+ }
+
+ break;
+ }
+ }
+
+ /* Free the inner node */
+ dsa_free(tree->dsa, ptr);
+}
+#endif
+
+/*
+ * Free the given radix tree.
+ */
+RT_SCOPE void
+RT_FREE(RT_RADIX_TREE *tree)
+{
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+ /* Free all memory used for radix tree nodes */
+ if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ RT_FREE_RECURSE(tree, tree->ctl->root);
+
+ /*
+ * Vandalize the control block to help catch programming error where
+ * other backends access the memory formerly occupied by this radix tree.
+ */
+ tree->ctl->magic = 0;
+ dsa_free(tree->dsa, tree->ctl->handle);
+#else
+ pfree(tree->ctl);
+
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ MemoryContextDelete(tree->inner_slabs[i]);
+ MemoryContextDelete(tree->leaf_slabs[i]);
+ }
+#endif
+
+ pfree(tree);
+}
+
+/*
+ * Set key to value. If the entry already exists, we update its value to 'value'
+ * and return true. Returns false if entry doesn't yet exist.
+ */
+RT_SCOPE bool
+RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+ int shift;
+ bool updated;
+ RT_PTR_LOCAL parent;
+ RT_PTR_ALLOC stored_child;
+ RT_PTR_LOCAL child;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ RT_LOCK_EXCLUSIVE(tree);
+
+ /* Empty tree, create the root */
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ RT_NEW_ROOT(tree, key);
+
+ /* Extend the tree if necessary */
+ if (key > tree->ctl->max_val)
+ RT_EXTEND(tree, key);
+
+ stored_child = tree->ctl->root;
+ parent = RT_PTR_GET_LOCAL(tree, stored_child);
+ shift = parent->shift;
+
+ /* Descend the tree until we reach a leaf node */
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC new_child = RT_INVALID_PTR_ALLOC;
+
+ child = RT_PTR_GET_LOCAL(tree, stored_child);
+
+ if (RT_NODE_IS_LEAF(child))
+ break;
+
+ if (!RT_NODE_SEARCH_INNER(child, key, &new_child))
+ {
+ RT_SET_EXTEND(tree, key, value_p, parent, stored_child, child);
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ parent = child;
+ stored_child = new_child;
+ shift -= RT_NODE_SPAN;
+ }
+
+ updated = RT_NODE_INSERT_LEAF(tree, parent, stored_child, child, key, value_p);
+
+ /* Update the statistics */
+ if (!updated)
+ tree->ctl->num_keys++;
+
+ RT_UNLOCK(tree);
+ return updated;
+}
+
+/*
+ * Search the given key in the radix tree. Return true if there is the key,
+ * otherwise return false. On success, we set the value to *value_p so it must
+ * not be NULL.
+ */
+RT_SCOPE bool
+RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+ RT_PTR_LOCAL node;
+ int shift;
+ bool found;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+ Assert(value_p != NULL);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ node = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+ shift = node->shift;
+
+ /* Descend the tree until a leaf node */
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+ if (RT_NODE_IS_LEAF(node))
+ break;
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ node = RT_PTR_GET_LOCAL(tree, child);
+ shift -= RT_NODE_SPAN;
+ }
+
+ found = RT_NODE_SEARCH_LEAF(node, key, value_p);
+
+ RT_UNLOCK(tree);
+ return found;
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Delete the given key from the radix tree. Return true if the key is found (and
+ * deleted), otherwise do nothing and return false.
+ */
+RT_SCOPE bool
+RT_DELETE(RT_RADIX_TREE *tree, uint64 key)
+{
+ RT_PTR_LOCAL node;
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_ALLOC stack[RT_MAX_LEVEL] = {0};
+ int shift;
+ int level;
+ bool deleted;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ RT_LOCK_EXCLUSIVE(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ /*
+ * Descend the tree to search the key while building a stack of nodes we
+ * visited.
+ */
+ allocnode = tree->ctl->root;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift = node->shift;
+ level = -1;
+ while (shift > 0)
+ {
+ RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+ /* Push the current node to the stack */
+ stack[++level] = allocnode;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ allocnode = child;
+ shift -= RT_NODE_SPAN;
+ }
+
+ /* Delete the key from the leaf node if exists */
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ deleted = RT_NODE_DELETE_LEAF(node, key);
+
+ if (!deleted)
+ {
+ /* no key is found in the leaf node */
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ /* Found the key to delete. Update the statistics */
+ tree->ctl->num_keys--;
+
+ /*
+ * Return if the leaf node still has keys and we don't need to delete the
+ * node.
+ */
+ if (node->count > 0)
+ {
+ RT_UNLOCK(tree);
+ return true;
+ }
+
+ /* Free the empty leaf node */
+ RT_FREE_NODE(tree, allocnode);
+
+ /* Delete the key in inner nodes recursively */
+ while (level >= 0)
+ {
+ allocnode = stack[level--];
+
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ deleted = RT_NODE_DELETE_INNER(node, key);
+ Assert(deleted);
+
+ /* If the node didn't become empty, we stop deleting the key */
+ if (node->count > 0)
+ break;
+
+ /* The node became empty */
+ RT_FREE_NODE(tree, allocnode);
+ }
+
+ RT_UNLOCK(tree);
+ return true;
+}
+#endif
+
+static inline void
+RT_ITER_UPDATE_KEY(RT_ITER *iter, uint8 chunk, uint8 shift)
+{
+ iter->key &= ~(((uint64) RT_CHUNK_MASK) << shift);
+ iter->key |= (((uint64) chunk) << shift);
+}
+
+/*
+ * Advance the slot in the inner node. Return the child if exists, otherwise
+ * null.
+ */
+static inline RT_PTR_LOCAL
+RT_NODE_INNER_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Advance the slot in the leaf node. On success, return true and the value
+ * is set to value_p, otherwise return false.
+ */
+static inline bool
+RT_NODE_LEAF_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter,
+ RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Update each node_iter for inner nodes in the iterator node stack.
+ */
+static void
+RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
+{
+ int level = from;
+ RT_PTR_LOCAL node = from_node;
+
+ for (;;)
+ {
+ RT_NODE_ITER *node_iter = &(iter->stack[level--]);
+
+ node_iter->node = node;
+ node_iter->current_idx = -1;
+
+ /* We don't advance the leaf node iterator here */
+ if (RT_NODE_IS_LEAF(node))
+ return;
+
+ /* Advance to the next slot in the inner node */
+ node = RT_NODE_INNER_ITERATE_NEXT(iter, node_iter);
+
+ /* We must find the first children in the node */
+ Assert(node);
+ }
+}
+
+/*
+ * Create and return the iterator for the given radix tree.
+ *
+ * The radix tree is locked in shared mode during the iteration, so
+ * RT_END_ITERATE needs to be called when finished to release the lock.
+ */
+RT_SCOPE RT_ITER *
+RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
+{
+ MemoryContext old_ctx;
+ RT_ITER *iter;
+ RT_PTR_LOCAL root;
+ int top_level;
+
+ old_ctx = MemoryContextSwitchTo(tree->context);
+
+ iter = (RT_ITER *) palloc0(sizeof(RT_ITER));
+ iter->tree = tree;
+
+ RT_LOCK_SHARED(tree);
+
+ /* empty tree */
+ if (!iter->tree->ctl->root)
+ return iter;
+
+ root = RT_PTR_GET_LOCAL(tree, iter->tree->ctl->root);
+ top_level = root->shift / RT_NODE_SPAN;
+ iter->stack_len = top_level;
+
+ /*
+ * Descend to the left most leaf node from the root. The key is being
+ * constructed while descending to the leaf.
+ */
+ RT_UPDATE_ITER_STACK(iter, root, top_level);
+
+ MemoryContextSwitchTo(old_ctx);
+
+ return iter;
+}
+
+/*
+ * Return true with setting key_p and value_p if there is next key. Otherwise
+ * return false.
+ */
+RT_SCOPE bool
+RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p)
+{
+ /* Empty tree */
+ if (!iter->tree->ctl->root)
+ return false;
+
+ for (;;)
+ {
+ RT_PTR_LOCAL child = NULL;
+ RT_VALUE_TYPE value;
+ int level;
+ bool found;
+
+ /* Advance the leaf node iterator to get next key-value pair */
+ found = RT_NODE_LEAF_ITERATE_NEXT(iter, &(iter->stack[0]), &value);
+
+ if (found)
+ {
+ *key_p = iter->key;
+ *value_p = value;
+ return true;
+ }
+
+ /*
+ * We've visited all values in the leaf node, so advance inner node
+ * iterators from the level=1 until we find the next child node.
+ */
+ for (level = 1; level <= iter->stack_len; level++)
+ {
+ child = RT_NODE_INNER_ITERATE_NEXT(iter, &(iter->stack[level]));
+
+ if (child)
+ break;
+ }
+
+ /* the iteration finished */
+ if (!child)
+ return false;
+
+ /*
+ * Set the node to the node iterator and update the iterator stack
+ * from this node.
+ */
+ RT_UPDATE_ITER_STACK(iter, child, level - 1);
+
+ /* Node iterators are updated, so try again from the leaf */
+ }
+
+ return false;
+}
+
+/*
+ * Terminate the iteration and release the lock.
+ *
+ * This function needs to be called after finishing or when exiting an
+ * iteration.
+ */
+RT_SCOPE void
+RT_END_ITERATE(RT_ITER *iter)
+{
+#ifdef RT_SHMEM
+ Assert(LWLockHeldByMe(&iter->tree->ctl->lock));
+#endif
+
+ RT_UNLOCK(iter->tree);
+ pfree(iter);
+}
+
+/*
+ * Return the statistics of the amount of memory used by the radix tree.
+ */
+RT_SCOPE uint64
+RT_MEMORY_USAGE(RT_RADIX_TREE *tree)
+{
+ Size total = 0;
+
+ RT_LOCK_SHARED(tree);
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ total = dsa_get_total_size(tree->dsa);
+#else
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ total += MemoryContextMemAllocated(tree->inner_slabs[i], true);
+ total += MemoryContextMemAllocated(tree->leaf_slabs[i], true);
+ }
+#endif
+
+ RT_UNLOCK(tree);
+ return total;
+}
+
+/*
+ * Verify the radix tree node.
+ */
+static void
+RT_VERIFY_NODE(RT_PTR_LOCAL node)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(node->count >= 0);
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE_BASE_3 *n3 = (RT_NODE_BASE_3 *) node;
+
+ for (int i = 1; i < n3->n.count; i++)
+ Assert(n3->chunks[i - 1] < n3->chunks[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE_BASE_32 *n32 = (RT_NODE_BASE_32 *) node;
+
+ for (int i = 1; i < n32->n.count; i++)
+ Assert(n32->chunks[i - 1] < n32->chunks[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+ int cnt = 0;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ uint8 slot = n125->slot_idxs[i];
+ int idx = RT_BM_IDX(slot);
+ int bitnum = RT_BM_BIT(slot);
+
+ if (!RT_NODE_125_IS_CHUNK_USED(n125, i))
+ continue;
+
+ /* Check if the corresponding slot is used */
+ Assert(slot < node->fanout);
+ Assert((n125->isset[idx] & ((bitmapword) 1 << bitnum)) != 0);
+
+ cnt++;
+ }
+
+ Assert(n125->n.count == cnt);
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+ int cnt = 0;
+
+ for (int i = 0; i < RT_BM_IDX(RT_NODE_MAX_SLOTS); i++)
+ cnt += bmw_popcount(n256->isset[i]);
+
+ /* Check if the number of used chunk matches */
+ Assert(n256->base.n.count == cnt);
+
+ break;
+ }
+ }
+ }
+#endif
+}
+
+/***************** DEBUG FUNCTIONS *****************/
+#ifdef RT_DEBUG
+
+#define RT_UINT64_FORMAT_HEX "%" INT64_MODIFIER "X"
+
+RT_SCOPE void
+RT_STATS(RT_RADIX_TREE *tree)
+{
+ RT_LOCK_SHARED(tree);
+
+ fprintf(stderr, "max_val = " UINT64_FORMAT "\n", tree->ctl->max_val);
+ fprintf(stderr, "num_keys = " UINT64_FORMAT "\n", tree->ctl->num_keys);
+
+#ifdef RT_SHMEM
+ fprintf(stderr, "handle = " UINT64_FORMAT "\n", tree->ctl->handle);
+#endif
+
+ if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+
+ fprintf(stderr, "height = %d, n3 = %u, n15 = %u, n32 = %u, n125 = %u, n256 = %u\n",
+ root->shift / RT_NODE_SPAN,
+ tree->ctl->cnt[RT_CLASS_3],
+ tree->ctl->cnt[RT_CLASS_32_MIN],
+ tree->ctl->cnt[RT_CLASS_32_MAX],
+ tree->ctl->cnt[RT_CLASS_125],
+ tree->ctl->cnt[RT_CLASS_256]);
+ }
+
+ RT_UNLOCK(tree);
+}
+
+static void
+RT_DUMP_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, int level,
+ bool recurse, StringInfo buf)
+{
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+ StringInfoData spaces;
+
+ initStringInfo(&spaces);
+ appendStringInfoSpaces(&spaces, (level * 4) + 1);
+
+ appendStringInfo(buf, "%s%s[%s] kind %d, fanout %d, count %u, shift %u:\n",
+ spaces.data,
+ level == 0 ? "" : "-> ",
+ RT_NODE_IS_LEAF(node) ? "LEAF" : "INNR",
+ (node->kind == RT_NODE_KIND_3) ? 3 :
+ (node->kind == RT_NODE_KIND_32) ? 32 :
+ (node->kind == RT_NODE_KIND_125) ? 125 : 256,
+ node->fanout == 0 ? 256 : node->fanout,
+ node->count, node->shift);
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ for (int i = 0; i < node->count; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_3 *n3 = (RT_NODE_LEAF_3 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+ spaces.data, i, n3->base.chunks[i]);
+ }
+ else
+ {
+ RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X",
+ spaces.data, i, n3->base.chunks[i]);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, n3->children[i], level + 1,
+ recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ for (int i = 0; i < node->count; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_32 *n32 = (RT_NODE_LEAF_32 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+ spaces.data, i, n32->base.chunks[i]);
+ }
+ else
+ {
+ RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X",
+ spaces.data, i, n32->base.chunks[i]);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, n32->children[i], level + 1,
+ recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_BASE_125 *b125 = (RT_NODE_BASE_125 *) node;
+ char *sep = "";
+
+ appendStringInfo(buf, "%sslot_idxs: ", spaces.data);
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+ continue;
+
+ appendStringInfo(buf, "%s[%d]=%d ",
+ sep, i, b125->slot_idxs[i]);
+ sep = ",";
+ }
+
+ appendStringInfo(buf, "\n%sisset-bitmap: ", spaces.data);
+ for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+ appendStringInfo(buf, "%X ", ((uint8 *) b125->isset)[i]);
+ appendStringInfo(buf, "\n");
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+ continue;
+
+ if (RT_NODE_IS_LEAF(node))
+ appendStringInfo(buf, "%schunk 0x%X\n",
+ spaces.data, i);
+ else
+ {
+ RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) b125;
+
+ appendStringInfo(buf, "%schunk 0x%X",
+ spaces.data, i);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i),
+ level + 1, recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+ appendStringInfo(buf, "%sisset-bitmap: ", spaces.data);
+ for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+ appendStringInfo(buf, "%X ", ((uint8 *) n256->isset)[i]);
+ appendStringInfo(buf, "\n");
+ }
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ appendStringInfo(buf, "%schunk 0x%X\n",
+ spaces.data, i);
+ }
+ else
+ {
+ RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ appendStringInfo(buf, "%schunk 0x%X",
+ spaces.data, i);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i),
+ level + 1, recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ }
+}
+
+RT_SCOPE void
+RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key)
+{
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL node;
+ StringInfoData buf;
+ int shift;
+ int level = 0;
+
+ RT_STATS(tree);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "empty tree\n");
+ return;
+ }
+
+ if (key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "key " UINT64_FORMAT "(0x" RT_UINT64_FORMAT_HEX ") is larger than max val\n",
+ key, key);
+ return;
+ }
+
+ initStringInfo(&buf);
+ allocnode = tree->ctl->root;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift = node->shift;
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC child;
+
+ RT_DUMP_NODE(tree, allocnode, level, false, &buf);
+
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_VALUE_TYPE dummy;
+
+ /* We reached at a leaf node, find the corresponding slot */
+ RT_NODE_SEARCH_LEAF(node, key, &dummy);
+
+ break;
+ }
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ break;
+
+ allocnode = child;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift -= RT_NODE_SPAN;
+ level++;
+ }
+ RT_UNLOCK(tree);
+
+ fprintf(stderr, "%s", buf.data);
+}
+
+RT_SCOPE void
+RT_DUMP(RT_RADIX_TREE *tree)
+{
+ StringInfoData buf;
+
+ RT_STATS(tree);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "empty tree\n");
+ return;
+ }
+
+ initStringInfo(&buf);
+
+ RT_DUMP_NODE(tree, tree->ctl->root, 0, true, &buf);
+ RT_UNLOCK(tree);
+
+ fprintf(stderr, "%s",buf.data);
+}
+#endif
+
+#endif /* RT_DEFINE */
+
+
+/* undefine external parameters, so next radix tree can be defined */
+#undef RT_PREFIX
+#undef RT_SCOPE
+#undef RT_DECLARE
+#undef RT_DEFINE
+#undef RT_VALUE_TYPE
+
+/* locally declared macros */
+#undef RT_MAKE_PREFIX
+#undef RT_MAKE_NAME
+#undef RT_MAKE_NAME_
+#undef RT_NODE_SPAN
+#undef RT_NODE_MAX_SLOTS
+#undef RT_CHUNK_MASK
+#undef RT_MAX_SHIFT
+#undef RT_MAX_LEVEL
+#undef RT_GET_KEY_CHUNK
+#undef RT_BM_IDX
+#undef RT_BM_BIT
+#undef RT_LOCK_EXCLUSIVE
+#undef RT_LOCK_SHARED
+#undef RT_UNLOCK
+#undef RT_NODE_IS_LEAF
+#undef RT_NODE_MUST_GROW
+#undef RT_NODE_KIND_COUNT
+#undef RT_SIZE_CLASS_COUNT
+#undef RT_SLOT_IDX_LIMIT
+#undef RT_INVALID_SLOT_IDX
+#undef RT_SLAB_BLOCK_SIZE
+#undef RT_RADIX_TREE_MAGIC
+#undef RT_UINT64_FORMAT_HEX
+
+/* type declarations */
+#undef RT_RADIX_TREE
+#undef RT_RADIX_TREE_CONTROL
+#undef RT_PTR_LOCAL
+#undef RT_PTR_ALLOC
+#undef RT_INVALID_PTR_ALLOC
+#undef RT_HANDLE
+#undef RT_ITER
+#undef RT_NODE
+#undef RT_NODE_ITER
+#undef RT_NODE_KIND_3
+#undef RT_NODE_KIND_32
+#undef RT_NODE_KIND_125
+#undef RT_NODE_KIND_256
+#undef RT_NODE_BASE_3
+#undef RT_NODE_BASE_32
+#undef RT_NODE_BASE_125
+#undef RT_NODE_BASE_256
+#undef RT_NODE_INNER_3
+#undef RT_NODE_INNER_32
+#undef RT_NODE_INNER_125
+#undef RT_NODE_INNER_256
+#undef RT_NODE_LEAF_3
+#undef RT_NODE_LEAF_32
+#undef RT_NODE_LEAF_125
+#undef RT_NODE_LEAF_256
+#undef RT_SIZE_CLASS
+#undef RT_SIZE_CLASS_ELEM
+#undef RT_SIZE_CLASS_INFO
+#undef RT_CLASS_3
+#undef RT_CLASS_32_MIN
+#undef RT_CLASS_32_MAX
+#undef RT_CLASS_125
+#undef RT_CLASS_256
+
+/* function declarations */
+#undef RT_CREATE
+#undef RT_FREE
+#undef RT_ATTACH
+#undef RT_DETACH
+#undef RT_GET_HANDLE
+#undef RT_SEARCH
+#undef RT_SET
+#undef RT_BEGIN_ITERATE
+#undef RT_ITERATE_NEXT
+#undef RT_END_ITERATE
+#undef RT_USE_DELETE
+#undef RT_DELETE
+#undef RT_MEMORY_USAGE
+#undef RT_DUMP
+#undef RT_DUMP_NODE
+#undef RT_DUMP_SEARCH
+#undef RT_STATS
+
+/* internal helper functions */
+#undef RT_NEW_ROOT
+#undef RT_ALLOC_NODE
+#undef RT_INIT_NODE
+#undef RT_FREE_NODE
+#undef RT_FREE_RECURSE
+#undef RT_EXTEND
+#undef RT_SET_EXTEND
+#undef RT_SWITCH_NODE_KIND
+#undef RT_COPY_NODE
+#undef RT_REPLACE_NODE
+#undef RT_PTR_GET_LOCAL
+#undef RT_PTR_ALLOC_IS_VALID
+#undef RT_NODE_3_SEARCH_EQ
+#undef RT_NODE_32_SEARCH_EQ
+#undef RT_NODE_3_GET_INSERTPOS
+#undef RT_NODE_32_GET_INSERTPOS
+#undef RT_CHUNK_CHILDREN_ARRAY_SHIFT
+#undef RT_CHUNK_VALUES_ARRAY_SHIFT
+#undef RT_CHUNK_CHILDREN_ARRAY_DELETE
+#undef RT_CHUNK_VALUES_ARRAY_DELETE
+#undef RT_CHUNK_CHILDREN_ARRAY_COPY
+#undef RT_CHUNK_VALUES_ARRAY_COPY
+#undef RT_NODE_125_IS_CHUNK_USED
+#undef RT_NODE_INNER_125_GET_CHILD
+#undef RT_NODE_LEAF_125_GET_VALUE
+#undef RT_NODE_INNER_256_IS_CHUNK_USED
+#undef RT_NODE_LEAF_256_IS_CHUNK_USED
+#undef RT_NODE_INNER_256_GET_CHILD
+#undef RT_NODE_LEAF_256_GET_VALUE
+#undef RT_NODE_INNER_256_SET
+#undef RT_NODE_LEAF_256_SET
+#undef RT_NODE_INNER_256_DELETE
+#undef RT_NODE_LEAF_256_DELETE
+#undef RT_KEY_GET_SHIFT
+#undef RT_SHIFT_GET_MAX_VAL
+#undef RT_NODE_SEARCH_INNER
+#undef RT_NODE_SEARCH_LEAF
+#undef RT_NODE_UPDATE_INNER
+#undef RT_NODE_DELETE_INNER
+#undef RT_NODE_DELETE_LEAF
+#undef RT_NODE_INSERT_INNER
+#undef RT_NODE_INSERT_LEAF
+#undef RT_NODE_INNER_ITERATE_NEXT
+#undef RT_NODE_LEAF_ITERATE_NEXT
+#undef RT_UPDATE_ITER_STACK
+#undef RT_ITER_UPDATE_KEY
+#undef RT_VERIFY_NODE
+
+#undef RT_DEBUG
diff --git a/src/include/lib/radixtree_delete_impl.h b/src/include/lib/radixtree_delete_impl.h
new file mode 100644
index 0000000000..5f6dda1f12
--- /dev/null
+++ b/src/include/lib/radixtree_delete_impl.h
@@ -0,0 +1,122 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_delete_impl.h
+ * Common implementation for deletion in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ * TODO: Shrink nodes when deletion would allow them to fit in a smaller
+ * size class.
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_delete_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_DELETE(n3->base.chunks, n3->values,
+ n3->base.n.count, idx);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_DELETE(n3->base.chunks, n3->children,
+ n3->base.n.count, idx);
+#endif
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_DELETE(n32->base.chunks, n32->values,
+ n32->base.n.count, idx);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_DELETE(n32->base.chunks, n32->children,
+ n32->base.n.count, idx);
+#endif
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+ int idx;
+ int bitnum;
+
+ if (slotpos == RT_INVALID_SLOT_IDX)
+ return false;
+
+ idx = RT_BM_IDX(slotpos);
+ bitnum = RT_BM_BIT(slotpos);
+ n125->base.isset[idx] &= ~((bitmapword) 1 << bitnum);
+ n125->base.slot_idxs[chunk] = RT_INVALID_SLOT_IDX;
+
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_DELETE(n256, chunk);
+#else
+ RT_NODE_INNER_256_DELETE(n256, chunk);
+#endif
+ break;
+ }
+ }
+
+ /* update statistics */
+ node->count--;
+
+ return true;
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
new file mode 100644
index 0000000000..d56e58dcac
--- /dev/null
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -0,0 +1,328 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_insert_impl.h
+ * Common implementation for insertion in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_insert_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ const bool is_leaf = true;
+ bool chunk_exists = false;
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+ const bool is_leaf = false;
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ int idx = RT_NODE_3_SEARCH_EQ(&n3->base, chunk);
+
+ if (idx != -1)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n3->values[idx] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n3)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE32_TYPE *new32;
+ const uint8 new_kind = RT_NODE_KIND_32;
+ const RT_SIZE_CLASS new_class = RT_CLASS_32_MIN;
+
+ /* grow node from 3 to 32 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_COPY(n3->base.chunks, n3->values,
+ new32->base.chunks, new32->values);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_COPY(n3->base.chunks, n3->children,
+ new32->base.chunks, new32->children);
+#endif
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
+ int count = n3->base.n.count;
+
+ /* shift chunks and children */
+ if (insertpos < count)
+ {
+ Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_SHIFT(n3->base.chunks, n3->values,
+ count, insertpos);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_SHIFT(n3->base.chunks, n3->children,
+ count, insertpos);
+#endif
+ }
+
+ n3->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+ n3->values[insertpos] = *value_p;
+#else
+ n3->children[insertpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_32:
+ {
+ const RT_SIZE_CLASS_ELEM class32_max = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX];
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ int idx = RT_NODE_32_SEARCH_EQ(&n32->base, chunk);
+
+ if (idx != -1)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n32->values[idx] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n32)) &&
+ n32->base.n.fanout < class32_max.fanout)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ const RT_SIZE_CLASS_ELEM class32_min = RT_SIZE_CLASS_INFO[RT_CLASS_32_MIN];
+ const RT_SIZE_CLASS new_class = RT_CLASS_32_MAX;
+
+ Assert(n32->base.n.fanout == class32_min.fanout);
+
+ /* grow to the next size class of this kind */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ n32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ memcpy(newnode, node, class32_min.leaf_size);
+#else
+ memcpy(newnode, node, class32_min.inner_size);
+#endif
+ newnode->fanout = class32_max.fanout;
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+
+ if (unlikely(RT_NODE_MUST_GROW(n32)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE125_TYPE *new125;
+ const uint8 new_kind = RT_NODE_KIND_125;
+ const RT_SIZE_CLASS new_class = RT_CLASS_125;
+
+ Assert(n32->base.n.fanout == class32_max.fanout);
+
+ /* grow node from 32 to 125 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new125 = (RT_NODE125_TYPE *) newnode;
+
+ for (int i = 0; i < class32_max.fanout; i++)
+ {
+ new125->base.slot_idxs[n32->base.chunks[i]] = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ new125->values[i] = n32->values[i];
+#else
+ new125->children[i] = n32->children[i];
+#endif
+ }
+
+ /*
+ * Since we just copied a dense array, we can set the bits
+ * using a single store, provided the length of that array
+ * is at most the number of bits in a bitmapword.
+ */
+ Assert(class32_max.fanout <= sizeof(bitmapword) * BITS_PER_BYTE);
+ new125->base.isset[0] = (bitmapword) (((uint64) 1 << class32_max.fanout) - 1);
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
+ int count = n32->base.n.count;
+
+ if (insertpos < count)
+ {
+ Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_SHIFT(n32->base.chunks, n32->values,
+ count, insertpos);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_SHIFT(n32->base.chunks, n32->children,
+ count, insertpos);
+#endif
+ }
+
+ n32->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+ n32->values[insertpos] = *value_p;
+#else
+ n32->children[insertpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos;
+ int cnt = 0;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ slotpos = n125->base.slot_idxs[chunk];
+ if (slotpos != RT_INVALID_SLOT_IDX)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n125->values[slotpos] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n125)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE256_TYPE *new256;
+ const uint8 new_kind = RT_NODE_KIND_256;
+ const RT_SIZE_CLASS new_class = RT_CLASS_256;
+
+ /* grow node from 125 to 256 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new256 = (RT_NODE256_TYPE *) newnode;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS && cnt < n125->base.n.count; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+ continue;
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_SET(new256, i, RT_NODE_LEAF_125_GET_VALUE(n125, i));
+#else
+ RT_NODE_INNER_256_SET(new256, i, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+ cnt++;
+ }
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int idx;
+ bitmapword inverse;
+
+ /* get the first word with at least one bit not set */
+ for (idx = 0; idx < RT_BM_IDX(RT_SLOT_IDX_LIMIT); idx++)
+ {
+ if (n125->base.isset[idx] < ~((bitmapword) 0))
+ break;
+ }
+
+ /* To get the first unset bit in X, get the first set bit in ~X */
+ inverse = ~(n125->base.isset[idx]);
+ slotpos = idx * BITS_PER_BITMAPWORD;
+ slotpos += bmw_rightmost_one_pos(inverse);
+ Assert(slotpos < node->fanout);
+
+ /* mark the slot used */
+ n125->base.isset[idx] |= bmw_rightmost_one(inverse);
+ n125->base.slot_idxs[chunk] = slotpos;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ n125->values[slotpos] = *value_p;
+#else
+ n125->children[slotpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ chunk_exists = RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk);
+ Assert(chunk_exists || node->count < RT_NODE_MAX_SLOTS);
+ RT_NODE_LEAF_256_SET(n256, chunk, *value_p);
+#else
+ Assert(node->count < RT_NODE_MAX_SLOTS);
+ RT_NODE_INNER_256_SET(n256, chunk, child);
+#endif
+ break;
+ }
+ }
+
+ /* Update statistics */
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!chunk_exists)
+ node->count++;
+#else
+ node->count++;
+#endif
+
+ /*
+ * Done. Finally, verify the chunk and value is inserted or replaced
+ * properly in the node.
+ */
+ RT_VERIFY_NODE(node);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ return chunk_exists;
+#else
+ return;
+#endif
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_iter_impl.h b/src/include/lib/radixtree_iter_impl.h
new file mode 100644
index 0000000000..98c78eb237
--- /dev/null
+++ b/src/include/lib/radixtree_iter_impl.h
@@ -0,0 +1,153 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_iter_impl.h
+ * Common implementation for iteration in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_iter_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ bool found = false;
+ uint8 key_chunk;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_VALUE_TYPE value;
+
+ Assert(RT_NODE_IS_LEAF(node_iter->node));
+#else
+ RT_PTR_LOCAL child = NULL;
+
+ Assert(!RT_NODE_IS_LEAF(node_iter->node));
+#endif
+
+#ifdef RT_SHMEM
+ Assert(iter->tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ switch (node_iter->node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node_iter->node;
+
+ node_iter->current_idx++;
+ if (node_iter->current_idx >= n3->base.n.count)
+ break;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = n3->values[node_iter->current_idx];
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, n3->children[node_iter->current_idx]);
+#endif
+ key_chunk = n3->base.chunks[node_iter->current_idx];
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node_iter->node;
+
+ node_iter->current_idx++;
+ if (node_iter->current_idx >= n32->base.n.count)
+ break;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ value = n32->values[node_iter->current_idx];
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, n32->children[node_iter->current_idx]);
+#endif
+ key_chunk = n32->base.chunks[node_iter->current_idx];
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node_iter->node;
+ int i;
+
+ for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (RT_NODE_125_IS_CHUNK_USED((RT_NODE_BASE_125 *) n125, i))
+ break;
+ }
+
+ if (i >= RT_NODE_MAX_SLOTS)
+ break;
+
+ node_iter->current_idx = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = RT_NODE_LEAF_125_GET_VALUE(n125, i);
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+ key_chunk = i;
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node_iter->node;
+ int i;
+
+ for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ {
+#ifdef RT_NODE_LEVEL_LEAF
+ if (RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+#else
+ if (RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+#endif
+ break;
+ }
+
+ if (i >= RT_NODE_MAX_SLOTS)
+ break;
+
+ node_iter->current_idx = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = RT_NODE_LEAF_256_GET_VALUE(n256, i);
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+#endif
+ key_chunk = i;
+ found = true;
+ break;
+ }
+ }
+
+ if (found)
+ {
+ RT_ITER_UPDATE_KEY(iter, key_chunk, node_iter->node->shift);
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = value;
+#endif
+ }
+
+#ifdef RT_NODE_LEVEL_LEAF
+ return found;
+#else
+ return child;
+#endif
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_search_impl.h b/src/include/lib/radixtree_search_impl.h
new file mode 100644
index 0000000000..a8925c75d0
--- /dev/null
+++ b/src/include/lib/radixtree_search_impl.h
@@ -0,0 +1,138 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_search_impl.h
+ * Common implementation for search in leaf and inner nodes, plus
+ * update for inner nodes only.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_search_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(value_p != NULL);
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+#ifndef RT_ACTION_UPDATE
+ Assert(child_p != NULL);
+#endif
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+#ifdef RT_ACTION_UPDATE
+ Assert(idx >= 0);
+ n3->children[idx] = new_child;
+#else
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n3->values[idx];
+#else
+ *child_p = n3->children[idx];
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+#ifdef RT_ACTION_UPDATE
+ Assert(idx >= 0);
+ n32->children[idx] = new_child;
+#else
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n32->values[idx];
+#else
+ *child_p = n32->children[idx];
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+
+#ifdef RT_ACTION_UPDATE
+ Assert(slotpos != RT_INVALID_SLOT_IDX);
+ n125->children[slotpos] = new_child;
+#else
+ if (slotpos == RT_INVALID_SLOT_IDX)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_125_GET_VALUE(n125, chunk);
+#else
+ *child_p = RT_NODE_INNER_125_GET_CHILD(n125, chunk);
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_ACTION_UPDATE
+ RT_NODE_INNER_256_SET(n256, chunk, new_child);
+#else
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_256_GET_VALUE(n256, chunk);
+#else
+ *child_p = RT_NODE_INNER_256_GET_CHILD(n256, chunk);
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ }
+
+#ifdef RT_ACTION_UPDATE
+ return;
+#else
+ return true;
+#endif /* RT_ACTION_UPDATE */
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/utils/dsa.h b/src/include/utils/dsa.h
index 3ce4ee300a..2af215484f 100644
--- a/src/include/utils/dsa.h
+++ b/src/include/utils/dsa.h
@@ -121,6 +121,7 @@ extern dsa_handle dsa_get_handle(dsa_area *area);
extern dsa_pointer dsa_allocate_extended(dsa_area *area, size_t size, int flags);
extern void dsa_free(dsa_area *area, dsa_pointer dp);
extern void *dsa_get_address(dsa_area *area, dsa_pointer dp);
+extern size_t dsa_get_total_size(dsa_area *area);
extern void dsa_trim(dsa_area *area);
extern void dsa_dump(dsa_area *area);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 79e3033ec2..89f42bf9e3 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -28,6 +28,7 @@ SUBDIRS = \
test_pg_db_role_setting \
test_pg_dump \
test_predtest \
+ test_radixtree \
test_rbtree \
test_regex \
test_rls_hooks \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index dcb82ed68f..beaf4080fb 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -25,6 +25,7 @@ subdir('test_parser')
subdir('test_pg_db_role_setting')
subdir('test_pg_dump')
subdir('test_predtest')
+subdir('test_radixtree')
subdir('test_rbtree')
subdir('test_regex')
subdir('test_rls_hooks')
diff --git a/src/test/modules/test_radixtree/.gitignore b/src/test/modules/test_radixtree/.gitignore
new file mode 100644
index 0000000000..5dcb3ff972
--- /dev/null
+++ b/src/test/modules/test_radixtree/.gitignore
@@ -0,0 +1,4 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
diff --git a/src/test/modules/test_radixtree/Makefile b/src/test/modules/test_radixtree/Makefile
new file mode 100644
index 0000000000..da06b93da3
--- /dev/null
+++ b/src/test/modules/test_radixtree/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_radixtree/Makefile
+
+MODULE_big = test_radixtree
+OBJS = \
+ $(WIN32RES) \
+ test_radixtree.o
+PGFILEDESC = "test_radixtree - test code for src/backend/lib/radixtree.c"
+
+EXTENSION = test_radixtree
+DATA = test_radixtree--1.0.sql
+
+REGRESS = test_radixtree
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_radixtree
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_radixtree/README b/src/test/modules/test_radixtree/README
new file mode 100644
index 0000000000..a8b271869a
--- /dev/null
+++ b/src/test/modules/test_radixtree/README
@@ -0,0 +1,7 @@
+test_integerset contains unit tests for testing the integer set implementation
+in src/backend/lib/integerset.c.
+
+The tests verify the correctness of the implementation, but they can also be
+used as a micro-benchmark. If you set the 'intset_test_stats' flag in
+test_integerset.c, the tests will print extra information about execution time
+and memory usage.
diff --git a/src/test/modules/test_radixtree/expected/test_radixtree.out b/src/test/modules/test_radixtree/expected/test_radixtree.out
new file mode 100644
index 0000000000..ce645cb8b5
--- /dev/null
+++ b/src/test/modules/test_radixtree/expected/test_radixtree.out
@@ -0,0 +1,36 @@
+CREATE EXTENSION test_radixtree;
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
+NOTICE: testing basic operations with leaf node 4
+NOTICE: testing basic operations with inner node 4
+NOTICE: testing basic operations with leaf node 32
+NOTICE: testing basic operations with inner node 32
+NOTICE: testing basic operations with leaf node 125
+NOTICE: testing basic operations with inner node 125
+NOTICE: testing basic operations with leaf node 256
+NOTICE: testing basic operations with inner node 256
+NOTICE: testing radix tree node types with shift "0"
+NOTICE: testing radix tree node types with shift "8"
+NOTICE: testing radix tree node types with shift "16"
+NOTICE: testing radix tree node types with shift "24"
+NOTICE: testing radix tree node types with shift "32"
+NOTICE: testing radix tree node types with shift "40"
+NOTICE: testing radix tree node types with shift "48"
+NOTICE: testing radix tree node types with shift "56"
+NOTICE: testing radix tree with pattern "all ones"
+NOTICE: testing radix tree with pattern "alternating bits"
+NOTICE: testing radix tree with pattern "clusters of ten"
+NOTICE: testing radix tree with pattern "clusters of hundred"
+NOTICE: testing radix tree with pattern "one-every-64k"
+NOTICE: testing radix tree with pattern "sparse"
+NOTICE: testing radix tree with pattern "single values, distance > 2^32"
+NOTICE: testing radix tree with pattern "clusters, distance > 2^32"
+NOTICE: testing radix tree with pattern "clusters, distance > 2^60"
+ test_radixtree
+----------------
+
+(1 row)
+
diff --git a/src/test/modules/test_radixtree/meson.build b/src/test/modules/test_radixtree/meson.build
new file mode 100644
index 0000000000..6add06bbdb
--- /dev/null
+++ b/src/test/modules/test_radixtree/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_radixtree_sources = files(
+ 'test_radixtree.c',
+)
+
+if host_system == 'windows'
+ test_radixtree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_radixtree',
+ '--FILEDESC', 'test_radixtree - test code for src/include//lib/radixtree.h',])
+endif
+
+test_radixtree = shared_module('test_radixtree',
+ test_radixtree_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += test_radixtree
+
+install_data(
+ 'test_radixtree.control',
+ 'test_radixtree--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'test_radixtree',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'test_radixtree',
+ ],
+ },
+}
diff --git a/src/test/modules/test_radixtree/sql/test_radixtree.sql b/src/test/modules/test_radixtree/sql/test_radixtree.sql
new file mode 100644
index 0000000000..41ece5e9f5
--- /dev/null
+++ b/src/test/modules/test_radixtree/sql/test_radixtree.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_radixtree;
+
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
diff --git a/src/test/modules/test_radixtree/test_radixtree--1.0.sql b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
new file mode 100644
index 0000000000..074a5a7ea7
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_radixtree/test_radixtree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_radixtree" to load this file. \quit
+
+CREATE FUNCTION test_radixtree()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
new file mode 100644
index 0000000000..afe53382f3
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -0,0 +1,681 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_radixtree.c
+ * Test radixtree set data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_radixtree/test_radixtree.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "storage/lwlock.h"
+#include "utils/memutils.h"
+#include "utils/timestamp.h"
+
+#define UINT64_HEX_FORMAT "%" INT64_MODIFIER "X"
+
+/*
+ * The tests pass with uint32, but build with warnings because the string
+ * format expects uint64.
+ */
+typedef uint64 TestValueType;
+
+/*
+ * If you enable this, the "pattern" tests will print information about
+ * how long populating, probing, and iterating the test set takes, and
+ * how much memory the test set consumed. That can be used as
+ * micro-benchmark of various operations and input patterns (you might
+ * want to increase the number of values used in each of the test, if
+ * you do that, to reduce noise).
+ *
+ * The information is printed to the server's stderr, mostly because
+ * that's where MemoryContextStats() output goes.
+ */
+static const bool rt_test_stats = false;
+
+static int rt_node_kind_fanouts[] = {
+ 0,
+ 4, /* RT_NODE_KIND_4 */
+ 32, /* RT_NODE_KIND_32 */
+ 125, /* RT_NODE_KIND_125 */
+ 256 /* RT_NODE_KIND_256 */
+};
+/*
+ * A struct to define a pattern of integers, for use with the test_pattern()
+ * function.
+ */
+typedef struct
+{
+ char *test_name; /* short name of the test, for humans */
+ char *pattern_str; /* a bit pattern */
+ uint64 spacing; /* pattern repeats at this interval */
+ uint64 num_values; /* number of integers to set in total */
+} test_spec;
+
+/* Test patterns borrowed from test_integerset.c */
+static const test_spec test_specs[] = {
+ {
+ "all ones", "1111111111",
+ 10, 1000000
+ },
+ {
+ "alternating bits", "0101010101",
+ 10, 1000000
+ },
+ {
+ "clusters of ten", "1111111111",
+ 10000, 1000000
+ },
+ {
+ "clusters of hundred",
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111",
+ 10000, 1000000
+ },
+ {
+ "one-every-64k", "1",
+ 65536, 1000000
+ },
+ {
+ "sparse", "100000000000000000000000000000001",
+ 10000000, 1000000
+ },
+ {
+ "single values, distance > 2^32", "1",
+ UINT64CONST(10000000000), 100000
+ },
+ {
+ "clusters, distance > 2^32", "10101010",
+ UINT64CONST(10000000000), 1000000
+ },
+ {
+ "clusters, distance > 2^60", "10101010",
+ UINT64CONST(2000000000000000000),
+ 23 /* can't be much higher than this, or we
+ * overflow uint64 */
+ }
+};
+
+/* define the radix tree implementation to test */
+#define RT_PREFIX rt
+#define RT_SCOPE
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE TestValueType
+/* #define RT_SHMEM */
+#include "lib/radixtree.h"
+
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+ return tree->ctl->num_keys;
+}
+
+PG_MODULE_MAGIC;
+
+PG_FUNCTION_INFO_V1(test_radixtree);
+
+static void
+test_empty(void)
+{
+ rt_radix_tree *radixtree;
+ rt_iter *iter;
+ TestValueType dummy;
+ uint64 key;
+ TestValueType val;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
+ dsa = dsa_create(tranche_id);
+
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ if (rt_search(radixtree, 0, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_search(radixtree, 1, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_search(radixtree, PG_UINT64_MAX, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_delete(radixtree, 0))
+ elog(ERROR, "rt_delete on empty tree returned true");
+
+ if (rt_num_entries(radixtree) != 0)
+ elog(ERROR, "rt_num_entries on empty tree return non-zero");
+
+ iter = rt_begin_iterate(radixtree);
+
+ if (rt_iterate_next(iter, &key, &val))
+ elog(ERROR, "rt_itereate_next on empty tree returned true");
+
+ rt_end_iterate(iter);
+
+ rt_free(radixtree);
+
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+static void
+test_basic(int children, bool test_inner)
+{
+ rt_radix_tree *radixtree;
+ uint64 *keys;
+ int shift = test_inner ? 8 : 0;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing basic operations with %s node %d",
+ test_inner ? "inner" : "leaf", children);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ /* prepare keys in order like 1, 32, 2, 31, 2, ... */
+ keys = palloc(sizeof(uint64) * children);
+ for (int i = 0; i < children; i++)
+ {
+ if (i % 2 == 0)
+ keys[i] = (uint64) ((i / 2) + 1) << shift;
+ else
+ keys[i] = (uint64) (children - (i / 2)) << shift;
+ }
+
+ /* insert keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (rt_set(radixtree, keys[i], (TestValueType*) &keys[i]))
+ elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+ }
+
+ /* look up keys */
+ for (int i = 0; i < children; i++)
+ {
+ TestValueType value;
+
+ if (!rt_search(radixtree, keys[i], &value))
+ elog(ERROR, "could not find key 0x" UINT64_HEX_FORMAT, keys[i]);
+ if (value != (TestValueType) keys[i])
+ elog(ERROR, "rt_search returned 0x" UINT64_HEX_FORMAT ", expected " UINT64_HEX_FORMAT,
+ value, (TestValueType) keys[i]);
+ }
+
+ /* update keys */
+ for (int i = 0; i < children; i++)
+ {
+ TestValueType update = keys[i] + 1;
+ if (!rt_set(radixtree, keys[i], (TestValueType*) &update))
+ elog(ERROR, "could not update key 0x" UINT64_HEX_FORMAT, keys[i]);
+ }
+
+ /* repeat deleting and inserting keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (!rt_delete(radixtree, keys[i]))
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, keys[i]);
+ if (rt_set(radixtree, keys[i], (TestValueType*) &keys[i]))
+ elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+ }
+
+ pfree(keys);
+ rt_free(radixtree);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Check if keys from start to end with the shift exist in the tree.
+ */
+static void
+check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end,
+ int incr)
+{
+ for (int i = start; i < end; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ TestValueType val;
+
+ if (!rt_search(radixtree, key, &val))
+ elog(ERROR, "key 0x" UINT64_HEX_FORMAT " is not found on node-%d",
+ key, end);
+ if (val != (TestValueType) key)
+ elog(ERROR, "rt_search with key 0x" UINT64_HEX_FORMAT " returns 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+ key, val, key);
+ }
+}
+
+static void
+test_node_types_insert(rt_radix_tree *radixtree, uint8 shift, bool insert_asc)
+{
+ uint64 num_entries;
+ int ninserted = 0;
+ int start = insert_asc ? 0 : 256;
+ int incr = insert_asc ? 1 : -1;
+ int end = insert_asc ? 256 : 0;
+ int node_kind_idx = 1;
+
+ for (int i = start; i != end; i += incr)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_set(radixtree, key, (TestValueType*) &key);
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " is found", key);
+
+ /*
+ * After filling all slots in each node type, check if the values
+ * are stored properly.
+ */
+ if (ninserted == rt_node_kind_fanouts[node_kind_idx] - 1)
+ {
+ int check_start = insert_asc
+ ? rt_node_kind_fanouts[node_kind_idx - 1]
+ : rt_node_kind_fanouts[node_kind_idx];
+ int check_end = insert_asc
+ ? rt_node_kind_fanouts[node_kind_idx]
+ : rt_node_kind_fanouts[node_kind_idx - 1];
+
+ check_search_on_node(radixtree, shift, check_start, check_end, incr);
+ node_kind_idx++;
+ }
+
+ ninserted++;
+ }
+
+ num_entries = rt_num_entries(radixtree);
+
+ if (num_entries != 256)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+static void
+test_node_types_delete(rt_radix_tree *radixtree, uint8 shift)
+{
+ uint64 num_entries;
+
+ for (int i = 0; i < 256; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_delete(radixtree, key);
+
+ if (!found)
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, key);
+ }
+
+ num_entries = rt_num_entries(radixtree);
+
+ /* The tree must be empty */
+ if (num_entries != 0)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+/*
+ * Test for inserting and deleting key-value pairs to each node type at the given shift
+ * level.
+ */
+static void
+test_node_types(uint8 shift)
+{
+ rt_radix_tree *radixtree;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing radix tree node types with shift \"%d\"", shift);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ /*
+ * Insert and search entries for every node type at the 'shift' level,
+ * then delete all entries to make it empty, and insert and search entries
+ * again.
+ */
+ test_node_types_insert(radixtree, shift, true);
+ test_node_types_delete(radixtree, shift);
+ test_node_types_insert(radixtree, shift, false);
+
+ rt_free(radixtree);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Test with a repeating pattern, defined by the 'spec'.
+ */
+static void
+test_pattern(const test_spec * spec)
+{
+ rt_radix_tree *radixtree;
+ rt_iter *iter;
+ MemoryContext radixtree_ctx;
+ TimestampTz starttime;
+ TimestampTz endtime;
+ uint64 n;
+ uint64 last_int;
+ uint64 ndeleted;
+ uint64 nbefore;
+ uint64 nafter;
+ int patternlen;
+ uint64 *pattern_values;
+ uint64 pattern_num_values;
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing radix tree with pattern \"%s\"", spec->test_name);
+ if (rt_test_stats)
+ fprintf(stderr, "-----\ntesting radix tree with pattern \"%s\"\n", spec->test_name);
+
+ /* Pre-process the pattern, creating an array of integers from it. */
+ patternlen = strlen(spec->pattern_str);
+ pattern_values = palloc(patternlen * sizeof(uint64));
+ pattern_num_values = 0;
+ for (int i = 0; i < patternlen; i++)
+ {
+ if (spec->pattern_str[i] == '1')
+ pattern_values[pattern_num_values++] = i;
+ }
+
+ /*
+ * Allocate the radix tree.
+ *
+ * Allocate it in a separate memory context, so that we can print its
+ * memory usage easily.
+ */
+ radixtree_ctx = AllocSetContextCreate(CurrentMemoryContext,
+ "radixtree test",
+ ALLOCSET_SMALL_SIZES);
+ MemoryContextSetIdentifier(radixtree_ctx, spec->test_name);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(radixtree_ctx, dsa, tranche_id);
+#else
+ radixtree = rt_create(radixtree_ctx);
+#endif
+
+
+ /*
+ * Add values to the set.
+ */
+ starttime = GetCurrentTimestamp();
+
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ uint64 x = 0;
+
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ bool found;
+
+ x = last_int + pattern_values[i];
+
+ found = rt_set(radixtree, x, (TestValueType*) &x);
+
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " found", x);
+
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+
+ endtime = GetCurrentTimestamp();
+
+ if (rt_test_stats)
+ fprintf(stderr, "added " UINT64_FORMAT " values in %d ms\n",
+ spec->num_values, (int) (endtime - starttime) / 1000);
+
+ /*
+ * Print stats on the amount of memory used.
+ *
+ * We print the usage reported by rt_memory_usage(), as well as the stats
+ * from the memory context. They should be in the same ballpark, but it's
+ * hard to automate testing that, so if you're making changes to the
+ * implementation, just observe that manually.
+ */
+ if (rt_test_stats)
+ {
+ uint64 mem_usage;
+
+ /*
+ * Also print memory usage as reported by rt_memory_usage(). It
+ * should be in the same ballpark as the usage reported by
+ * MemoryContextStats().
+ */
+ mem_usage = rt_memory_usage(radixtree);
+ fprintf(stderr, "rt_memory_usage() reported " UINT64_FORMAT " (%0.2f bytes / integer)\n",
+ mem_usage, (double) mem_usage / spec->num_values);
+
+ MemoryContextStats(radixtree_ctx);
+ }
+
+ /* Check that rt_num_entries works */
+ n = rt_num_entries(radixtree);
+ if (n != spec->num_values)
+ elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT, n, spec->num_values);
+
+ /*
+ * Test random-access probes with rt_search()
+ */
+ starttime = GetCurrentTimestamp();
+
+ for (n = 0; n < 100000; n++)
+ {
+ bool found;
+ bool expected;
+ uint64 x;
+ TestValueType v;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit values that are actually in the set.
+ */
+ x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+ /* Do we expect this value to be present in the set? */
+ if (x >= last_int)
+ expected = false;
+ else
+ {
+ uint64 idx = x % spec->spacing;
+
+ if (idx >= patternlen)
+ expected = false;
+ else if (spec->pattern_str[idx] == '1')
+ expected = true;
+ else
+ expected = false;
+ }
+
+ /* Is it present according to rt_search() ? */
+ found = rt_search(radixtree, x, &v);
+
+ if (found != expected)
+ elog(ERROR, "mismatch at 0x" UINT64_HEX_FORMAT ": %d vs %d", x, found, expected);
+ if (found && (v != (TestValueType) x))
+ elog(ERROR, "found 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+ v, x);
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "probed " UINT64_FORMAT " values in %d ms\n",
+ n, (int) (endtime - starttime) / 1000);
+
+ /*
+ * Test iterator
+ */
+ starttime = GetCurrentTimestamp();
+
+ iter = rt_begin_iterate(radixtree);
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ uint64 expected = last_int + pattern_values[i];
+ uint64 x;
+ TestValueType val;
+
+ if (!rt_iterate_next(iter, &x, &val))
+ break;
+
+ if (x != expected)
+ elog(ERROR,
+ "iterate returned wrong key; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d",
+ x, expected, i);
+ if (val != (TestValueType) expected)
+ elog(ERROR,
+ "iterate returned wrong value; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d", x, expected, i);
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "iterated " UINT64_FORMAT " values in %d ms\n",
+ n, (int) (endtime - starttime) / 1000);
+
+ rt_end_iterate(iter);
+
+ if (n < spec->num_values)
+ elog(ERROR, "iterator stopped short after " UINT64_FORMAT " entries, expected " UINT64_FORMAT, n, spec->num_values);
+ if (n > spec->num_values)
+ elog(ERROR, "iterator returned " UINT64_FORMAT " entries, " UINT64_FORMAT " was expected", n, spec->num_values);
+
+ /*
+ * Test random-access probes with rt_delete()
+ */
+ starttime = GetCurrentTimestamp();
+
+ nbefore = rt_num_entries(radixtree);
+ ndeleted = 0;
+ for (n = 0; n < 1; n++)
+ {
+ bool found;
+ uint64 x;
+ TestValueType v;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit values that are actually in the set.
+ */
+ x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+ /* Is it present according to rt_search() ? */
+ found = rt_search(radixtree, x, &v);
+
+ if (!found)
+ continue;
+
+ /* If the key is found, delete it and check again */
+ if (!rt_delete(radixtree, x))
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, x);
+ if (rt_search(radixtree, x, &v))
+ elog(ERROR, "found deleted key 0x" UINT64_HEX_FORMAT, x);
+ if (rt_delete(radixtree, x))
+ elog(ERROR, "deleted already-deleted key 0x" UINT64_HEX_FORMAT, x);
+
+ ndeleted++;
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "deleted " UINT64_FORMAT " values in %d ms\n",
+ ndeleted, (int) (endtime - starttime) / 1000);
+
+ nafter = rt_num_entries(radixtree);
+
+ /* Check that rt_num_entries works */
+ if ((nbefore - ndeleted) != nafter)
+ elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT "after " UINT64_FORMAT " deletion",
+ nafter, (nbefore - ndeleted), ndeleted);
+
+ rt_free(radixtree);
+ MemoryContextDelete(radixtree_ctx);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+Datum
+test_radixtree(PG_FUNCTION_ARGS)
+{
+ test_empty();
+
+ for (int i = 1; i < lengthof(rt_node_kind_fanouts); i++)
+ {
+ test_basic(rt_node_kind_fanouts[i], false);
+ test_basic(rt_node_kind_fanouts[i], true);
+ }
+
+ for (int shift = 0; shift <= (64 - 8); shift += 8)
+ test_node_types(shift);
+
+ /* Test different test patterns, with lots of entries */
+ for (int i = 0; i < lengthof(test_specs); i++)
+ test_pattern(&test_specs[i]);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_radixtree/test_radixtree.control b/src/test/modules/test_radixtree/test_radixtree.control
new file mode 100644
index 0000000000..e53f2a3e0c
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.control
@@ -0,0 +1,4 @@
+comment = 'Test code for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/test_radixtree'
+relocatable = true
diff --git a/src/tools/pginclude/cpluspluscheck b/src/tools/pginclude/cpluspluscheck
index 4e09c4686b..202bf1c04e 100755
--- a/src/tools/pginclude/cpluspluscheck
+++ b/src/tools/pginclude/cpluspluscheck
@@ -101,6 +101,12 @@ do
test "$f" = src/include/nodes/nodetags.h && continue
test "$f" = src/backend/nodes/nodetags.h && continue
+ # radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+ test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+ test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+ test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+ test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
# These files are not meant to be included standalone, because
# they contain lists that might have multiple use-cases.
test "$f" = src/include/access/rmgrlist.h && continue
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index 8dee1b5670..133313255c 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -96,6 +96,12 @@ do
test "$f" = src/include/nodes/nodetags.h && continue
test "$f" = src/backend/nodes/nodetags.h && continue
+ # radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+ test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+ test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+ test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+ test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
# These files are not meant to be included standalone, because
# they contain lists that might have multiple use-cases.
test "$f" = src/include/access/rmgrlist.h && continue
--
2.31.1
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-03-11 15:54 ` John Naylor <[email protected]>
2023-03-13 01:41 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-04-07 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
1 sibling, 2 replies; 78+ messages in thread
From: John Naylor @ 2023-03-11 15:54 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Fri, Mar 10, 2023 at 9:30 PM Masahiko Sawada <[email protected]>
wrote:
>
> On Fri, Mar 10, 2023 at 3:42 PM John Naylor
> <[email protected]> wrote:
> > I'd suggest sharing your todo list in the meanwhile, it'd be good to
discuss what's worth doing and what is not.
>
> Apart from more rounds of reviews and tests, my todo items that need
> discussion and possibly implementation are:
Quick thoughts on these:
> * The memory measurement in radix trees and the memory limit in
> tidstores. I've implemented it in v30-0007 through 0009 but we need to
> review it. This is the highest priority for me.
Agreed.
> * Additional size classes. It's important for an alternative of path
> compression as well as supporting our decoupling approach. Middle
> priority.
I'm going to push back a bit and claim this doesn't bring much gain, while
it does have a complexity cost. The node1 from Andres's prototype is 32
bytes in size, same as our node3, so it's roughly equivalent as a way to
ameliorate the lack of path compression. I say "roughly" because the loop
in node3 is probably noticeably slower. A new size class will by definition
still use that loop.
About a smaller node125-type class: I'm actually not even sure we need to
have any sub-max node bigger about 64 (node size 768 bytes). I'd just let
65+ go to the max node -- there won't be many of them, at least in
synthetic workloads we've seen so far.
> * Node shrinking support. Low priority.
This is an architectural wart that's been neglected since the tid store
doesn't perform deletion. We'll need it sometime. If we're not going to
make this work, why ship a deletion API at all?
I took a look at this a couple weeks ago, and fixing it wouldn't be that
hard. I even had an idea of how to detect when to shrink size class within
a node kind, while keeping the header at 5 bytes. I'd be willing to put
effort into that, but to have a chance of succeeding, I'm unwilling to make
it more difficult by adding more size classes at this point.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-11 15:54 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-03-13 01:41 ` Masahiko Sawada <[email protected]>
2023-03-13 13:28 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
1 sibling, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-03-13 01:41 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Sun, Mar 12, 2023 at 12:54 AM John Naylor
<[email protected]> wrote:
>
> On Fri, Mar 10, 2023 at 9:30 PM Masahiko Sawada <[email protected]> wrote:
> >
> > On Fri, Mar 10, 2023 at 3:42 PM John Naylor
> > <[email protected]> wrote:
>
> > > I'd suggest sharing your todo list in the meanwhile, it'd be good to discuss what's worth doing and what is not.
> >
> > Apart from more rounds of reviews and tests, my todo items that need
> > discussion and possibly implementation are:
>
> Quick thoughts on these:
>
> > * The memory measurement in radix trees and the memory limit in
> > tidstores. I've implemented it in v30-0007 through 0009 but we need to
> > review it. This is the highest priority for me.
>
> Agreed.
>
> > * Additional size classes. It's important for an alternative of path
> > compression as well as supporting our decoupling approach. Middle
> > priority.
>
> I'm going to push back a bit and claim this doesn't bring much gain, while it does have a complexity cost. The node1 from Andres's prototype is 32 bytes in size, same as our node3, so it's roughly equivalent as a way to ameliorate the lack of path compression.
But does it mean that our node1 would help reduce the memory further
since since our base node type (i.e. RT_NODE) is smaller than the base
node type of Andres's prototype? The result I shared before showed
1.2GB vs. 1.9GB.
> I say "roughly" because the loop in node3 is probably noticeably slower. A new size class will by definition still use that loop.
I've evaluated the performance of node1 but the result seems to show
the opposite. I used the test query:
select * from bench_search_random_nodes(100 * 1000 * 1000,
'0xFF000000000000FF');
Which make the radix tree that has node1 like:
max_val = 18446744073709551615
num_keys = 65536
height = 7, n1 = 1536, n3 = 0, n15 = 0, n32 = 0, n61 = 0, n256 = 257
All internal nodes except for the root node are node1. The radix tree
that doesn't have node1 is:
max_val = 18446744073709551615
num_keys = 65536
height = 7, n3 = 1536, n15 = 0, n32 = 0, n125 = 0, n256 = 257
Here is the result:
* w/ node1
mem_allocated | load_ms | search_ms
---------------+---------+-----------
573448 | 1848 | 1707
(1 row)
* w/o node1
mem_allocated | load_ms | search_ms
---------------+---------+-----------
598024 | 2014 | 1825
(1 row)
Am I missing something?
>
> About a smaller node125-type class: I'm actually not even sure we need to have any sub-max node bigger about 64 (node size 768 bytes). I'd just let 65+ go to the max node -- there won't be many of them, at least in synthetic workloads we've seen so far.
Makes sense to me.
>
> > * Node shrinking support. Low priority.
>
> This is an architectural wart that's been neglected since the tid store doesn't perform deletion. We'll need it sometime. If we're not going to make this work, why ship a deletion API at all?
>
> I took a look at this a couple weeks ago, and fixing it wouldn't be that hard. I even had an idea of how to detect when to shrink size class within a node kind, while keeping the header at 5 bytes. I'd be willing to put effort into that, but to have a chance of succeeding, I'm unwilling to make it more difficult by adding more size classes at this point.
I think that the deletion (and locking support) doesn't have use cases
in the core (i.e. tidstore) but is implemented so that external
extensions can use it. There might not be such extensions. Given the
lack of use cases in the core (and the rest of time), I think it's
okay even if the implementation of such API is minimal and not
optimized enough. For instance, the implementation of dshash.c is
minimalist, and doesn't have resizing. We can improve them in the
future if extensions or other core features want.
Personally I think we should focus on addressing feedback that we
would get and improving the existing use cases for the rest of time.
That's why considering min-max size class has a higher priority than
the node shrinking support in my todo list.
FYI, I've run TPC-C workload over the weekend, and didn't get any
failures of the assertion proving tidstore and the current tid lookup
return the same result.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-11 15:54 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-13 01:41 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-03-13 13:28 ` John Naylor <[email protected]>
2023-03-13 14:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: John Naylor @ 2023-03-13 13:28 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Mon, Mar 13, 2023 at 8:41 AM Masahiko Sawada <[email protected]>
wrote:
>
> On Sun, Mar 12, 2023 at 12:54 AM John Naylor
> <[email protected]> wrote:
> >
> > On Fri, Mar 10, 2023 at 9:30 PM Masahiko Sawada <[email protected]>
wrote:
> > > * Additional size classes. It's important for an alternative of path
> > > compression as well as supporting our decoupling approach. Middle
> > > priority.
> >
> > I'm going to push back a bit and claim this doesn't bring much gain,
while it does have a complexity cost. The node1 from Andres's prototype is
32 bytes in size, same as our node3, so it's roughly equivalent as a way to
ameliorate the lack of path compression.
>
> But does it mean that our node1 would help reduce the memory further
> since since our base node type (i.e. RT_NODE) is smaller than the base
> node type of Andres's prototype? The result I shared before showed
> 1.2GB vs. 1.9GB.
The benefit is found in a synthetic benchmark with random integers. I
highly doubt that anyone would be willing to force us to keep
binary-searching the 1GB array for one more cycle on account of not adding
a size class here. I'll repeat myself and say that there are also
maintenance costs.
In contrast, I'm fairly certain that our attempts thus far at memory
accounting/limiting are not quite up to par, and lacking enough to
jeopardize the feature. We're already discussing that, so I'll say no more.
> > I say "roughly" because the loop in node3 is probably noticeably
slower. A new size class will by definition still use that loop.
>
> I've evaluated the performance of node1 but the result seems to show
> the opposite.
As an aside, I meant the loop in our node3 might make your node1 slower
than the prototype's node1, which was coded for 1 member only.
> > > * Node shrinking support. Low priority.
> >
> > This is an architectural wart that's been neglected since the tid store
doesn't perform deletion. We'll need it sometime. If we're not going to
make this work, why ship a deletion API at all?
> >
> > I took a look at this a couple weeks ago, and fixing it wouldn't be
that hard. I even had an idea of how to detect when to shrink size class
within a node kind, while keeping the header at 5 bytes. I'd be willing to
put effort into that, but to have a chance of succeeding, I'm unwilling to
make it more difficult by adding more size classes at this point.
>
> I think that the deletion (and locking support) doesn't have use cases
> in the core (i.e. tidstore) but is implemented so that external
> extensions can use it.
I think these cases are a bit different: Doing anything with a data
structure stored in shared memory without a synchronization scheme is
completely unthinkable and insane. I'm not yet sure if
deleting-without-shrinking is a showstopper, or if it's preferable in v16
than no deletion at all.
Anything we don't implement now is a limit on future use cases, and thus a
cause for objection. On the other hand, anything we implement also
represents more stuff that will have to be rewritten for high-concurrency.
> FYI, I've run TPC-C workload over the weekend, and didn't get any
> failures of the assertion proving tidstore and the current tid lookup
> return the same result.
Great!
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-11 15:54 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-13 01:41 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-13 13:28 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-03-13 14:55 ` Masahiko Sawada <[email protected]>
2023-03-14 11:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-03-13 14:55 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Mon, Mar 13, 2023 at 10:28 PM John Naylor
<[email protected]> wrote:
>
> On Mon, Mar 13, 2023 at 8:41 AM Masahiko Sawada <[email protected]> wrote:
> >
> > On Sun, Mar 12, 2023 at 12:54 AM John Naylor
> > <[email protected]> wrote:
> > >
> > > On Fri, Mar 10, 2023 at 9:30 PM Masahiko Sawada <[email protected]> wrote:
>
> > > > * Additional size classes. It's important for an alternative of path
> > > > compression as well as supporting our decoupling approach. Middle
> > > > priority.
> > >
> > > I'm going to push back a bit and claim this doesn't bring much gain, while it does have a complexity cost. The node1 from Andres's prototype is 32 bytes in size, same as our node3, so it's roughly equivalent as a way to ameliorate the lack of path compression.
> >
> > But does it mean that our node1 would help reduce the memory further
> > since since our base node type (i.e. RT_NODE) is smaller than the base
> > node type of Andres's prototype? The result I shared before showed
> > 1.2GB vs. 1.9GB.
>
> The benefit is found in a synthetic benchmark with random integers. I highly doubt that anyone would be willing to force us to keep binary-searching the 1GB array for one more cycle on account of not adding a size class here. I'll repeat myself and say that there are also maintenance costs.
>
> In contrast, I'm fairly certain that our attempts thus far at memory accounting/limiting are not quite up to par, and lacking enough to jeopardize the feature. We're already discussing that, so I'll say no more.
I agree that memory accounting/limiting stuff is the highest priority.
So what kinds of size classes do you think we need? node3, 15, 32, 61
and 256?
>
> > > I say "roughly" because the loop in node3 is probably noticeably slower. A new size class will by definition still use that loop.
> >
> > I've evaluated the performance of node1 but the result seems to show
> > the opposite.
>
> As an aside, I meant the loop in our node3 might make your node1 slower than the prototype's node1, which was coded for 1 member only.
Agreed.
>
> > > > * Node shrinking support. Low priority.
> > >
> > > This is an architectural wart that's been neglected since the tid store doesn't perform deletion. We'll need it sometime. If we're not going to make this work, why ship a deletion API at all?
> > >
> > > I took a look at this a couple weeks ago, and fixing it wouldn't be that hard. I even had an idea of how to detect when to shrink size class within a node kind, while keeping the header at 5 bytes. I'd be willing to put effort into that, but to have a chance of succeeding, I'm unwilling to make it more difficult by adding more size classes at this point.
> >
> > I think that the deletion (and locking support) doesn't have use cases
> > in the core (i.e. tidstore) but is implemented so that external
> > extensions can use it.
>
> I think these cases are a bit different: Doing anything with a data structure stored in shared memory without a synchronization scheme is completely unthinkable and insane.
Right.
> I'm not yet sure if deleting-without-shrinking is a showstopper, or if it's preferable in v16 than no deletion at all.
>
> Anything we don't implement now is a limit on future use cases, and thus a cause for objection. On the other hand, anything we implement also represents more stuff that will have to be rewritten for high-concurrency.
Okay. Given that adding shrinking support also requires maintenance
costs (and probably new test cases?) and there are no use cases in the
core, I'm not sure it's worth supporting it at this stage. So I prefer
either shipping the deletion API as it is and removing the deletion
API. I think that it's a discussion point that we'd like to hear
feedback from other hackers.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-11 15:54 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-13 01:41 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-13 13:28 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-13 14:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-03-14 11:27 ` John Naylor <[email protected]>
2023-03-15 02:32 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: John Naylor @ 2023-03-14 11:27 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
I wrote:
> > > Since the block-level measurement is likely overestimating quite a
bit, I propose to simply reverse the order of the actions here, effectively
reporting progress for the *last page* and not the current one: First
update progress with the current memory usage, then add tids for this page.
If this allocated a new block, only a small bit of that will be written to.
If this block pushes it over the limit, we will detect that up at the top
of the loop. It's kind of like our earlier attempts at a "fudge factor",
but simpler and less brittle. And, as far as OS pages we have actually
written to, I think it'll effectively respect the memory limit, at least in
the local mem case. And the numbers will make sense.
> > >
> > > Thoughts?
> >
> > It looks to work but it still doesn't work in a case where a shared
> > tidstore is created with a 64kB memory limit, right?
> > TidStoreMemoryUsage() returns 1MB and TidStoreIsFull() returns true
> > from the beginning.
>
> I have two ideas:
>
> 1. Make it optional to track chunk memory space by a template parameter.
It might be tiny compared to everything else that vacuum does. That would
allow other users to avoid that overhead.
> 2. When context block usage exceeds the limit (rare), make the additional
effort to get the precise usage -- I'm not sure such a top-down facility
exists, and I'm not feeling well enough today to study this further.
Since then, Masahiko incorporated #1 into v31, and that's what I'm looking
at now. Unfortunately, If I had spent five minutes reminding myself what
the original objections were to this approach, I could have saved us some
effort. Back in July (!), Andres raised two points: GetMemoryChunkSpace()
is slow [1], and fragmentation [2] (leading to underestimation).
In v31, in the local case at least, the underestimation is actually worse
than tracking chunk space, since it ignores chunk header and alignment.
I'm not sure about the DSA case. This doesn't seem great.
It shouldn't be a surprise why a simple increment of raw allocation size is
comparable in speed -- GetMemoryChunkSpace() calls the right function
through a pointer, which is slower. If we were willing to underestimate for
the sake of speed, that takes away the reason for making memory tracking
optional.
Further, if the option is not specified, in v31 there is no way to get the
memory use at all, which seems odd. Surely the caller should be able to ask
the context/area, if it wants to.
I still like my idea at the top of the page -- at least for vacuum and
m_w_m. It's still not completely clear if it's right but I've got nothing
better. It also ignores the work_mem issue, but I've given up anticipating
all future cases at the moment.
I'll put this item and a couple other things together in a separate email
tomorrow.
[1]
https://www.postgresql.org/message-id/20220704211822.kfxtzpcdmslzm2dy%40awork3.anarazel.de
[2]
https://www.postgresql.org/message-id/20220704220038.at2ane5xkymzzssb%40awork3.anarazel.de
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-11 15:54 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-13 01:41 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-13 13:28 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-13 14:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-14 11:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-03-15 02:32 ` Masahiko Sawada <[email protected]>
2023-03-17 07:02 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-03-15 02:32 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Tue, Mar 14, 2023 at 8:27 PM John Naylor
<[email protected]> wrote:
>
> I wrote:
>
> > > > Since the block-level measurement is likely overestimating quite a bit, I propose to simply reverse the order of the actions here, effectively reporting progress for the *last page* and not the current one: First update progress with the current memory usage, then add tids for this page. If this allocated a new block, only a small bit of that will be written to. If this block pushes it over the limit, we will detect that up at the top of the loop. It's kind of like our earlier attempts at a "fudge factor", but simpler and less brittle. And, as far as OS pages we have actually written to, I think it'll effectively respect the memory limit, at least in the local mem case. And the numbers will make sense.
> > > >
> > > > Thoughts?
> > >
> > > It looks to work but it still doesn't work in a case where a shared
> > > tidstore is created with a 64kB memory limit, right?
> > > TidStoreMemoryUsage() returns 1MB and TidStoreIsFull() returns true
> > > from the beginning.
> >
> > I have two ideas:
> >
> > 1. Make it optional to track chunk memory space by a template parameter. It might be tiny compared to everything else that vacuum does. That would allow other users to avoid that overhead.
> > 2. When context block usage exceeds the limit (rare), make the additional effort to get the precise usage -- I'm not sure such a top-down facility exists, and I'm not feeling well enough today to study this further.
>
> Since then, Masahiko incorporated #1 into v31, and that's what I'm looking at now. Unfortunately, If I had spent five minutes reminding myself what the original objections were to this approach, I could have saved us some effort. Back in July (!), Andres raised two points: GetMemoryChunkSpace() is slow [1], and fragmentation [2] (leading to underestimation).
>
> In v31, in the local case at least, the underestimation is actually worse than tracking chunk space, since it ignores chunk header and alignment. I'm not sure about the DSA case. This doesn't seem great.
Right.
>
> It shouldn't be a surprise why a simple increment of raw allocation size is comparable in speed -- GetMemoryChunkSpace() calls the right function through a pointer, which is slower. If we were willing to underestimate for the sake of speed, that takes away the reason for making memory tracking optional.
>
> Further, if the option is not specified, in v31 there is no way to get the memory use at all, which seems odd. Surely the caller should be able to ask the context/area, if it wants to.
There are precedents that don't provide a way to return memory usage,
such as simplehash.h and dshash.c.
>
> I still like my idea at the top of the page -- at least for vacuum and m_w_m. It's still not completely clear if it's right but I've got nothing better. It also ignores the work_mem issue, but I've given up anticipating all future cases at the moment.
>
What does it mean by "the precise usage" in your idea? Quoting from
the email you referred to, Andres said:
---
One thing I was wondering about is trying to choose node types in
roughly-power-of-two struct sizes. It's pretty easy to end up with significant
fragmentation in the slabs right now when inserting as you go, because some of
the smaller node types will be freed but not enough to actually free blocks of
memory. If we instead have ~power-of-two sizes we could just use a single slab
of the max size, and carve out the smaller node types out of that largest
allocation.
Btw, that fragmentation is another reason why I think it's better to track
memory usage via memory contexts, rather than doing so based on
GetMemoryChunkSpace().
---
IIUC he suggested measuring memory usage in block-level in order to
count blocks that are not actually freed but some of its chunks are
freed. That's why we used MemoryContextMemAllocated(). On the other
hand, recently you pointed out[1]:
---
I think we're trying to solve the wrong problem here. I need to study
this more, but it seems that code that needs to stay within a memory
limit only needs to track what's been allocated in chunks within a
block, since writing there is what invokes a page fault.
---
IIUC you suggested measuring memory usage by tracking how much memory
chunks are allocated within a block. If your idea at the top of the
page follows this method, it still doesn't deal with the point Andres
mentioned.
> I'll put this item and a couple other things together in a separate email tomorrow.
Thanks!
Regards,
[1] https://www.postgresql.org/message-id/CAFBsxsEnzivaJ13iCGdDoUMsXJVGOaahuBe_y%3Dq6ow%3DLTzyDvA%40mail...
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-11 15:54 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-13 01:41 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-13 13:28 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-13 14:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-14 11:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-15 02:32 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-03-17 07:02 ` John Naylor <[email protected]>
2023-03-17 07:49 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: John Naylor @ 2023-03-17 07:02 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Wed, Mar 15, 2023 at 9:32 AM Masahiko Sawada <[email protected]>
wrote:
>
> On Tue, Mar 14, 2023 at 8:27 PM John Naylor
> <[email protected]> wrote:
> >
> > I wrote:
> >
> > > > > Since the block-level measurement is likely overestimating quite
a bit, I propose to simply reverse the order of the actions here,
effectively reporting progress for the *last page* and not the current one:
First update progress with the current memory usage, then add tids for this
page. If this allocated a new block, only a small bit of that will be
written to. If this block pushes it over the limit, we will detect that up
at the top of the loop. It's kind of like our earlier attempts at a "fudge
factor", but simpler and less brittle. And, as far as OS pages we have
actually written to, I think it'll effectively respect the memory limit, at
least in the local mem case. And the numbers will make sense.
> > I still like my idea at the top of the page -- at least for vacuum and
m_w_m. It's still not completely clear if it's right but I've got nothing
better. It also ignores the work_mem issue, but I've given up anticipating
all future cases at the moment.
> IIUC you suggested measuring memory usage by tracking how much memory
> chunks are allocated within a block. If your idea at the top of the
> page follows this method, it still doesn't deal with the point Andres
> mentioned.
Right, but that idea was orthogonal to how we measure memory use, and in
fact mentions blocks specifically. The re-ordering was just to make sure
that progress reporting didn't show current-use > max-use.
However, the big question remains DSA, since a new segment can be as large
as the entire previous set of allocations. It seems it just wasn't designed
for things where memory growth is unpredictable.
I'm starting to wonder if we need to give DSA a bit more info at the start.
Imagine a "soft" limit given to the DSA area when it is initialized. If the
total segment usage exceeds this, it stops doubling and instead new
segments get smaller. Modifying an example we used for the fudge-factor
idea some time ago:
m_w_m = 1GB, so calculate the soft limit to be 512MB and pass it to the DSA
area.
2*(1+2+4+8+16+32+64+128) + 256 = 766MB (74.8% of 1GB) -> hit soft limit, so
"stairstep down" the new segment sizes:
766 + 2*(128) + 64 = 1086MB -> stop
That's just an undeveloped idea, however, so likely v17 development, even
assuming it's not a bad idea (could be).
And sadly, unless we find some other, simpler answer soon for tracking and
limiting shared memory, the tid store is looking like v17 material.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-11 15:54 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-13 01:41 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-13 13:28 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-13 14:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-14 11:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-15 02:32 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-17 07:02 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-03-17 07:49 ` Masahiko Sawada <[email protected]>
2023-03-20 05:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-03-17 07:49 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Fri, Mar 17, 2023 at 4:03 PM John Naylor
<[email protected]> wrote:
>
> On Wed, Mar 15, 2023 at 9:32 AM Masahiko Sawada <[email protected]> wrote:
> >
> > On Tue, Mar 14, 2023 at 8:27 PM John Naylor
> > <[email protected]> wrote:
> > >
> > > I wrote:
> > >
> > > > > > Since the block-level measurement is likely overestimating quite a bit, I propose to simply reverse the order of the actions here, effectively reporting progress for the *last page* and not the current one: First update progress with the current memory usage, then add tids for this page. If this allocated a new block, only a small bit of that will be written to. If this block pushes it over the limit, we will detect that up at the top of the loop. It's kind of like our earlier attempts at a "fudge factor", but simpler and less brittle. And, as far as OS pages we have actually written to, I think it'll effectively respect the memory limit, at least in the local mem case. And the numbers will make sense.
>
> > > I still like my idea at the top of the page -- at least for vacuum and m_w_m. It's still not completely clear if it's right but I've got nothing better. It also ignores the work_mem issue, but I've given up anticipating all future cases at the moment.
>
> > IIUC you suggested measuring memory usage by tracking how much memory
> > chunks are allocated within a block. If your idea at the top of the
> > page follows this method, it still doesn't deal with the point Andres
> > mentioned.
>
> Right, but that idea was orthogonal to how we measure memory use, and in fact mentions blocks specifically. The re-ordering was just to make sure that progress reporting didn't show current-use > max-use.
Right. I still like your re-ordering idea. It's true that the most
area of the last allocated block before heap scanning stops is not
actually used yet. I'm guessing we can just check if the context
memory has gone over the limit. But I'm concerned it might not work
well in systems where overcommit memory is disabled.
>
> However, the big question remains DSA, since a new segment can be as large as the entire previous set of allocations. It seems it just wasn't designed for things where memory growth is unpredictable.
>
> I'm starting to wonder if we need to give DSA a bit more info at the start. Imagine a "soft" limit given to the DSA area when it is initialized. If the total segment usage exceeds this, it stops doubling and instead new segments get smaller. Modifying an example we used for the fudge-factor idea some time ago:
>
> m_w_m = 1GB, so calculate the soft limit to be 512MB and pass it to the DSA area.
>
> 2*(1+2+4+8+16+32+64+128) + 256 = 766MB (74.8% of 1GB) -> hit soft limit, so "stairstep down" the new segment sizes:
>
> 766 + 2*(128) + 64 = 1086MB -> stop
>
> That's just an undeveloped idea, however, so likely v17 development, even assuming it's not a bad idea (could be).
This is an interesting idea. But I'm concerned we don't have enough
time to get confident with adding this new concept to DSA.
>
> And sadly, unless we find some other, simpler answer soon for tracking and limiting shared memory, the tid store is looking like v17 material.
Another problem we need to deal with is the supported minimum memory
in shared tidstore cases. Since the initial DSA segment size is 1MB,
memory usage of a shared tidstore will start from 1MB+. This is higher
than the minimum values of both work_mem and maintenance_work_mem,
64kB and 1MB respectively. Increasing the minimum m_w_m to 2MB seems
to be acceptable in the community but not for work_mem. One idea is to
deny the memory limit less than 2MB so it won't work with small m_w_m
settings. While it might be an acceptable restriction at this stage
(where there is no use case of using tidstore with work_mem in the
core) but it will be a blocker for the future adoptions such as
unifying with tidbitmap.c. Another idea is that the process can
specify the initial segment size at dsa_create() so that DSA can start
with a smaller segment, say 32kB. That way, a tidstore with a 32kB
limit gets full once it allocates the next DSA segment, 32kB. . But a
downside of this idea is to increase the number of segments behind
DSA. Assuming it's a relatively rare case where we use such a low
work_mem, it might be acceptable. FYI, the total number of DSM
segments available on the system is calculated by:
#define PG_DYNSHMEM_FIXED_SLOTS 64
#define PG_DYNSHMEM_SLOTS_PER_BACKEND 5
maxitems = PG_DYNSHMEM_FIXED_SLOTS
+ PG_DYNSHMEM_SLOTS_PER_BACKEND * MaxBackends;
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-11 15:54 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-13 01:41 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-13 13:28 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-13 14:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-14 11:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-15 02:32 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-17 07:02 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-17 07:49 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-03-20 05:24 ` Masahiko Sawada <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Masahiko Sawada @ 2023-03-20 05:24 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Fri, Mar 17, 2023 at 4:49 PM Masahiko Sawada <[email protected]> wrote:
>
> On Fri, Mar 17, 2023 at 4:03 PM John Naylor
> <[email protected]> wrote:
> >
> > On Wed, Mar 15, 2023 at 9:32 AM Masahiko Sawada <[email protected]> wrote:
> > >
> > > On Tue, Mar 14, 2023 at 8:27 PM John Naylor
> > > <[email protected]> wrote:
> > > >
> > > > I wrote:
> > > >
> > > > > > > Since the block-level measurement is likely overestimating quite a bit, I propose to simply reverse the order of the actions here, effectively reporting progress for the *last page* and not the current one: First update progress with the current memory usage, then add tids for this page. If this allocated a new block, only a small bit of that will be written to. If this block pushes it over the limit, we will detect that up at the top of the loop. It's kind of like our earlier attempts at a "fudge factor", but simpler and less brittle. And, as far as OS pages we have actually written to, I think it'll effectively respect the memory limit, at least in the local mem case. And the numbers will make sense.
> >
> > > > I still like my idea at the top of the page -- at least for vacuum and m_w_m. It's still not completely clear if it's right but I've got nothing better. It also ignores the work_mem issue, but I've given up anticipating all future cases at the moment.
> >
> > > IIUC you suggested measuring memory usage by tracking how much memory
> > > chunks are allocated within a block. If your idea at the top of the
> > > page follows this method, it still doesn't deal with the point Andres
> > > mentioned.
> >
> > Right, but that idea was orthogonal to how we measure memory use, and in fact mentions blocks specifically. The re-ordering was just to make sure that progress reporting didn't show current-use > max-use.
>
> Right. I still like your re-ordering idea. It's true that the most
> area of the last allocated block before heap scanning stops is not
> actually used yet. I'm guessing we can just check if the context
> memory has gone over the limit. But I'm concerned it might not work
> well in systems where overcommit memory is disabled.
>
> >
> > However, the big question remains DSA, since a new segment can be as large as the entire previous set of allocations. It seems it just wasn't designed for things where memory growth is unpredictable.
aset.c also has a similar characteristic; allocates an 8K block upon
the first allocation in a context, and doubles that size for each
successive block request. But we can specify the initial block size
and max blocksize. This made me think of another idea to specify both
to DSA and both values are calculated based on m_w_m. For example, we
can create a DSA in parallel_vacuum_init() as follows:
initial block size = min(m_w_m / 4, 1MB)
max block size = max(m_w_m / 8, 8MB)
In most cases, we can start with a 1MB initial segment, the same as
before. For small memory cases, say 1MB, we start with a 256KB initial
segment and heap scanning stops after DSA allocated 1.5MB (= 256kB +
256kB + 512kB + 512kB). For larger memory, we can have heap scan stop
after DSA allocates 1.25 times more memory than m_w_m. For example, if
m_w_m = 1GB, the both initial and maximum segment sizes are 1MB and
128MB respectively, and then DSA allocates the segments as follows
until heap scanning stops:
2 * (1 + 2 + 4 + 8 + 16 + 32 + 64 + 128) + (128 * 5) = 1150MB
dsa_allocate() will be extended to have the initial and maximum block
sizes like AllocSetContextCreate().
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-11 15:54 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-04-07 09:55 ` John Naylor <[email protected]>
2023-04-17 13:48 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-05-08 10:23 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
1 sibling, 2 replies; 78+ messages in thread
From: John Naylor @ 2023-04-07 09:55 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Thu, Feb 16, 2023 at 11:44 PM Andres Freund <[email protected]> wrote:
>
> We really ought to replace the tid bitmap used for bitmap heap scans. The
> hashtable we use is a pretty awful data structure for it. And that's not
> filled in-order, for example.
I spent some time studying tidbitmap.c, and not only does it make sense to
use a radix tree there, but since it has more complex behavior and stricter
runtime requirements, it should really be the thing driving the design and
tradeoffs, not vacuum:
- With lazy expansion and single-value leaves, the root of a radix tree can
point to a single leaf. That might get rid of the need to track TBMStatus,
since setting a single-leaf tree should be cheap.
- Fixed-size PagetableEntry's are pretty large, but the tid compression
scheme used in this thread (in addition to being complex) is not a great
fit for tidbitmap because it makes it more difficult to track per-block
metadata (see also next point). With the "combined pointer-value slots"
technique, if a page's max tid offset is 63 or less, the offsets can be
stored directly in the pointer for the exact case. The lowest bit can tag
to indicate a pointer to a single-value leaf. That would complicate
operations like union/intersection and tracking "needs recheck", but it
would reduce memory use and node-traversal in common cases.
- Managing lossy storage. With pure blocknumber keys, replacing exact
storage for a range of 256 pages amounts to replacing a last-level node
with a single leaf containing one lossy PagetableEntry. The leader could
iterate over the nodes, and rank the last-level nodes by how much storage
they (possibly with leaf children) are using, and come up with an optimal
lossy-conversion plan.
The above would address the points (not including better iteration and
parallel bitmap index scans) raised in
https://www.postgresql.org/message-id/[email protected]...
Ironically, by targeting a more difficult use case, it's easier since there
is less freedom. There are many ways to beat a binary search, but fewer
good ways to improve bitmap heap scan. I'd like to put aside vacuum for
some time and try killing two birds with one stone, building upon our work
thus far.
Note: I've moved the CF entry to the next CF, and set to waiting on
author for now. Since no action is currently required from Masahiko, I've
added myself as author as well. If tackling bitmap heap scan shows promise,
we could RWF and resurrect at a later time.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-11 15:54 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-04-07 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-04-17 13:48 ` Masahiko Sawada <[email protected]>
2023-04-19 07:02 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
1 sibling, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-04-17 13:48 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Fri, Apr 7, 2023 at 6:55 PM John Naylor <[email protected]> wrote:
>
> On Thu, Feb 16, 2023 at 11:44 PM Andres Freund <[email protected]> wrote:
> >
> > We really ought to replace the tid bitmap used for bitmap heap scans. The
> > hashtable we use is a pretty awful data structure for it. And that's not
> > filled in-order, for example.
>
> I spent some time studying tidbitmap.c, and not only does it make sense to use a radix tree there, but since it has more complex behavior and stricter runtime requirements, it should really be the thing driving the design and tradeoffs, not vacuum:
>
> - With lazy expansion and single-value leaves, the root of a radix tree can point to a single leaf. That might get rid of the need to track TBMStatus, since setting a single-leaf tree should be cheap.
>
Instead of introducing single-value leaves to the radix tree as
another structure, can we store pointers to PagetableEntry as values?
> - Fixed-size PagetableEntry's are pretty large, but the tid compression scheme used in this thread (in addition to being complex) is not a great fit for tidbitmap because it makes it more difficult to track per-block metadata (see also next point). With the "combined pointer-value slots" technique, if a page's max tid offset is 63 or less, the offsets can be stored directly in the pointer for the exact case. The lowest bit can tag to indicate a pointer to a single-value leaf. That would complicate operations like union/intersection and tracking "needs recheck", but it would reduce memory use and node-traversal in common cases.
>
> - Managing lossy storage. With pure blocknumber keys, replacing exact storage for a range of 256 pages amounts to replacing a last-level node with a single leaf containing one lossy PagetableEntry. The leader could iterate over the nodes, and rank the last-level nodes by how much storage they (possibly with leaf children) are using, and come up with an optimal lossy-conversion plan.
>
> The above would address the points (not including better iteration and parallel bitmap index scans) raised in
>
> https://www.postgresql.org/message-id/[email protected]...
>
> Ironically, by targeting a more difficult use case, it's easier since there is less freedom. There are many ways to beat a binary search, but fewer good ways to improve bitmap heap scan. I'd like to put aside vacuum for some time and try killing two birds with one stone, building upon our work thus far.
>
> Note: I've moved the CF entry to the next CF, and set to waiting on author for now. Since no action is currently required from Masahiko, I've added myself as author as well. If tackling bitmap heap scan shows promise, we could RWF and resurrect at a later time.
Thanks. I'm going to continue researching the memory limitation and
try lazy path expansion until PG17 development begins.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-11 15:54 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-04-07 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-04-17 13:48 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-04-19 07:02 ` John Naylor <[email protected]>
2023-04-24 05:45 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-05-23 23:16 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
0 siblings, 2 replies; 78+ messages in thread
From: John Naylor @ 2023-04-19 07:02 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Mon, Apr 17, 2023 at 8:49 PM Masahiko Sawada <[email protected]>
wrote:
> > - With lazy expansion and single-value leaves, the root of a radix tree
can point to a single leaf. That might get rid of the need to track
TBMStatus, since setting a single-leaf tree should be cheap.
> >
>
> Instead of introducing single-value leaves to the radix tree as
> another structure, can we store pointers to PagetableEntry as values?
Well, that's pretty much what a single-value leaf is. Now that I've had
time to pause and regroup, I've looked into some aspects we previously put
off for future work, and this is one of them.
The concept is really quite trivial, and it's the simplest and most
flexible way to implement ART. Our, or at least my, documented reason not
to go that route was due to "an extra pointer traversal", but that's
partially mitigated by "lazy expansion", which is actually fairly easy to
do with single-value leaves. The two techniques complement each other in a
natural way. (Path compression, on the other hand, is much more complex.)
> > Note: I've moved the CF entry to the next CF, and set to waiting on
author for now. Since no action is currently required from Masahiko, I've
added myself as author as well. If tackling bitmap heap scan shows promise,
we could RWF and resurrect at a later time.
>
> Thanks. I'm going to continue researching the memory limitation and
Sounds like the best thing to nail down at this point.
> try lazy path expansion until PG17 development begins.
This doesn't seem like a useful thing to try and attach into the current
patch (if that's what you mean), as the current insert/delete paths are
quite complex. Using bitmap heap scan as a motivating use case, I hope to
refocus complexity to where it's most needed, and aggressively simplify
where possible.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-11 15:54 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-04-07 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-04-17 13:48 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-04-19 07:02 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-04-24 05:45 ` Masahiko Sawada <[email protected]>
1 sibling, 0 replies; 78+ messages in thread
From: Masahiko Sawada @ 2023-04-24 05:45 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Wed, Apr 19, 2023 at 4:02 PM John Naylor
<[email protected]> wrote:
>
> On Mon, Apr 17, 2023 at 8:49 PM Masahiko Sawada <[email protected]> wrote:
>
> > > - With lazy expansion and single-value leaves, the root of a radix tree can point to a single leaf. That might get rid of the need to track TBMStatus, since setting a single-leaf tree should be cheap.
> > >
> >
> > Instead of introducing single-value leaves to the radix tree as
> > another structure, can we store pointers to PagetableEntry as values?
>
> Well, that's pretty much what a single-value leaf is. Now that I've had time to pause and regroup, I've looked into some aspects we previously put off for future work, and this is one of them.
>
> The concept is really quite trivial, and it's the simplest and most flexible way to implement ART. Our, or at least my, documented reason not to go that route was due to "an extra pointer traversal", but that's partially mitigated by "lazy expansion", which is actually fairly easy to do with single-value leaves. The two techniques complement each other in a natural way. (Path compression, on the other hand, is much more complex.)
>
> > > Note: I've moved the CF entry to the next CF, and set to waiting on author for now. Since no action is currently required from Masahiko, I've added myself as author as well. If tackling bitmap heap scan shows promise, we could RWF and resurrect at a later time.
> >
> > Thanks. I'm going to continue researching the memory limitation and
>
> Sounds like the best thing to nail down at this point.
>
> > try lazy path expansion until PG17 development begins.
>
> This doesn't seem like a useful thing to try and attach into the current patch (if that's what you mean), as the current insert/delete paths are quite complex. Using bitmap heap scan as a motivating use case, I hope to refocus complexity to where it's most needed, and aggressively simplify where possible.
>
I agree that we don't want to make the current patch complex further.
Thinking about the memory limitation more, I think that combination of
the idea of specifying the initial and max DSA segment size and
dsa_set_size_limit() works well. There are two points in terms of
memory limitation; when the memory usage reaches the limit we want (1)
to minimize the last allocated memory block that is allocated but not
used yet and (2) to minimize the amount of memory that exceeds the
memory limit. Since we can specify the maximum DSA segment size, the
last allocated block before reaching the memory limit is small. Also,
thanks to dsa_set_size_limit(), the total DSA size will stop at the
limit, so (memory_usage >= memory_limit) returns true without any
exceeding memory.
Given that we need to configure the initial and maximum DSA segment
size and set the DSA limit for TidStore memory accounting and
limiting, it would be better to create the DSA for TidStore by
TidStoreCreate() API, rather than creating DSA in the caller and pass
it to TidStoreCreate().
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-11 15:54 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-04-07 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-04-17 13:48 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-04-19 07:02 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-05-23 23:16 ` John Naylor <[email protected]>
2023-06-05 10:31 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
1 sibling, 1 reply; 78+ messages in thread
From: John Naylor @ 2023-05-23 23:16 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
I wrote:
> the current insert/delete paths are quite complex. Using bitmap heap scan
as a motivating use case, I hope to refocus complexity to where it's most
needed, and aggressively simplify where possible.
Sometime in the not-too-distant future, I will start a new thread focusing
on bitmap heap scan, but for now, I just want to share some progress on
making the radix tree usable not only for that, but hopefully a wider range
of applications, while making the code simpler and the binary smaller. The
attached patches are incomplete (e.g. no iteration) and quite a bit messy,
so tar'd and gzip'd for the curious (should apply on top of v32 0001-03 +
0007-09 ).
0001
This combines a few concepts that I didn't bother separating out after the
fact:
- Split insert_impl.h into multiple functions for improved readability and
maintainability.
- Use single-value leaves as the basis for storing values, with the goal to
get to "combined pointer-value slots" for efficiency and flexibility.
- With the latter in mind, searching the child within a node now returns
the address of the slot. This allows the same interface whether the slot
contains a child pointer or a value.
- Starting with RT_SET, start turning some iterative algorithms into
recursive ones. This is a more natural way to traverse a tree structure,
and we already see an advantage: Previously when growing a node, we
searched within the parent to update its reference to the new node, because
we didn't know the slot we descended from. Now we can simply update a
single variable.
- Since we recursively pass the "shift" down the stack, it doesn't have to
be stored in any node -- only the "top-level" start shift is stored in the
tree control struct. This was easy to code since the node's shift value was
hardly ever accessed anyway! The node header shrinks from 5 bytes to 4.
0002
Back in v15, we tried keeping DSA/local pointers as members of a struct. I
did not like the result, but still thought it was a good idea. RT_DELETE is
a complex function and I didn't want to try rewriting it without a pointer
abstraction, so I've resurrected this idea, but in a simpler, less
intrusive way. A key difference from v15 is using a union type for the
non-shmem case.
0004
Rewrite RT_DELETE using recursion. I find this simpler than the previous
open-coded stack.
0005-06
Deletion has an inefficiency: One function searches for the child to see if
it's there, then another function searches for it again to delete it. Since
0001, a successful child search returns the address of the slot, so we can
save it. For the two smaller "linear search" node kinds we can then use a
single subtraction to compute the chunk/slot index for deletion. Also,
split RT_NODE_DELETE_INNER into separate functions, for a similar reason as
the insert case in 0001.
0007
Anticipate node shrinking: If only one node-kind needs to be freed, we can
move a branch to that one code path, rather than every place where RT_FREE
is inlined.
0009
Teach node256 how to shrink *. Since we know the number of children in a
node256 can't possibly be zero, we can use uint8 to store the count and
interpret an overflow to zero as 256 for this node. The node header shrinks
from 4 bytes to 3.
* Other nodes will follow in due time, but only after I figure out how to
do it nicely (ideas welcome!) -- currently node32's two size classes work
fine for growing, but the code should be simplified before extending to
other cases.)
0010
Limited support for "combined pointer-value slots". At compile-time, choose
either that or "single-value leaves" based on the size of the value type
template parameter. Values that are pointer-sized or less can fit in the
last-level child slots of nominal "inner nodes" without duplicated
leaf-node code. Node256 now must act like the previous 'node256 leaf',
since zero is a valid value. Aside from that, this was a small change.
What I've shared here could work (in principal, since it uses uint64
values) for tidstore, possibly faster (untested) because of better code
density, but as mentioned I want to shoot for higher. For tidbitmap.c, I
want to extend this idea and branch at run-time on a per-value basis, so
that a page-table entry that fits in a pointer can go there, and if not,
it'll be a full leaf. (This technique enables more flexibility in
lossifying pages as well.) Run-time info will require e.g. an additional
bit per slot. Since the node header is now 3 bytes, we can spare one more
byte in the node3 case. In addition, we can and should also bump it back up
to node4, still keeping the metadata within 8 bytes (no struct padding).
I've started in this patchset to refer to the node kinds as "4/16/48/256",
regardless of their actual fanout. This is for readability (by matching the
language in the paper) and maintainability (should *not* ever change
again). The size classes (including multiple classes per kind) could be
determined by macros and #ifdef's. For example, in non-SIMD architectures,
it's likely slow to search an array of 32 key chunks, so in that case the
compiler should choose size classes similar to these four nominal kinds.
--
John Naylor
EDB: http://www.enterprisedb.com
Attachments:
[application/gzip] v33-ART.tar.gz (26.9K, ../../CAFBsxsFyWLxweHVDtKb7otOCR4XdQGYR4b+9svxpVFnJs08BmQ@mail.gmail.com/3-v33-ART.tar.gz)
download
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-11 15:54 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-04-07 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-04-17 13:48 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-04-19 07:02 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-05-23 23:16 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-06-05 10:31 ` Masahiko Sawada <[email protected]>
2023-06-06 05:13 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-06-05 10:31 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
Hi,
On Tue, May 23, 2023 at 7:17 PM John Naylor
<[email protected]> wrote:
>
> I wrote:
> > the current insert/delete paths are quite complex. Using bitmap heap scan as a motivating use case, I hope to refocus complexity to where it's most needed, and aggressively simplify where possible.
>
> Sometime in the not-too-distant future, I will start a new thread focusing on bitmap heap scan, but for now, I just want to share some progress on making the radix tree usable not only for that, but hopefully a wider range of applications, while making the code simpler and the binary smaller. The attached patches are incomplete (e.g. no iteration) and quite a bit messy, so tar'd and gzip'd for the curious (should apply on top of v32 0001-03 + 0007-09 ).
>
Thank you for making progress on this. I agree with these directions
overall. I have some comments and questions:
> - With the latter in mind, searching the child within a node now returns the address of the slot. This allows the same interface whether the slot contains a child pointer or a value.
Probably we can apply similar changes to the iteration as well.
> * Other nodes will follow in due time, but only after I figure out how to do it nicely (ideas welcome!) -- currently node32's two size classes work fine for growing, but the code should be simplified before extending to other cases.)
Within the size class, we just alloc a new node of lower size class
and do memcpy(). I guess it will be almost same as what we do for
growing. It might be a good idea to support node shrinking within the
size class for node32 (and node125 if we support). I don't think
shrinking class-3 to class-1 makes sense.
>
> Limited support for "combined pointer-value slots". At compile-time, choose either that or "single-value leaves" based on the size of the value type template parameter. Values that are pointer-sized or less can fit in the last-level child slots of nominal "inner nodes" without duplicated leaf-node code. Node256 now must act like the previous 'node256 leaf', since zero is a valid value. Aside from that, this was a small change.
Yes, but it also means that we use pointer-sized value anyway even if
the value size is less than that, which wastes the memory, no?
>
> What I've shared here could work (in principal, since it uses uint64 values) for tidstore, possibly faster (untested) because of better code density, but as mentioned I want to shoot for higher. For tidbitmap.c, I want to extend this idea and branch at run-time on a per-value basis, so that a page-table entry that fits in a pointer can go there, and if not, it'll be a full leaf. (This technique enables more flexibility in lossifying pages as well.) Run-time info will require e.g. an additional bit per slot. Since the node header is now 3 bytes, we can spare one more byte in the node3 case. In addition, we can and should also bump it back up to node4, still keeping the metadata within 8 bytes (no struct padding).
Sounds good.
> I've started in this patchset to refer to the node kinds as "4/16/48/256", regardless of their actual fanout. This is for readability (by matching the language in the paper) and maintainability (should *not* ever change again). The size classes (including multiple classes per kind) could be determined by macros and #ifdef's. For example, in non-SIMD architectures, it's likely slow to search an array of 32 key chunks, so in that case the compiler should choose size classes similar to these four nominal kinds.
If we want to use the node kinds used in the paper, I think we should
change the number in RT_NODE_KIND_X too. Otherwise, it would be
confusing when reading the code without referring to the paper.
Particularly, this part is very confusing:
case RT_NODE_KIND_3:
RT_ADD_CHILD_4(tree, ref, node, chunk, child);
break;
case RT_NODE_KIND_32:
RT_ADD_CHILD_16(tree, ref, node, chunk, child);
break;
case RT_NODE_KIND_125:
RT_ADD_CHILD_48(tree, ref, node, chunk, child);
break;
case RT_NODE_KIND_256:
RT_ADD_CHILD_256(tree, ref, node, chunk, child);
break;
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-11 15:54 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-04-07 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-04-17 13:48 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-04-19 07:02 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-05-23 23:16 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-06-05 10:31 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-06-06 05:13 ` John Naylor <[email protected]>
2023-06-13 05:46 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: John Naylor @ 2023-06-06 05:13 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Mon, Jun 5, 2023 at 5:32 PM Masahiko Sawada <[email protected]>
wrote:
>
> > Sometime in the not-too-distant future, I will start a new thread
focusing on bitmap heap scan, but for now, I just want to share some
progress on making the radix tree usable not only for that, but hopefully a
wider range of applications, while making the code simpler and the binary
smaller. The attached patches are incomplete (e.g. no iteration) and quite
a bit messy, so tar'd and gzip'd for the curious (should apply on top of
v32 0001-03 + 0007-09 ).
> >
>
> Thank you for making progress on this. I agree with these directions
> overall. I have some comments and questions:
Glad to hear it and thanks for looking!
> > * Other nodes will follow in due time, but only after I figure out how
to do it nicely (ideas welcome!) -- currently node32's two size classes
work fine for growing, but the code should be simplified before extending
to other cases.)
>
> Within the size class, we just alloc a new node of lower size class
> and do memcpy(). I guess it will be almost same as what we do for
> growing.
Oh, the memcpy part is great, very simple. I mean the (compile-time) "class
info" table lookups are a bit awkward. I'm thinking the hard-coded numbers
like this:
.fanout = 3,
.inner_size = sizeof(RT_NODE_INNER_3) + 3 * sizeof(RT_PTR_ALLOC),
...may be better with a #defined symbol that can also be used elsewhere.
> I don't think
> shrinking class-3 to class-1 makes sense.
Agreed. The smallest kind should just be freed when empty.
> > Limited support for "combined pointer-value slots". At compile-time,
choose either that or "single-value leaves" based on the size of the value
type template parameter. Values that are pointer-sized or less can fit in
the last-level child slots of nominal "inner nodes" without duplicated
leaf-node code. Node256 now must act like the previous 'node256 leaf',
since zero is a valid value. Aside from that, this was a small change.
>
> Yes, but it also means that we use pointer-sized value anyway even if
> the value size is less than that, which wastes the memory, no?
At a low-level, that makes sense, but I've found an interesting global
effect showing the opposite: _less_ memory, which may compensate:
psql -c "select * from bench_search_random_nodes(1*1000*1000)"
num_keys = 992660
(using a low enough number that the experimental change n125->n63 doesn't
affect anything)
height = 4, n3 = 375258, n15 = 137490, n32 = 0, n63 = 0, n256 = 1025
v31:
mem_allocated | load_ms | search_ms
---------------+---------+-----------
47800768 | 253 | 134
(unreleased code "similar" to v33, but among other things restores the
separate "extend down" function)
mem_allocated | load_ms | search_ms
---------------+---------+-----------
42926048 | 221 | 127
I'd need to make sure, but apparently just going from 6 non-empty memory
contexts to 3 (remember all values are embedded here) reduces memory
fragmentation significantly in this test. (That should also serve as a
demonstration that additional size classes have both runtime costs as well
as benefits. We need to have a balance.)
So, I'm inclined to think the only reason to prefer "multi-value leaves" is
if 1) the value type is _bigger_ than a pointer 2) there is no convenient
abbreviation (like tid bitmaps have) and 3) the use case really needs to
avoid another memory access. Under those circumstances, though, the new
code plus lazy expansion etc might suit and be easier to maintain. That
said, I've mostly left alone the "leaf" types and functions, as well as
added some detritus like "const bool = false;". It would look a *lot* nicer
if we gave up on multi-value leaves entirely, but there's no rush and I
don't want to close that door entirely just yet.
> > What I've shared here could work (in principal, since it uses uint64
values) for tidstore, possibly faster (untested) because of better code
density, but as mentioned I want to shoot for higher. For tidbitmap.c, I
want to extend this idea and branch at run-time on a per-value basis, so
that a page-table entry that fits in a pointer can go there, and if not,
it'll be a full leaf. (This technique enables more flexibility in
lossifying pages as well.) Run-time info will require e.g. an additional
bit per slot. Since the node header is now 3 bytes, we can spare one more
byte in the node3 case. In addition, we can and should also bump it back up
to node4, still keeping the metadata within 8 bytes (no struct padding).
>
> Sounds good.
The additional bit per slot would require per-node logic and additional
branches, which is not great. I'm now thinking a much easier way to get
there is to give up (at least for now) on promising that "run-time
embeddable values" can use the full pointer-size (unlike value types found
embeddable at compile-time). Reserving the lowest pointer bit for a tag
"value or pointer-to-leaf" would have a much smaller code footprint. That
also has a curious side-effect for TID offsets: They are one-based so
reserving the zero bit would actually simplify things: getting rid of the
+1/-1 logic when converting bits to/from offsets.
In addition, without a new bitmap, the smallest node can actually be up to
a node5 with no struct padding, with a node2 as a subclass. (Those numbers
coincidentally were also one scenario in the paper, when calculating
worst-case memory usage). That's worth considering.
> > I've started in this patchset to refer to the node kinds as
"4/16/48/256", regardless of their actual fanout.
> If we want to use the node kinds used in the paper, I think we should
> change the number in RT_NODE_KIND_X too.
Oh absolutely, this is nowhere near ready for cosmetic review :-)
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-11 15:54 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-04-07 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-04-17 13:48 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-04-19 07:02 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-05-23 23:16 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-06-05 10:31 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-06-06 05:13 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-06-13 05:46 ` Masahiko Sawada <[email protected]>
2023-06-14 06:23 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
0 siblings, 1 reply; 78+ messages in thread
From: Masahiko Sawada @ 2023-06-13 05:46 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Tue, Jun 6, 2023 at 2:13 PM John Naylor <[email protected]> wrote:
>
> On Mon, Jun 5, 2023 at 5:32 PM Masahiko Sawada <[email protected]> wrote:
> >
> > > Sometime in the not-too-distant future, I will start a new thread focusing on bitmap heap scan, but for now, I just want to share some progress on making the radix tree usable not only for that, but hopefully a wider range of applications, while making the code simpler and the binary smaller. The attached patches are incomplete (e.g. no iteration) and quite a bit messy, so tar'd and gzip'd for the curious (should apply on top of v32 0001-03 + 0007-09 ).
> > >
> >
> > Thank you for making progress on this. I agree with these directions
> > overall. I have some comments and questions:
>
> Glad to hear it and thanks for looking!
>
> > > * Other nodes will follow in due time, but only after I figure out how to do it nicely (ideas welcome!) -- currently node32's two size classes work fine for growing, but the code should be simplified before extending to other cases.)
> >
> > Within the size class, we just alloc a new node of lower size class
> > and do memcpy(). I guess it will be almost same as what we do for
> > growing.
>
> Oh, the memcpy part is great, very simple. I mean the (compile-time) "class info" table lookups are a bit awkward. I'm thinking the hard-coded numbers like this:
>
> .fanout = 3,
> .inner_size = sizeof(RT_NODE_INNER_3) + 3 * sizeof(RT_PTR_ALLOC),
>
> ...may be better with a #defined symbol that can also be used elsewhere.
FWIW, exposing these definitions would be good in terms of testing too
since we can use them in regression tests.
>
> > I don't think
> > shrinking class-3 to class-1 makes sense.
>
> Agreed. The smallest kind should just be freed when empty.
>
> > > Limited support for "combined pointer-value slots". At compile-time, choose either that or "single-value leaves" based on the size of the value type template parameter. Values that are pointer-sized or less can fit in the last-level child slots of nominal "inner nodes" without duplicated leaf-node code. Node256 now must act like the previous 'node256 leaf', since zero is a valid value. Aside from that, this was a small change.
> >
> > Yes, but it also means that we use pointer-sized value anyway even if
> > the value size is less than that, which wastes the memory, no?
>
> At a low-level, that makes sense, but I've found an interesting global effect showing the opposite: _less_ memory, which may compensate:
>
> psql -c "select * from bench_search_random_nodes(1*1000*1000)"
> num_keys = 992660
>
> (using a low enough number that the experimental change n125->n63 doesn't affect anything)
> height = 4, n3 = 375258, n15 = 137490, n32 = 0, n63 = 0, n256 = 1025
>
> v31:
> mem_allocated | load_ms | search_ms
> ---------------+---------+-----------
> 47800768 | 253 | 134
>
> (unreleased code "similar" to v33, but among other things restores the separate "extend down" function)
> mem_allocated | load_ms | search_ms
> ---------------+---------+-----------
> 42926048 | 221 | 127
>
> I'd need to make sure, but apparently just going from 6 non-empty memory contexts to 3 (remember all values are embedded here) reduces memory fragmentation significantly in this test. (That should also serve as a demonstration that additional size classes have both runtime costs as well as benefits. We need to have a balance.)
Interesting. The result would probably vary if we change the slab
block sizes. I'd like to experiment if the code is available.
>
> So, I'm inclined to think the only reason to prefer "multi-value leaves" is if 1) the value type is _bigger_ than a pointer 2) there is no convenient abbreviation (like tid bitmaps have) and 3) the use case really needs to avoid another memory access. Under those circumstances, though, the new code plus lazy expansion etc might suit and be easier to maintain.
Indeed.
>
> > > What I've shared here could work (in principal, since it uses uint64 values) for tidstore, possibly faster (untested) because of better code density, but as mentioned I want to shoot for higher. For tidbitmap.c, I want to extend this idea and branch at run-time on a per-value basis, so that a page-table entry that fits in a pointer can go there, and if not, it'll be a full leaf. (This technique enables more flexibility in lossifying pages as well.) Run-time info will require e.g. an additional bit per slot. Since the node header is now 3 bytes, we can spare one more byte in the node3 case. In addition, we can and should also bump it back up to node4, still keeping the metadata within 8 bytes (no struct padding).
> >
> > Sounds good.
>
> The additional bit per slot would require per-node logic and additional branches, which is not great. I'm now thinking a much easier way to get there is to give up (at least for now) on promising that "run-time embeddable values" can use the full pointer-size (unlike value types found embeddable at compile-time). Reserving the lowest pointer bit for a tag "value or pointer-to-leaf" would have a much smaller code footprint.
Do you mean we can make sure that the value doesn't set the lowest
bit? Or is it an optimization for TIDStore?
> In addition, without a new bitmap, the smallest node can actually be up to a node5 with no struct padding, with a node2 as a subclass. (Those numbers coincidentally were also one scenario in the paper, when calculating worst-case memory usage). That's worth considering.
Agreed.
FWIW please let me know if there are some experiments I can help with.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-11 15:54 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-04-07 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-04-17 13:48 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-04-19 07:02 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-05-23 23:16 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-06-05 10:31 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-06-06 05:13 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-06-13 05:46 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
@ 2023-06-14 06:23 ` John Naylor <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: John Naylor @ 2023-06-14 06:23 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Tue, Jun 13, 2023 at 12:47 PM Masahiko Sawada <[email protected]>
wrote:
>
> On Tue, Jun 6, 2023 at 2:13 PM John Naylor <[email protected]>
wrote:
> >
> > I'd need to make sure, but apparently just going from 6 non-empty
memory contexts to 3 (remember all values are embedded here) reduces memory
fragmentation significantly in this test. (That should also serve as a
demonstration that additional size classes have both runtime costs as well
as benefits. We need to have a balance.)
>
> Interesting. The result would probably vary if we change the slab
> block sizes. I'd like to experiment if the code is available.
I cleaned up a few things and attached v34 so you can do that if you like.
(Note: what I said about node63/n125 not making a difference in that one
test is not quite true since slab keeps a few empty blocks around. I did
some rough mental math and I think it doesn't change the conclusion any.)
0001-0007 is basically v33, but can apply on master.
0008 just adds back RT_EXTEND_DOWN. I left it out to simplify moving to
recursion.
> > Oh, the memcpy part is great, very simple. I mean the (compile-time)
"class info" table lookups are a bit awkward. I'm thinking the hard-coded
numbers like this:
> >
> > .fanout = 3,
> > .inner_size = sizeof(RT_NODE_INNER_3) + 3 * sizeof(RT_PTR_ALLOC),
> >
> > ...may be better with a #defined symbol that can also be used elsewhere.
>
> FWIW, exposing these definitions would be good in terms of testing too
> since we can use them in regression tests.
I added some definitions in 0012. It kind of doesn't matter now what sizes
are the test unless it also can test that it stays within the expected
size, if that makes sense. It is helpful during debugging to force growth
to stop at a certain size.
> > > Within the size class, we just alloc a new node of lower size class
> > > and do memcpy().
Not anymore. ;-) To be technical, it didn't "just" memcpy(), since it then
fell through to find the insert position and memmove(). In some parts of
Andres' prototype, no memmove() is necessary, because it memcpy()'s around
the insert position, and puts the new child in the right place. I've done
this in 0009.
The memcpy you mention was done for 1) simplicity 2) to avoid memset'ing.
Well, it was never necessary to memset the whole node in the first place.
Only the header, slot index array, and isset arrays need to be zeroed, so
in 0011 we always do only that. That combines alloc and init functionality,
and it's simple everywhere.
In 0010 I restored iteration functionality -- it can no longer get the
shift from the node, because it's not there as of v33. I was not
particularly impressed that there were no basic iteration tests, and in
fact the test_pattern test relied on functioning iteration. I added some
basic tests. I'm not entirely pleased with testing overall, but I think
it's at least sufficient for the job. I had the idea to replace "shift"
everywhere and use "level" as a fundamental concept. This is clearer. I do
want to make sure the compiler can compute the shift efficiently where
necessary. I think that can wait until much later.
0013 standardizes (mostly) on 4/16/48/256 for naming convention, regardless
of actual size, as I started to do earlier.
0014 is part cleanup of shrinking, and part making grow-node-48 more
consistent with the rest.
> > The additional bit per slot would require per-node logic and additional
branches, which is not great. I'm now thinking a much easier way to get
there is to give up (at least for now) on promising that "run-time
embeddable values" can use the full pointer-size (unlike value types found
embeddable at compile-time). Reserving the lowest pointer bit for a tag
"value or pointer-to-leaf" would have a much smaller code footprint.
>
> Do you mean we can make sure that the value doesn't set the lowest
> bit? Or is it an optimization for TIDStore?
It will be up to the caller (the user of the template) -- if an
abbreviation is possible that fits in the upper 63 bits (with something to
guard for 32-bit platforms), the developer will be able to specify a
conversion function so that the caller only sees the full value when
searching and setting. Without such a function, the template will fall back
to the size of the value type to determine how the value is stored.
--
John Naylor
EDB: http://www.enterprisedb.com
Attachments:
[application/gzip] v34-ART.tar.gz (75.4K, ../../CAFBsxsFsd_WnS7c-aRX9xEEhRnA3u9aBumH+_HmmmY7bCjOoBw@mail.gmail.com/3-v34-ART.tar.gz)
download
^ permalink raw reply [nested|flat] 78+ messages in thread
* Re: [PoC] Improve dead tuple storage for lazy vacuum
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-09 07:08 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-13 07:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-16 03:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-20 05:56 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-22 08:29 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-24 05:50 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-02-28 13:20 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-01 11:58 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-06 06:27 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-07 01:24 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-09 06:51 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-03-10 14:30 ` Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-03-11 15:54 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
2023-04-07 09:55 ` Re: [PoC] Improve dead tuple storage for lazy vacuum John Naylor <[email protected]>
@ 2023-05-08 10:23 ` John Naylor <[email protected]>
1 sibling, 0 replies; 78+ messages in thread
From: John Naylor @ 2023-05-08 10:23 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Matthias van de Meent <[email protected]>; Yura Sokolov <[email protected]>; pgsql-hackers
On Fri, Apr 7, 2023 at 4:55 PM John Naylor <[email protected]>
wrote:
> - Fixed-size PagetableEntry's are pretty large, but the tid compression
scheme used in this thread (in addition to being complex) is not a great
fit for tidbitmap because it makes it more difficult to track per-block
metadata (see also next point). With the "combined pointer-value slots"
technique, if a page's max tid offset is 63 or less, the offsets can be
stored directly in the pointer for the exact case. The lowest bit can tag
to indicate a pointer to a single-value leaf. That would complicate
operations like union/intersection and tracking "needs recheck", but it
would reduce memory use and node-traversal in common cases.
[just getting some thoughts out there before I have something concrete]
Thinking some more, this needn't be complicated at all. We'd just need to
reserve some bits of a bitmapword for the tag, as well as flags for
"ischunk" and "recheck". The other bits can be used for offsets.
Getting/storing the offsets basically amounts to adjusting the shift by a
constant. That way, this "embeddable PTE" could serve as both "PTE embedded
in a node pointer" and also the first member of a full PTE. A full PTE is
now just an array of embedded PTEs, except only the first one has the flags
we need. That reduces the number of places that have to be different.
Storing any set of offsets all less than ~60 would save
allocation/traversal in a large number of real cases. Furthermore, that
would reduce a full PTE to 40 bytes because there would be no padding.
This all assumes the key (block number) is no longer stored in the PTE,
whether embedded or not. That would mean this technique:
> - With lazy expansion and single-value leaves, the root of a radix tree
can point to a single leaf. That might get rid of the need to track
TBMStatus, since setting a single-leaf tree should be cheap.
...is not a good trade off because it requires each leaf to have the key,
and would thus reduce the utility of embedded leaves. We just need to make
sure storing a single value is not costly, and I suspect it's not.
(Currently the overhead avoided is allocating and zeroing a few kilobytes
for a hash table). If it is not, then we don't need a special case in
tidbitmap, which would be a great simplification. If it is, there are other
ways to mitigate.
--
John Naylor
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v15 2/2] handle relation statistics correctly during rewrites
@ 2025-11-04 13:52 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Bertrand Drouvot @ 2025-11-04 13:52 UTC (permalink / raw)
Now that PGSTAT_KIND_RELATION is keyed by refilenode, we need to handle rewrites.
To do so, this patch:
- Adds PgStat_PendingRewrite, a new struct to track rewrite operations within
a transaction, storing the old locator, new locator, and original locator (for
rewrite chains). This allows stats to be copied from the original location to
the final location at commit time.
- Adds a new function, pgstat_mark_rewrite(), called when a table rewrite begins.
It records the rewrite operation in a local list and detects rewrite chains by
checking if the old_locator matches any existing new_locator, preserving the
chain's original_locator.
- Modifies pgstat_copy_relation_stats(), to accept RelFileLocators instead of
Relations, with a new increment parameter to accumulate stats (needed for rewrite
chains with DML between rewrites).
- Ensures that AtEOXact_PgStat_Relations(), AtPrepare_PgStat_Relations(),
pgstat_twophase_postcommit()/postabort() pgstat_drop_relation() handle the
PgStat_PendingRewrite list correctly.
Note that due to the new flush call in pgstat_twophase_postcommit() we can not
call GetCurrentTransactionStopTimestamp() in pgstat_relation_flush_cb(). So,
adding a check to handle this special case and call GetCurrentTimestamp() instead.
Note that we'd call GetCurrentTimestamp() only if there is a rewrite, so that
the GetCurrentTimestamp() extra cost should be negligible. Another solution
could be to trigger the flush from FinishPreparedTransaction() but that's not
worth the extra complexity.
The new pending_rewrites list is traversed in multiple places. The overhead
should be negligible in comparison to a rewrite and the list should not contain
a lot of rewrites in practice.
The pending_rewrites list is traversed in multiple places. In typical usage,
the list will contain only a few entries so the traversal cost is negligible (
furthermore in comparison to a rewrite).
---
src/backend/catalog/index.c | 2 +-
src/backend/commands/cluster.c | 5 +
src/backend/commands/tablecmds.c | 6 +
src/backend/utils/activity/pgstat_relation.c | 391 ++++++++++++++++++-
src/backend/utils/activity/pgstat_xact.c | 25 +-
src/backend/utils/cache/relcache.c | 6 +
src/include/pgstat.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 424 insertions(+), 17 deletions(-)
92.8% src/backend/utils/activity/
4.9% src/backend/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 1ccfa687f05..e292199ca7c 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1796,7 +1796,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId);
/* copy over statistics from old to new index */
- pgstat_copy_relation_stats(newClassRel, oldClassRel);
+ pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 09066db0956..c491daceb0f 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1206,6 +1206,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
rel1 = relation_open(r1, NoLock);
rel2 = relation_open(r2, NoLock);
+
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind))
+ pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator);
+
rel2->rd_createSubid = rel1->rd_createSubid;
rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid;
rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 8b4ebc6f226..49de04d9697 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -17102,6 +17102,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
Oid reltoastrelid;
RelFileNumber newrelfilenumber;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator;
List *reltoastidxids = NIL;
ListCell *lc;
@@ -17140,6 +17141,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
newrlocator = rel->rd_locator;
newrlocator.relNumber = newrelfilenumber;
newrlocator.spcOid = newTableSpace;
+ oldrlocator = rel->rd_locator;
/* hand off to AM to actually create new rel storage and copy the data */
if (rel->rd_rel->relkind == RELKIND_INDEX)
@@ -17152,6 +17154,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
table_relation_copy_data(rel, &newrlocator);
}
+ /* mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Update the pg_class row.
*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 89bf0cbed56..4d929972fab 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -30,6 +30,19 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
+/* Pending rewrite operations for stats copying */
+typedef struct PgStat_PendingRewrite
+{
+ RelFileLocator old_locator;
+ RelFileLocator new_locator;
+ RelFileLocator original_locator;
+ int nest_level; /* Transaction nesting level where rewrite
+ * occurred */
+ struct PgStat_PendingRewrite *next;
+} PgStat_PendingRewrite;
+
+/* The pending rewrites list for current transaction */
+static PgStat_PendingRewrite *pending_rewrites = NULL;
/* Record that's written to 2PC state file when pgstat state is persisted */
typedef struct TwoPhasePgStatRecord
@@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter deleted_pre_truncdrop;
RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
+ RelFileLocator rewrite_old_locator;
+ int rewrite_nest_level;
} TwoPhasePgStatRecord;
@@ -54,27 +69,70 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
/*
- * Copy stats between relations. This is used for things like REINDEX
+ * Copy stats between RelFileLocator. This is used for things like REINDEX
* CONCURRENTLY.
*/
void
-pgstat_copy_relation_stats(Relation dst, Relation src)
+pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment)
{
PgStat_StatTabEntry *srcstats;
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src));
+ srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION,
+ src.dbOid,
+ RelFileLocatorToPgStatObjid(src));
if (!srcstats)
return;
dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
- dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(dst),
+ dst.dbOid,
+ RelFileLocatorToPgStatObjid(dst),
false);
dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats;
- dstshstats->stats = *srcstats;
+
+ if (!increment)
+ dstshstats->stats = *srcstats;
+ else
+ {
+ /* Increment those statistics */
+#define RELFSTAT_ACC(fld, stats_to_add) \
+ (dstshstats->stats.fld += stats_to_add->fld)
+ RELFSTAT_ACC(numscans, srcstats);
+ RELFSTAT_ACC(tuples_returned, srcstats);
+ RELFSTAT_ACC(tuples_fetched, srcstats);
+ RELFSTAT_ACC(tuples_inserted, srcstats);
+ RELFSTAT_ACC(tuples_updated, srcstats);
+ RELFSTAT_ACC(tuples_deleted, srcstats);
+ RELFSTAT_ACC(tuples_hot_updated, srcstats);
+ RELFSTAT_ACC(tuples_newpage_updated, srcstats);
+ RELFSTAT_ACC(live_tuples, srcstats);
+ RELFSTAT_ACC(dead_tuples, srcstats);
+ RELFSTAT_ACC(mod_since_analyze, srcstats);
+ RELFSTAT_ACC(ins_since_vacuum, srcstats);
+ RELFSTAT_ACC(blocks_fetched, srcstats);
+ RELFSTAT_ACC(blocks_hit, srcstats);
+ RELFSTAT_ACC(vacuum_count, srcstats);
+ RELFSTAT_ACC(autovacuum_count, srcstats);
+ RELFSTAT_ACC(analyze_count, srcstats);
+ RELFSTAT_ACC(autoanalyze_count, srcstats);
+ RELFSTAT_ACC(total_vacuum_time, srcstats);
+ RELFSTAT_ACC(total_autovacuum_time, srcstats);
+ RELFSTAT_ACC(total_analyze_time, srcstats);
+ RELFSTAT_ACC(total_autoanalyze_time, srcstats);
+#undef RELFSTAT_ACC
+
+ /* Replace those statistics */
+#define RELFSTAT_REP(fld, stats_to_rep) \
+ (dstshstats->stats.fld = stats_to_rep->fld)
+ RELFSTAT_REP(lastscan, srcstats);
+ RELFSTAT_REP(last_vacuum_time, srcstats);
+ RELFSTAT_REP(last_autovacuum_time, srcstats);
+ RELFSTAT_REP(last_analyze_time, srcstats);
+ RELFSTAT_REP(last_autoanalyze_time, srcstats);
+#undef RELFSTAT_REP
+ }
pgstat_unlock_entry(dst_ref);
}
@@ -136,6 +194,7 @@ void
pgstat_assoc_relation(Relation rel)
{
RelFileLocator locator;
+ PgStat_TableStatus *pgstat_info;
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
@@ -164,14 +223,54 @@ pgstat_assoc_relation(Relation rel)
locator.relNumber = rel->rd_id;
}
+ /*
+ * If this relation was rewritten during the current transaction we may be
+ * reopening it with its new RelFileLocator. In that case, continue using
+ * the stats entry associated with the old locator rather than creating a
+ * new one. This ensures all stats from before and after the rewrite are
+ * tracked in a single entry which will be properly copied to the new
+ * locator at transaction commit.
+ */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (locator.dbOid == rewrite->new_locator.dbOid &&
+ locator.spcOid == rewrite->new_locator.spcOid &&
+ locator.relNumber == rewrite->new_locator.relNumber)
+ {
+ pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator);
+ goto found_entry;
+ }
+ }
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(locator);
+ pgstat_info = pgstat_prep_relation_pending(locator);
+
+found_entry:
+ rel->pgstat_info = pgstat_info;
+
+ /*
+ * For relations stats, we key by physical file location, not by relation
+ * OID. This means during operations like ALTER TYPE it's possible that
+ * the relation OID changes but the relfilenode stays the same (no actual
+ * rewrite needed). Unlink the old relation first.
+ */
+ if (pgstat_info->relation != NULL &&
+ pgstat_info->relation != rel)
+ {
+ pgstat_info->relation->pgstat_info = NULL;
+ pgstat_info->relation = NULL;
+ }
/* don't allow link a stats to multiple relcache entries */
- Assert(rel->pgstat_info->relation == NULL);
+ Assert(pgstat_info->relation == NULL);
/* mark this relation as the owner */
- rel->pgstat_info->relation = rel;
+ pgstat_info->relation = rel;
}
/*
@@ -214,14 +313,37 @@ pgstat_drop_relation(Relation rel)
{
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ bool skip_transactional_drop = false;
/* don't track stats for relations without storage */
if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
return;
- pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_locator.dbOid,
- RelFileLocatorToPgStatObjid(rel->rd_locator));
+ /* Check if this drop is part of a pending rewrite */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid &&
+ rel->rd_locator.spcOid == rewrite->old_locator.spcOid &&
+ rel->rd_locator.relNumber == rewrite->old_locator.relNumber)
+ {
+ skip_transactional_drop = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * If it is part of a rewrite, drop its stats later, for example in
+ * AtEOXact_PgStat_Relations(), so skip it here.
+ */
+ if (!skip_transactional_drop)
+ pgstat_drop_transactional(PGSTAT_KIND_RELATION,
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -666,6 +788,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit)
}
tabstat->trans = NULL;
}
+
+ /* preserve the stats in case of rewrite */
+ if (isCommit && pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *prev = NULL;
+ PgStat_PendingRewrite *current = pending_rewrites;
+ PgStat_PendingRewrite *next;
+
+ /* reverse the rewrites list to process in chronological order */
+ while (current != NULL)
+ {
+ next = current->next;
+ current->next = prev;
+ prev = current;
+ current = next;
+ }
+
+ /* now process rewrites in chronological order */
+ for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ pgstat_copy_relation_stats(rewrite->new_locator,
+ rewrite->old_locator, true);
+
+ /* drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
+ }
+
+ pending_rewrites = NULL;
}
/*
@@ -681,6 +845,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in
PgStat_TableXactStatus *trans;
PgStat_TableXactStatus *next_trans;
+ /*
+ * If we don't commit then remove the associated rewrites if any, to keep
+ * the rewrite chain in sync with what will be eventually committed.
+ */
+ if (!isCommit)
+ {
+ PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites;
+
+ while (*rewrite_ptr != NULL)
+ {
+ if ((*rewrite_ptr)->nest_level >= nestDepth)
+ {
+ PgStat_PendingRewrite *to_remove = *rewrite_ptr;
+
+ *rewrite_ptr = (*rewrite_ptr)->next;
+ pfree(to_remove);
+ }
+ else
+ {
+ rewrite_ptr = &((*rewrite_ptr)->next);
+ }
+ }
+ }
+
for (trans = xact_state->first; trans != NULL; trans = next_trans)
{
PgStat_TableStatus *tabstat;
@@ -760,11 +948,19 @@ void
AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
{
PgStat_TableXactStatus *trans;
+ PgStat_PendingRewrite *rewrite;
+ /*
+ * For each tabstat, find its matching rewrite and remove it from the
+ * pending rewrites list. This way, after processing all tabstats, pending
+ * rewrites will only contain rewrite only transactions.
+ */
for (trans = xact_state->first; trans != NULL; trans = trans->next)
{
PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY;
TwoPhasePgStatRecord record;
+ PgStat_PendingRewrite **rewrite_ptr;
+ bool found_rewrite = false;
Assert(trans->nest_level == 1);
Assert(trans->upper == NULL);
@@ -784,10 +980,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.locator = tabstat->locator;
record.truncdropped = trans->truncdropped;
+ record.rewrite_nest_level = 0;
+
+ /*
+ * Look for a matching rewrite and remove it from pending rewrites. We
+ * check three possible matches:
+ *
+ * The new_locator when stats have been added after the rewrite. The
+ * old_locator when stats have been added before the rewrite but not
+ * after. The original_locator when this tabstat is part of a rewrite
+ * chain.
+ */
+ rewrite_ptr = &pending_rewrites;
+ while (*rewrite_ptr != NULL)
+ {
+ rewrite = *rewrite_ptr;
+
+ if ((record.locator.dbOid == rewrite->new_locator.dbOid &&
+ record.locator.spcOid == rewrite->new_locator.spcOid &&
+ record.locator.relNumber == rewrite->new_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->old_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->old_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->old_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->original_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->original_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->original_locator.relNumber))
+ {
+ /*
+ * Found matching rewrite. Record the rewrite information and
+ * remove this rewrite from the list since it's now handled.
+ */
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.locator = rewrite->new_locator;
+ found_rewrite = true;
+
+ /* Remove from pending_rewrites list */
+ *rewrite_ptr = rewrite->next;
+ pfree(rewrite);
+ break;
+ }
+ else
+ {
+ /* Move to next rewrite in the list */
+ rewrite_ptr = &(rewrite->next);
+ }
+ }
+
+ /* If no rewrite found, clear the rewrite fields */
+ if (!found_rewrite)
+ {
+ memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator));
+ }
+
+ RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
+ &record, sizeof(TwoPhasePgStatRecord));
+ }
+
+ /*
+ * Now process any rewrites still pending. These are rewrite only
+ * transactions. We need to preserve their stats even though there's no
+ * tabstat entry for them.
+ */
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ TwoPhasePgStatRecord record;
+
+ memset(&record, 0, sizeof(TwoPhasePgStatRecord));
+ record.locator = rewrite->new_locator;
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.truncdropped = false;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
&record, sizeof(TwoPhasePgStatRecord));
}
+
+ pending_rewrites = NULL;
}
/*
@@ -810,6 +1079,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
tabstat = trans->parent;
tabstat->trans = NULL;
}
+
+ pending_rewrites = NULL;
}
/*
@@ -845,6 +1116,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
pgstat_info->counts.changed_tuples +=
rec->tuples_inserted + rec->tuples_updated +
rec->tuples_deleted;
+
+ if (rec->rewrite_nest_level > 0)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ /* Flush any pending stats for old locator first */
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ /* Copy stats from old to new locator */
+ pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator,
+ true);
+
+ /* Drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
}
/*
@@ -859,9 +1153,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
{
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
+ RelFileLocator target_locator;
+
+ /*
+ * For aborted transactions with rewrites (like TRUNCATE), we need to
+ * restore stats to the old locator, not the new one. The new locator
+ * should be dropped since the rewrite is being rolled back.
+ */
+ if (rec->rewrite_nest_level > 0)
+ {
+ /* Use the old locator */
+ target_locator = rec->rewrite_old_locator;
+ }
+ else
+ {
+ /* No rewrite, use the original locator */
+ target_locator = rec->locator;
+ }
/* Find or create a tabstat entry for the target locator */
- pgstat_info = pgstat_prep_relation_pending(rec->locator);
+ pgstat_info = pgstat_prep_relation_pending(target_locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -916,7 +1227,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
+ TimestampTz t;
+
+ /*
+ * Checking the transaction state due to the flush call in
+ * pgstat_twophase_postcommit() that would break the assertion on the
+ * state in GetCurrentTransactionStopTimestamp().
+ */
+ if (!IsTransactionState())
+ t = GetCurrentTransactionStopTimestamp();
+ else
+ t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
@@ -1167,3 +1488,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
ReleaseSysCache(tuple);
return result;
}
+
+/*
+ * Mark that a relation rewrite has occurred, preserving the original locator
+ * so stats can be copied at transaction commit.
+ */
+void
+pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator)
+{
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *existing;
+ RelFileLocator original_locator = old_locator;
+
+ for (existing = pending_rewrites; existing != NULL; existing = existing->next)
+ {
+ if (old_locator.dbOid == existing->new_locator.dbOid &&
+ old_locator.spcOid == existing->new_locator.spcOid &&
+ old_locator.relNumber == existing->new_locator.relNumber)
+ {
+ original_locator = existing->original_locator;
+ break;
+ }
+ }
+
+ /* Allocate in TopTransactionContext memory context */
+ rewrite = MemoryContextAlloc(TopTransactionContext,
+ sizeof(PgStat_PendingRewrite));
+
+ rewrite->old_locator = old_locator;
+ rewrite->new_locator = new_locator;
+ rewrite->original_locator = original_locator;
+ rewrite->nest_level = GetCurrentTransactionNestLevel();
+
+ /* Add to the list */
+ rewrite->next = pending_rewrites;
+ pending_rewrites = rewrite;
+}
+
+void
+pgstat_clear_rewrite(void)
+{
+ pending_rewrites = NULL;
+}
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index 5e2d69e6297..8ed8f5317f3 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel)
}
pgStatXactStack = NULL;
+ pgstat_clear_rewrite();
+
/* Make sure any stats snapshot is thrown away */
pgstat_clear_snapshot();
}
@@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo
void
pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid)
{
- if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL))
+ PgStat_EntryRef *entry_ref;
+
+ entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL);
+
+ if (entry_ref)
{
+ /*
+ * For relations stats, we key by physical file location, not by
+ * relation OID. This means during operations like ALTER TYPE where
+ * the relation OID changes but the relfilenode stays the same (no
+ * actual rewrite needed), we'll find an existing entry.
+ *
+ * This is expected behavior, we want to preserve stats across the
+ * catalog change. Simply reset and recreate the entry for the new
+ * relation OID without warning.
+ */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ pgstat_reset(kind, dboid, objid);
+ create_drop_transactional_internal(kind, dboid, objid, true);
+ return;
+ }
+
ereport(WARNING,
errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64,
(pgstat_get_kind_info(kind))->name, dboid,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index e19f0d3e51c..bf49830dba2 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -87,6 +87,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/relmapper.h"
#include "utils/resowner.h"
#include "utils/snapmgr.h"
@@ -3772,6 +3773,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
MultiXactId minmulti = InvalidMultiXactId;
TransactionId freezeXid = InvalidTransactionId;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator = relation->rd_locator;
if (!IsBinaryUpgrade)
{
@@ -3943,6 +3945,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
table_close(pg_class, RowExclusiveLock);
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Make the pg_class row change or relation map change visible. This will
* cause the relcache entry to get updated, too.
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index ef740aff26e..4d3130ae99e 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -701,7 +701,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id);
extern void pgstat_create_relation(Relation rel);
extern void pgstat_drop_relation(Relation rel);
-extern void pgstat_copy_relation_stats(Relation dst, Relation src);
+extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment);
extern void pgstat_init_relation(Relation rel);
extern void pgstat_assoc_relation(Relation rel);
@@ -713,6 +713,9 @@ extern void pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_mark_rewrite(RelFileLocator old_locator,
+ RelFileLocator new_locator);
+extern void pgstat_clear_rewrite(void);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index c9da1f91cb9..53c94382a8d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2314,6 +2314,7 @@ PgStat_LockEntry
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
PgStat_PendingLock
+PgStat_PendingRewrite
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
--Og8T0Lf2JVsuKOzn--
^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v7 3/3] handle relation statistics correctly during rewrites
@ 2025-11-04 13:52 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Bertrand Drouvot @ 2025-11-04 13:52 UTC (permalink / raw)
Now that PGSTAT_KIND_RELATION is keyed by refilenode, we need to handle rewrites.
To do so, this patch:
- Adds PgStat_PendingRewrite, a new struct to track rewrite operations within
a transaction, storing the old locator, new locator, and original locator (for
rewrite chains). This allows stats to be copied from the original location to
the final location at commit time.
- Adds a new function, pgstat_mark_rewrite(), called when a table rewrite begins.
It records the rewrite operation in a local list and detects rewrite chains by
checking if the old_locator matches any existing new_locator, preserving the
chain's original_locator.
- Modifies pgstat_copy_relation_stats(), to accept RelFileLocators instead of
Relations, with a new increment parameter to accumulate stats (needed for rewrite
chains with DML between rewrites).
- Ensures that AtEOXact_PgStat_Relations(), AtPrepare_PgStat_Relations(),
pgstat_twophase_postcommit()/postabort() pgstat_drop_relation() handle the
PgStat_PendingRewrite list correctly.
Note that due to the new flush call in pgstat_twophase_postcommit() we can not
call GetCurrentTransactionStopTimestamp() in pgstat_relation_flush_cb(). So,
adding a check to handle this special case and call GetCurrentTimestamp() instead.
Note that we'd call GetCurrentTimestamp() only if there is a rewrite, so that
the GetCurrentTimestamp() extra cost should be negligible. Another solution
could be to trigger the flush from FinishPreparedTransaction() but that's not
worth the extra complexity.
The new pending_rewrites list is traversed in multiple places. The overhead
should be negligible in comparison to a rewrite and the list should not contain
a lot of rewrites in practice.
The pending_rewrites list is traversed in multiple places. In typical usage,
the list will contain only a few entries so the traversal cost is negligible (
furthermore in comparison to a rewrite).
---
src/backend/catalog/index.c | 2 +-
src/backend/commands/cluster.c | 5 +
src/backend/commands/tablecmds.c | 6 +
src/backend/utils/activity/pgstat_relation.c | 391 ++++++++++++++++++-
src/backend/utils/activity/pgstat_xact.c | 25 +-
src/backend/utils/cache/relcache.c | 6 +
src/include/pgstat.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 424 insertions(+), 17 deletions(-)
92.8% src/backend/utils/activity/
4.9% src/backend/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 5d9db167e59..8b6a7652fcf 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1795,7 +1795,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId);
/* copy over statistics from old to new index */
- pgstat_copy_relation_stats(newClassRel, oldClassRel);
+ pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index b55221d44cd..da75dfa6ab8 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1196,6 +1196,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
rel1 = relation_open(r1, NoLock);
rel2 = relation_open(r2, NoLock);
+
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind))
+ pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator);
+
rel2->rd_createSubid = rel1->rd_createSubid;
rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid;
rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 3aac459e483..540923452fb 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -16848,6 +16848,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
Oid reltoastrelid;
RelFileNumber newrelfilenumber;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator;
List *reltoastidxids = NIL;
ListCell *lc;
@@ -16886,6 +16887,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
newrlocator = rel->rd_locator;
newrlocator.relNumber = newrelfilenumber;
newrlocator.spcOid = newTableSpace;
+ oldrlocator = rel->rd_locator;
/* hand off to AM to actually create new rel storage and copy the data */
if (rel->rd_rel->relkind == RELKIND_INDEX)
@@ -16898,6 +16900,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
table_relation_copy_data(rel, &newrlocator);
}
+ /* mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Update the pg_class row.
*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 7debb14bb5d..15b4663eb77 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -30,6 +30,19 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
+/* Pending rewrite operations for stats copying */
+typedef struct PgStat_PendingRewrite
+{
+ RelFileLocator old_locator;
+ RelFileLocator new_locator;
+ RelFileLocator original_locator;
+ int nest_level; /* Transaction nesting level where rewrite
+ * occurred */
+ struct PgStat_PendingRewrite *next;
+} PgStat_PendingRewrite;
+
+/* The pending rewrites list for current transaction */
+static PgStat_PendingRewrite *pending_rewrites = NULL;
/* Record that's written to 2PC state file when pgstat state is persisted */
typedef struct TwoPhasePgStatRecord
@@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter deleted_pre_truncdrop;
RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
+ RelFileLocator rewrite_old_locator;
+ int rewrite_nest_level;
} TwoPhasePgStatRecord;
@@ -54,27 +69,70 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
/*
- * Copy stats between relations. This is used for things like REINDEX
+ * Copy stats between RelFileLocator. This is used for things like REINDEX
* CONCURRENTLY.
*/
void
-pgstat_copy_relation_stats(Relation dst, Relation src)
+pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment)
{
PgStat_StatTabEntry *srcstats;
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src));
+ srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION,
+ src.dbOid,
+ RelFileLocatorToPgStatObjid(src));
if (!srcstats)
return;
dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
- dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(dst),
+ dst.dbOid,
+ RelFileLocatorToPgStatObjid(dst),
false);
dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats;
- dstshstats->stats = *srcstats;
+
+ if (!increment)
+ dstshstats->stats = *srcstats;
+ else
+ {
+ /* Increment those statistics */
+#define RELFSTAT_ACC(fld, stats_to_add) \
+ (dstshstats->stats.fld += stats_to_add->fld)
+ RELFSTAT_ACC(numscans, srcstats);
+ RELFSTAT_ACC(tuples_returned, srcstats);
+ RELFSTAT_ACC(tuples_fetched, srcstats);
+ RELFSTAT_ACC(tuples_inserted, srcstats);
+ RELFSTAT_ACC(tuples_updated, srcstats);
+ RELFSTAT_ACC(tuples_deleted, srcstats);
+ RELFSTAT_ACC(tuples_hot_updated, srcstats);
+ RELFSTAT_ACC(tuples_newpage_updated, srcstats);
+ RELFSTAT_ACC(live_tuples, srcstats);
+ RELFSTAT_ACC(dead_tuples, srcstats);
+ RELFSTAT_ACC(mod_since_analyze, srcstats);
+ RELFSTAT_ACC(ins_since_vacuum, srcstats);
+ RELFSTAT_ACC(blocks_fetched, srcstats);
+ RELFSTAT_ACC(blocks_hit, srcstats);
+ RELFSTAT_ACC(vacuum_count, srcstats);
+ RELFSTAT_ACC(autovacuum_count, srcstats);
+ RELFSTAT_ACC(analyze_count, srcstats);
+ RELFSTAT_ACC(autoanalyze_count, srcstats);
+ RELFSTAT_ACC(total_vacuum_time, srcstats);
+ RELFSTAT_ACC(total_autovacuum_time, srcstats);
+ RELFSTAT_ACC(total_analyze_time, srcstats);
+ RELFSTAT_ACC(total_autoanalyze_time, srcstats);
+#undef RELFSTAT_ACC
+
+ /* Replace those statistics */
+#define RELFSTAT_REP(fld, stats_to_rep) \
+ (dstshstats->stats.fld = stats_to_rep->fld)
+ RELFSTAT_REP(lastscan, srcstats);
+ RELFSTAT_REP(last_vacuum_time, srcstats);
+ RELFSTAT_REP(last_autovacuum_time, srcstats);
+ RELFSTAT_REP(last_analyze_time, srcstats);
+ RELFSTAT_REP(last_autoanalyze_time, srcstats);
+#undef RELFSTAT_REP
+ }
pgstat_unlock_entry(dst_ref);
}
@@ -136,6 +194,7 @@ void
pgstat_assoc_relation(Relation rel)
{
RelFileLocator locator;
+ PgStat_TableStatus *pgstat_info;
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
@@ -165,14 +224,54 @@ pgstat_assoc_relation(Relation rel)
locator.relNumber = rel->rd_id;
}
+ /*
+ * If this relation was rewritten during the current transaction we may be
+ * reopening it with its new RelFileLocator. In that case, continue using
+ * the stats entry associated with the old locator rather than creating a
+ * new one. This ensures all stats from before and after the rewrite are
+ * tracked in a single entry which will be properly copied to the new
+ * locator at transaction commit.
+ */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (locator.dbOid == rewrite->new_locator.dbOid &&
+ locator.spcOid == rewrite->new_locator.spcOid &&
+ locator.relNumber == rewrite->new_locator.relNumber)
+ {
+ pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator);
+ goto found_entry;
+ }
+ }
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(locator);
+ pgstat_info = pgstat_prep_relation_pending(locator);
+
+found_entry:
+ rel->pgstat_info = pgstat_info;
+
+ /*
+ * For relations stats, we key by physical file location, not by relation
+ * OID. This means during operations like ALTER TYPE it's possible that
+ * the relation OID changes but the relfilenode stays the same (no actual
+ * rewrite needed). Unlink the old relation first.
+ */
+ if (pgstat_info->relation != NULL &&
+ pgstat_info->relation != rel)
+ {
+ pgstat_info->relation->pgstat_info = NULL;
+ pgstat_info->relation = NULL;
+ }
/* don't allow link a stats to multiple relcache entries */
- Assert(rel->pgstat_info->relation == NULL);
+ Assert(pgstat_info->relation == NULL);
/* mark this relation as the owner */
- rel->pgstat_info->relation = rel;
+ pgstat_info->relation = rel;
}
/*
@@ -215,14 +314,37 @@ pgstat_drop_relation(Relation rel)
{
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ bool skip_transactional_drop = false;
/* don't track stats for relations without storage */
if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
return;
- pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_locator.dbOid,
- RelFileLocatorToPgStatObjid(rel->rd_locator));
+ /* Check if this drop is part of a pending rewrite */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid &&
+ rel->rd_locator.spcOid == rewrite->old_locator.spcOid &&
+ rel->rd_locator.relNumber == rewrite->old_locator.relNumber)
+ {
+ skip_transactional_drop = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * If it is part of a rewrite, drop its stats later, for example in
+ * AtEOXact_PgStat_Relations(), so skip it here.
+ */
+ if (!skip_transactional_drop)
+ pgstat_drop_transactional(PGSTAT_KIND_RELATION,
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -660,6 +782,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit)
}
tabstat->trans = NULL;
}
+
+ /* preserve the stats in case of rewrite */
+ if (isCommit && pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *prev = NULL;
+ PgStat_PendingRewrite *current = pending_rewrites;
+ PgStat_PendingRewrite *next;
+
+ /* reverse the rewrites list to process in chronological order */
+ while (current != NULL)
+ {
+ next = current->next;
+ current->next = prev;
+ prev = current;
+ current = next;
+ }
+
+ /* now process rewrites in chronological order */
+ for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ pgstat_copy_relation_stats(rewrite->new_locator,
+ rewrite->old_locator, true);
+
+ /* drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
+ }
+
+ pending_rewrites = NULL;
}
/*
@@ -675,6 +839,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in
PgStat_TableXactStatus *trans;
PgStat_TableXactStatus *next_trans;
+ /*
+ * If we don't commit then remove the associated rewrites if any, to keep
+ * the rewrite chain in sync with what will be eventually committed.
+ */
+ if (!isCommit)
+ {
+ PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites;
+
+ while (*rewrite_ptr != NULL)
+ {
+ if ((*rewrite_ptr)->nest_level >= nestDepth)
+ {
+ PgStat_PendingRewrite *to_remove = *rewrite_ptr;
+
+ *rewrite_ptr = (*rewrite_ptr)->next;
+ pfree(to_remove);
+ }
+ else
+ {
+ rewrite_ptr = &((*rewrite_ptr)->next);
+ }
+ }
+ }
+
for (trans = xact_state->first; trans != NULL; trans = next_trans)
{
PgStat_TableStatus *tabstat;
@@ -754,11 +942,19 @@ void
AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
{
PgStat_TableXactStatus *trans;
+ PgStat_PendingRewrite *rewrite;
+ /*
+ * For each tabstat, find its matching rewrite and remove it from the
+ * pending rewrites list. This way, after processing all tabstats, pending
+ * rewrites will only contain rewrite only transactions.
+ */
for (trans = xact_state->first; trans != NULL; trans = trans->next)
{
PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY;
TwoPhasePgStatRecord record;
+ PgStat_PendingRewrite **rewrite_ptr;
+ bool found_rewrite = false;
Assert(trans->nest_level == 1);
Assert(trans->upper == NULL);
@@ -778,10 +974,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.locator = tabstat->locator;
record.truncdropped = trans->truncdropped;
+ record.rewrite_nest_level = 0;
+
+ /*
+ * Look for a matching rewrite and remove it from pending rewrites. We
+ * check three possible matches:
+ *
+ * The new_locator when stats have been added after the rewrite. The
+ * old_locator when stats have been added before the rewrite but not
+ * after. The original_locator when this tabstat is part of a rewrite
+ * chain.
+ */
+ rewrite_ptr = &pending_rewrites;
+ while (*rewrite_ptr != NULL)
+ {
+ rewrite = *rewrite_ptr;
+
+ if ((record.locator.dbOid == rewrite->new_locator.dbOid &&
+ record.locator.spcOid == rewrite->new_locator.spcOid &&
+ record.locator.relNumber == rewrite->new_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->old_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->old_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->old_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->original_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->original_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->original_locator.relNumber))
+ {
+ /*
+ * Found matching rewrite. Record the rewrite information and
+ * remove this rewrite from the list since it's now handled.
+ */
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.locator = rewrite->new_locator;
+ found_rewrite = true;
+
+ /* Remove from pending_rewrites list */
+ *rewrite_ptr = rewrite->next;
+ pfree(rewrite);
+ break;
+ }
+ else
+ {
+ /* Move to next rewrite in the list */
+ rewrite_ptr = &(rewrite->next);
+ }
+ }
+
+ /* If no rewrite found, clear the rewrite fields */
+ if (!found_rewrite)
+ {
+ memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator));
+ }
+
+ RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
+ &record, sizeof(TwoPhasePgStatRecord));
+ }
+
+ /*
+ * Now process any rewrites still pending. These are rewrite only
+ * transactions. We need to preserve their stats even though there's no
+ * tabstat entry for them.
+ */
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ TwoPhasePgStatRecord record;
+
+ memset(&record, 0, sizeof(TwoPhasePgStatRecord));
+ record.locator = rewrite->new_locator;
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.truncdropped = false;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
&record, sizeof(TwoPhasePgStatRecord));
}
+
+ pending_rewrites = NULL;
}
/*
@@ -804,6 +1073,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
tabstat = trans->parent;
tabstat->trans = NULL;
}
+
+ pending_rewrites = NULL;
}
/*
@@ -839,6 +1110,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
pgstat_info->counts.changed_tuples +=
rec->tuples_inserted + rec->tuples_updated +
rec->tuples_deleted;
+
+ if (rec->rewrite_nest_level > 0)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ /* Flush any pending stats for old locator first */
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ /* Copy stats from old to new locator */
+ pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator,
+ true);
+
+ /* Drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
}
/*
@@ -853,9 +1147,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
{
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
+ RelFileLocator target_locator;
+
+ /*
+ * For aborted transactions with rewrites (like TRUNCATE), we need to
+ * restore stats to the old locator, not the new one. The new locator
+ * should be dropped since the rewrite is being rolled back.
+ */
+ if (rec->rewrite_nest_level > 0)
+ {
+ /* Use the old locator */
+ target_locator = rec->rewrite_old_locator;
+ }
+ else
+ {
+ /* No rewrite, use the original locator */
+ target_locator = rec->locator;
+ }
/* Find or create a tabstat entry for the target locator */
- pgstat_info = pgstat_prep_relation_pending(rec->locator);
+ pgstat_info = pgstat_prep_relation_pending(target_locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -910,7 +1221,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
+ TimestampTz t;
+
+ /*
+ * Checking the transaction state due to the flush call in
+ * pgstat_twophase_postcommit() that would break the assertion on the
+ * state in GetCurrentTransactionStopTimestamp().
+ */
+ if (!IsTransactionState())
+ t = GetCurrentTransactionStopTimestamp();
+ else
+ t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
@@ -1162,3 +1483,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
ReleaseSysCache(tuple);
return result;
}
+
+/*
+ * Mark that a relation rewrite has occurred, preserving the original locator
+ * so stats can be copied at transaction commit.
+ */
+void
+pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator)
+{
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *existing;
+ RelFileLocator original_locator = old_locator;
+
+ for (existing = pending_rewrites; existing != NULL; existing = existing->next)
+ {
+ if (old_locator.dbOid == existing->new_locator.dbOid &&
+ old_locator.spcOid == existing->new_locator.spcOid &&
+ old_locator.relNumber == existing->new_locator.relNumber)
+ {
+ original_locator = existing->original_locator;
+ break;
+ }
+ }
+
+ /* Allocate in TopTransactionContext memory context */
+ rewrite = MemoryContextAlloc(TopTransactionContext,
+ sizeof(PgStat_PendingRewrite));
+
+ rewrite->old_locator = old_locator;
+ rewrite->new_locator = new_locator;
+ rewrite->original_locator = original_locator;
+ rewrite->nest_level = GetCurrentTransactionNestLevel();
+
+ /* Add to the list */
+ rewrite->next = pending_rewrites;
+ pending_rewrites = rewrite;
+}
+
+void
+pgstat_clear_rewrite(void)
+{
+ pending_rewrites = NULL;
+}
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index bc9864bd8d9..f8cf3755ce2 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel)
}
pgStatXactStack = NULL;
+ pgstat_clear_rewrite();
+
/* Make sure any stats snapshot is thrown away */
pgstat_clear_snapshot();
}
@@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo
void
pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid)
{
- if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL))
+ PgStat_EntryRef *entry_ref;
+
+ entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL);
+
+ if (entry_ref)
{
+ /*
+ * For relations stats, we key by physical file location, not by
+ * relation OID. This means during operations like ALTER TYPE where
+ * the relation OID changes but the relfilenode stays the same (no
+ * actual rewrite needed), we'll find an existing entry.
+ *
+ * This is expected behavior, we want to preserve stats across the
+ * catalog change. Simply reset and recreate the entry for the new
+ * relation OID without warning.
+ */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ pgstat_reset(kind, dboid, objid);
+ create_drop_transactional_internal(kind, dboid, objid, true);
+ return;
+ }
+
ereport(WARNING,
errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64,
(pgstat_get_kind_info(kind))->name, dboid,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 915d0bc9084..7a7f8023eb3 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -85,6 +85,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/relmapper.h"
#include "utils/resowner.h"
#include "utils/snapmgr.h"
@@ -3780,6 +3781,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
MultiXactId minmulti = InvalidMultiXactId;
TransactionId freezeXid = InvalidTransactionId;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator = relation->rd_locator;
if (!IsBinaryUpgrade)
{
@@ -3951,6 +3953,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
table_close(pg_class, RowExclusiveLock);
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Make the pg_class row change or relation map change visible. This will
* cause the relcache entry to get updated, too.
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 5d0fe79f7e3..332dffde400 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -665,7 +665,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id);
extern void pgstat_create_relation(Relation rel);
extern void pgstat_drop_relation(Relation rel);
-extern void pgstat_copy_relation_stats(Relation dst, Relation src);
+extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment);
extern void pgstat_init_relation(Relation rel);
extern void pgstat_assoc_relation(Relation rel);
@@ -677,6 +677,9 @@ extern void pgstat_report_vacuum(RelFileLocator locator, PgStat_Counter livetupl
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_mark_rewrite(RelFileLocator old_locator,
+ RelFileLocator new_locator);
+extern void pgstat_clear_rewrite(void);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 432509277c9..bd8fcb16dcf 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2248,6 +2248,7 @@ PgStat_KindInfo
PgStat_LocalState
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
+PgStat_PendingRewrite
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
--KMYjwlsACqdYx2P8--
^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v11 2/2] handle relation statistics correctly during rewrites
@ 2025-11-04 13:52 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Bertrand Drouvot @ 2025-11-04 13:52 UTC (permalink / raw)
Now that PGSTAT_KIND_RELATION is keyed by refilenode, we need to handle rewrites.
To do so, this patch:
- Adds PgStat_PendingRewrite, a new struct to track rewrite operations within
a transaction, storing the old locator, new locator, and original locator (for
rewrite chains). This allows stats to be copied from the original location to
the final location at commit time.
- Adds a new function, pgstat_mark_rewrite(), called when a table rewrite begins.
It records the rewrite operation in a local list and detects rewrite chains by
checking if the old_locator matches any existing new_locator, preserving the
chain's original_locator.
- Modifies pgstat_copy_relation_stats(), to accept RelFileLocators instead of
Relations, with a new increment parameter to accumulate stats (needed for rewrite
chains with DML between rewrites).
- Ensures that AtEOXact_PgStat_Relations(), AtPrepare_PgStat_Relations(),
pgstat_twophase_postcommit()/postabort() pgstat_drop_relation() handle the
PgStat_PendingRewrite list correctly.
Note that due to the new flush call in pgstat_twophase_postcommit() we can not
call GetCurrentTransactionStopTimestamp() in pgstat_relation_flush_cb(). So,
adding a check to handle this special case and call GetCurrentTimestamp() instead.
Note that we'd call GetCurrentTimestamp() only if there is a rewrite, so that
the GetCurrentTimestamp() extra cost should be negligible. Another solution
could be to trigger the flush from FinishPreparedTransaction() but that's not
worth the extra complexity.
The new pending_rewrites list is traversed in multiple places. The overhead
should be negligible in comparison to a rewrite and the list should not contain
a lot of rewrites in practice.
The pending_rewrites list is traversed in multiple places. In typical usage,
the list will contain only a few entries so the traversal cost is negligible (
furthermore in comparison to a rewrite).
---
src/backend/catalog/index.c | 2 +-
src/backend/commands/cluster.c | 5 +
src/backend/commands/tablecmds.c | 6 +
src/backend/utils/activity/pgstat_relation.c | 391 ++++++++++++++++++-
src/backend/utils/activity/pgstat_xact.c | 25 +-
src/backend/utils/cache/relcache.c | 6 +
src/include/pgstat.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 424 insertions(+), 17 deletions(-)
92.8% src/backend/utils/activity/
4.9% src/backend/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 43de42ce39e..88ef00609db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1793,7 +1793,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId);
/* copy over statistics from old to new index */
- pgstat_copy_relation_stats(newClassRel, oldClassRel);
+ pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 60a4617a585..3a446f32921 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1196,6 +1196,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
rel1 = relation_open(r1, NoLock);
rel2 = relation_open(r2, NoLock);
+
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind))
+ pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator);
+
rel2->rd_createSubid = rel1->rd_createSubid;
rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid;
rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index df1ba112b35..6244d04ab12 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -16896,6 +16896,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
Oid reltoastrelid;
RelFileNumber newrelfilenumber;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator;
List *reltoastidxids = NIL;
ListCell *lc;
@@ -16934,6 +16935,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
newrlocator = rel->rd_locator;
newrlocator.relNumber = newrelfilenumber;
newrlocator.spcOid = newTableSpace;
+ oldrlocator = rel->rd_locator;
/* hand off to AM to actually create new rel storage and copy the data */
if (rel->rd_rel->relkind == RELKIND_INDEX)
@@ -16946,6 +16948,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
table_relation_copy_data(rel, &newrlocator);
}
+ /* mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Update the pg_class row.
*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 89bf0cbed56..4d929972fab 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -30,6 +30,19 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
+/* Pending rewrite operations for stats copying */
+typedef struct PgStat_PendingRewrite
+{
+ RelFileLocator old_locator;
+ RelFileLocator new_locator;
+ RelFileLocator original_locator;
+ int nest_level; /* Transaction nesting level where rewrite
+ * occurred */
+ struct PgStat_PendingRewrite *next;
+} PgStat_PendingRewrite;
+
+/* The pending rewrites list for current transaction */
+static PgStat_PendingRewrite *pending_rewrites = NULL;
/* Record that's written to 2PC state file when pgstat state is persisted */
typedef struct TwoPhasePgStatRecord
@@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter deleted_pre_truncdrop;
RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
+ RelFileLocator rewrite_old_locator;
+ int rewrite_nest_level;
} TwoPhasePgStatRecord;
@@ -54,27 +69,70 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
/*
- * Copy stats between relations. This is used for things like REINDEX
+ * Copy stats between RelFileLocator. This is used for things like REINDEX
* CONCURRENTLY.
*/
void
-pgstat_copy_relation_stats(Relation dst, Relation src)
+pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment)
{
PgStat_StatTabEntry *srcstats;
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src));
+ srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION,
+ src.dbOid,
+ RelFileLocatorToPgStatObjid(src));
if (!srcstats)
return;
dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
- dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(dst),
+ dst.dbOid,
+ RelFileLocatorToPgStatObjid(dst),
false);
dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats;
- dstshstats->stats = *srcstats;
+
+ if (!increment)
+ dstshstats->stats = *srcstats;
+ else
+ {
+ /* Increment those statistics */
+#define RELFSTAT_ACC(fld, stats_to_add) \
+ (dstshstats->stats.fld += stats_to_add->fld)
+ RELFSTAT_ACC(numscans, srcstats);
+ RELFSTAT_ACC(tuples_returned, srcstats);
+ RELFSTAT_ACC(tuples_fetched, srcstats);
+ RELFSTAT_ACC(tuples_inserted, srcstats);
+ RELFSTAT_ACC(tuples_updated, srcstats);
+ RELFSTAT_ACC(tuples_deleted, srcstats);
+ RELFSTAT_ACC(tuples_hot_updated, srcstats);
+ RELFSTAT_ACC(tuples_newpage_updated, srcstats);
+ RELFSTAT_ACC(live_tuples, srcstats);
+ RELFSTAT_ACC(dead_tuples, srcstats);
+ RELFSTAT_ACC(mod_since_analyze, srcstats);
+ RELFSTAT_ACC(ins_since_vacuum, srcstats);
+ RELFSTAT_ACC(blocks_fetched, srcstats);
+ RELFSTAT_ACC(blocks_hit, srcstats);
+ RELFSTAT_ACC(vacuum_count, srcstats);
+ RELFSTAT_ACC(autovacuum_count, srcstats);
+ RELFSTAT_ACC(analyze_count, srcstats);
+ RELFSTAT_ACC(autoanalyze_count, srcstats);
+ RELFSTAT_ACC(total_vacuum_time, srcstats);
+ RELFSTAT_ACC(total_autovacuum_time, srcstats);
+ RELFSTAT_ACC(total_analyze_time, srcstats);
+ RELFSTAT_ACC(total_autoanalyze_time, srcstats);
+#undef RELFSTAT_ACC
+
+ /* Replace those statistics */
+#define RELFSTAT_REP(fld, stats_to_rep) \
+ (dstshstats->stats.fld = stats_to_rep->fld)
+ RELFSTAT_REP(lastscan, srcstats);
+ RELFSTAT_REP(last_vacuum_time, srcstats);
+ RELFSTAT_REP(last_autovacuum_time, srcstats);
+ RELFSTAT_REP(last_analyze_time, srcstats);
+ RELFSTAT_REP(last_autoanalyze_time, srcstats);
+#undef RELFSTAT_REP
+ }
pgstat_unlock_entry(dst_ref);
}
@@ -136,6 +194,7 @@ void
pgstat_assoc_relation(Relation rel)
{
RelFileLocator locator;
+ PgStat_TableStatus *pgstat_info;
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
@@ -164,14 +223,54 @@ pgstat_assoc_relation(Relation rel)
locator.relNumber = rel->rd_id;
}
+ /*
+ * If this relation was rewritten during the current transaction we may be
+ * reopening it with its new RelFileLocator. In that case, continue using
+ * the stats entry associated with the old locator rather than creating a
+ * new one. This ensures all stats from before and after the rewrite are
+ * tracked in a single entry which will be properly copied to the new
+ * locator at transaction commit.
+ */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (locator.dbOid == rewrite->new_locator.dbOid &&
+ locator.spcOid == rewrite->new_locator.spcOid &&
+ locator.relNumber == rewrite->new_locator.relNumber)
+ {
+ pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator);
+ goto found_entry;
+ }
+ }
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(locator);
+ pgstat_info = pgstat_prep_relation_pending(locator);
+
+found_entry:
+ rel->pgstat_info = pgstat_info;
+
+ /*
+ * For relations stats, we key by physical file location, not by relation
+ * OID. This means during operations like ALTER TYPE it's possible that
+ * the relation OID changes but the relfilenode stays the same (no actual
+ * rewrite needed). Unlink the old relation first.
+ */
+ if (pgstat_info->relation != NULL &&
+ pgstat_info->relation != rel)
+ {
+ pgstat_info->relation->pgstat_info = NULL;
+ pgstat_info->relation = NULL;
+ }
/* don't allow link a stats to multiple relcache entries */
- Assert(rel->pgstat_info->relation == NULL);
+ Assert(pgstat_info->relation == NULL);
/* mark this relation as the owner */
- rel->pgstat_info->relation = rel;
+ pgstat_info->relation = rel;
}
/*
@@ -214,14 +313,37 @@ pgstat_drop_relation(Relation rel)
{
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ bool skip_transactional_drop = false;
/* don't track stats for relations without storage */
if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
return;
- pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_locator.dbOid,
- RelFileLocatorToPgStatObjid(rel->rd_locator));
+ /* Check if this drop is part of a pending rewrite */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid &&
+ rel->rd_locator.spcOid == rewrite->old_locator.spcOid &&
+ rel->rd_locator.relNumber == rewrite->old_locator.relNumber)
+ {
+ skip_transactional_drop = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * If it is part of a rewrite, drop its stats later, for example in
+ * AtEOXact_PgStat_Relations(), so skip it here.
+ */
+ if (!skip_transactional_drop)
+ pgstat_drop_transactional(PGSTAT_KIND_RELATION,
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -666,6 +788,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit)
}
tabstat->trans = NULL;
}
+
+ /* preserve the stats in case of rewrite */
+ if (isCommit && pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *prev = NULL;
+ PgStat_PendingRewrite *current = pending_rewrites;
+ PgStat_PendingRewrite *next;
+
+ /* reverse the rewrites list to process in chronological order */
+ while (current != NULL)
+ {
+ next = current->next;
+ current->next = prev;
+ prev = current;
+ current = next;
+ }
+
+ /* now process rewrites in chronological order */
+ for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ pgstat_copy_relation_stats(rewrite->new_locator,
+ rewrite->old_locator, true);
+
+ /* drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
+ }
+
+ pending_rewrites = NULL;
}
/*
@@ -681,6 +845,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in
PgStat_TableXactStatus *trans;
PgStat_TableXactStatus *next_trans;
+ /*
+ * If we don't commit then remove the associated rewrites if any, to keep
+ * the rewrite chain in sync with what will be eventually committed.
+ */
+ if (!isCommit)
+ {
+ PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites;
+
+ while (*rewrite_ptr != NULL)
+ {
+ if ((*rewrite_ptr)->nest_level >= nestDepth)
+ {
+ PgStat_PendingRewrite *to_remove = *rewrite_ptr;
+
+ *rewrite_ptr = (*rewrite_ptr)->next;
+ pfree(to_remove);
+ }
+ else
+ {
+ rewrite_ptr = &((*rewrite_ptr)->next);
+ }
+ }
+ }
+
for (trans = xact_state->first; trans != NULL; trans = next_trans)
{
PgStat_TableStatus *tabstat;
@@ -760,11 +948,19 @@ void
AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
{
PgStat_TableXactStatus *trans;
+ PgStat_PendingRewrite *rewrite;
+ /*
+ * For each tabstat, find its matching rewrite and remove it from the
+ * pending rewrites list. This way, after processing all tabstats, pending
+ * rewrites will only contain rewrite only transactions.
+ */
for (trans = xact_state->first; trans != NULL; trans = trans->next)
{
PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY;
TwoPhasePgStatRecord record;
+ PgStat_PendingRewrite **rewrite_ptr;
+ bool found_rewrite = false;
Assert(trans->nest_level == 1);
Assert(trans->upper == NULL);
@@ -784,10 +980,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.locator = tabstat->locator;
record.truncdropped = trans->truncdropped;
+ record.rewrite_nest_level = 0;
+
+ /*
+ * Look for a matching rewrite and remove it from pending rewrites. We
+ * check three possible matches:
+ *
+ * The new_locator when stats have been added after the rewrite. The
+ * old_locator when stats have been added before the rewrite but not
+ * after. The original_locator when this tabstat is part of a rewrite
+ * chain.
+ */
+ rewrite_ptr = &pending_rewrites;
+ while (*rewrite_ptr != NULL)
+ {
+ rewrite = *rewrite_ptr;
+
+ if ((record.locator.dbOid == rewrite->new_locator.dbOid &&
+ record.locator.spcOid == rewrite->new_locator.spcOid &&
+ record.locator.relNumber == rewrite->new_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->old_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->old_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->old_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->original_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->original_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->original_locator.relNumber))
+ {
+ /*
+ * Found matching rewrite. Record the rewrite information and
+ * remove this rewrite from the list since it's now handled.
+ */
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.locator = rewrite->new_locator;
+ found_rewrite = true;
+
+ /* Remove from pending_rewrites list */
+ *rewrite_ptr = rewrite->next;
+ pfree(rewrite);
+ break;
+ }
+ else
+ {
+ /* Move to next rewrite in the list */
+ rewrite_ptr = &(rewrite->next);
+ }
+ }
+
+ /* If no rewrite found, clear the rewrite fields */
+ if (!found_rewrite)
+ {
+ memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator));
+ }
+
+ RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
+ &record, sizeof(TwoPhasePgStatRecord));
+ }
+
+ /*
+ * Now process any rewrites still pending. These are rewrite only
+ * transactions. We need to preserve their stats even though there's no
+ * tabstat entry for them.
+ */
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ TwoPhasePgStatRecord record;
+
+ memset(&record, 0, sizeof(TwoPhasePgStatRecord));
+ record.locator = rewrite->new_locator;
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.truncdropped = false;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
&record, sizeof(TwoPhasePgStatRecord));
}
+
+ pending_rewrites = NULL;
}
/*
@@ -810,6 +1079,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
tabstat = trans->parent;
tabstat->trans = NULL;
}
+
+ pending_rewrites = NULL;
}
/*
@@ -845,6 +1116,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
pgstat_info->counts.changed_tuples +=
rec->tuples_inserted + rec->tuples_updated +
rec->tuples_deleted;
+
+ if (rec->rewrite_nest_level > 0)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ /* Flush any pending stats for old locator first */
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ /* Copy stats from old to new locator */
+ pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator,
+ true);
+
+ /* Drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
}
/*
@@ -859,9 +1153,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
{
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
+ RelFileLocator target_locator;
+
+ /*
+ * For aborted transactions with rewrites (like TRUNCATE), we need to
+ * restore stats to the old locator, not the new one. The new locator
+ * should be dropped since the rewrite is being rolled back.
+ */
+ if (rec->rewrite_nest_level > 0)
+ {
+ /* Use the old locator */
+ target_locator = rec->rewrite_old_locator;
+ }
+ else
+ {
+ /* No rewrite, use the original locator */
+ target_locator = rec->locator;
+ }
/* Find or create a tabstat entry for the target locator */
- pgstat_info = pgstat_prep_relation_pending(rec->locator);
+ pgstat_info = pgstat_prep_relation_pending(target_locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -916,7 +1227,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
+ TimestampTz t;
+
+ /*
+ * Checking the transaction state due to the flush call in
+ * pgstat_twophase_postcommit() that would break the assertion on the
+ * state in GetCurrentTransactionStopTimestamp().
+ */
+ if (!IsTransactionState())
+ t = GetCurrentTransactionStopTimestamp();
+ else
+ t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
@@ -1167,3 +1488,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
ReleaseSysCache(tuple);
return result;
}
+
+/*
+ * Mark that a relation rewrite has occurred, preserving the original locator
+ * so stats can be copied at transaction commit.
+ */
+void
+pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator)
+{
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *existing;
+ RelFileLocator original_locator = old_locator;
+
+ for (existing = pending_rewrites; existing != NULL; existing = existing->next)
+ {
+ if (old_locator.dbOid == existing->new_locator.dbOid &&
+ old_locator.spcOid == existing->new_locator.spcOid &&
+ old_locator.relNumber == existing->new_locator.relNumber)
+ {
+ original_locator = existing->original_locator;
+ break;
+ }
+ }
+
+ /* Allocate in TopTransactionContext memory context */
+ rewrite = MemoryContextAlloc(TopTransactionContext,
+ sizeof(PgStat_PendingRewrite));
+
+ rewrite->old_locator = old_locator;
+ rewrite->new_locator = new_locator;
+ rewrite->original_locator = original_locator;
+ rewrite->nest_level = GetCurrentTransactionNestLevel();
+
+ /* Add to the list */
+ rewrite->next = pending_rewrites;
+ pending_rewrites = rewrite;
+}
+
+void
+pgstat_clear_rewrite(void)
+{
+ pending_rewrites = NULL;
+}
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index 5e2d69e6297..8ed8f5317f3 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel)
}
pgStatXactStack = NULL;
+ pgstat_clear_rewrite();
+
/* Make sure any stats snapshot is thrown away */
pgstat_clear_snapshot();
}
@@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo
void
pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid)
{
- if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL))
+ PgStat_EntryRef *entry_ref;
+
+ entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL);
+
+ if (entry_ref)
{
+ /*
+ * For relations stats, we key by physical file location, not by
+ * relation OID. This means during operations like ALTER TYPE where
+ * the relation OID changes but the relfilenode stays the same (no
+ * actual rewrite needed), we'll find an existing entry.
+ *
+ * This is expected behavior, we want to preserve stats across the
+ * catalog change. Simply reset and recreate the entry for the new
+ * relation OID without warning.
+ */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ pgstat_reset(kind, dboid, objid);
+ create_drop_transactional_internal(kind, dboid, objid, true);
+ return;
+ }
+
ereport(WARNING,
errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64,
(pgstat_get_kind_info(kind))->name, dboid,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 6b634c9fff1..692d084acb2 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -85,6 +85,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/relmapper.h"
#include "utils/resowner.h"
#include "utils/snapmgr.h"
@@ -3775,6 +3776,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
MultiXactId minmulti = InvalidMultiXactId;
TransactionId freezeXid = InvalidTransactionId;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator = relation->rd_locator;
if (!IsBinaryUpgrade)
{
@@ -3946,6 +3948,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
table_close(pg_class, RowExclusiveLock);
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Make the pg_class row change or relation map change visible. This will
* cause the relcache entry to get updated, too.
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index e9e4d32c3b8..a9a4fa9f8f2 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -669,7 +669,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id);
extern void pgstat_create_relation(Relation rel);
extern void pgstat_drop_relation(Relation rel);
-extern void pgstat_copy_relation_stats(Relation dst, Relation src);
+extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment);
extern void pgstat_init_relation(Relation rel);
extern void pgstat_assoc_relation(Relation rel);
@@ -681,6 +681,9 @@ extern void pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_mark_rewrite(RelFileLocator old_locator,
+ RelFileLocator new_locator);
+extern void pgstat_clear_rewrite(void);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 241945734ec..05b1380c08c 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2279,6 +2279,7 @@ PgStat_KindInfo
PgStat_LocalState
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
+PgStat_PendingRewrite
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
--zLA8DXafBEP889wv--
^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v13 2/2] handle relation statistics correctly during rewrites
@ 2025-11-04 13:52 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Bertrand Drouvot @ 2025-11-04 13:52 UTC (permalink / raw)
Now that PGSTAT_KIND_RELATION is keyed by refilenode, we need to handle rewrites.
To do so, this patch:
- Adds PgStat_PendingRewrite, a new struct to track rewrite operations within
a transaction, storing the old locator, new locator, and original locator (for
rewrite chains). This allows stats to be copied from the original location to
the final location at commit time.
- Adds a new function, pgstat_mark_rewrite(), called when a table rewrite begins.
It records the rewrite operation in a local list and detects rewrite chains by
checking if the old_locator matches any existing new_locator, preserving the
chain's original_locator.
- Modifies pgstat_copy_relation_stats(), to accept RelFileLocators instead of
Relations, with a new increment parameter to accumulate stats (needed for rewrite
chains with DML between rewrites).
- Ensures that AtEOXact_PgStat_Relations(), AtPrepare_PgStat_Relations(),
pgstat_twophase_postcommit()/postabort() pgstat_drop_relation() handle the
PgStat_PendingRewrite list correctly.
Note that due to the new flush call in pgstat_twophase_postcommit() we can not
call GetCurrentTransactionStopTimestamp() in pgstat_relation_flush_cb(). So,
adding a check to handle this special case and call GetCurrentTimestamp() instead.
Note that we'd call GetCurrentTimestamp() only if there is a rewrite, so that
the GetCurrentTimestamp() extra cost should be negligible. Another solution
could be to trigger the flush from FinishPreparedTransaction() but that's not
worth the extra complexity.
The new pending_rewrites list is traversed in multiple places. The overhead
should be negligible in comparison to a rewrite and the list should not contain
a lot of rewrites in practice.
The pending_rewrites list is traversed in multiple places. In typical usage,
the list will contain only a few entries so the traversal cost is negligible (
furthermore in comparison to a rewrite).
---
src/backend/catalog/index.c | 2 +-
src/backend/commands/cluster.c | 5 +
src/backend/commands/tablecmds.c | 6 +
src/backend/utils/activity/pgstat_relation.c | 391 ++++++++++++++++++-
src/backend/utils/activity/pgstat_xact.c | 25 +-
src/backend/utils/cache/relcache.c | 6 +
src/include/pgstat.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 424 insertions(+), 17 deletions(-)
92.8% src/backend/utils/activity/
4.9% src/backend/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 8b3c60d91f9..7c7d0fb8e7a 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1796,7 +1796,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId);
/* copy over statistics from old to new index */
- pgstat_copy_relation_stats(newClassRel, oldClassRel);
+ pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 09066db0956..c491daceb0f 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1206,6 +1206,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
rel1 = relation_open(r1, NoLock);
rel2 = relation_open(r2, NoLock);
+
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind))
+ pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator);
+
rel2->rd_createSubid = rel1->rd_createSubid;
rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid;
rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 67e42e5df29..1715af2c0d8 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -17099,6 +17099,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
Oid reltoastrelid;
RelFileNumber newrelfilenumber;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator;
List *reltoastidxids = NIL;
ListCell *lc;
@@ -17137,6 +17138,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
newrlocator = rel->rd_locator;
newrlocator.relNumber = newrelfilenumber;
newrlocator.spcOid = newTableSpace;
+ oldrlocator = rel->rd_locator;
/* hand off to AM to actually create new rel storage and copy the data */
if (rel->rd_rel->relkind == RELKIND_INDEX)
@@ -17149,6 +17151,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
table_relation_copy_data(rel, &newrlocator);
}
+ /* mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Update the pg_class row.
*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 89bf0cbed56..4d929972fab 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -30,6 +30,19 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
+/* Pending rewrite operations for stats copying */
+typedef struct PgStat_PendingRewrite
+{
+ RelFileLocator old_locator;
+ RelFileLocator new_locator;
+ RelFileLocator original_locator;
+ int nest_level; /* Transaction nesting level where rewrite
+ * occurred */
+ struct PgStat_PendingRewrite *next;
+} PgStat_PendingRewrite;
+
+/* The pending rewrites list for current transaction */
+static PgStat_PendingRewrite *pending_rewrites = NULL;
/* Record that's written to 2PC state file when pgstat state is persisted */
typedef struct TwoPhasePgStatRecord
@@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter deleted_pre_truncdrop;
RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
+ RelFileLocator rewrite_old_locator;
+ int rewrite_nest_level;
} TwoPhasePgStatRecord;
@@ -54,27 +69,70 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
/*
- * Copy stats between relations. This is used for things like REINDEX
+ * Copy stats between RelFileLocator. This is used for things like REINDEX
* CONCURRENTLY.
*/
void
-pgstat_copy_relation_stats(Relation dst, Relation src)
+pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment)
{
PgStat_StatTabEntry *srcstats;
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src));
+ srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION,
+ src.dbOid,
+ RelFileLocatorToPgStatObjid(src));
if (!srcstats)
return;
dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
- dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(dst),
+ dst.dbOid,
+ RelFileLocatorToPgStatObjid(dst),
false);
dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats;
- dstshstats->stats = *srcstats;
+
+ if (!increment)
+ dstshstats->stats = *srcstats;
+ else
+ {
+ /* Increment those statistics */
+#define RELFSTAT_ACC(fld, stats_to_add) \
+ (dstshstats->stats.fld += stats_to_add->fld)
+ RELFSTAT_ACC(numscans, srcstats);
+ RELFSTAT_ACC(tuples_returned, srcstats);
+ RELFSTAT_ACC(tuples_fetched, srcstats);
+ RELFSTAT_ACC(tuples_inserted, srcstats);
+ RELFSTAT_ACC(tuples_updated, srcstats);
+ RELFSTAT_ACC(tuples_deleted, srcstats);
+ RELFSTAT_ACC(tuples_hot_updated, srcstats);
+ RELFSTAT_ACC(tuples_newpage_updated, srcstats);
+ RELFSTAT_ACC(live_tuples, srcstats);
+ RELFSTAT_ACC(dead_tuples, srcstats);
+ RELFSTAT_ACC(mod_since_analyze, srcstats);
+ RELFSTAT_ACC(ins_since_vacuum, srcstats);
+ RELFSTAT_ACC(blocks_fetched, srcstats);
+ RELFSTAT_ACC(blocks_hit, srcstats);
+ RELFSTAT_ACC(vacuum_count, srcstats);
+ RELFSTAT_ACC(autovacuum_count, srcstats);
+ RELFSTAT_ACC(analyze_count, srcstats);
+ RELFSTAT_ACC(autoanalyze_count, srcstats);
+ RELFSTAT_ACC(total_vacuum_time, srcstats);
+ RELFSTAT_ACC(total_autovacuum_time, srcstats);
+ RELFSTAT_ACC(total_analyze_time, srcstats);
+ RELFSTAT_ACC(total_autoanalyze_time, srcstats);
+#undef RELFSTAT_ACC
+
+ /* Replace those statistics */
+#define RELFSTAT_REP(fld, stats_to_rep) \
+ (dstshstats->stats.fld = stats_to_rep->fld)
+ RELFSTAT_REP(lastscan, srcstats);
+ RELFSTAT_REP(last_vacuum_time, srcstats);
+ RELFSTAT_REP(last_autovacuum_time, srcstats);
+ RELFSTAT_REP(last_analyze_time, srcstats);
+ RELFSTAT_REP(last_autoanalyze_time, srcstats);
+#undef RELFSTAT_REP
+ }
pgstat_unlock_entry(dst_ref);
}
@@ -136,6 +194,7 @@ void
pgstat_assoc_relation(Relation rel)
{
RelFileLocator locator;
+ PgStat_TableStatus *pgstat_info;
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
@@ -164,14 +223,54 @@ pgstat_assoc_relation(Relation rel)
locator.relNumber = rel->rd_id;
}
+ /*
+ * If this relation was rewritten during the current transaction we may be
+ * reopening it with its new RelFileLocator. In that case, continue using
+ * the stats entry associated with the old locator rather than creating a
+ * new one. This ensures all stats from before and after the rewrite are
+ * tracked in a single entry which will be properly copied to the new
+ * locator at transaction commit.
+ */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (locator.dbOid == rewrite->new_locator.dbOid &&
+ locator.spcOid == rewrite->new_locator.spcOid &&
+ locator.relNumber == rewrite->new_locator.relNumber)
+ {
+ pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator);
+ goto found_entry;
+ }
+ }
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(locator);
+ pgstat_info = pgstat_prep_relation_pending(locator);
+
+found_entry:
+ rel->pgstat_info = pgstat_info;
+
+ /*
+ * For relations stats, we key by physical file location, not by relation
+ * OID. This means during operations like ALTER TYPE it's possible that
+ * the relation OID changes but the relfilenode stays the same (no actual
+ * rewrite needed). Unlink the old relation first.
+ */
+ if (pgstat_info->relation != NULL &&
+ pgstat_info->relation != rel)
+ {
+ pgstat_info->relation->pgstat_info = NULL;
+ pgstat_info->relation = NULL;
+ }
/* don't allow link a stats to multiple relcache entries */
- Assert(rel->pgstat_info->relation == NULL);
+ Assert(pgstat_info->relation == NULL);
/* mark this relation as the owner */
- rel->pgstat_info->relation = rel;
+ pgstat_info->relation = rel;
}
/*
@@ -214,14 +313,37 @@ pgstat_drop_relation(Relation rel)
{
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ bool skip_transactional_drop = false;
/* don't track stats for relations without storage */
if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
return;
- pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_locator.dbOid,
- RelFileLocatorToPgStatObjid(rel->rd_locator));
+ /* Check if this drop is part of a pending rewrite */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid &&
+ rel->rd_locator.spcOid == rewrite->old_locator.spcOid &&
+ rel->rd_locator.relNumber == rewrite->old_locator.relNumber)
+ {
+ skip_transactional_drop = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * If it is part of a rewrite, drop its stats later, for example in
+ * AtEOXact_PgStat_Relations(), so skip it here.
+ */
+ if (!skip_transactional_drop)
+ pgstat_drop_transactional(PGSTAT_KIND_RELATION,
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -666,6 +788,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit)
}
tabstat->trans = NULL;
}
+
+ /* preserve the stats in case of rewrite */
+ if (isCommit && pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *prev = NULL;
+ PgStat_PendingRewrite *current = pending_rewrites;
+ PgStat_PendingRewrite *next;
+
+ /* reverse the rewrites list to process in chronological order */
+ while (current != NULL)
+ {
+ next = current->next;
+ current->next = prev;
+ prev = current;
+ current = next;
+ }
+
+ /* now process rewrites in chronological order */
+ for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ pgstat_copy_relation_stats(rewrite->new_locator,
+ rewrite->old_locator, true);
+
+ /* drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
+ }
+
+ pending_rewrites = NULL;
}
/*
@@ -681,6 +845,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in
PgStat_TableXactStatus *trans;
PgStat_TableXactStatus *next_trans;
+ /*
+ * If we don't commit then remove the associated rewrites if any, to keep
+ * the rewrite chain in sync with what will be eventually committed.
+ */
+ if (!isCommit)
+ {
+ PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites;
+
+ while (*rewrite_ptr != NULL)
+ {
+ if ((*rewrite_ptr)->nest_level >= nestDepth)
+ {
+ PgStat_PendingRewrite *to_remove = *rewrite_ptr;
+
+ *rewrite_ptr = (*rewrite_ptr)->next;
+ pfree(to_remove);
+ }
+ else
+ {
+ rewrite_ptr = &((*rewrite_ptr)->next);
+ }
+ }
+ }
+
for (trans = xact_state->first; trans != NULL; trans = next_trans)
{
PgStat_TableStatus *tabstat;
@@ -760,11 +948,19 @@ void
AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
{
PgStat_TableXactStatus *trans;
+ PgStat_PendingRewrite *rewrite;
+ /*
+ * For each tabstat, find its matching rewrite and remove it from the
+ * pending rewrites list. This way, after processing all tabstats, pending
+ * rewrites will only contain rewrite only transactions.
+ */
for (trans = xact_state->first; trans != NULL; trans = trans->next)
{
PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY;
TwoPhasePgStatRecord record;
+ PgStat_PendingRewrite **rewrite_ptr;
+ bool found_rewrite = false;
Assert(trans->nest_level == 1);
Assert(trans->upper == NULL);
@@ -784,10 +980,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.locator = tabstat->locator;
record.truncdropped = trans->truncdropped;
+ record.rewrite_nest_level = 0;
+
+ /*
+ * Look for a matching rewrite and remove it from pending rewrites. We
+ * check three possible matches:
+ *
+ * The new_locator when stats have been added after the rewrite. The
+ * old_locator when stats have been added before the rewrite but not
+ * after. The original_locator when this tabstat is part of a rewrite
+ * chain.
+ */
+ rewrite_ptr = &pending_rewrites;
+ while (*rewrite_ptr != NULL)
+ {
+ rewrite = *rewrite_ptr;
+
+ if ((record.locator.dbOid == rewrite->new_locator.dbOid &&
+ record.locator.spcOid == rewrite->new_locator.spcOid &&
+ record.locator.relNumber == rewrite->new_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->old_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->old_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->old_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->original_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->original_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->original_locator.relNumber))
+ {
+ /*
+ * Found matching rewrite. Record the rewrite information and
+ * remove this rewrite from the list since it's now handled.
+ */
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.locator = rewrite->new_locator;
+ found_rewrite = true;
+
+ /* Remove from pending_rewrites list */
+ *rewrite_ptr = rewrite->next;
+ pfree(rewrite);
+ break;
+ }
+ else
+ {
+ /* Move to next rewrite in the list */
+ rewrite_ptr = &(rewrite->next);
+ }
+ }
+
+ /* If no rewrite found, clear the rewrite fields */
+ if (!found_rewrite)
+ {
+ memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator));
+ }
+
+ RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
+ &record, sizeof(TwoPhasePgStatRecord));
+ }
+
+ /*
+ * Now process any rewrites still pending. These are rewrite only
+ * transactions. We need to preserve their stats even though there's no
+ * tabstat entry for them.
+ */
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ TwoPhasePgStatRecord record;
+
+ memset(&record, 0, sizeof(TwoPhasePgStatRecord));
+ record.locator = rewrite->new_locator;
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.truncdropped = false;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
&record, sizeof(TwoPhasePgStatRecord));
}
+
+ pending_rewrites = NULL;
}
/*
@@ -810,6 +1079,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
tabstat = trans->parent;
tabstat->trans = NULL;
}
+
+ pending_rewrites = NULL;
}
/*
@@ -845,6 +1116,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
pgstat_info->counts.changed_tuples +=
rec->tuples_inserted + rec->tuples_updated +
rec->tuples_deleted;
+
+ if (rec->rewrite_nest_level > 0)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ /* Flush any pending stats for old locator first */
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ /* Copy stats from old to new locator */
+ pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator,
+ true);
+
+ /* Drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
}
/*
@@ -859,9 +1153,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
{
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
+ RelFileLocator target_locator;
+
+ /*
+ * For aborted transactions with rewrites (like TRUNCATE), we need to
+ * restore stats to the old locator, not the new one. The new locator
+ * should be dropped since the rewrite is being rolled back.
+ */
+ if (rec->rewrite_nest_level > 0)
+ {
+ /* Use the old locator */
+ target_locator = rec->rewrite_old_locator;
+ }
+ else
+ {
+ /* No rewrite, use the original locator */
+ target_locator = rec->locator;
+ }
/* Find or create a tabstat entry for the target locator */
- pgstat_info = pgstat_prep_relation_pending(rec->locator);
+ pgstat_info = pgstat_prep_relation_pending(target_locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -916,7 +1227,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
+ TimestampTz t;
+
+ /*
+ * Checking the transaction state due to the flush call in
+ * pgstat_twophase_postcommit() that would break the assertion on the
+ * state in GetCurrentTransactionStopTimestamp().
+ */
+ if (!IsTransactionState())
+ t = GetCurrentTransactionStopTimestamp();
+ else
+ t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
@@ -1167,3 +1488,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
ReleaseSysCache(tuple);
return result;
}
+
+/*
+ * Mark that a relation rewrite has occurred, preserving the original locator
+ * so stats can be copied at transaction commit.
+ */
+void
+pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator)
+{
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *existing;
+ RelFileLocator original_locator = old_locator;
+
+ for (existing = pending_rewrites; existing != NULL; existing = existing->next)
+ {
+ if (old_locator.dbOid == existing->new_locator.dbOid &&
+ old_locator.spcOid == existing->new_locator.spcOid &&
+ old_locator.relNumber == existing->new_locator.relNumber)
+ {
+ original_locator = existing->original_locator;
+ break;
+ }
+ }
+
+ /* Allocate in TopTransactionContext memory context */
+ rewrite = MemoryContextAlloc(TopTransactionContext,
+ sizeof(PgStat_PendingRewrite));
+
+ rewrite->old_locator = old_locator;
+ rewrite->new_locator = new_locator;
+ rewrite->original_locator = original_locator;
+ rewrite->nest_level = GetCurrentTransactionNestLevel();
+
+ /* Add to the list */
+ rewrite->next = pending_rewrites;
+ pending_rewrites = rewrite;
+}
+
+void
+pgstat_clear_rewrite(void)
+{
+ pending_rewrites = NULL;
+}
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index 5e2d69e6297..8ed8f5317f3 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel)
}
pgStatXactStack = NULL;
+ pgstat_clear_rewrite();
+
/* Make sure any stats snapshot is thrown away */
pgstat_clear_snapshot();
}
@@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo
void
pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid)
{
- if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL))
+ PgStat_EntryRef *entry_ref;
+
+ entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL);
+
+ if (entry_ref)
{
+ /*
+ * For relations stats, we key by physical file location, not by
+ * relation OID. This means during operations like ALTER TYPE where
+ * the relation OID changes but the relfilenode stays the same (no
+ * actual rewrite needed), we'll find an existing entry.
+ *
+ * This is expected behavior, we want to preserve stats across the
+ * catalog change. Simply reset and recreate the entry for the new
+ * relation OID without warning.
+ */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ pgstat_reset(kind, dboid, objid);
+ create_drop_transactional_internal(kind, dboid, objid, true);
+ return;
+ }
+
ereport(WARNING,
errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64,
(pgstat_get_kind_info(kind))->name, dboid,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 3a4f19e8d58..60491962668 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -86,6 +86,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/relmapper.h"
#include "utils/resowner.h"
#include "utils/snapmgr.h"
@@ -3771,6 +3772,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
MultiXactId minmulti = InvalidMultiXactId;
TransactionId freezeXid = InvalidTransactionId;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator = relation->rd_locator;
if (!IsBinaryUpgrade)
{
@@ -3942,6 +3944,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
table_close(pg_class, RowExclusiveLock);
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Make the pg_class row change or relation map change visible. This will
* cause the relcache entry to get updated, too.
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 689c624f373..d0d5473f8a5 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -673,7 +673,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id);
extern void pgstat_create_relation(Relation rel);
extern void pgstat_drop_relation(Relation rel);
-extern void pgstat_copy_relation_stats(Relation dst, Relation src);
+extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment);
extern void pgstat_init_relation(Relation rel);
extern void pgstat_assoc_relation(Relation rel);
@@ -685,6 +685,9 @@ extern void pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_mark_rewrite(RelFileLocator old_locator,
+ RelFileLocator new_locator);
+extern void pgstat_clear_rewrite(void);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 52f8603a7be..8b84e3a0bcf 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2298,6 +2298,7 @@ PgStat_KindInfo
PgStat_LocalState
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
+PgStat_PendingRewrite
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
--USlF2Vr25us5UpvY--
^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v8 2/2] handle relation statistics correctly during rewrites
@ 2025-11-04 13:52 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Bertrand Drouvot @ 2025-11-04 13:52 UTC (permalink / raw)
Now that PGSTAT_KIND_RELATION is keyed by refilenode, we need to handle rewrites.
To do so, this patch:
- Adds PgStat_PendingRewrite, a new struct to track rewrite operations within
a transaction, storing the old locator, new locator, and original locator (for
rewrite chains). This allows stats to be copied from the original location to
the final location at commit time.
- Adds a new function, pgstat_mark_rewrite(), called when a table rewrite begins.
It records the rewrite operation in a local list and detects rewrite chains by
checking if the old_locator matches any existing new_locator, preserving the
chain's original_locator.
- Modifies pgstat_copy_relation_stats(), to accept RelFileLocators instead of
Relations, with a new increment parameter to accumulate stats (needed for rewrite
chains with DML between rewrites).
- Ensures that AtEOXact_PgStat_Relations(), AtPrepare_PgStat_Relations(),
pgstat_twophase_postcommit()/postabort() pgstat_drop_relation() handle the
PgStat_PendingRewrite list correctly.
Note that due to the new flush call in pgstat_twophase_postcommit() we can not
call GetCurrentTransactionStopTimestamp() in pgstat_relation_flush_cb(). So,
adding a check to handle this special case and call GetCurrentTimestamp() instead.
Note that we'd call GetCurrentTimestamp() only if there is a rewrite, so that
the GetCurrentTimestamp() extra cost should be negligible. Another solution
could be to trigger the flush from FinishPreparedTransaction() but that's not
worth the extra complexity.
The new pending_rewrites list is traversed in multiple places. The overhead
should be negligible in comparison to a rewrite and the list should not contain
a lot of rewrites in practice.
The pending_rewrites list is traversed in multiple places. In typical usage,
the list will contain only a few entries so the traversal cost is negligible (
furthermore in comparison to a rewrite).
---
src/backend/catalog/index.c | 2 +-
src/backend/commands/cluster.c | 5 +
src/backend/commands/tablecmds.c | 6 +
src/backend/utils/activity/pgstat_relation.c | 391 ++++++++++++++++++-
src/backend/utils/activity/pgstat_xact.c | 25 +-
src/backend/utils/cache/relcache.c | 6 +
src/include/pgstat.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 424 insertions(+), 17 deletions(-)
92.8% src/backend/utils/activity/
4.9% src/backend/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 8dea58ad96b..b71925a22c3 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1795,7 +1795,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId);
/* copy over statistics from old to new index */
- pgstat_copy_relation_stats(newClassRel, oldClassRel);
+ pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 2120c85ccb4..6155b12afab 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1196,6 +1196,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
rel1 = relation_open(r1, NoLock);
rel2 = relation_open(r2, NoLock);
+
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind))
+ pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator);
+
rel2->rd_createSubid = rel1->rd_createSubid;
rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid;
rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 6b1a00ed477..9de70f321ed 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -16884,6 +16884,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
Oid reltoastrelid;
RelFileNumber newrelfilenumber;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator;
List *reltoastidxids = NIL;
ListCell *lc;
@@ -16922,6 +16923,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
newrlocator = rel->rd_locator;
newrlocator.relNumber = newrelfilenumber;
newrlocator.spcOid = newTableSpace;
+ oldrlocator = rel->rd_locator;
/* hand off to AM to actually create new rel storage and copy the data */
if (rel->rd_rel->relkind == RELKIND_INDEX)
@@ -16934,6 +16936,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
table_relation_copy_data(rel, &newrlocator);
}
+ /* mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Update the pg_class row.
*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index ee6f2eb2bdb..83103189fb5 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -30,6 +30,19 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
+/* Pending rewrite operations for stats copying */
+typedef struct PgStat_PendingRewrite
+{
+ RelFileLocator old_locator;
+ RelFileLocator new_locator;
+ RelFileLocator original_locator;
+ int nest_level; /* Transaction nesting level where rewrite
+ * occurred */
+ struct PgStat_PendingRewrite *next;
+} PgStat_PendingRewrite;
+
+/* The pending rewrites list for current transaction */
+static PgStat_PendingRewrite *pending_rewrites = NULL;
/* Record that's written to 2PC state file when pgstat state is persisted */
typedef struct TwoPhasePgStatRecord
@@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter deleted_pre_truncdrop;
RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
+ RelFileLocator rewrite_old_locator;
+ int rewrite_nest_level;
} TwoPhasePgStatRecord;
@@ -54,27 +69,70 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
/*
- * Copy stats between relations. This is used for things like REINDEX
+ * Copy stats between RelFileLocator. This is used for things like REINDEX
* CONCURRENTLY.
*/
void
-pgstat_copy_relation_stats(Relation dst, Relation src)
+pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment)
{
PgStat_StatTabEntry *srcstats;
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src));
+ srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION,
+ src.dbOid,
+ RelFileLocatorToPgStatObjid(src));
if (!srcstats)
return;
dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
- dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(dst),
+ dst.dbOid,
+ RelFileLocatorToPgStatObjid(dst),
false);
dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats;
- dstshstats->stats = *srcstats;
+
+ if (!increment)
+ dstshstats->stats = *srcstats;
+ else
+ {
+ /* Increment those statistics */
+#define RELFSTAT_ACC(fld, stats_to_add) \
+ (dstshstats->stats.fld += stats_to_add->fld)
+ RELFSTAT_ACC(numscans, srcstats);
+ RELFSTAT_ACC(tuples_returned, srcstats);
+ RELFSTAT_ACC(tuples_fetched, srcstats);
+ RELFSTAT_ACC(tuples_inserted, srcstats);
+ RELFSTAT_ACC(tuples_updated, srcstats);
+ RELFSTAT_ACC(tuples_deleted, srcstats);
+ RELFSTAT_ACC(tuples_hot_updated, srcstats);
+ RELFSTAT_ACC(tuples_newpage_updated, srcstats);
+ RELFSTAT_ACC(live_tuples, srcstats);
+ RELFSTAT_ACC(dead_tuples, srcstats);
+ RELFSTAT_ACC(mod_since_analyze, srcstats);
+ RELFSTAT_ACC(ins_since_vacuum, srcstats);
+ RELFSTAT_ACC(blocks_fetched, srcstats);
+ RELFSTAT_ACC(blocks_hit, srcstats);
+ RELFSTAT_ACC(vacuum_count, srcstats);
+ RELFSTAT_ACC(autovacuum_count, srcstats);
+ RELFSTAT_ACC(analyze_count, srcstats);
+ RELFSTAT_ACC(autoanalyze_count, srcstats);
+ RELFSTAT_ACC(total_vacuum_time, srcstats);
+ RELFSTAT_ACC(total_autovacuum_time, srcstats);
+ RELFSTAT_ACC(total_analyze_time, srcstats);
+ RELFSTAT_ACC(total_autoanalyze_time, srcstats);
+#undef RELFSTAT_ACC
+
+ /* Replace those statistics */
+#define RELFSTAT_REP(fld, stats_to_rep) \
+ (dstshstats->stats.fld = stats_to_rep->fld)
+ RELFSTAT_REP(lastscan, srcstats);
+ RELFSTAT_REP(last_vacuum_time, srcstats);
+ RELFSTAT_REP(last_autovacuum_time, srcstats);
+ RELFSTAT_REP(last_analyze_time, srcstats);
+ RELFSTAT_REP(last_autoanalyze_time, srcstats);
+#undef RELFSTAT_REP
+ }
pgstat_unlock_entry(dst_ref);
}
@@ -136,6 +194,7 @@ void
pgstat_assoc_relation(Relation rel)
{
RelFileLocator locator;
+ PgStat_TableStatus *pgstat_info;
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
@@ -165,14 +224,54 @@ pgstat_assoc_relation(Relation rel)
locator.relNumber = rel->rd_id;
}
+ /*
+ * If this relation was rewritten during the current transaction we may be
+ * reopening it with its new RelFileLocator. In that case, continue using
+ * the stats entry associated with the old locator rather than creating a
+ * new one. This ensures all stats from before and after the rewrite are
+ * tracked in a single entry which will be properly copied to the new
+ * locator at transaction commit.
+ */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (locator.dbOid == rewrite->new_locator.dbOid &&
+ locator.spcOid == rewrite->new_locator.spcOid &&
+ locator.relNumber == rewrite->new_locator.relNumber)
+ {
+ pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator);
+ goto found_entry;
+ }
+ }
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(locator);
+ pgstat_info = pgstat_prep_relation_pending(locator);
+
+found_entry:
+ rel->pgstat_info = pgstat_info;
+
+ /*
+ * For relations stats, we key by physical file location, not by relation
+ * OID. This means during operations like ALTER TYPE it's possible that
+ * the relation OID changes but the relfilenode stays the same (no actual
+ * rewrite needed). Unlink the old relation first.
+ */
+ if (pgstat_info->relation != NULL &&
+ pgstat_info->relation != rel)
+ {
+ pgstat_info->relation->pgstat_info = NULL;
+ pgstat_info->relation = NULL;
+ }
/* don't allow link a stats to multiple relcache entries */
- Assert(rel->pgstat_info->relation == NULL);
+ Assert(pgstat_info->relation == NULL);
/* mark this relation as the owner */
- rel->pgstat_info->relation = rel;
+ pgstat_info->relation = rel;
}
/*
@@ -215,14 +314,37 @@ pgstat_drop_relation(Relation rel)
{
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ bool skip_transactional_drop = false;
/* don't track stats for relations without storage */
if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
return;
- pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_locator.dbOid,
- RelFileLocatorToPgStatObjid(rel->rd_locator));
+ /* Check if this drop is part of a pending rewrite */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid &&
+ rel->rd_locator.spcOid == rewrite->old_locator.spcOid &&
+ rel->rd_locator.relNumber == rewrite->old_locator.relNumber)
+ {
+ skip_transactional_drop = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * If it is part of a rewrite, drop its stats later, for example in
+ * AtEOXact_PgStat_Relations(), so skip it here.
+ */
+ if (!skip_transactional_drop)
+ pgstat_drop_transactional(PGSTAT_KIND_RELATION,
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -660,6 +782,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit)
}
tabstat->trans = NULL;
}
+
+ /* preserve the stats in case of rewrite */
+ if (isCommit && pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *prev = NULL;
+ PgStat_PendingRewrite *current = pending_rewrites;
+ PgStat_PendingRewrite *next;
+
+ /* reverse the rewrites list to process in chronological order */
+ while (current != NULL)
+ {
+ next = current->next;
+ current->next = prev;
+ prev = current;
+ current = next;
+ }
+
+ /* now process rewrites in chronological order */
+ for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ pgstat_copy_relation_stats(rewrite->new_locator,
+ rewrite->old_locator, true);
+
+ /* drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
+ }
+
+ pending_rewrites = NULL;
}
/*
@@ -675,6 +839,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in
PgStat_TableXactStatus *trans;
PgStat_TableXactStatus *next_trans;
+ /*
+ * If we don't commit then remove the associated rewrites if any, to keep
+ * the rewrite chain in sync with what will be eventually committed.
+ */
+ if (!isCommit)
+ {
+ PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites;
+
+ while (*rewrite_ptr != NULL)
+ {
+ if ((*rewrite_ptr)->nest_level >= nestDepth)
+ {
+ PgStat_PendingRewrite *to_remove = *rewrite_ptr;
+
+ *rewrite_ptr = (*rewrite_ptr)->next;
+ pfree(to_remove);
+ }
+ else
+ {
+ rewrite_ptr = &((*rewrite_ptr)->next);
+ }
+ }
+ }
+
for (trans = xact_state->first; trans != NULL; trans = next_trans)
{
PgStat_TableStatus *tabstat;
@@ -754,11 +942,19 @@ void
AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
{
PgStat_TableXactStatus *trans;
+ PgStat_PendingRewrite *rewrite;
+ /*
+ * For each tabstat, find its matching rewrite and remove it from the
+ * pending rewrites list. This way, after processing all tabstats, pending
+ * rewrites will only contain rewrite only transactions.
+ */
for (trans = xact_state->first; trans != NULL; trans = trans->next)
{
PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY;
TwoPhasePgStatRecord record;
+ PgStat_PendingRewrite **rewrite_ptr;
+ bool found_rewrite = false;
Assert(trans->nest_level == 1);
Assert(trans->upper == NULL);
@@ -778,10 +974,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.locator = tabstat->locator;
record.truncdropped = trans->truncdropped;
+ record.rewrite_nest_level = 0;
+
+ /*
+ * Look for a matching rewrite and remove it from pending rewrites. We
+ * check three possible matches:
+ *
+ * The new_locator when stats have been added after the rewrite. The
+ * old_locator when stats have been added before the rewrite but not
+ * after. The original_locator when this tabstat is part of a rewrite
+ * chain.
+ */
+ rewrite_ptr = &pending_rewrites;
+ while (*rewrite_ptr != NULL)
+ {
+ rewrite = *rewrite_ptr;
+
+ if ((record.locator.dbOid == rewrite->new_locator.dbOid &&
+ record.locator.spcOid == rewrite->new_locator.spcOid &&
+ record.locator.relNumber == rewrite->new_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->old_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->old_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->old_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->original_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->original_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->original_locator.relNumber))
+ {
+ /*
+ * Found matching rewrite. Record the rewrite information and
+ * remove this rewrite from the list since it's now handled.
+ */
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.locator = rewrite->new_locator;
+ found_rewrite = true;
+
+ /* Remove from pending_rewrites list */
+ *rewrite_ptr = rewrite->next;
+ pfree(rewrite);
+ break;
+ }
+ else
+ {
+ /* Move to next rewrite in the list */
+ rewrite_ptr = &(rewrite->next);
+ }
+ }
+
+ /* If no rewrite found, clear the rewrite fields */
+ if (!found_rewrite)
+ {
+ memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator));
+ }
+
+ RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
+ &record, sizeof(TwoPhasePgStatRecord));
+ }
+
+ /*
+ * Now process any rewrites still pending. These are rewrite only
+ * transactions. We need to preserve their stats even though there's no
+ * tabstat entry for them.
+ */
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ TwoPhasePgStatRecord record;
+
+ memset(&record, 0, sizeof(TwoPhasePgStatRecord));
+ record.locator = rewrite->new_locator;
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.truncdropped = false;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
&record, sizeof(TwoPhasePgStatRecord));
}
+
+ pending_rewrites = NULL;
}
/*
@@ -804,6 +1073,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
tabstat = trans->parent;
tabstat->trans = NULL;
}
+
+ pending_rewrites = NULL;
}
/*
@@ -839,6 +1110,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
pgstat_info->counts.changed_tuples +=
rec->tuples_inserted + rec->tuples_updated +
rec->tuples_deleted;
+
+ if (rec->rewrite_nest_level > 0)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ /* Flush any pending stats for old locator first */
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ /* Copy stats from old to new locator */
+ pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator,
+ true);
+
+ /* Drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
}
/*
@@ -853,9 +1147,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
{
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
+ RelFileLocator target_locator;
+
+ /*
+ * For aborted transactions with rewrites (like TRUNCATE), we need to
+ * restore stats to the old locator, not the new one. The new locator
+ * should be dropped since the rewrite is being rolled back.
+ */
+ if (rec->rewrite_nest_level > 0)
+ {
+ /* Use the old locator */
+ target_locator = rec->rewrite_old_locator;
+ }
+ else
+ {
+ /* No rewrite, use the original locator */
+ target_locator = rec->locator;
+ }
/* Find or create a tabstat entry for the target locator */
- pgstat_info = pgstat_prep_relation_pending(rec->locator);
+ pgstat_info = pgstat_prep_relation_pending(target_locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -910,7 +1221,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
+ TimestampTz t;
+
+ /*
+ * Checking the transaction state due to the flush call in
+ * pgstat_twophase_postcommit() that would break the assertion on the
+ * state in GetCurrentTransactionStopTimestamp().
+ */
+ if (!IsTransactionState())
+ t = GetCurrentTransactionStopTimestamp();
+ else
+ t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
@@ -1162,3 +1483,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
ReleaseSysCache(tuple);
return result;
}
+
+/*
+ * Mark that a relation rewrite has occurred, preserving the original locator
+ * so stats can be copied at transaction commit.
+ */
+void
+pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator)
+{
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *existing;
+ RelFileLocator original_locator = old_locator;
+
+ for (existing = pending_rewrites; existing != NULL; existing = existing->next)
+ {
+ if (old_locator.dbOid == existing->new_locator.dbOid &&
+ old_locator.spcOid == existing->new_locator.spcOid &&
+ old_locator.relNumber == existing->new_locator.relNumber)
+ {
+ original_locator = existing->original_locator;
+ break;
+ }
+ }
+
+ /* Allocate in TopTransactionContext memory context */
+ rewrite = MemoryContextAlloc(TopTransactionContext,
+ sizeof(PgStat_PendingRewrite));
+
+ rewrite->old_locator = old_locator;
+ rewrite->new_locator = new_locator;
+ rewrite->original_locator = original_locator;
+ rewrite->nest_level = GetCurrentTransactionNestLevel();
+
+ /* Add to the list */
+ rewrite->next = pending_rewrites;
+ pending_rewrites = rewrite;
+}
+
+void
+pgstat_clear_rewrite(void)
+{
+ pending_rewrites = NULL;
+}
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index bc9864bd8d9..f8cf3755ce2 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel)
}
pgStatXactStack = NULL;
+ pgstat_clear_rewrite();
+
/* Make sure any stats snapshot is thrown away */
pgstat_clear_snapshot();
}
@@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo
void
pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid)
{
- if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL))
+ PgStat_EntryRef *entry_ref;
+
+ entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL);
+
+ if (entry_ref)
{
+ /*
+ * For relations stats, we key by physical file location, not by
+ * relation OID. This means during operations like ALTER TYPE where
+ * the relation OID changes but the relfilenode stays the same (no
+ * actual rewrite needed), we'll find an existing entry.
+ *
+ * This is expected behavior, we want to preserve stats across the
+ * catalog change. Simply reset and recreate the entry for the new
+ * relation OID without warning.
+ */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ pgstat_reset(kind, dboid, objid);
+ create_drop_transactional_internal(kind, dboid, objid, true);
+ return;
+ }
+
ereport(WARNING,
errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64,
(pgstat_get_kind_info(kind))->name, dboid,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 2d0cb7bcfd4..c98e5c51d63 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -85,6 +85,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/relmapper.h"
#include "utils/resowner.h"
#include "utils/snapmgr.h"
@@ -3780,6 +3781,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
MultiXactId minmulti = InvalidMultiXactId;
TransactionId freezeXid = InvalidTransactionId;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator = relation->rd_locator;
if (!IsBinaryUpgrade)
{
@@ -3951,6 +3953,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
table_close(pg_class, RowExclusiveLock);
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Make the pg_class row change or relation map change visible. This will
* cause the relcache entry to get updated, too.
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index c9e451094f3..33d530740d0 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -670,7 +670,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id);
extern void pgstat_create_relation(Relation rel);
extern void pgstat_drop_relation(Relation rel);
-extern void pgstat_copy_relation_stats(Relation dst, Relation src);
+extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment);
extern void pgstat_init_relation(Relation rel);
extern void pgstat_assoc_relation(Relation rel);
@@ -682,6 +682,9 @@ extern void pgstat_report_vacuum(RelFileLocator locator, PgStat_Counter livetupl
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_mark_rewrite(RelFileLocator old_locator,
+ RelFileLocator new_locator);
+extern void pgstat_clear_rewrite(void);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3451538565e..0ca3eae8026 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2268,6 +2268,7 @@ PgStat_KindInfo
PgStat_LocalState
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
+PgStat_PendingRewrite
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
--BHD/2LBqtxMXWADJ--
^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v14 2/2] handle relation statistics correctly during rewrites
@ 2025-11-04 13:52 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Bertrand Drouvot @ 2025-11-04 13:52 UTC (permalink / raw)
Now that PGSTAT_KIND_RELATION is keyed by refilenode, we need to handle rewrites.
To do so, this patch:
- Adds PgStat_PendingRewrite, a new struct to track rewrite operations within
a transaction, storing the old locator, new locator, and original locator (for
rewrite chains). This allows stats to be copied from the original location to
the final location at commit time.
- Adds a new function, pgstat_mark_rewrite(), called when a table rewrite begins.
It records the rewrite operation in a local list and detects rewrite chains by
checking if the old_locator matches any existing new_locator, preserving the
chain's original_locator.
- Modifies pgstat_copy_relation_stats(), to accept RelFileLocators instead of
Relations, with a new increment parameter to accumulate stats (needed for rewrite
chains with DML between rewrites).
- Ensures that AtEOXact_PgStat_Relations(), AtPrepare_PgStat_Relations(),
pgstat_twophase_postcommit()/postabort() pgstat_drop_relation() handle the
PgStat_PendingRewrite list correctly.
Note that due to the new flush call in pgstat_twophase_postcommit() we can not
call GetCurrentTransactionStopTimestamp() in pgstat_relation_flush_cb(). So,
adding a check to handle this special case and call GetCurrentTimestamp() instead.
Note that we'd call GetCurrentTimestamp() only if there is a rewrite, so that
the GetCurrentTimestamp() extra cost should be negligible. Another solution
could be to trigger the flush from FinishPreparedTransaction() but that's not
worth the extra complexity.
The new pending_rewrites list is traversed in multiple places. The overhead
should be negligible in comparison to a rewrite and the list should not contain
a lot of rewrites in practice.
The pending_rewrites list is traversed in multiple places. In typical usage,
the list will contain only a few entries so the traversal cost is negligible (
furthermore in comparison to a rewrite).
---
src/backend/catalog/index.c | 2 +-
src/backend/commands/cluster.c | 5 +
src/backend/commands/tablecmds.c | 6 +
src/backend/utils/activity/pgstat_relation.c | 391 ++++++++++++++++++-
src/backend/utils/activity/pgstat_xact.c | 25 +-
src/backend/utils/cache/relcache.c | 6 +
src/include/pgstat.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 424 insertions(+), 17 deletions(-)
92.8% src/backend/utils/activity/
4.9% src/backend/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index d8219b18c48..079e504b96f 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1796,7 +1796,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId);
/* copy over statistics from old to new index */
- pgstat_copy_relation_stats(newClassRel, oldClassRel);
+ pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 09066db0956..c491daceb0f 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1206,6 +1206,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
rel1 = relation_open(r1, NoLock);
rel2 = relation_open(r2, NoLock);
+
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind))
+ pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator);
+
rel2->rd_createSubid = rel1->rd_createSubid;
rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid;
rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index c69c12dc014..13630846116 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -17099,6 +17099,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
Oid reltoastrelid;
RelFileNumber newrelfilenumber;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator;
List *reltoastidxids = NIL;
ListCell *lc;
@@ -17137,6 +17138,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
newrlocator = rel->rd_locator;
newrlocator.relNumber = newrelfilenumber;
newrlocator.spcOid = newTableSpace;
+ oldrlocator = rel->rd_locator;
/* hand off to AM to actually create new rel storage and copy the data */
if (rel->rd_rel->relkind == RELKIND_INDEX)
@@ -17149,6 +17151,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
table_relation_copy_data(rel, &newrlocator);
}
+ /* mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Update the pg_class row.
*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 89bf0cbed56..4d929972fab 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -30,6 +30,19 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
+/* Pending rewrite operations for stats copying */
+typedef struct PgStat_PendingRewrite
+{
+ RelFileLocator old_locator;
+ RelFileLocator new_locator;
+ RelFileLocator original_locator;
+ int nest_level; /* Transaction nesting level where rewrite
+ * occurred */
+ struct PgStat_PendingRewrite *next;
+} PgStat_PendingRewrite;
+
+/* The pending rewrites list for current transaction */
+static PgStat_PendingRewrite *pending_rewrites = NULL;
/* Record that's written to 2PC state file when pgstat state is persisted */
typedef struct TwoPhasePgStatRecord
@@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter deleted_pre_truncdrop;
RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
+ RelFileLocator rewrite_old_locator;
+ int rewrite_nest_level;
} TwoPhasePgStatRecord;
@@ -54,27 +69,70 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
/*
- * Copy stats between relations. This is used for things like REINDEX
+ * Copy stats between RelFileLocator. This is used for things like REINDEX
* CONCURRENTLY.
*/
void
-pgstat_copy_relation_stats(Relation dst, Relation src)
+pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment)
{
PgStat_StatTabEntry *srcstats;
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src));
+ srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION,
+ src.dbOid,
+ RelFileLocatorToPgStatObjid(src));
if (!srcstats)
return;
dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
- dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(dst),
+ dst.dbOid,
+ RelFileLocatorToPgStatObjid(dst),
false);
dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats;
- dstshstats->stats = *srcstats;
+
+ if (!increment)
+ dstshstats->stats = *srcstats;
+ else
+ {
+ /* Increment those statistics */
+#define RELFSTAT_ACC(fld, stats_to_add) \
+ (dstshstats->stats.fld += stats_to_add->fld)
+ RELFSTAT_ACC(numscans, srcstats);
+ RELFSTAT_ACC(tuples_returned, srcstats);
+ RELFSTAT_ACC(tuples_fetched, srcstats);
+ RELFSTAT_ACC(tuples_inserted, srcstats);
+ RELFSTAT_ACC(tuples_updated, srcstats);
+ RELFSTAT_ACC(tuples_deleted, srcstats);
+ RELFSTAT_ACC(tuples_hot_updated, srcstats);
+ RELFSTAT_ACC(tuples_newpage_updated, srcstats);
+ RELFSTAT_ACC(live_tuples, srcstats);
+ RELFSTAT_ACC(dead_tuples, srcstats);
+ RELFSTAT_ACC(mod_since_analyze, srcstats);
+ RELFSTAT_ACC(ins_since_vacuum, srcstats);
+ RELFSTAT_ACC(blocks_fetched, srcstats);
+ RELFSTAT_ACC(blocks_hit, srcstats);
+ RELFSTAT_ACC(vacuum_count, srcstats);
+ RELFSTAT_ACC(autovacuum_count, srcstats);
+ RELFSTAT_ACC(analyze_count, srcstats);
+ RELFSTAT_ACC(autoanalyze_count, srcstats);
+ RELFSTAT_ACC(total_vacuum_time, srcstats);
+ RELFSTAT_ACC(total_autovacuum_time, srcstats);
+ RELFSTAT_ACC(total_analyze_time, srcstats);
+ RELFSTAT_ACC(total_autoanalyze_time, srcstats);
+#undef RELFSTAT_ACC
+
+ /* Replace those statistics */
+#define RELFSTAT_REP(fld, stats_to_rep) \
+ (dstshstats->stats.fld = stats_to_rep->fld)
+ RELFSTAT_REP(lastscan, srcstats);
+ RELFSTAT_REP(last_vacuum_time, srcstats);
+ RELFSTAT_REP(last_autovacuum_time, srcstats);
+ RELFSTAT_REP(last_analyze_time, srcstats);
+ RELFSTAT_REP(last_autoanalyze_time, srcstats);
+#undef RELFSTAT_REP
+ }
pgstat_unlock_entry(dst_ref);
}
@@ -136,6 +194,7 @@ void
pgstat_assoc_relation(Relation rel)
{
RelFileLocator locator;
+ PgStat_TableStatus *pgstat_info;
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
@@ -164,14 +223,54 @@ pgstat_assoc_relation(Relation rel)
locator.relNumber = rel->rd_id;
}
+ /*
+ * If this relation was rewritten during the current transaction we may be
+ * reopening it with its new RelFileLocator. In that case, continue using
+ * the stats entry associated with the old locator rather than creating a
+ * new one. This ensures all stats from before and after the rewrite are
+ * tracked in a single entry which will be properly copied to the new
+ * locator at transaction commit.
+ */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (locator.dbOid == rewrite->new_locator.dbOid &&
+ locator.spcOid == rewrite->new_locator.spcOid &&
+ locator.relNumber == rewrite->new_locator.relNumber)
+ {
+ pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator);
+ goto found_entry;
+ }
+ }
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(locator);
+ pgstat_info = pgstat_prep_relation_pending(locator);
+
+found_entry:
+ rel->pgstat_info = pgstat_info;
+
+ /*
+ * For relations stats, we key by physical file location, not by relation
+ * OID. This means during operations like ALTER TYPE it's possible that
+ * the relation OID changes but the relfilenode stays the same (no actual
+ * rewrite needed). Unlink the old relation first.
+ */
+ if (pgstat_info->relation != NULL &&
+ pgstat_info->relation != rel)
+ {
+ pgstat_info->relation->pgstat_info = NULL;
+ pgstat_info->relation = NULL;
+ }
/* don't allow link a stats to multiple relcache entries */
- Assert(rel->pgstat_info->relation == NULL);
+ Assert(pgstat_info->relation == NULL);
/* mark this relation as the owner */
- rel->pgstat_info->relation = rel;
+ pgstat_info->relation = rel;
}
/*
@@ -214,14 +313,37 @@ pgstat_drop_relation(Relation rel)
{
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ bool skip_transactional_drop = false;
/* don't track stats for relations without storage */
if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
return;
- pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_locator.dbOid,
- RelFileLocatorToPgStatObjid(rel->rd_locator));
+ /* Check if this drop is part of a pending rewrite */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid &&
+ rel->rd_locator.spcOid == rewrite->old_locator.spcOid &&
+ rel->rd_locator.relNumber == rewrite->old_locator.relNumber)
+ {
+ skip_transactional_drop = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * If it is part of a rewrite, drop its stats later, for example in
+ * AtEOXact_PgStat_Relations(), so skip it here.
+ */
+ if (!skip_transactional_drop)
+ pgstat_drop_transactional(PGSTAT_KIND_RELATION,
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -666,6 +788,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit)
}
tabstat->trans = NULL;
}
+
+ /* preserve the stats in case of rewrite */
+ if (isCommit && pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *prev = NULL;
+ PgStat_PendingRewrite *current = pending_rewrites;
+ PgStat_PendingRewrite *next;
+
+ /* reverse the rewrites list to process in chronological order */
+ while (current != NULL)
+ {
+ next = current->next;
+ current->next = prev;
+ prev = current;
+ current = next;
+ }
+
+ /* now process rewrites in chronological order */
+ for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ pgstat_copy_relation_stats(rewrite->new_locator,
+ rewrite->old_locator, true);
+
+ /* drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
+ }
+
+ pending_rewrites = NULL;
}
/*
@@ -681,6 +845,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in
PgStat_TableXactStatus *trans;
PgStat_TableXactStatus *next_trans;
+ /*
+ * If we don't commit then remove the associated rewrites if any, to keep
+ * the rewrite chain in sync with what will be eventually committed.
+ */
+ if (!isCommit)
+ {
+ PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites;
+
+ while (*rewrite_ptr != NULL)
+ {
+ if ((*rewrite_ptr)->nest_level >= nestDepth)
+ {
+ PgStat_PendingRewrite *to_remove = *rewrite_ptr;
+
+ *rewrite_ptr = (*rewrite_ptr)->next;
+ pfree(to_remove);
+ }
+ else
+ {
+ rewrite_ptr = &((*rewrite_ptr)->next);
+ }
+ }
+ }
+
for (trans = xact_state->first; trans != NULL; trans = next_trans)
{
PgStat_TableStatus *tabstat;
@@ -760,11 +948,19 @@ void
AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
{
PgStat_TableXactStatus *trans;
+ PgStat_PendingRewrite *rewrite;
+ /*
+ * For each tabstat, find its matching rewrite and remove it from the
+ * pending rewrites list. This way, after processing all tabstats, pending
+ * rewrites will only contain rewrite only transactions.
+ */
for (trans = xact_state->first; trans != NULL; trans = trans->next)
{
PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY;
TwoPhasePgStatRecord record;
+ PgStat_PendingRewrite **rewrite_ptr;
+ bool found_rewrite = false;
Assert(trans->nest_level == 1);
Assert(trans->upper == NULL);
@@ -784,10 +980,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.locator = tabstat->locator;
record.truncdropped = trans->truncdropped;
+ record.rewrite_nest_level = 0;
+
+ /*
+ * Look for a matching rewrite and remove it from pending rewrites. We
+ * check three possible matches:
+ *
+ * The new_locator when stats have been added after the rewrite. The
+ * old_locator when stats have been added before the rewrite but not
+ * after. The original_locator when this tabstat is part of a rewrite
+ * chain.
+ */
+ rewrite_ptr = &pending_rewrites;
+ while (*rewrite_ptr != NULL)
+ {
+ rewrite = *rewrite_ptr;
+
+ if ((record.locator.dbOid == rewrite->new_locator.dbOid &&
+ record.locator.spcOid == rewrite->new_locator.spcOid &&
+ record.locator.relNumber == rewrite->new_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->old_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->old_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->old_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->original_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->original_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->original_locator.relNumber))
+ {
+ /*
+ * Found matching rewrite. Record the rewrite information and
+ * remove this rewrite from the list since it's now handled.
+ */
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.locator = rewrite->new_locator;
+ found_rewrite = true;
+
+ /* Remove from pending_rewrites list */
+ *rewrite_ptr = rewrite->next;
+ pfree(rewrite);
+ break;
+ }
+ else
+ {
+ /* Move to next rewrite in the list */
+ rewrite_ptr = &(rewrite->next);
+ }
+ }
+
+ /* If no rewrite found, clear the rewrite fields */
+ if (!found_rewrite)
+ {
+ memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator));
+ }
+
+ RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
+ &record, sizeof(TwoPhasePgStatRecord));
+ }
+
+ /*
+ * Now process any rewrites still pending. These are rewrite only
+ * transactions. We need to preserve their stats even though there's no
+ * tabstat entry for them.
+ */
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ TwoPhasePgStatRecord record;
+
+ memset(&record, 0, sizeof(TwoPhasePgStatRecord));
+ record.locator = rewrite->new_locator;
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.truncdropped = false;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
&record, sizeof(TwoPhasePgStatRecord));
}
+
+ pending_rewrites = NULL;
}
/*
@@ -810,6 +1079,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
tabstat = trans->parent;
tabstat->trans = NULL;
}
+
+ pending_rewrites = NULL;
}
/*
@@ -845,6 +1116,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
pgstat_info->counts.changed_tuples +=
rec->tuples_inserted + rec->tuples_updated +
rec->tuples_deleted;
+
+ if (rec->rewrite_nest_level > 0)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ /* Flush any pending stats for old locator first */
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ /* Copy stats from old to new locator */
+ pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator,
+ true);
+
+ /* Drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
}
/*
@@ -859,9 +1153,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
{
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
+ RelFileLocator target_locator;
+
+ /*
+ * For aborted transactions with rewrites (like TRUNCATE), we need to
+ * restore stats to the old locator, not the new one. The new locator
+ * should be dropped since the rewrite is being rolled back.
+ */
+ if (rec->rewrite_nest_level > 0)
+ {
+ /* Use the old locator */
+ target_locator = rec->rewrite_old_locator;
+ }
+ else
+ {
+ /* No rewrite, use the original locator */
+ target_locator = rec->locator;
+ }
/* Find or create a tabstat entry for the target locator */
- pgstat_info = pgstat_prep_relation_pending(rec->locator);
+ pgstat_info = pgstat_prep_relation_pending(target_locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -916,7 +1227,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
+ TimestampTz t;
+
+ /*
+ * Checking the transaction state due to the flush call in
+ * pgstat_twophase_postcommit() that would break the assertion on the
+ * state in GetCurrentTransactionStopTimestamp().
+ */
+ if (!IsTransactionState())
+ t = GetCurrentTransactionStopTimestamp();
+ else
+ t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
@@ -1167,3 +1488,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
ReleaseSysCache(tuple);
return result;
}
+
+/*
+ * Mark that a relation rewrite has occurred, preserving the original locator
+ * so stats can be copied at transaction commit.
+ */
+void
+pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator)
+{
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *existing;
+ RelFileLocator original_locator = old_locator;
+
+ for (existing = pending_rewrites; existing != NULL; existing = existing->next)
+ {
+ if (old_locator.dbOid == existing->new_locator.dbOid &&
+ old_locator.spcOid == existing->new_locator.spcOid &&
+ old_locator.relNumber == existing->new_locator.relNumber)
+ {
+ original_locator = existing->original_locator;
+ break;
+ }
+ }
+
+ /* Allocate in TopTransactionContext memory context */
+ rewrite = MemoryContextAlloc(TopTransactionContext,
+ sizeof(PgStat_PendingRewrite));
+
+ rewrite->old_locator = old_locator;
+ rewrite->new_locator = new_locator;
+ rewrite->original_locator = original_locator;
+ rewrite->nest_level = GetCurrentTransactionNestLevel();
+
+ /* Add to the list */
+ rewrite->next = pending_rewrites;
+ pending_rewrites = rewrite;
+}
+
+void
+pgstat_clear_rewrite(void)
+{
+ pending_rewrites = NULL;
+}
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index 5e2d69e6297..8ed8f5317f3 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel)
}
pgStatXactStack = NULL;
+ pgstat_clear_rewrite();
+
/* Make sure any stats snapshot is thrown away */
pgstat_clear_snapshot();
}
@@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo
void
pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid)
{
- if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL))
+ PgStat_EntryRef *entry_ref;
+
+ entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL);
+
+ if (entry_ref)
{
+ /*
+ * For relations stats, we key by physical file location, not by
+ * relation OID. This means during operations like ALTER TYPE where
+ * the relation OID changes but the relfilenode stays the same (no
+ * actual rewrite needed), we'll find an existing entry.
+ *
+ * This is expected behavior, we want to preserve stats across the
+ * catalog change. Simply reset and recreate the entry for the new
+ * relation OID without warning.
+ */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ pgstat_reset(kind, dboid, objid);
+ create_drop_transactional_internal(kind, dboid, objid, true);
+ return;
+ }
+
ereport(WARNING,
errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64,
(pgstat_get_kind_info(kind))->name, dboid,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index e19f0d3e51c..bf49830dba2 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -87,6 +87,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/relmapper.h"
#include "utils/resowner.h"
#include "utils/snapmgr.h"
@@ -3772,6 +3773,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
MultiXactId minmulti = InvalidMultiXactId;
TransactionId freezeXid = InvalidTransactionId;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator = relation->rd_locator;
if (!IsBinaryUpgrade)
{
@@ -3943,6 +3945,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
table_close(pg_class, RowExclusiveLock);
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Make the pg_class row change or relation map change visible. This will
* cause the relcache entry to get updated, too.
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index ef740aff26e..4d3130ae99e 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -701,7 +701,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id);
extern void pgstat_create_relation(Relation rel);
extern void pgstat_drop_relation(Relation rel);
-extern void pgstat_copy_relation_stats(Relation dst, Relation src);
+extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment);
extern void pgstat_init_relation(Relation rel);
extern void pgstat_assoc_relation(Relation rel);
@@ -713,6 +713,9 @@ extern void pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_mark_rewrite(RelFileLocator old_locator,
+ RelFileLocator new_locator);
+extern void pgstat_clear_rewrite(void);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index dbbec84b222..bae04e95989 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2310,6 +2310,7 @@ PgStat_LockEntry
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
PgStat_PendingLock
+PgStat_PendingRewrite
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
--96iU9DKeu+W+YlSa--
^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v14 2/2] handle relation statistics correctly during rewrites
@ 2025-11-04 13:52 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Bertrand Drouvot @ 2025-11-04 13:52 UTC (permalink / raw)
Now that PGSTAT_KIND_RELATION is keyed by refilenode, we need to handle rewrites.
To do so, this patch:
- Adds PgStat_PendingRewrite, a new struct to track rewrite operations within
a transaction, storing the old locator, new locator, and original locator (for
rewrite chains). This allows stats to be copied from the original location to
the final location at commit time.
- Adds a new function, pgstat_mark_rewrite(), called when a table rewrite begins.
It records the rewrite operation in a local list and detects rewrite chains by
checking if the old_locator matches any existing new_locator, preserving the
chain's original_locator.
- Modifies pgstat_copy_relation_stats(), to accept RelFileLocators instead of
Relations, with a new increment parameter to accumulate stats (needed for rewrite
chains with DML between rewrites).
- Ensures that AtEOXact_PgStat_Relations(), AtPrepare_PgStat_Relations(),
pgstat_twophase_postcommit()/postabort() pgstat_drop_relation() handle the
PgStat_PendingRewrite list correctly.
Note that due to the new flush call in pgstat_twophase_postcommit() we can not
call GetCurrentTransactionStopTimestamp() in pgstat_relation_flush_cb(). So,
adding a check to handle this special case and call GetCurrentTimestamp() instead.
Note that we'd call GetCurrentTimestamp() only if there is a rewrite, so that
the GetCurrentTimestamp() extra cost should be negligible. Another solution
could be to trigger the flush from FinishPreparedTransaction() but that's not
worth the extra complexity.
The new pending_rewrites list is traversed in multiple places. The overhead
should be negligible in comparison to a rewrite and the list should not contain
a lot of rewrites in practice.
The pending_rewrites list is traversed in multiple places. In typical usage,
the list will contain only a few entries so the traversal cost is negligible (
furthermore in comparison to a rewrite).
---
src/backend/catalog/index.c | 2 +-
src/backend/commands/cluster.c | 5 +
src/backend/commands/tablecmds.c | 6 +
src/backend/utils/activity/pgstat_relation.c | 391 ++++++++++++++++++-
src/backend/utils/activity/pgstat_xact.c | 25 +-
src/backend/utils/cache/relcache.c | 6 +
src/include/pgstat.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 424 insertions(+), 17 deletions(-)
92.8% src/backend/utils/activity/
4.9% src/backend/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index d8219b18c48..079e504b96f 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1796,7 +1796,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId);
/* copy over statistics from old to new index */
- pgstat_copy_relation_stats(newClassRel, oldClassRel);
+ pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 09066db0956..c491daceb0f 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1206,6 +1206,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
rel1 = relation_open(r1, NoLock);
rel2 = relation_open(r2, NoLock);
+
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind))
+ pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator);
+
rel2->rd_createSubid = rel1->rd_createSubid;
rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid;
rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index c69c12dc014..13630846116 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -17099,6 +17099,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
Oid reltoastrelid;
RelFileNumber newrelfilenumber;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator;
List *reltoastidxids = NIL;
ListCell *lc;
@@ -17137,6 +17138,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
newrlocator = rel->rd_locator;
newrlocator.relNumber = newrelfilenumber;
newrlocator.spcOid = newTableSpace;
+ oldrlocator = rel->rd_locator;
/* hand off to AM to actually create new rel storage and copy the data */
if (rel->rd_rel->relkind == RELKIND_INDEX)
@@ -17149,6 +17151,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
table_relation_copy_data(rel, &newrlocator);
}
+ /* mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Update the pg_class row.
*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 89bf0cbed56..4d929972fab 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -30,6 +30,19 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
+/* Pending rewrite operations for stats copying */
+typedef struct PgStat_PendingRewrite
+{
+ RelFileLocator old_locator;
+ RelFileLocator new_locator;
+ RelFileLocator original_locator;
+ int nest_level; /* Transaction nesting level where rewrite
+ * occurred */
+ struct PgStat_PendingRewrite *next;
+} PgStat_PendingRewrite;
+
+/* The pending rewrites list for current transaction */
+static PgStat_PendingRewrite *pending_rewrites = NULL;
/* Record that's written to 2PC state file when pgstat state is persisted */
typedef struct TwoPhasePgStatRecord
@@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter deleted_pre_truncdrop;
RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
+ RelFileLocator rewrite_old_locator;
+ int rewrite_nest_level;
} TwoPhasePgStatRecord;
@@ -54,27 +69,70 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
/*
- * Copy stats between relations. This is used for things like REINDEX
+ * Copy stats between RelFileLocator. This is used for things like REINDEX
* CONCURRENTLY.
*/
void
-pgstat_copy_relation_stats(Relation dst, Relation src)
+pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment)
{
PgStat_StatTabEntry *srcstats;
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src));
+ srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION,
+ src.dbOid,
+ RelFileLocatorToPgStatObjid(src));
if (!srcstats)
return;
dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
- dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(dst),
+ dst.dbOid,
+ RelFileLocatorToPgStatObjid(dst),
false);
dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats;
- dstshstats->stats = *srcstats;
+
+ if (!increment)
+ dstshstats->stats = *srcstats;
+ else
+ {
+ /* Increment those statistics */
+#define RELFSTAT_ACC(fld, stats_to_add) \
+ (dstshstats->stats.fld += stats_to_add->fld)
+ RELFSTAT_ACC(numscans, srcstats);
+ RELFSTAT_ACC(tuples_returned, srcstats);
+ RELFSTAT_ACC(tuples_fetched, srcstats);
+ RELFSTAT_ACC(tuples_inserted, srcstats);
+ RELFSTAT_ACC(tuples_updated, srcstats);
+ RELFSTAT_ACC(tuples_deleted, srcstats);
+ RELFSTAT_ACC(tuples_hot_updated, srcstats);
+ RELFSTAT_ACC(tuples_newpage_updated, srcstats);
+ RELFSTAT_ACC(live_tuples, srcstats);
+ RELFSTAT_ACC(dead_tuples, srcstats);
+ RELFSTAT_ACC(mod_since_analyze, srcstats);
+ RELFSTAT_ACC(ins_since_vacuum, srcstats);
+ RELFSTAT_ACC(blocks_fetched, srcstats);
+ RELFSTAT_ACC(blocks_hit, srcstats);
+ RELFSTAT_ACC(vacuum_count, srcstats);
+ RELFSTAT_ACC(autovacuum_count, srcstats);
+ RELFSTAT_ACC(analyze_count, srcstats);
+ RELFSTAT_ACC(autoanalyze_count, srcstats);
+ RELFSTAT_ACC(total_vacuum_time, srcstats);
+ RELFSTAT_ACC(total_autovacuum_time, srcstats);
+ RELFSTAT_ACC(total_analyze_time, srcstats);
+ RELFSTAT_ACC(total_autoanalyze_time, srcstats);
+#undef RELFSTAT_ACC
+
+ /* Replace those statistics */
+#define RELFSTAT_REP(fld, stats_to_rep) \
+ (dstshstats->stats.fld = stats_to_rep->fld)
+ RELFSTAT_REP(lastscan, srcstats);
+ RELFSTAT_REP(last_vacuum_time, srcstats);
+ RELFSTAT_REP(last_autovacuum_time, srcstats);
+ RELFSTAT_REP(last_analyze_time, srcstats);
+ RELFSTAT_REP(last_autoanalyze_time, srcstats);
+#undef RELFSTAT_REP
+ }
pgstat_unlock_entry(dst_ref);
}
@@ -136,6 +194,7 @@ void
pgstat_assoc_relation(Relation rel)
{
RelFileLocator locator;
+ PgStat_TableStatus *pgstat_info;
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
@@ -164,14 +223,54 @@ pgstat_assoc_relation(Relation rel)
locator.relNumber = rel->rd_id;
}
+ /*
+ * If this relation was rewritten during the current transaction we may be
+ * reopening it with its new RelFileLocator. In that case, continue using
+ * the stats entry associated with the old locator rather than creating a
+ * new one. This ensures all stats from before and after the rewrite are
+ * tracked in a single entry which will be properly copied to the new
+ * locator at transaction commit.
+ */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (locator.dbOid == rewrite->new_locator.dbOid &&
+ locator.spcOid == rewrite->new_locator.spcOid &&
+ locator.relNumber == rewrite->new_locator.relNumber)
+ {
+ pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator);
+ goto found_entry;
+ }
+ }
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(locator);
+ pgstat_info = pgstat_prep_relation_pending(locator);
+
+found_entry:
+ rel->pgstat_info = pgstat_info;
+
+ /*
+ * For relations stats, we key by physical file location, not by relation
+ * OID. This means during operations like ALTER TYPE it's possible that
+ * the relation OID changes but the relfilenode stays the same (no actual
+ * rewrite needed). Unlink the old relation first.
+ */
+ if (pgstat_info->relation != NULL &&
+ pgstat_info->relation != rel)
+ {
+ pgstat_info->relation->pgstat_info = NULL;
+ pgstat_info->relation = NULL;
+ }
/* don't allow link a stats to multiple relcache entries */
- Assert(rel->pgstat_info->relation == NULL);
+ Assert(pgstat_info->relation == NULL);
/* mark this relation as the owner */
- rel->pgstat_info->relation = rel;
+ pgstat_info->relation = rel;
}
/*
@@ -214,14 +313,37 @@ pgstat_drop_relation(Relation rel)
{
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ bool skip_transactional_drop = false;
/* don't track stats for relations without storage */
if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
return;
- pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_locator.dbOid,
- RelFileLocatorToPgStatObjid(rel->rd_locator));
+ /* Check if this drop is part of a pending rewrite */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid &&
+ rel->rd_locator.spcOid == rewrite->old_locator.spcOid &&
+ rel->rd_locator.relNumber == rewrite->old_locator.relNumber)
+ {
+ skip_transactional_drop = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * If it is part of a rewrite, drop its stats later, for example in
+ * AtEOXact_PgStat_Relations(), so skip it here.
+ */
+ if (!skip_transactional_drop)
+ pgstat_drop_transactional(PGSTAT_KIND_RELATION,
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -666,6 +788,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit)
}
tabstat->trans = NULL;
}
+
+ /* preserve the stats in case of rewrite */
+ if (isCommit && pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *prev = NULL;
+ PgStat_PendingRewrite *current = pending_rewrites;
+ PgStat_PendingRewrite *next;
+
+ /* reverse the rewrites list to process in chronological order */
+ while (current != NULL)
+ {
+ next = current->next;
+ current->next = prev;
+ prev = current;
+ current = next;
+ }
+
+ /* now process rewrites in chronological order */
+ for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ pgstat_copy_relation_stats(rewrite->new_locator,
+ rewrite->old_locator, true);
+
+ /* drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
+ }
+
+ pending_rewrites = NULL;
}
/*
@@ -681,6 +845,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in
PgStat_TableXactStatus *trans;
PgStat_TableXactStatus *next_trans;
+ /*
+ * If we don't commit then remove the associated rewrites if any, to keep
+ * the rewrite chain in sync with what will be eventually committed.
+ */
+ if (!isCommit)
+ {
+ PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites;
+
+ while (*rewrite_ptr != NULL)
+ {
+ if ((*rewrite_ptr)->nest_level >= nestDepth)
+ {
+ PgStat_PendingRewrite *to_remove = *rewrite_ptr;
+
+ *rewrite_ptr = (*rewrite_ptr)->next;
+ pfree(to_remove);
+ }
+ else
+ {
+ rewrite_ptr = &((*rewrite_ptr)->next);
+ }
+ }
+ }
+
for (trans = xact_state->first; trans != NULL; trans = next_trans)
{
PgStat_TableStatus *tabstat;
@@ -760,11 +948,19 @@ void
AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
{
PgStat_TableXactStatus *trans;
+ PgStat_PendingRewrite *rewrite;
+ /*
+ * For each tabstat, find its matching rewrite and remove it from the
+ * pending rewrites list. This way, after processing all tabstats, pending
+ * rewrites will only contain rewrite only transactions.
+ */
for (trans = xact_state->first; trans != NULL; trans = trans->next)
{
PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY;
TwoPhasePgStatRecord record;
+ PgStat_PendingRewrite **rewrite_ptr;
+ bool found_rewrite = false;
Assert(trans->nest_level == 1);
Assert(trans->upper == NULL);
@@ -784,10 +980,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.locator = tabstat->locator;
record.truncdropped = trans->truncdropped;
+ record.rewrite_nest_level = 0;
+
+ /*
+ * Look for a matching rewrite and remove it from pending rewrites. We
+ * check three possible matches:
+ *
+ * The new_locator when stats have been added after the rewrite. The
+ * old_locator when stats have been added before the rewrite but not
+ * after. The original_locator when this tabstat is part of a rewrite
+ * chain.
+ */
+ rewrite_ptr = &pending_rewrites;
+ while (*rewrite_ptr != NULL)
+ {
+ rewrite = *rewrite_ptr;
+
+ if ((record.locator.dbOid == rewrite->new_locator.dbOid &&
+ record.locator.spcOid == rewrite->new_locator.spcOid &&
+ record.locator.relNumber == rewrite->new_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->old_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->old_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->old_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->original_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->original_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->original_locator.relNumber))
+ {
+ /*
+ * Found matching rewrite. Record the rewrite information and
+ * remove this rewrite from the list since it's now handled.
+ */
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.locator = rewrite->new_locator;
+ found_rewrite = true;
+
+ /* Remove from pending_rewrites list */
+ *rewrite_ptr = rewrite->next;
+ pfree(rewrite);
+ break;
+ }
+ else
+ {
+ /* Move to next rewrite in the list */
+ rewrite_ptr = &(rewrite->next);
+ }
+ }
+
+ /* If no rewrite found, clear the rewrite fields */
+ if (!found_rewrite)
+ {
+ memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator));
+ }
+
+ RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
+ &record, sizeof(TwoPhasePgStatRecord));
+ }
+
+ /*
+ * Now process any rewrites still pending. These are rewrite only
+ * transactions. We need to preserve their stats even though there's no
+ * tabstat entry for them.
+ */
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ TwoPhasePgStatRecord record;
+
+ memset(&record, 0, sizeof(TwoPhasePgStatRecord));
+ record.locator = rewrite->new_locator;
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.truncdropped = false;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
&record, sizeof(TwoPhasePgStatRecord));
}
+
+ pending_rewrites = NULL;
}
/*
@@ -810,6 +1079,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
tabstat = trans->parent;
tabstat->trans = NULL;
}
+
+ pending_rewrites = NULL;
}
/*
@@ -845,6 +1116,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
pgstat_info->counts.changed_tuples +=
rec->tuples_inserted + rec->tuples_updated +
rec->tuples_deleted;
+
+ if (rec->rewrite_nest_level > 0)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ /* Flush any pending stats for old locator first */
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ /* Copy stats from old to new locator */
+ pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator,
+ true);
+
+ /* Drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
}
/*
@@ -859,9 +1153,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
{
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
+ RelFileLocator target_locator;
+
+ /*
+ * For aborted transactions with rewrites (like TRUNCATE), we need to
+ * restore stats to the old locator, not the new one. The new locator
+ * should be dropped since the rewrite is being rolled back.
+ */
+ if (rec->rewrite_nest_level > 0)
+ {
+ /* Use the old locator */
+ target_locator = rec->rewrite_old_locator;
+ }
+ else
+ {
+ /* No rewrite, use the original locator */
+ target_locator = rec->locator;
+ }
/* Find or create a tabstat entry for the target locator */
- pgstat_info = pgstat_prep_relation_pending(rec->locator);
+ pgstat_info = pgstat_prep_relation_pending(target_locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -916,7 +1227,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
+ TimestampTz t;
+
+ /*
+ * Checking the transaction state due to the flush call in
+ * pgstat_twophase_postcommit() that would break the assertion on the
+ * state in GetCurrentTransactionStopTimestamp().
+ */
+ if (!IsTransactionState())
+ t = GetCurrentTransactionStopTimestamp();
+ else
+ t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
@@ -1167,3 +1488,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
ReleaseSysCache(tuple);
return result;
}
+
+/*
+ * Mark that a relation rewrite has occurred, preserving the original locator
+ * so stats can be copied at transaction commit.
+ */
+void
+pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator)
+{
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *existing;
+ RelFileLocator original_locator = old_locator;
+
+ for (existing = pending_rewrites; existing != NULL; existing = existing->next)
+ {
+ if (old_locator.dbOid == existing->new_locator.dbOid &&
+ old_locator.spcOid == existing->new_locator.spcOid &&
+ old_locator.relNumber == existing->new_locator.relNumber)
+ {
+ original_locator = existing->original_locator;
+ break;
+ }
+ }
+
+ /* Allocate in TopTransactionContext memory context */
+ rewrite = MemoryContextAlloc(TopTransactionContext,
+ sizeof(PgStat_PendingRewrite));
+
+ rewrite->old_locator = old_locator;
+ rewrite->new_locator = new_locator;
+ rewrite->original_locator = original_locator;
+ rewrite->nest_level = GetCurrentTransactionNestLevel();
+
+ /* Add to the list */
+ rewrite->next = pending_rewrites;
+ pending_rewrites = rewrite;
+}
+
+void
+pgstat_clear_rewrite(void)
+{
+ pending_rewrites = NULL;
+}
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index 5e2d69e6297..8ed8f5317f3 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel)
}
pgStatXactStack = NULL;
+ pgstat_clear_rewrite();
+
/* Make sure any stats snapshot is thrown away */
pgstat_clear_snapshot();
}
@@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo
void
pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid)
{
- if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL))
+ PgStat_EntryRef *entry_ref;
+
+ entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL);
+
+ if (entry_ref)
{
+ /*
+ * For relations stats, we key by physical file location, not by
+ * relation OID. This means during operations like ALTER TYPE where
+ * the relation OID changes but the relfilenode stays the same (no
+ * actual rewrite needed), we'll find an existing entry.
+ *
+ * This is expected behavior, we want to preserve stats across the
+ * catalog change. Simply reset and recreate the entry for the new
+ * relation OID without warning.
+ */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ pgstat_reset(kind, dboid, objid);
+ create_drop_transactional_internal(kind, dboid, objid, true);
+ return;
+ }
+
ereport(WARNING,
errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64,
(pgstat_get_kind_info(kind))->name, dboid,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index e19f0d3e51c..bf49830dba2 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -87,6 +87,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/relmapper.h"
#include "utils/resowner.h"
#include "utils/snapmgr.h"
@@ -3772,6 +3773,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
MultiXactId minmulti = InvalidMultiXactId;
TransactionId freezeXid = InvalidTransactionId;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator = relation->rd_locator;
if (!IsBinaryUpgrade)
{
@@ -3943,6 +3945,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
table_close(pg_class, RowExclusiveLock);
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Make the pg_class row change or relation map change visible. This will
* cause the relcache entry to get updated, too.
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index ef740aff26e..4d3130ae99e 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -701,7 +701,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id);
extern void pgstat_create_relation(Relation rel);
extern void pgstat_drop_relation(Relation rel);
-extern void pgstat_copy_relation_stats(Relation dst, Relation src);
+extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment);
extern void pgstat_init_relation(Relation rel);
extern void pgstat_assoc_relation(Relation rel);
@@ -713,6 +713,9 @@ extern void pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_mark_rewrite(RelFileLocator old_locator,
+ RelFileLocator new_locator);
+extern void pgstat_clear_rewrite(void);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index dbbec84b222..bae04e95989 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2310,6 +2310,7 @@ PgStat_LockEntry
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
PgStat_PendingLock
+PgStat_PendingRewrite
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
--96iU9DKeu+W+YlSa--
^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v10 2/2] handle relation statistics correctly during rewrites
@ 2025-11-04 13:52 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Bertrand Drouvot @ 2025-11-04 13:52 UTC (permalink / raw)
Now that PGSTAT_KIND_RELATION is keyed by refilenode, we need to handle rewrites.
To do so, this patch:
- Adds PgStat_PendingRewrite, a new struct to track rewrite operations within
a transaction, storing the old locator, new locator, and original locator (for
rewrite chains). This allows stats to be copied from the original location to
the final location at commit time.
- Adds a new function, pgstat_mark_rewrite(), called when a table rewrite begins.
It records the rewrite operation in a local list and detects rewrite chains by
checking if the old_locator matches any existing new_locator, preserving the
chain's original_locator.
- Modifies pgstat_copy_relation_stats(), to accept RelFileLocators instead of
Relations, with a new increment parameter to accumulate stats (needed for rewrite
chains with DML between rewrites).
- Ensures that AtEOXact_PgStat_Relations(), AtPrepare_PgStat_Relations(),
pgstat_twophase_postcommit()/postabort() pgstat_drop_relation() handle the
PgStat_PendingRewrite list correctly.
Note that due to the new flush call in pgstat_twophase_postcommit() we can not
call GetCurrentTransactionStopTimestamp() in pgstat_relation_flush_cb(). So,
adding a check to handle this special case and call GetCurrentTimestamp() instead.
Note that we'd call GetCurrentTimestamp() only if there is a rewrite, so that
the GetCurrentTimestamp() extra cost should be negligible. Another solution
could be to trigger the flush from FinishPreparedTransaction() but that's not
worth the extra complexity.
The new pending_rewrites list is traversed in multiple places. The overhead
should be negligible in comparison to a rewrite and the list should not contain
a lot of rewrites in practice.
The pending_rewrites list is traversed in multiple places. In typical usage,
the list will contain only a few entries so the traversal cost is negligible (
furthermore in comparison to a rewrite).
---
src/backend/catalog/index.c | 2 +-
src/backend/commands/cluster.c | 5 +
src/backend/commands/tablecmds.c | 6 +
src/backend/utils/activity/pgstat_relation.c | 391 ++++++++++++++++++-
src/backend/utils/activity/pgstat_xact.c | 25 +-
src/backend/utils/cache/relcache.c | 6 +
src/include/pgstat.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 424 insertions(+), 17 deletions(-)
92.8% src/backend/utils/activity/
4.9% src/backend/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 8dea58ad96b..b71925a22c3 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1795,7 +1795,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId);
/* copy over statistics from old to new index */
- pgstat_copy_relation_stats(newClassRel, oldClassRel);
+ pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 2120c85ccb4..6155b12afab 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1196,6 +1196,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
rel1 = relation_open(r1, NoLock);
rel2 = relation_open(r2, NoLock);
+
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind))
+ pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator);
+
rel2->rd_createSubid = rel1->rd_createSubid;
rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid;
rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 6b1a00ed477..9de70f321ed 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -16884,6 +16884,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
Oid reltoastrelid;
RelFileNumber newrelfilenumber;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator;
List *reltoastidxids = NIL;
ListCell *lc;
@@ -16922,6 +16923,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
newrlocator = rel->rd_locator;
newrlocator.relNumber = newrelfilenumber;
newrlocator.spcOid = newTableSpace;
+ oldrlocator = rel->rd_locator;
/* hand off to AM to actually create new rel storage and copy the data */
if (rel->rd_rel->relkind == RELKIND_INDEX)
@@ -16934,6 +16936,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
table_relation_copy_data(rel, &newrlocator);
}
+ /* mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Update the pg_class row.
*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index a267e93f8be..123fb50d98f 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -30,6 +30,19 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
+/* Pending rewrite operations for stats copying */
+typedef struct PgStat_PendingRewrite
+{
+ RelFileLocator old_locator;
+ RelFileLocator new_locator;
+ RelFileLocator original_locator;
+ int nest_level; /* Transaction nesting level where rewrite
+ * occurred */
+ struct PgStat_PendingRewrite *next;
+} PgStat_PendingRewrite;
+
+/* The pending rewrites list for current transaction */
+static PgStat_PendingRewrite *pending_rewrites = NULL;
/* Record that's written to 2PC state file when pgstat state is persisted */
typedef struct TwoPhasePgStatRecord
@@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter deleted_pre_truncdrop;
RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
+ RelFileLocator rewrite_old_locator;
+ int rewrite_nest_level;
} TwoPhasePgStatRecord;
@@ -54,27 +69,70 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
/*
- * Copy stats between relations. This is used for things like REINDEX
+ * Copy stats between RelFileLocator. This is used for things like REINDEX
* CONCURRENTLY.
*/
void
-pgstat_copy_relation_stats(Relation dst, Relation src)
+pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment)
{
PgStat_StatTabEntry *srcstats;
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src));
+ srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION,
+ src.dbOid,
+ RelFileLocatorToPgStatObjid(src));
if (!srcstats)
return;
dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
- dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(dst),
+ dst.dbOid,
+ RelFileLocatorToPgStatObjid(dst),
false);
dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats;
- dstshstats->stats = *srcstats;
+
+ if (!increment)
+ dstshstats->stats = *srcstats;
+ else
+ {
+ /* Increment those statistics */
+#define RELFSTAT_ACC(fld, stats_to_add) \
+ (dstshstats->stats.fld += stats_to_add->fld)
+ RELFSTAT_ACC(numscans, srcstats);
+ RELFSTAT_ACC(tuples_returned, srcstats);
+ RELFSTAT_ACC(tuples_fetched, srcstats);
+ RELFSTAT_ACC(tuples_inserted, srcstats);
+ RELFSTAT_ACC(tuples_updated, srcstats);
+ RELFSTAT_ACC(tuples_deleted, srcstats);
+ RELFSTAT_ACC(tuples_hot_updated, srcstats);
+ RELFSTAT_ACC(tuples_newpage_updated, srcstats);
+ RELFSTAT_ACC(live_tuples, srcstats);
+ RELFSTAT_ACC(dead_tuples, srcstats);
+ RELFSTAT_ACC(mod_since_analyze, srcstats);
+ RELFSTAT_ACC(ins_since_vacuum, srcstats);
+ RELFSTAT_ACC(blocks_fetched, srcstats);
+ RELFSTAT_ACC(blocks_hit, srcstats);
+ RELFSTAT_ACC(vacuum_count, srcstats);
+ RELFSTAT_ACC(autovacuum_count, srcstats);
+ RELFSTAT_ACC(analyze_count, srcstats);
+ RELFSTAT_ACC(autoanalyze_count, srcstats);
+ RELFSTAT_ACC(total_vacuum_time, srcstats);
+ RELFSTAT_ACC(total_autovacuum_time, srcstats);
+ RELFSTAT_ACC(total_analyze_time, srcstats);
+ RELFSTAT_ACC(total_autoanalyze_time, srcstats);
+#undef RELFSTAT_ACC
+
+ /* Replace those statistics */
+#define RELFSTAT_REP(fld, stats_to_rep) \
+ (dstshstats->stats.fld = stats_to_rep->fld)
+ RELFSTAT_REP(lastscan, srcstats);
+ RELFSTAT_REP(last_vacuum_time, srcstats);
+ RELFSTAT_REP(last_autovacuum_time, srcstats);
+ RELFSTAT_REP(last_analyze_time, srcstats);
+ RELFSTAT_REP(last_autoanalyze_time, srcstats);
+#undef RELFSTAT_REP
+ }
pgstat_unlock_entry(dst_ref);
}
@@ -136,6 +194,7 @@ void
pgstat_assoc_relation(Relation rel)
{
RelFileLocator locator;
+ PgStat_TableStatus *pgstat_info;
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
@@ -164,14 +223,54 @@ pgstat_assoc_relation(Relation rel)
locator.relNumber = rel->rd_id;
}
+ /*
+ * If this relation was rewritten during the current transaction we may be
+ * reopening it with its new RelFileLocator. In that case, continue using
+ * the stats entry associated with the old locator rather than creating a
+ * new one. This ensures all stats from before and after the rewrite are
+ * tracked in a single entry which will be properly copied to the new
+ * locator at transaction commit.
+ */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (locator.dbOid == rewrite->new_locator.dbOid &&
+ locator.spcOid == rewrite->new_locator.spcOid &&
+ locator.relNumber == rewrite->new_locator.relNumber)
+ {
+ pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator);
+ goto found_entry;
+ }
+ }
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(locator);
+ pgstat_info = pgstat_prep_relation_pending(locator);
+
+found_entry:
+ rel->pgstat_info = pgstat_info;
+
+ /*
+ * For relations stats, we key by physical file location, not by relation
+ * OID. This means during operations like ALTER TYPE it's possible that
+ * the relation OID changes but the relfilenode stays the same (no actual
+ * rewrite needed). Unlink the old relation first.
+ */
+ if (pgstat_info->relation != NULL &&
+ pgstat_info->relation != rel)
+ {
+ pgstat_info->relation->pgstat_info = NULL;
+ pgstat_info->relation = NULL;
+ }
/* don't allow link a stats to multiple relcache entries */
- Assert(rel->pgstat_info->relation == NULL);
+ Assert(pgstat_info->relation == NULL);
/* mark this relation as the owner */
- rel->pgstat_info->relation = rel;
+ pgstat_info->relation = rel;
}
/*
@@ -214,14 +313,37 @@ pgstat_drop_relation(Relation rel)
{
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ bool skip_transactional_drop = false;
/* don't track stats for relations without storage */
if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
return;
- pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_locator.dbOid,
- RelFileLocatorToPgStatObjid(rel->rd_locator));
+ /* Check if this drop is part of a pending rewrite */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid &&
+ rel->rd_locator.spcOid == rewrite->old_locator.spcOid &&
+ rel->rd_locator.relNumber == rewrite->old_locator.relNumber)
+ {
+ skip_transactional_drop = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * If it is part of a rewrite, drop its stats later, for example in
+ * AtEOXact_PgStat_Relations(), so skip it here.
+ */
+ if (!skip_transactional_drop)
+ pgstat_drop_transactional(PGSTAT_KIND_RELATION,
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -666,6 +788,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit)
}
tabstat->trans = NULL;
}
+
+ /* preserve the stats in case of rewrite */
+ if (isCommit && pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *prev = NULL;
+ PgStat_PendingRewrite *current = pending_rewrites;
+ PgStat_PendingRewrite *next;
+
+ /* reverse the rewrites list to process in chronological order */
+ while (current != NULL)
+ {
+ next = current->next;
+ current->next = prev;
+ prev = current;
+ current = next;
+ }
+
+ /* now process rewrites in chronological order */
+ for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ pgstat_copy_relation_stats(rewrite->new_locator,
+ rewrite->old_locator, true);
+
+ /* drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
+ }
+
+ pending_rewrites = NULL;
}
/*
@@ -681,6 +845,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in
PgStat_TableXactStatus *trans;
PgStat_TableXactStatus *next_trans;
+ /*
+ * If we don't commit then remove the associated rewrites if any, to keep
+ * the rewrite chain in sync with what will be eventually committed.
+ */
+ if (!isCommit)
+ {
+ PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites;
+
+ while (*rewrite_ptr != NULL)
+ {
+ if ((*rewrite_ptr)->nest_level >= nestDepth)
+ {
+ PgStat_PendingRewrite *to_remove = *rewrite_ptr;
+
+ *rewrite_ptr = (*rewrite_ptr)->next;
+ pfree(to_remove);
+ }
+ else
+ {
+ rewrite_ptr = &((*rewrite_ptr)->next);
+ }
+ }
+ }
+
for (trans = xact_state->first; trans != NULL; trans = next_trans)
{
PgStat_TableStatus *tabstat;
@@ -760,11 +948,19 @@ void
AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
{
PgStat_TableXactStatus *trans;
+ PgStat_PendingRewrite *rewrite;
+ /*
+ * For each tabstat, find its matching rewrite and remove it from the
+ * pending rewrites list. This way, after processing all tabstats, pending
+ * rewrites will only contain rewrite only transactions.
+ */
for (trans = xact_state->first; trans != NULL; trans = trans->next)
{
PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY;
TwoPhasePgStatRecord record;
+ PgStat_PendingRewrite **rewrite_ptr;
+ bool found_rewrite = false;
Assert(trans->nest_level == 1);
Assert(trans->upper == NULL);
@@ -784,10 +980,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.locator = tabstat->locator;
record.truncdropped = trans->truncdropped;
+ record.rewrite_nest_level = 0;
+
+ /*
+ * Look for a matching rewrite and remove it from pending rewrites. We
+ * check three possible matches:
+ *
+ * The new_locator when stats have been added after the rewrite. The
+ * old_locator when stats have been added before the rewrite but not
+ * after. The original_locator when this tabstat is part of a rewrite
+ * chain.
+ */
+ rewrite_ptr = &pending_rewrites;
+ while (*rewrite_ptr != NULL)
+ {
+ rewrite = *rewrite_ptr;
+
+ if ((record.locator.dbOid == rewrite->new_locator.dbOid &&
+ record.locator.spcOid == rewrite->new_locator.spcOid &&
+ record.locator.relNumber == rewrite->new_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->old_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->old_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->old_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->original_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->original_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->original_locator.relNumber))
+ {
+ /*
+ * Found matching rewrite. Record the rewrite information and
+ * remove this rewrite from the list since it's now handled.
+ */
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.locator = rewrite->new_locator;
+ found_rewrite = true;
+
+ /* Remove from pending_rewrites list */
+ *rewrite_ptr = rewrite->next;
+ pfree(rewrite);
+ break;
+ }
+ else
+ {
+ /* Move to next rewrite in the list */
+ rewrite_ptr = &(rewrite->next);
+ }
+ }
+
+ /* If no rewrite found, clear the rewrite fields */
+ if (!found_rewrite)
+ {
+ memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator));
+ }
+
+ RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
+ &record, sizeof(TwoPhasePgStatRecord));
+ }
+
+ /*
+ * Now process any rewrites still pending. These are rewrite only
+ * transactions. We need to preserve their stats even though there's no
+ * tabstat entry for them.
+ */
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ TwoPhasePgStatRecord record;
+
+ memset(&record, 0, sizeof(TwoPhasePgStatRecord));
+ record.locator = rewrite->new_locator;
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.truncdropped = false;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
&record, sizeof(TwoPhasePgStatRecord));
}
+
+ pending_rewrites = NULL;
}
/*
@@ -810,6 +1079,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
tabstat = trans->parent;
tabstat->trans = NULL;
}
+
+ pending_rewrites = NULL;
}
/*
@@ -845,6 +1116,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
pgstat_info->counts.changed_tuples +=
rec->tuples_inserted + rec->tuples_updated +
rec->tuples_deleted;
+
+ if (rec->rewrite_nest_level > 0)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ /* Flush any pending stats for old locator first */
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ /* Copy stats from old to new locator */
+ pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator,
+ true);
+
+ /* Drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
}
/*
@@ -859,9 +1153,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
{
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
+ RelFileLocator target_locator;
+
+ /*
+ * For aborted transactions with rewrites (like TRUNCATE), we need to
+ * restore stats to the old locator, not the new one. The new locator
+ * should be dropped since the rewrite is being rolled back.
+ */
+ if (rec->rewrite_nest_level > 0)
+ {
+ /* Use the old locator */
+ target_locator = rec->rewrite_old_locator;
+ }
+ else
+ {
+ /* No rewrite, use the original locator */
+ target_locator = rec->locator;
+ }
/* Find or create a tabstat entry for the target locator */
- pgstat_info = pgstat_prep_relation_pending(rec->locator);
+ pgstat_info = pgstat_prep_relation_pending(target_locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -916,7 +1227,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
+ TimestampTz t;
+
+ /*
+ * Checking the transaction state due to the flush call in
+ * pgstat_twophase_postcommit() that would break the assertion on the
+ * state in GetCurrentTransactionStopTimestamp().
+ */
+ if (!IsTransactionState())
+ t = GetCurrentTransactionStopTimestamp();
+ else
+ t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
@@ -1167,3 +1488,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
ReleaseSysCache(tuple);
return result;
}
+
+/*
+ * Mark that a relation rewrite has occurred, preserving the original locator
+ * so stats can be copied at transaction commit.
+ */
+void
+pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator)
+{
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *existing;
+ RelFileLocator original_locator = old_locator;
+
+ for (existing = pending_rewrites; existing != NULL; existing = existing->next)
+ {
+ if (old_locator.dbOid == existing->new_locator.dbOid &&
+ old_locator.spcOid == existing->new_locator.spcOid &&
+ old_locator.relNumber == existing->new_locator.relNumber)
+ {
+ original_locator = existing->original_locator;
+ break;
+ }
+ }
+
+ /* Allocate in TopTransactionContext memory context */
+ rewrite = MemoryContextAlloc(TopTransactionContext,
+ sizeof(PgStat_PendingRewrite));
+
+ rewrite->old_locator = old_locator;
+ rewrite->new_locator = new_locator;
+ rewrite->original_locator = original_locator;
+ rewrite->nest_level = GetCurrentTransactionNestLevel();
+
+ /* Add to the list */
+ rewrite->next = pending_rewrites;
+ pending_rewrites = rewrite;
+}
+
+void
+pgstat_clear_rewrite(void)
+{
+ pending_rewrites = NULL;
+}
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index bc9864bd8d9..f8cf3755ce2 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel)
}
pgStatXactStack = NULL;
+ pgstat_clear_rewrite();
+
/* Make sure any stats snapshot is thrown away */
pgstat_clear_snapshot();
}
@@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo
void
pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid)
{
- if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL))
+ PgStat_EntryRef *entry_ref;
+
+ entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL);
+
+ if (entry_ref)
{
+ /*
+ * For relations stats, we key by physical file location, not by
+ * relation OID. This means during operations like ALTER TYPE where
+ * the relation OID changes but the relfilenode stays the same (no
+ * actual rewrite needed), we'll find an existing entry.
+ *
+ * This is expected behavior, we want to preserve stats across the
+ * catalog change. Simply reset and recreate the entry for the new
+ * relation OID without warning.
+ */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ pgstat_reset(kind, dboid, objid);
+ create_drop_transactional_internal(kind, dboid, objid, true);
+ return;
+ }
+
ereport(WARNING,
errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64,
(pgstat_get_kind_info(kind))->name, dboid,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 2d0cb7bcfd4..c98e5c51d63 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -85,6 +85,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/relmapper.h"
#include "utils/resowner.h"
#include "utils/snapmgr.h"
@@ -3780,6 +3781,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
MultiXactId minmulti = InvalidMultiXactId;
TransactionId freezeXid = InvalidTransactionId;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator = relation->rd_locator;
if (!IsBinaryUpgrade)
{
@@ -3951,6 +3953,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
table_close(pg_class, RowExclusiveLock);
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Make the pg_class row change or relation map change visible. This will
* cause the relcache entry to get updated, too.
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 3102f86aa24..8750c025bbe 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -670,7 +670,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id);
extern void pgstat_create_relation(Relation rel);
extern void pgstat_drop_relation(Relation rel);
-extern void pgstat_copy_relation_stats(Relation dst, Relation src);
+extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment);
extern void pgstat_init_relation(Relation rel);
extern void pgstat_assoc_relation(Relation rel);
@@ -682,6 +682,9 @@ extern void pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_mark_rewrite(RelFileLocator old_locator,
+ RelFileLocator new_locator);
+extern void pgstat_clear_rewrite(void);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 04845d5e680..3184e9c464b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2269,6 +2269,7 @@ PgStat_KindInfo
PgStat_LocalState
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
+PgStat_PendingRewrite
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
--xJsrX25x3tm7Tn81--
^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v7 3/3] handle relation statistics correctly during rewrites
@ 2025-11-04 13:52 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Bertrand Drouvot @ 2025-11-04 13:52 UTC (permalink / raw)
Now that PGSTAT_KIND_RELATION is keyed by refilenode, we need to handle rewrites.
To do so, this patch:
- Adds PgStat_PendingRewrite, a new struct to track rewrite operations within
a transaction, storing the old locator, new locator, and original locator (for
rewrite chains). This allows stats to be copied from the original location to
the final location at commit time.
- Adds a new function, pgstat_mark_rewrite(), called when a table rewrite begins.
It records the rewrite operation in a local list and detects rewrite chains by
checking if the old_locator matches any existing new_locator, preserving the
chain's original_locator.
- Modifies pgstat_copy_relation_stats(), to accept RelFileLocators instead of
Relations, with a new increment parameter to accumulate stats (needed for rewrite
chains with DML between rewrites).
- Ensures that AtEOXact_PgStat_Relations(), AtPrepare_PgStat_Relations(),
pgstat_twophase_postcommit()/postabort() pgstat_drop_relation() handle the
PgStat_PendingRewrite list correctly.
Note that due to the new flush call in pgstat_twophase_postcommit() we can not
call GetCurrentTransactionStopTimestamp() in pgstat_relation_flush_cb(). So,
adding a check to handle this special case and call GetCurrentTimestamp() instead.
Note that we'd call GetCurrentTimestamp() only if there is a rewrite, so that
the GetCurrentTimestamp() extra cost should be negligible. Another solution
could be to trigger the flush from FinishPreparedTransaction() but that's not
worth the extra complexity.
The new pending_rewrites list is traversed in multiple places. The overhead
should be negligible in comparison to a rewrite and the list should not contain
a lot of rewrites in practice.
The pending_rewrites list is traversed in multiple places. In typical usage,
the list will contain only a few entries so the traversal cost is negligible (
furthermore in comparison to a rewrite).
---
src/backend/catalog/index.c | 2 +-
src/backend/commands/cluster.c | 5 +
src/backend/commands/tablecmds.c | 6 +
src/backend/utils/activity/pgstat_relation.c | 391 ++++++++++++++++++-
src/backend/utils/activity/pgstat_xact.c | 25 +-
src/backend/utils/cache/relcache.c | 6 +
src/include/pgstat.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 424 insertions(+), 17 deletions(-)
92.8% src/backend/utils/activity/
4.9% src/backend/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 5d9db167e59..8b6a7652fcf 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1795,7 +1795,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId);
/* copy over statistics from old to new index */
- pgstat_copy_relation_stats(newClassRel, oldClassRel);
+ pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index b55221d44cd..da75dfa6ab8 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1196,6 +1196,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
rel1 = relation_open(r1, NoLock);
rel2 = relation_open(r2, NoLock);
+
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind))
+ pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator);
+
rel2->rd_createSubid = rel1->rd_createSubid;
rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid;
rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 3aac459e483..540923452fb 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -16848,6 +16848,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
Oid reltoastrelid;
RelFileNumber newrelfilenumber;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator;
List *reltoastidxids = NIL;
ListCell *lc;
@@ -16886,6 +16887,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
newrlocator = rel->rd_locator;
newrlocator.relNumber = newrelfilenumber;
newrlocator.spcOid = newTableSpace;
+ oldrlocator = rel->rd_locator;
/* hand off to AM to actually create new rel storage and copy the data */
if (rel->rd_rel->relkind == RELKIND_INDEX)
@@ -16898,6 +16900,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
table_relation_copy_data(rel, &newrlocator);
}
+ /* mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Update the pg_class row.
*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 7debb14bb5d..15b4663eb77 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -30,6 +30,19 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
+/* Pending rewrite operations for stats copying */
+typedef struct PgStat_PendingRewrite
+{
+ RelFileLocator old_locator;
+ RelFileLocator new_locator;
+ RelFileLocator original_locator;
+ int nest_level; /* Transaction nesting level where rewrite
+ * occurred */
+ struct PgStat_PendingRewrite *next;
+} PgStat_PendingRewrite;
+
+/* The pending rewrites list for current transaction */
+static PgStat_PendingRewrite *pending_rewrites = NULL;
/* Record that's written to 2PC state file when pgstat state is persisted */
typedef struct TwoPhasePgStatRecord
@@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter deleted_pre_truncdrop;
RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
+ RelFileLocator rewrite_old_locator;
+ int rewrite_nest_level;
} TwoPhasePgStatRecord;
@@ -54,27 +69,70 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
/*
- * Copy stats between relations. This is used for things like REINDEX
+ * Copy stats between RelFileLocator. This is used for things like REINDEX
* CONCURRENTLY.
*/
void
-pgstat_copy_relation_stats(Relation dst, Relation src)
+pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment)
{
PgStat_StatTabEntry *srcstats;
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src));
+ srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION,
+ src.dbOid,
+ RelFileLocatorToPgStatObjid(src));
if (!srcstats)
return;
dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
- dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(dst),
+ dst.dbOid,
+ RelFileLocatorToPgStatObjid(dst),
false);
dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats;
- dstshstats->stats = *srcstats;
+
+ if (!increment)
+ dstshstats->stats = *srcstats;
+ else
+ {
+ /* Increment those statistics */
+#define RELFSTAT_ACC(fld, stats_to_add) \
+ (dstshstats->stats.fld += stats_to_add->fld)
+ RELFSTAT_ACC(numscans, srcstats);
+ RELFSTAT_ACC(tuples_returned, srcstats);
+ RELFSTAT_ACC(tuples_fetched, srcstats);
+ RELFSTAT_ACC(tuples_inserted, srcstats);
+ RELFSTAT_ACC(tuples_updated, srcstats);
+ RELFSTAT_ACC(tuples_deleted, srcstats);
+ RELFSTAT_ACC(tuples_hot_updated, srcstats);
+ RELFSTAT_ACC(tuples_newpage_updated, srcstats);
+ RELFSTAT_ACC(live_tuples, srcstats);
+ RELFSTAT_ACC(dead_tuples, srcstats);
+ RELFSTAT_ACC(mod_since_analyze, srcstats);
+ RELFSTAT_ACC(ins_since_vacuum, srcstats);
+ RELFSTAT_ACC(blocks_fetched, srcstats);
+ RELFSTAT_ACC(blocks_hit, srcstats);
+ RELFSTAT_ACC(vacuum_count, srcstats);
+ RELFSTAT_ACC(autovacuum_count, srcstats);
+ RELFSTAT_ACC(analyze_count, srcstats);
+ RELFSTAT_ACC(autoanalyze_count, srcstats);
+ RELFSTAT_ACC(total_vacuum_time, srcstats);
+ RELFSTAT_ACC(total_autovacuum_time, srcstats);
+ RELFSTAT_ACC(total_analyze_time, srcstats);
+ RELFSTAT_ACC(total_autoanalyze_time, srcstats);
+#undef RELFSTAT_ACC
+
+ /* Replace those statistics */
+#define RELFSTAT_REP(fld, stats_to_rep) \
+ (dstshstats->stats.fld = stats_to_rep->fld)
+ RELFSTAT_REP(lastscan, srcstats);
+ RELFSTAT_REP(last_vacuum_time, srcstats);
+ RELFSTAT_REP(last_autovacuum_time, srcstats);
+ RELFSTAT_REP(last_analyze_time, srcstats);
+ RELFSTAT_REP(last_autoanalyze_time, srcstats);
+#undef RELFSTAT_REP
+ }
pgstat_unlock_entry(dst_ref);
}
@@ -136,6 +194,7 @@ void
pgstat_assoc_relation(Relation rel)
{
RelFileLocator locator;
+ PgStat_TableStatus *pgstat_info;
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
@@ -165,14 +224,54 @@ pgstat_assoc_relation(Relation rel)
locator.relNumber = rel->rd_id;
}
+ /*
+ * If this relation was rewritten during the current transaction we may be
+ * reopening it with its new RelFileLocator. In that case, continue using
+ * the stats entry associated with the old locator rather than creating a
+ * new one. This ensures all stats from before and after the rewrite are
+ * tracked in a single entry which will be properly copied to the new
+ * locator at transaction commit.
+ */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (locator.dbOid == rewrite->new_locator.dbOid &&
+ locator.spcOid == rewrite->new_locator.spcOid &&
+ locator.relNumber == rewrite->new_locator.relNumber)
+ {
+ pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator);
+ goto found_entry;
+ }
+ }
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(locator);
+ pgstat_info = pgstat_prep_relation_pending(locator);
+
+found_entry:
+ rel->pgstat_info = pgstat_info;
+
+ /*
+ * For relations stats, we key by physical file location, not by relation
+ * OID. This means during operations like ALTER TYPE it's possible that
+ * the relation OID changes but the relfilenode stays the same (no actual
+ * rewrite needed). Unlink the old relation first.
+ */
+ if (pgstat_info->relation != NULL &&
+ pgstat_info->relation != rel)
+ {
+ pgstat_info->relation->pgstat_info = NULL;
+ pgstat_info->relation = NULL;
+ }
/* don't allow link a stats to multiple relcache entries */
- Assert(rel->pgstat_info->relation == NULL);
+ Assert(pgstat_info->relation == NULL);
/* mark this relation as the owner */
- rel->pgstat_info->relation = rel;
+ pgstat_info->relation = rel;
}
/*
@@ -215,14 +314,37 @@ pgstat_drop_relation(Relation rel)
{
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ bool skip_transactional_drop = false;
/* don't track stats for relations without storage */
if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
return;
- pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_locator.dbOid,
- RelFileLocatorToPgStatObjid(rel->rd_locator));
+ /* Check if this drop is part of a pending rewrite */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid &&
+ rel->rd_locator.spcOid == rewrite->old_locator.spcOid &&
+ rel->rd_locator.relNumber == rewrite->old_locator.relNumber)
+ {
+ skip_transactional_drop = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * If it is part of a rewrite, drop its stats later, for example in
+ * AtEOXact_PgStat_Relations(), so skip it here.
+ */
+ if (!skip_transactional_drop)
+ pgstat_drop_transactional(PGSTAT_KIND_RELATION,
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -660,6 +782,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit)
}
tabstat->trans = NULL;
}
+
+ /* preserve the stats in case of rewrite */
+ if (isCommit && pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *prev = NULL;
+ PgStat_PendingRewrite *current = pending_rewrites;
+ PgStat_PendingRewrite *next;
+
+ /* reverse the rewrites list to process in chronological order */
+ while (current != NULL)
+ {
+ next = current->next;
+ current->next = prev;
+ prev = current;
+ current = next;
+ }
+
+ /* now process rewrites in chronological order */
+ for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ pgstat_copy_relation_stats(rewrite->new_locator,
+ rewrite->old_locator, true);
+
+ /* drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
+ }
+
+ pending_rewrites = NULL;
}
/*
@@ -675,6 +839,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in
PgStat_TableXactStatus *trans;
PgStat_TableXactStatus *next_trans;
+ /*
+ * If we don't commit then remove the associated rewrites if any, to keep
+ * the rewrite chain in sync with what will be eventually committed.
+ */
+ if (!isCommit)
+ {
+ PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites;
+
+ while (*rewrite_ptr != NULL)
+ {
+ if ((*rewrite_ptr)->nest_level >= nestDepth)
+ {
+ PgStat_PendingRewrite *to_remove = *rewrite_ptr;
+
+ *rewrite_ptr = (*rewrite_ptr)->next;
+ pfree(to_remove);
+ }
+ else
+ {
+ rewrite_ptr = &((*rewrite_ptr)->next);
+ }
+ }
+ }
+
for (trans = xact_state->first; trans != NULL; trans = next_trans)
{
PgStat_TableStatus *tabstat;
@@ -754,11 +942,19 @@ void
AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
{
PgStat_TableXactStatus *trans;
+ PgStat_PendingRewrite *rewrite;
+ /*
+ * For each tabstat, find its matching rewrite and remove it from the
+ * pending rewrites list. This way, after processing all tabstats, pending
+ * rewrites will only contain rewrite only transactions.
+ */
for (trans = xact_state->first; trans != NULL; trans = trans->next)
{
PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY;
TwoPhasePgStatRecord record;
+ PgStat_PendingRewrite **rewrite_ptr;
+ bool found_rewrite = false;
Assert(trans->nest_level == 1);
Assert(trans->upper == NULL);
@@ -778,10 +974,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.locator = tabstat->locator;
record.truncdropped = trans->truncdropped;
+ record.rewrite_nest_level = 0;
+
+ /*
+ * Look for a matching rewrite and remove it from pending rewrites. We
+ * check three possible matches:
+ *
+ * The new_locator when stats have been added after the rewrite. The
+ * old_locator when stats have been added before the rewrite but not
+ * after. The original_locator when this tabstat is part of a rewrite
+ * chain.
+ */
+ rewrite_ptr = &pending_rewrites;
+ while (*rewrite_ptr != NULL)
+ {
+ rewrite = *rewrite_ptr;
+
+ if ((record.locator.dbOid == rewrite->new_locator.dbOid &&
+ record.locator.spcOid == rewrite->new_locator.spcOid &&
+ record.locator.relNumber == rewrite->new_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->old_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->old_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->old_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->original_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->original_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->original_locator.relNumber))
+ {
+ /*
+ * Found matching rewrite. Record the rewrite information and
+ * remove this rewrite from the list since it's now handled.
+ */
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.locator = rewrite->new_locator;
+ found_rewrite = true;
+
+ /* Remove from pending_rewrites list */
+ *rewrite_ptr = rewrite->next;
+ pfree(rewrite);
+ break;
+ }
+ else
+ {
+ /* Move to next rewrite in the list */
+ rewrite_ptr = &(rewrite->next);
+ }
+ }
+
+ /* If no rewrite found, clear the rewrite fields */
+ if (!found_rewrite)
+ {
+ memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator));
+ }
+
+ RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
+ &record, sizeof(TwoPhasePgStatRecord));
+ }
+
+ /*
+ * Now process any rewrites still pending. These are rewrite only
+ * transactions. We need to preserve their stats even though there's no
+ * tabstat entry for them.
+ */
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ TwoPhasePgStatRecord record;
+
+ memset(&record, 0, sizeof(TwoPhasePgStatRecord));
+ record.locator = rewrite->new_locator;
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.truncdropped = false;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
&record, sizeof(TwoPhasePgStatRecord));
}
+
+ pending_rewrites = NULL;
}
/*
@@ -804,6 +1073,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
tabstat = trans->parent;
tabstat->trans = NULL;
}
+
+ pending_rewrites = NULL;
}
/*
@@ -839,6 +1110,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
pgstat_info->counts.changed_tuples +=
rec->tuples_inserted + rec->tuples_updated +
rec->tuples_deleted;
+
+ if (rec->rewrite_nest_level > 0)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ /* Flush any pending stats for old locator first */
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ /* Copy stats from old to new locator */
+ pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator,
+ true);
+
+ /* Drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
}
/*
@@ -853,9 +1147,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
{
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
+ RelFileLocator target_locator;
+
+ /*
+ * For aborted transactions with rewrites (like TRUNCATE), we need to
+ * restore stats to the old locator, not the new one. The new locator
+ * should be dropped since the rewrite is being rolled back.
+ */
+ if (rec->rewrite_nest_level > 0)
+ {
+ /* Use the old locator */
+ target_locator = rec->rewrite_old_locator;
+ }
+ else
+ {
+ /* No rewrite, use the original locator */
+ target_locator = rec->locator;
+ }
/* Find or create a tabstat entry for the target locator */
- pgstat_info = pgstat_prep_relation_pending(rec->locator);
+ pgstat_info = pgstat_prep_relation_pending(target_locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -910,7 +1221,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
+ TimestampTz t;
+
+ /*
+ * Checking the transaction state due to the flush call in
+ * pgstat_twophase_postcommit() that would break the assertion on the
+ * state in GetCurrentTransactionStopTimestamp().
+ */
+ if (!IsTransactionState())
+ t = GetCurrentTransactionStopTimestamp();
+ else
+ t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
@@ -1162,3 +1483,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
ReleaseSysCache(tuple);
return result;
}
+
+/*
+ * Mark that a relation rewrite has occurred, preserving the original locator
+ * so stats can be copied at transaction commit.
+ */
+void
+pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator)
+{
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *existing;
+ RelFileLocator original_locator = old_locator;
+
+ for (existing = pending_rewrites; existing != NULL; existing = existing->next)
+ {
+ if (old_locator.dbOid == existing->new_locator.dbOid &&
+ old_locator.spcOid == existing->new_locator.spcOid &&
+ old_locator.relNumber == existing->new_locator.relNumber)
+ {
+ original_locator = existing->original_locator;
+ break;
+ }
+ }
+
+ /* Allocate in TopTransactionContext memory context */
+ rewrite = MemoryContextAlloc(TopTransactionContext,
+ sizeof(PgStat_PendingRewrite));
+
+ rewrite->old_locator = old_locator;
+ rewrite->new_locator = new_locator;
+ rewrite->original_locator = original_locator;
+ rewrite->nest_level = GetCurrentTransactionNestLevel();
+
+ /* Add to the list */
+ rewrite->next = pending_rewrites;
+ pending_rewrites = rewrite;
+}
+
+void
+pgstat_clear_rewrite(void)
+{
+ pending_rewrites = NULL;
+}
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index bc9864bd8d9..f8cf3755ce2 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel)
}
pgStatXactStack = NULL;
+ pgstat_clear_rewrite();
+
/* Make sure any stats snapshot is thrown away */
pgstat_clear_snapshot();
}
@@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo
void
pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid)
{
- if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL))
+ PgStat_EntryRef *entry_ref;
+
+ entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL);
+
+ if (entry_ref)
{
+ /*
+ * For relations stats, we key by physical file location, not by
+ * relation OID. This means during operations like ALTER TYPE where
+ * the relation OID changes but the relfilenode stays the same (no
+ * actual rewrite needed), we'll find an existing entry.
+ *
+ * This is expected behavior, we want to preserve stats across the
+ * catalog change. Simply reset and recreate the entry for the new
+ * relation OID without warning.
+ */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ pgstat_reset(kind, dboid, objid);
+ create_drop_transactional_internal(kind, dboid, objid, true);
+ return;
+ }
+
ereport(WARNING,
errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64,
(pgstat_get_kind_info(kind))->name, dboid,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 915d0bc9084..7a7f8023eb3 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -85,6 +85,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/relmapper.h"
#include "utils/resowner.h"
#include "utils/snapmgr.h"
@@ -3780,6 +3781,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
MultiXactId minmulti = InvalidMultiXactId;
TransactionId freezeXid = InvalidTransactionId;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator = relation->rd_locator;
if (!IsBinaryUpgrade)
{
@@ -3951,6 +3953,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
table_close(pg_class, RowExclusiveLock);
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Make the pg_class row change or relation map change visible. This will
* cause the relcache entry to get updated, too.
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 5d0fe79f7e3..332dffde400 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -665,7 +665,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id);
extern void pgstat_create_relation(Relation rel);
extern void pgstat_drop_relation(Relation rel);
-extern void pgstat_copy_relation_stats(Relation dst, Relation src);
+extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment);
extern void pgstat_init_relation(Relation rel);
extern void pgstat_assoc_relation(Relation rel);
@@ -677,6 +677,9 @@ extern void pgstat_report_vacuum(RelFileLocator locator, PgStat_Counter livetupl
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_mark_rewrite(RelFileLocator old_locator,
+ RelFileLocator new_locator);
+extern void pgstat_clear_rewrite(void);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 432509277c9..bd8fcb16dcf 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2248,6 +2248,7 @@ PgStat_KindInfo
PgStat_LocalState
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
+PgStat_PendingRewrite
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
--KMYjwlsACqdYx2P8--
^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v15 2/2] handle relation statistics correctly during rewrites
@ 2025-11-04 13:52 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Bertrand Drouvot @ 2025-11-04 13:52 UTC (permalink / raw)
Now that PGSTAT_KIND_RELATION is keyed by refilenode, we need to handle rewrites.
To do so, this patch:
- Adds PgStat_PendingRewrite, a new struct to track rewrite operations within
a transaction, storing the old locator, new locator, and original locator (for
rewrite chains). This allows stats to be copied from the original location to
the final location at commit time.
- Adds a new function, pgstat_mark_rewrite(), called when a table rewrite begins.
It records the rewrite operation in a local list and detects rewrite chains by
checking if the old_locator matches any existing new_locator, preserving the
chain's original_locator.
- Modifies pgstat_copy_relation_stats(), to accept RelFileLocators instead of
Relations, with a new increment parameter to accumulate stats (needed for rewrite
chains with DML between rewrites).
- Ensures that AtEOXact_PgStat_Relations(), AtPrepare_PgStat_Relations(),
pgstat_twophase_postcommit()/postabort() pgstat_drop_relation() handle the
PgStat_PendingRewrite list correctly.
Note that due to the new flush call in pgstat_twophase_postcommit() we can not
call GetCurrentTransactionStopTimestamp() in pgstat_relation_flush_cb(). So,
adding a check to handle this special case and call GetCurrentTimestamp() instead.
Note that we'd call GetCurrentTimestamp() only if there is a rewrite, so that
the GetCurrentTimestamp() extra cost should be negligible. Another solution
could be to trigger the flush from FinishPreparedTransaction() but that's not
worth the extra complexity.
The new pending_rewrites list is traversed in multiple places. The overhead
should be negligible in comparison to a rewrite and the list should not contain
a lot of rewrites in practice.
The pending_rewrites list is traversed in multiple places. In typical usage,
the list will contain only a few entries so the traversal cost is negligible (
furthermore in comparison to a rewrite).
---
src/backend/catalog/index.c | 2 +-
src/backend/commands/cluster.c | 5 +
src/backend/commands/tablecmds.c | 6 +
src/backend/utils/activity/pgstat_relation.c | 391 ++++++++++++++++++-
src/backend/utils/activity/pgstat_xact.c | 25 +-
src/backend/utils/cache/relcache.c | 6 +
src/include/pgstat.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 424 insertions(+), 17 deletions(-)
92.8% src/backend/utils/activity/
4.9% src/backend/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 1ccfa687f05..e292199ca7c 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1796,7 +1796,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId);
/* copy over statistics from old to new index */
- pgstat_copy_relation_stats(newClassRel, oldClassRel);
+ pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 09066db0956..c491daceb0f 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1206,6 +1206,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
rel1 = relation_open(r1, NoLock);
rel2 = relation_open(r2, NoLock);
+
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind))
+ pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator);
+
rel2->rd_createSubid = rel1->rd_createSubid;
rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid;
rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 8b4ebc6f226..49de04d9697 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -17102,6 +17102,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
Oid reltoastrelid;
RelFileNumber newrelfilenumber;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator;
List *reltoastidxids = NIL;
ListCell *lc;
@@ -17140,6 +17141,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
newrlocator = rel->rd_locator;
newrlocator.relNumber = newrelfilenumber;
newrlocator.spcOid = newTableSpace;
+ oldrlocator = rel->rd_locator;
/* hand off to AM to actually create new rel storage and copy the data */
if (rel->rd_rel->relkind == RELKIND_INDEX)
@@ -17152,6 +17154,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
table_relation_copy_data(rel, &newrlocator);
}
+ /* mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Update the pg_class row.
*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 89bf0cbed56..4d929972fab 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -30,6 +30,19 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
+/* Pending rewrite operations for stats copying */
+typedef struct PgStat_PendingRewrite
+{
+ RelFileLocator old_locator;
+ RelFileLocator new_locator;
+ RelFileLocator original_locator;
+ int nest_level; /* Transaction nesting level where rewrite
+ * occurred */
+ struct PgStat_PendingRewrite *next;
+} PgStat_PendingRewrite;
+
+/* The pending rewrites list for current transaction */
+static PgStat_PendingRewrite *pending_rewrites = NULL;
/* Record that's written to 2PC state file when pgstat state is persisted */
typedef struct TwoPhasePgStatRecord
@@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter deleted_pre_truncdrop;
RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
+ RelFileLocator rewrite_old_locator;
+ int rewrite_nest_level;
} TwoPhasePgStatRecord;
@@ -54,27 +69,70 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
/*
- * Copy stats between relations. This is used for things like REINDEX
+ * Copy stats between RelFileLocator. This is used for things like REINDEX
* CONCURRENTLY.
*/
void
-pgstat_copy_relation_stats(Relation dst, Relation src)
+pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment)
{
PgStat_StatTabEntry *srcstats;
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src));
+ srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION,
+ src.dbOid,
+ RelFileLocatorToPgStatObjid(src));
if (!srcstats)
return;
dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
- dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(dst),
+ dst.dbOid,
+ RelFileLocatorToPgStatObjid(dst),
false);
dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats;
- dstshstats->stats = *srcstats;
+
+ if (!increment)
+ dstshstats->stats = *srcstats;
+ else
+ {
+ /* Increment those statistics */
+#define RELFSTAT_ACC(fld, stats_to_add) \
+ (dstshstats->stats.fld += stats_to_add->fld)
+ RELFSTAT_ACC(numscans, srcstats);
+ RELFSTAT_ACC(tuples_returned, srcstats);
+ RELFSTAT_ACC(tuples_fetched, srcstats);
+ RELFSTAT_ACC(tuples_inserted, srcstats);
+ RELFSTAT_ACC(tuples_updated, srcstats);
+ RELFSTAT_ACC(tuples_deleted, srcstats);
+ RELFSTAT_ACC(tuples_hot_updated, srcstats);
+ RELFSTAT_ACC(tuples_newpage_updated, srcstats);
+ RELFSTAT_ACC(live_tuples, srcstats);
+ RELFSTAT_ACC(dead_tuples, srcstats);
+ RELFSTAT_ACC(mod_since_analyze, srcstats);
+ RELFSTAT_ACC(ins_since_vacuum, srcstats);
+ RELFSTAT_ACC(blocks_fetched, srcstats);
+ RELFSTAT_ACC(blocks_hit, srcstats);
+ RELFSTAT_ACC(vacuum_count, srcstats);
+ RELFSTAT_ACC(autovacuum_count, srcstats);
+ RELFSTAT_ACC(analyze_count, srcstats);
+ RELFSTAT_ACC(autoanalyze_count, srcstats);
+ RELFSTAT_ACC(total_vacuum_time, srcstats);
+ RELFSTAT_ACC(total_autovacuum_time, srcstats);
+ RELFSTAT_ACC(total_analyze_time, srcstats);
+ RELFSTAT_ACC(total_autoanalyze_time, srcstats);
+#undef RELFSTAT_ACC
+
+ /* Replace those statistics */
+#define RELFSTAT_REP(fld, stats_to_rep) \
+ (dstshstats->stats.fld = stats_to_rep->fld)
+ RELFSTAT_REP(lastscan, srcstats);
+ RELFSTAT_REP(last_vacuum_time, srcstats);
+ RELFSTAT_REP(last_autovacuum_time, srcstats);
+ RELFSTAT_REP(last_analyze_time, srcstats);
+ RELFSTAT_REP(last_autoanalyze_time, srcstats);
+#undef RELFSTAT_REP
+ }
pgstat_unlock_entry(dst_ref);
}
@@ -136,6 +194,7 @@ void
pgstat_assoc_relation(Relation rel)
{
RelFileLocator locator;
+ PgStat_TableStatus *pgstat_info;
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
@@ -164,14 +223,54 @@ pgstat_assoc_relation(Relation rel)
locator.relNumber = rel->rd_id;
}
+ /*
+ * If this relation was rewritten during the current transaction we may be
+ * reopening it with its new RelFileLocator. In that case, continue using
+ * the stats entry associated with the old locator rather than creating a
+ * new one. This ensures all stats from before and after the rewrite are
+ * tracked in a single entry which will be properly copied to the new
+ * locator at transaction commit.
+ */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (locator.dbOid == rewrite->new_locator.dbOid &&
+ locator.spcOid == rewrite->new_locator.spcOid &&
+ locator.relNumber == rewrite->new_locator.relNumber)
+ {
+ pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator);
+ goto found_entry;
+ }
+ }
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(locator);
+ pgstat_info = pgstat_prep_relation_pending(locator);
+
+found_entry:
+ rel->pgstat_info = pgstat_info;
+
+ /*
+ * For relations stats, we key by physical file location, not by relation
+ * OID. This means during operations like ALTER TYPE it's possible that
+ * the relation OID changes but the relfilenode stays the same (no actual
+ * rewrite needed). Unlink the old relation first.
+ */
+ if (pgstat_info->relation != NULL &&
+ pgstat_info->relation != rel)
+ {
+ pgstat_info->relation->pgstat_info = NULL;
+ pgstat_info->relation = NULL;
+ }
/* don't allow link a stats to multiple relcache entries */
- Assert(rel->pgstat_info->relation == NULL);
+ Assert(pgstat_info->relation == NULL);
/* mark this relation as the owner */
- rel->pgstat_info->relation = rel;
+ pgstat_info->relation = rel;
}
/*
@@ -214,14 +313,37 @@ pgstat_drop_relation(Relation rel)
{
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ bool skip_transactional_drop = false;
/* don't track stats for relations without storage */
if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
return;
- pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_locator.dbOid,
- RelFileLocatorToPgStatObjid(rel->rd_locator));
+ /* Check if this drop is part of a pending rewrite */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid &&
+ rel->rd_locator.spcOid == rewrite->old_locator.spcOid &&
+ rel->rd_locator.relNumber == rewrite->old_locator.relNumber)
+ {
+ skip_transactional_drop = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * If it is part of a rewrite, drop its stats later, for example in
+ * AtEOXact_PgStat_Relations(), so skip it here.
+ */
+ if (!skip_transactional_drop)
+ pgstat_drop_transactional(PGSTAT_KIND_RELATION,
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -666,6 +788,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit)
}
tabstat->trans = NULL;
}
+
+ /* preserve the stats in case of rewrite */
+ if (isCommit && pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *prev = NULL;
+ PgStat_PendingRewrite *current = pending_rewrites;
+ PgStat_PendingRewrite *next;
+
+ /* reverse the rewrites list to process in chronological order */
+ while (current != NULL)
+ {
+ next = current->next;
+ current->next = prev;
+ prev = current;
+ current = next;
+ }
+
+ /* now process rewrites in chronological order */
+ for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ pgstat_copy_relation_stats(rewrite->new_locator,
+ rewrite->old_locator, true);
+
+ /* drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
+ }
+
+ pending_rewrites = NULL;
}
/*
@@ -681,6 +845,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in
PgStat_TableXactStatus *trans;
PgStat_TableXactStatus *next_trans;
+ /*
+ * If we don't commit then remove the associated rewrites if any, to keep
+ * the rewrite chain in sync with what will be eventually committed.
+ */
+ if (!isCommit)
+ {
+ PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites;
+
+ while (*rewrite_ptr != NULL)
+ {
+ if ((*rewrite_ptr)->nest_level >= nestDepth)
+ {
+ PgStat_PendingRewrite *to_remove = *rewrite_ptr;
+
+ *rewrite_ptr = (*rewrite_ptr)->next;
+ pfree(to_remove);
+ }
+ else
+ {
+ rewrite_ptr = &((*rewrite_ptr)->next);
+ }
+ }
+ }
+
for (trans = xact_state->first; trans != NULL; trans = next_trans)
{
PgStat_TableStatus *tabstat;
@@ -760,11 +948,19 @@ void
AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
{
PgStat_TableXactStatus *trans;
+ PgStat_PendingRewrite *rewrite;
+ /*
+ * For each tabstat, find its matching rewrite and remove it from the
+ * pending rewrites list. This way, after processing all tabstats, pending
+ * rewrites will only contain rewrite only transactions.
+ */
for (trans = xact_state->first; trans != NULL; trans = trans->next)
{
PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY;
TwoPhasePgStatRecord record;
+ PgStat_PendingRewrite **rewrite_ptr;
+ bool found_rewrite = false;
Assert(trans->nest_level == 1);
Assert(trans->upper == NULL);
@@ -784,10 +980,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.locator = tabstat->locator;
record.truncdropped = trans->truncdropped;
+ record.rewrite_nest_level = 0;
+
+ /*
+ * Look for a matching rewrite and remove it from pending rewrites. We
+ * check three possible matches:
+ *
+ * The new_locator when stats have been added after the rewrite. The
+ * old_locator when stats have been added before the rewrite but not
+ * after. The original_locator when this tabstat is part of a rewrite
+ * chain.
+ */
+ rewrite_ptr = &pending_rewrites;
+ while (*rewrite_ptr != NULL)
+ {
+ rewrite = *rewrite_ptr;
+
+ if ((record.locator.dbOid == rewrite->new_locator.dbOid &&
+ record.locator.spcOid == rewrite->new_locator.spcOid &&
+ record.locator.relNumber == rewrite->new_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->old_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->old_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->old_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->original_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->original_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->original_locator.relNumber))
+ {
+ /*
+ * Found matching rewrite. Record the rewrite information and
+ * remove this rewrite from the list since it's now handled.
+ */
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.locator = rewrite->new_locator;
+ found_rewrite = true;
+
+ /* Remove from pending_rewrites list */
+ *rewrite_ptr = rewrite->next;
+ pfree(rewrite);
+ break;
+ }
+ else
+ {
+ /* Move to next rewrite in the list */
+ rewrite_ptr = &(rewrite->next);
+ }
+ }
+
+ /* If no rewrite found, clear the rewrite fields */
+ if (!found_rewrite)
+ {
+ memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator));
+ }
+
+ RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
+ &record, sizeof(TwoPhasePgStatRecord));
+ }
+
+ /*
+ * Now process any rewrites still pending. These are rewrite only
+ * transactions. We need to preserve their stats even though there's no
+ * tabstat entry for them.
+ */
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ TwoPhasePgStatRecord record;
+
+ memset(&record, 0, sizeof(TwoPhasePgStatRecord));
+ record.locator = rewrite->new_locator;
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.truncdropped = false;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
&record, sizeof(TwoPhasePgStatRecord));
}
+
+ pending_rewrites = NULL;
}
/*
@@ -810,6 +1079,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
tabstat = trans->parent;
tabstat->trans = NULL;
}
+
+ pending_rewrites = NULL;
}
/*
@@ -845,6 +1116,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
pgstat_info->counts.changed_tuples +=
rec->tuples_inserted + rec->tuples_updated +
rec->tuples_deleted;
+
+ if (rec->rewrite_nest_level > 0)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ /* Flush any pending stats for old locator first */
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ /* Copy stats from old to new locator */
+ pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator,
+ true);
+
+ /* Drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
}
/*
@@ -859,9 +1153,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
{
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
+ RelFileLocator target_locator;
+
+ /*
+ * For aborted transactions with rewrites (like TRUNCATE), we need to
+ * restore stats to the old locator, not the new one. The new locator
+ * should be dropped since the rewrite is being rolled back.
+ */
+ if (rec->rewrite_nest_level > 0)
+ {
+ /* Use the old locator */
+ target_locator = rec->rewrite_old_locator;
+ }
+ else
+ {
+ /* No rewrite, use the original locator */
+ target_locator = rec->locator;
+ }
/* Find or create a tabstat entry for the target locator */
- pgstat_info = pgstat_prep_relation_pending(rec->locator);
+ pgstat_info = pgstat_prep_relation_pending(target_locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -916,7 +1227,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
+ TimestampTz t;
+
+ /*
+ * Checking the transaction state due to the flush call in
+ * pgstat_twophase_postcommit() that would break the assertion on the
+ * state in GetCurrentTransactionStopTimestamp().
+ */
+ if (!IsTransactionState())
+ t = GetCurrentTransactionStopTimestamp();
+ else
+ t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
@@ -1167,3 +1488,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
ReleaseSysCache(tuple);
return result;
}
+
+/*
+ * Mark that a relation rewrite has occurred, preserving the original locator
+ * so stats can be copied at transaction commit.
+ */
+void
+pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator)
+{
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *existing;
+ RelFileLocator original_locator = old_locator;
+
+ for (existing = pending_rewrites; existing != NULL; existing = existing->next)
+ {
+ if (old_locator.dbOid == existing->new_locator.dbOid &&
+ old_locator.spcOid == existing->new_locator.spcOid &&
+ old_locator.relNumber == existing->new_locator.relNumber)
+ {
+ original_locator = existing->original_locator;
+ break;
+ }
+ }
+
+ /* Allocate in TopTransactionContext memory context */
+ rewrite = MemoryContextAlloc(TopTransactionContext,
+ sizeof(PgStat_PendingRewrite));
+
+ rewrite->old_locator = old_locator;
+ rewrite->new_locator = new_locator;
+ rewrite->original_locator = original_locator;
+ rewrite->nest_level = GetCurrentTransactionNestLevel();
+
+ /* Add to the list */
+ rewrite->next = pending_rewrites;
+ pending_rewrites = rewrite;
+}
+
+void
+pgstat_clear_rewrite(void)
+{
+ pending_rewrites = NULL;
+}
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index 5e2d69e6297..8ed8f5317f3 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel)
}
pgStatXactStack = NULL;
+ pgstat_clear_rewrite();
+
/* Make sure any stats snapshot is thrown away */
pgstat_clear_snapshot();
}
@@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo
void
pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid)
{
- if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL))
+ PgStat_EntryRef *entry_ref;
+
+ entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL);
+
+ if (entry_ref)
{
+ /*
+ * For relations stats, we key by physical file location, not by
+ * relation OID. This means during operations like ALTER TYPE where
+ * the relation OID changes but the relfilenode stays the same (no
+ * actual rewrite needed), we'll find an existing entry.
+ *
+ * This is expected behavior, we want to preserve stats across the
+ * catalog change. Simply reset and recreate the entry for the new
+ * relation OID without warning.
+ */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ pgstat_reset(kind, dboid, objid);
+ create_drop_transactional_internal(kind, dboid, objid, true);
+ return;
+ }
+
ereport(WARNING,
errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64,
(pgstat_get_kind_info(kind))->name, dboid,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index e19f0d3e51c..bf49830dba2 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -87,6 +87,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/relmapper.h"
#include "utils/resowner.h"
#include "utils/snapmgr.h"
@@ -3772,6 +3773,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
MultiXactId minmulti = InvalidMultiXactId;
TransactionId freezeXid = InvalidTransactionId;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator = relation->rd_locator;
if (!IsBinaryUpgrade)
{
@@ -3943,6 +3945,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
table_close(pg_class, RowExclusiveLock);
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Make the pg_class row change or relation map change visible. This will
* cause the relcache entry to get updated, too.
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index ef740aff26e..4d3130ae99e 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -701,7 +701,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id);
extern void pgstat_create_relation(Relation rel);
extern void pgstat_drop_relation(Relation rel);
-extern void pgstat_copy_relation_stats(Relation dst, Relation src);
+extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment);
extern void pgstat_init_relation(Relation rel);
extern void pgstat_assoc_relation(Relation rel);
@@ -713,6 +713,9 @@ extern void pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_mark_rewrite(RelFileLocator old_locator,
+ RelFileLocator new_locator);
+extern void pgstat_clear_rewrite(void);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index c9da1f91cb9..53c94382a8d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2314,6 +2314,7 @@ PgStat_LockEntry
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
PgStat_PendingLock
+PgStat_PendingRewrite
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
--Og8T0Lf2JVsuKOzn--
^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v13 2/2] handle relation statistics correctly during rewrites
@ 2025-11-04 13:52 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Bertrand Drouvot @ 2025-11-04 13:52 UTC (permalink / raw)
Now that PGSTAT_KIND_RELATION is keyed by refilenode, we need to handle rewrites.
To do so, this patch:
- Adds PgStat_PendingRewrite, a new struct to track rewrite operations within
a transaction, storing the old locator, new locator, and original locator (for
rewrite chains). This allows stats to be copied from the original location to
the final location at commit time.
- Adds a new function, pgstat_mark_rewrite(), called when a table rewrite begins.
It records the rewrite operation in a local list and detects rewrite chains by
checking if the old_locator matches any existing new_locator, preserving the
chain's original_locator.
- Modifies pgstat_copy_relation_stats(), to accept RelFileLocators instead of
Relations, with a new increment parameter to accumulate stats (needed for rewrite
chains with DML between rewrites).
- Ensures that AtEOXact_PgStat_Relations(), AtPrepare_PgStat_Relations(),
pgstat_twophase_postcommit()/postabort() pgstat_drop_relation() handle the
PgStat_PendingRewrite list correctly.
Note that due to the new flush call in pgstat_twophase_postcommit() we can not
call GetCurrentTransactionStopTimestamp() in pgstat_relation_flush_cb(). So,
adding a check to handle this special case and call GetCurrentTimestamp() instead.
Note that we'd call GetCurrentTimestamp() only if there is a rewrite, so that
the GetCurrentTimestamp() extra cost should be negligible. Another solution
could be to trigger the flush from FinishPreparedTransaction() but that's not
worth the extra complexity.
The new pending_rewrites list is traversed in multiple places. The overhead
should be negligible in comparison to a rewrite and the list should not contain
a lot of rewrites in practice.
The pending_rewrites list is traversed in multiple places. In typical usage,
the list will contain only a few entries so the traversal cost is negligible (
furthermore in comparison to a rewrite).
---
src/backend/catalog/index.c | 2 +-
src/backend/commands/cluster.c | 5 +
src/backend/commands/tablecmds.c | 6 +
src/backend/utils/activity/pgstat_relation.c | 391 ++++++++++++++++++-
src/backend/utils/activity/pgstat_xact.c | 25 +-
src/backend/utils/cache/relcache.c | 6 +
src/include/pgstat.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 424 insertions(+), 17 deletions(-)
92.8% src/backend/utils/activity/
4.9% src/backend/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 8b3c60d91f9..7c7d0fb8e7a 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1796,7 +1796,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId);
/* copy over statistics from old to new index */
- pgstat_copy_relation_stats(newClassRel, oldClassRel);
+ pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 09066db0956..c491daceb0f 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1206,6 +1206,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
rel1 = relation_open(r1, NoLock);
rel2 = relation_open(r2, NoLock);
+
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind))
+ pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator);
+
rel2->rd_createSubid = rel1->rd_createSubid;
rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid;
rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 67e42e5df29..1715af2c0d8 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -17099,6 +17099,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
Oid reltoastrelid;
RelFileNumber newrelfilenumber;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator;
List *reltoastidxids = NIL;
ListCell *lc;
@@ -17137,6 +17138,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
newrlocator = rel->rd_locator;
newrlocator.relNumber = newrelfilenumber;
newrlocator.spcOid = newTableSpace;
+ oldrlocator = rel->rd_locator;
/* hand off to AM to actually create new rel storage and copy the data */
if (rel->rd_rel->relkind == RELKIND_INDEX)
@@ -17149,6 +17151,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
table_relation_copy_data(rel, &newrlocator);
}
+ /* mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Update the pg_class row.
*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 89bf0cbed56..4d929972fab 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -30,6 +30,19 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
+/* Pending rewrite operations for stats copying */
+typedef struct PgStat_PendingRewrite
+{
+ RelFileLocator old_locator;
+ RelFileLocator new_locator;
+ RelFileLocator original_locator;
+ int nest_level; /* Transaction nesting level where rewrite
+ * occurred */
+ struct PgStat_PendingRewrite *next;
+} PgStat_PendingRewrite;
+
+/* The pending rewrites list for current transaction */
+static PgStat_PendingRewrite *pending_rewrites = NULL;
/* Record that's written to 2PC state file when pgstat state is persisted */
typedef struct TwoPhasePgStatRecord
@@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter deleted_pre_truncdrop;
RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
+ RelFileLocator rewrite_old_locator;
+ int rewrite_nest_level;
} TwoPhasePgStatRecord;
@@ -54,27 +69,70 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
/*
- * Copy stats between relations. This is used for things like REINDEX
+ * Copy stats between RelFileLocator. This is used for things like REINDEX
* CONCURRENTLY.
*/
void
-pgstat_copy_relation_stats(Relation dst, Relation src)
+pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment)
{
PgStat_StatTabEntry *srcstats;
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src));
+ srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION,
+ src.dbOid,
+ RelFileLocatorToPgStatObjid(src));
if (!srcstats)
return;
dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
- dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(dst),
+ dst.dbOid,
+ RelFileLocatorToPgStatObjid(dst),
false);
dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats;
- dstshstats->stats = *srcstats;
+
+ if (!increment)
+ dstshstats->stats = *srcstats;
+ else
+ {
+ /* Increment those statistics */
+#define RELFSTAT_ACC(fld, stats_to_add) \
+ (dstshstats->stats.fld += stats_to_add->fld)
+ RELFSTAT_ACC(numscans, srcstats);
+ RELFSTAT_ACC(tuples_returned, srcstats);
+ RELFSTAT_ACC(tuples_fetched, srcstats);
+ RELFSTAT_ACC(tuples_inserted, srcstats);
+ RELFSTAT_ACC(tuples_updated, srcstats);
+ RELFSTAT_ACC(tuples_deleted, srcstats);
+ RELFSTAT_ACC(tuples_hot_updated, srcstats);
+ RELFSTAT_ACC(tuples_newpage_updated, srcstats);
+ RELFSTAT_ACC(live_tuples, srcstats);
+ RELFSTAT_ACC(dead_tuples, srcstats);
+ RELFSTAT_ACC(mod_since_analyze, srcstats);
+ RELFSTAT_ACC(ins_since_vacuum, srcstats);
+ RELFSTAT_ACC(blocks_fetched, srcstats);
+ RELFSTAT_ACC(blocks_hit, srcstats);
+ RELFSTAT_ACC(vacuum_count, srcstats);
+ RELFSTAT_ACC(autovacuum_count, srcstats);
+ RELFSTAT_ACC(analyze_count, srcstats);
+ RELFSTAT_ACC(autoanalyze_count, srcstats);
+ RELFSTAT_ACC(total_vacuum_time, srcstats);
+ RELFSTAT_ACC(total_autovacuum_time, srcstats);
+ RELFSTAT_ACC(total_analyze_time, srcstats);
+ RELFSTAT_ACC(total_autoanalyze_time, srcstats);
+#undef RELFSTAT_ACC
+
+ /* Replace those statistics */
+#define RELFSTAT_REP(fld, stats_to_rep) \
+ (dstshstats->stats.fld = stats_to_rep->fld)
+ RELFSTAT_REP(lastscan, srcstats);
+ RELFSTAT_REP(last_vacuum_time, srcstats);
+ RELFSTAT_REP(last_autovacuum_time, srcstats);
+ RELFSTAT_REP(last_analyze_time, srcstats);
+ RELFSTAT_REP(last_autoanalyze_time, srcstats);
+#undef RELFSTAT_REP
+ }
pgstat_unlock_entry(dst_ref);
}
@@ -136,6 +194,7 @@ void
pgstat_assoc_relation(Relation rel)
{
RelFileLocator locator;
+ PgStat_TableStatus *pgstat_info;
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
@@ -164,14 +223,54 @@ pgstat_assoc_relation(Relation rel)
locator.relNumber = rel->rd_id;
}
+ /*
+ * If this relation was rewritten during the current transaction we may be
+ * reopening it with its new RelFileLocator. In that case, continue using
+ * the stats entry associated with the old locator rather than creating a
+ * new one. This ensures all stats from before and after the rewrite are
+ * tracked in a single entry which will be properly copied to the new
+ * locator at transaction commit.
+ */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (locator.dbOid == rewrite->new_locator.dbOid &&
+ locator.spcOid == rewrite->new_locator.spcOid &&
+ locator.relNumber == rewrite->new_locator.relNumber)
+ {
+ pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator);
+ goto found_entry;
+ }
+ }
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(locator);
+ pgstat_info = pgstat_prep_relation_pending(locator);
+
+found_entry:
+ rel->pgstat_info = pgstat_info;
+
+ /*
+ * For relations stats, we key by physical file location, not by relation
+ * OID. This means during operations like ALTER TYPE it's possible that
+ * the relation OID changes but the relfilenode stays the same (no actual
+ * rewrite needed). Unlink the old relation first.
+ */
+ if (pgstat_info->relation != NULL &&
+ pgstat_info->relation != rel)
+ {
+ pgstat_info->relation->pgstat_info = NULL;
+ pgstat_info->relation = NULL;
+ }
/* don't allow link a stats to multiple relcache entries */
- Assert(rel->pgstat_info->relation == NULL);
+ Assert(pgstat_info->relation == NULL);
/* mark this relation as the owner */
- rel->pgstat_info->relation = rel;
+ pgstat_info->relation = rel;
}
/*
@@ -214,14 +313,37 @@ pgstat_drop_relation(Relation rel)
{
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ bool skip_transactional_drop = false;
/* don't track stats for relations without storage */
if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
return;
- pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_locator.dbOid,
- RelFileLocatorToPgStatObjid(rel->rd_locator));
+ /* Check if this drop is part of a pending rewrite */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid &&
+ rel->rd_locator.spcOid == rewrite->old_locator.spcOid &&
+ rel->rd_locator.relNumber == rewrite->old_locator.relNumber)
+ {
+ skip_transactional_drop = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * If it is part of a rewrite, drop its stats later, for example in
+ * AtEOXact_PgStat_Relations(), so skip it here.
+ */
+ if (!skip_transactional_drop)
+ pgstat_drop_transactional(PGSTAT_KIND_RELATION,
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -666,6 +788,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit)
}
tabstat->trans = NULL;
}
+
+ /* preserve the stats in case of rewrite */
+ if (isCommit && pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *prev = NULL;
+ PgStat_PendingRewrite *current = pending_rewrites;
+ PgStat_PendingRewrite *next;
+
+ /* reverse the rewrites list to process in chronological order */
+ while (current != NULL)
+ {
+ next = current->next;
+ current->next = prev;
+ prev = current;
+ current = next;
+ }
+
+ /* now process rewrites in chronological order */
+ for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ pgstat_copy_relation_stats(rewrite->new_locator,
+ rewrite->old_locator, true);
+
+ /* drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
+ }
+
+ pending_rewrites = NULL;
}
/*
@@ -681,6 +845,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in
PgStat_TableXactStatus *trans;
PgStat_TableXactStatus *next_trans;
+ /*
+ * If we don't commit then remove the associated rewrites if any, to keep
+ * the rewrite chain in sync with what will be eventually committed.
+ */
+ if (!isCommit)
+ {
+ PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites;
+
+ while (*rewrite_ptr != NULL)
+ {
+ if ((*rewrite_ptr)->nest_level >= nestDepth)
+ {
+ PgStat_PendingRewrite *to_remove = *rewrite_ptr;
+
+ *rewrite_ptr = (*rewrite_ptr)->next;
+ pfree(to_remove);
+ }
+ else
+ {
+ rewrite_ptr = &((*rewrite_ptr)->next);
+ }
+ }
+ }
+
for (trans = xact_state->first; trans != NULL; trans = next_trans)
{
PgStat_TableStatus *tabstat;
@@ -760,11 +948,19 @@ void
AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
{
PgStat_TableXactStatus *trans;
+ PgStat_PendingRewrite *rewrite;
+ /*
+ * For each tabstat, find its matching rewrite and remove it from the
+ * pending rewrites list. This way, after processing all tabstats, pending
+ * rewrites will only contain rewrite only transactions.
+ */
for (trans = xact_state->first; trans != NULL; trans = trans->next)
{
PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY;
TwoPhasePgStatRecord record;
+ PgStat_PendingRewrite **rewrite_ptr;
+ bool found_rewrite = false;
Assert(trans->nest_level == 1);
Assert(trans->upper == NULL);
@@ -784,10 +980,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.locator = tabstat->locator;
record.truncdropped = trans->truncdropped;
+ record.rewrite_nest_level = 0;
+
+ /*
+ * Look for a matching rewrite and remove it from pending rewrites. We
+ * check three possible matches:
+ *
+ * The new_locator when stats have been added after the rewrite. The
+ * old_locator when stats have been added before the rewrite but not
+ * after. The original_locator when this tabstat is part of a rewrite
+ * chain.
+ */
+ rewrite_ptr = &pending_rewrites;
+ while (*rewrite_ptr != NULL)
+ {
+ rewrite = *rewrite_ptr;
+
+ if ((record.locator.dbOid == rewrite->new_locator.dbOid &&
+ record.locator.spcOid == rewrite->new_locator.spcOid &&
+ record.locator.relNumber == rewrite->new_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->old_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->old_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->old_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->original_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->original_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->original_locator.relNumber))
+ {
+ /*
+ * Found matching rewrite. Record the rewrite information and
+ * remove this rewrite from the list since it's now handled.
+ */
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.locator = rewrite->new_locator;
+ found_rewrite = true;
+
+ /* Remove from pending_rewrites list */
+ *rewrite_ptr = rewrite->next;
+ pfree(rewrite);
+ break;
+ }
+ else
+ {
+ /* Move to next rewrite in the list */
+ rewrite_ptr = &(rewrite->next);
+ }
+ }
+
+ /* If no rewrite found, clear the rewrite fields */
+ if (!found_rewrite)
+ {
+ memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator));
+ }
+
+ RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
+ &record, sizeof(TwoPhasePgStatRecord));
+ }
+
+ /*
+ * Now process any rewrites still pending. These are rewrite only
+ * transactions. We need to preserve their stats even though there's no
+ * tabstat entry for them.
+ */
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ TwoPhasePgStatRecord record;
+
+ memset(&record, 0, sizeof(TwoPhasePgStatRecord));
+ record.locator = rewrite->new_locator;
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.truncdropped = false;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
&record, sizeof(TwoPhasePgStatRecord));
}
+
+ pending_rewrites = NULL;
}
/*
@@ -810,6 +1079,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
tabstat = trans->parent;
tabstat->trans = NULL;
}
+
+ pending_rewrites = NULL;
}
/*
@@ -845,6 +1116,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
pgstat_info->counts.changed_tuples +=
rec->tuples_inserted + rec->tuples_updated +
rec->tuples_deleted;
+
+ if (rec->rewrite_nest_level > 0)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ /* Flush any pending stats for old locator first */
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ /* Copy stats from old to new locator */
+ pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator,
+ true);
+
+ /* Drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
}
/*
@@ -859,9 +1153,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
{
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
+ RelFileLocator target_locator;
+
+ /*
+ * For aborted transactions with rewrites (like TRUNCATE), we need to
+ * restore stats to the old locator, not the new one. The new locator
+ * should be dropped since the rewrite is being rolled back.
+ */
+ if (rec->rewrite_nest_level > 0)
+ {
+ /* Use the old locator */
+ target_locator = rec->rewrite_old_locator;
+ }
+ else
+ {
+ /* No rewrite, use the original locator */
+ target_locator = rec->locator;
+ }
/* Find or create a tabstat entry for the target locator */
- pgstat_info = pgstat_prep_relation_pending(rec->locator);
+ pgstat_info = pgstat_prep_relation_pending(target_locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -916,7 +1227,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
+ TimestampTz t;
+
+ /*
+ * Checking the transaction state due to the flush call in
+ * pgstat_twophase_postcommit() that would break the assertion on the
+ * state in GetCurrentTransactionStopTimestamp().
+ */
+ if (!IsTransactionState())
+ t = GetCurrentTransactionStopTimestamp();
+ else
+ t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
@@ -1167,3 +1488,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
ReleaseSysCache(tuple);
return result;
}
+
+/*
+ * Mark that a relation rewrite has occurred, preserving the original locator
+ * so stats can be copied at transaction commit.
+ */
+void
+pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator)
+{
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *existing;
+ RelFileLocator original_locator = old_locator;
+
+ for (existing = pending_rewrites; existing != NULL; existing = existing->next)
+ {
+ if (old_locator.dbOid == existing->new_locator.dbOid &&
+ old_locator.spcOid == existing->new_locator.spcOid &&
+ old_locator.relNumber == existing->new_locator.relNumber)
+ {
+ original_locator = existing->original_locator;
+ break;
+ }
+ }
+
+ /* Allocate in TopTransactionContext memory context */
+ rewrite = MemoryContextAlloc(TopTransactionContext,
+ sizeof(PgStat_PendingRewrite));
+
+ rewrite->old_locator = old_locator;
+ rewrite->new_locator = new_locator;
+ rewrite->original_locator = original_locator;
+ rewrite->nest_level = GetCurrentTransactionNestLevel();
+
+ /* Add to the list */
+ rewrite->next = pending_rewrites;
+ pending_rewrites = rewrite;
+}
+
+void
+pgstat_clear_rewrite(void)
+{
+ pending_rewrites = NULL;
+}
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index 5e2d69e6297..8ed8f5317f3 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel)
}
pgStatXactStack = NULL;
+ pgstat_clear_rewrite();
+
/* Make sure any stats snapshot is thrown away */
pgstat_clear_snapshot();
}
@@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo
void
pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid)
{
- if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL))
+ PgStat_EntryRef *entry_ref;
+
+ entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL);
+
+ if (entry_ref)
{
+ /*
+ * For relations stats, we key by physical file location, not by
+ * relation OID. This means during operations like ALTER TYPE where
+ * the relation OID changes but the relfilenode stays the same (no
+ * actual rewrite needed), we'll find an existing entry.
+ *
+ * This is expected behavior, we want to preserve stats across the
+ * catalog change. Simply reset and recreate the entry for the new
+ * relation OID without warning.
+ */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ pgstat_reset(kind, dboid, objid);
+ create_drop_transactional_internal(kind, dboid, objid, true);
+ return;
+ }
+
ereport(WARNING,
errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64,
(pgstat_get_kind_info(kind))->name, dboid,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 3a4f19e8d58..60491962668 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -86,6 +86,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/relmapper.h"
#include "utils/resowner.h"
#include "utils/snapmgr.h"
@@ -3771,6 +3772,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
MultiXactId minmulti = InvalidMultiXactId;
TransactionId freezeXid = InvalidTransactionId;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator = relation->rd_locator;
if (!IsBinaryUpgrade)
{
@@ -3942,6 +3944,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
table_close(pg_class, RowExclusiveLock);
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Make the pg_class row change or relation map change visible. This will
* cause the relcache entry to get updated, too.
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 689c624f373..d0d5473f8a5 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -673,7 +673,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id);
extern void pgstat_create_relation(Relation rel);
extern void pgstat_drop_relation(Relation rel);
-extern void pgstat_copy_relation_stats(Relation dst, Relation src);
+extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment);
extern void pgstat_init_relation(Relation rel);
extern void pgstat_assoc_relation(Relation rel);
@@ -685,6 +685,9 @@ extern void pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_mark_rewrite(RelFileLocator old_locator,
+ RelFileLocator new_locator);
+extern void pgstat_clear_rewrite(void);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 52f8603a7be..8b84e3a0bcf 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2298,6 +2298,7 @@ PgStat_KindInfo
PgStat_LocalState
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
+PgStat_PendingRewrite
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
--USlF2Vr25us5UpvY--
^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v9 2/2] handle relation statistics correctly during rewrites
@ 2025-11-04 13:52 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Bertrand Drouvot @ 2025-11-04 13:52 UTC (permalink / raw)
Now that PGSTAT_KIND_RELATION is keyed by refilenode, we need to handle rewrites.
To do so, this patch:
- Adds PgStat_PendingRewrite, a new struct to track rewrite operations within
a transaction, storing the old locator, new locator, and original locator (for
rewrite chains). This allows stats to be copied from the original location to
the final location at commit time.
- Adds a new function, pgstat_mark_rewrite(), called when a table rewrite begins.
It records the rewrite operation in a local list and detects rewrite chains by
checking if the old_locator matches any existing new_locator, preserving the
chain's original_locator.
- Modifies pgstat_copy_relation_stats(), to accept RelFileLocators instead of
Relations, with a new increment parameter to accumulate stats (needed for rewrite
chains with DML between rewrites).
- Ensures that AtEOXact_PgStat_Relations(), AtPrepare_PgStat_Relations(),
pgstat_twophase_postcommit()/postabort() pgstat_drop_relation() handle the
PgStat_PendingRewrite list correctly.
Note that due to the new flush call in pgstat_twophase_postcommit() we can not
call GetCurrentTransactionStopTimestamp() in pgstat_relation_flush_cb(). So,
adding a check to handle this special case and call GetCurrentTimestamp() instead.
Note that we'd call GetCurrentTimestamp() only if there is a rewrite, so that
the GetCurrentTimestamp() extra cost should be negligible. Another solution
could be to trigger the flush from FinishPreparedTransaction() but that's not
worth the extra complexity.
The new pending_rewrites list is traversed in multiple places. The overhead
should be negligible in comparison to a rewrite and the list should not contain
a lot of rewrites in practice.
The pending_rewrites list is traversed in multiple places. In typical usage,
the list will contain only a few entries so the traversal cost is negligible (
furthermore in comparison to a rewrite).
---
src/backend/catalog/index.c | 2 +-
src/backend/commands/cluster.c | 5 +
src/backend/commands/tablecmds.c | 6 +
src/backend/utils/activity/pgstat_relation.c | 391 ++++++++++++++++++-
src/backend/utils/activity/pgstat_xact.c | 25 +-
src/backend/utils/cache/relcache.c | 6 +
src/include/pgstat.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 424 insertions(+), 17 deletions(-)
92.8% src/backend/utils/activity/
4.9% src/backend/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 8dea58ad96b..b71925a22c3 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1795,7 +1795,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId);
/* copy over statistics from old to new index */
- pgstat_copy_relation_stats(newClassRel, oldClassRel);
+ pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 2120c85ccb4..6155b12afab 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1196,6 +1196,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
rel1 = relation_open(r1, NoLock);
rel2 = relation_open(r2, NoLock);
+
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind))
+ pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator);
+
rel2->rd_createSubid = rel1->rd_createSubid;
rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid;
rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 6b1a00ed477..9de70f321ed 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -16884,6 +16884,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
Oid reltoastrelid;
RelFileNumber newrelfilenumber;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator;
List *reltoastidxids = NIL;
ListCell *lc;
@@ -16922,6 +16923,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
newrlocator = rel->rd_locator;
newrlocator.relNumber = newrelfilenumber;
newrlocator.spcOid = newTableSpace;
+ oldrlocator = rel->rd_locator;
/* hand off to AM to actually create new rel storage and copy the data */
if (rel->rd_rel->relkind == RELKIND_INDEX)
@@ -16934,6 +16936,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
table_relation_copy_data(rel, &newrlocator);
}
+ /* mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Update the pg_class row.
*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 48bf93cae6e..b6ba8fdf440 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -30,6 +30,19 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
+/* Pending rewrite operations for stats copying */
+typedef struct PgStat_PendingRewrite
+{
+ RelFileLocator old_locator;
+ RelFileLocator new_locator;
+ RelFileLocator original_locator;
+ int nest_level; /* Transaction nesting level where rewrite
+ * occurred */
+ struct PgStat_PendingRewrite *next;
+} PgStat_PendingRewrite;
+
+/* The pending rewrites list for current transaction */
+static PgStat_PendingRewrite *pending_rewrites = NULL;
/* Record that's written to 2PC state file when pgstat state is persisted */
typedef struct TwoPhasePgStatRecord
@@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter deleted_pre_truncdrop;
RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
+ RelFileLocator rewrite_old_locator;
+ int rewrite_nest_level;
} TwoPhasePgStatRecord;
@@ -54,27 +69,70 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
/*
- * Copy stats between relations. This is used for things like REINDEX
+ * Copy stats between RelFileLocator. This is used for things like REINDEX
* CONCURRENTLY.
*/
void
-pgstat_copy_relation_stats(Relation dst, Relation src)
+pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment)
{
PgStat_StatTabEntry *srcstats;
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src));
+ srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION,
+ src.dbOid,
+ RelFileLocatorToPgStatObjid(src));
if (!srcstats)
return;
dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
- dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(dst),
+ dst.dbOid,
+ RelFileLocatorToPgStatObjid(dst),
false);
dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats;
- dstshstats->stats = *srcstats;
+
+ if (!increment)
+ dstshstats->stats = *srcstats;
+ else
+ {
+ /* Increment those statistics */
+#define RELFSTAT_ACC(fld, stats_to_add) \
+ (dstshstats->stats.fld += stats_to_add->fld)
+ RELFSTAT_ACC(numscans, srcstats);
+ RELFSTAT_ACC(tuples_returned, srcstats);
+ RELFSTAT_ACC(tuples_fetched, srcstats);
+ RELFSTAT_ACC(tuples_inserted, srcstats);
+ RELFSTAT_ACC(tuples_updated, srcstats);
+ RELFSTAT_ACC(tuples_deleted, srcstats);
+ RELFSTAT_ACC(tuples_hot_updated, srcstats);
+ RELFSTAT_ACC(tuples_newpage_updated, srcstats);
+ RELFSTAT_ACC(live_tuples, srcstats);
+ RELFSTAT_ACC(dead_tuples, srcstats);
+ RELFSTAT_ACC(mod_since_analyze, srcstats);
+ RELFSTAT_ACC(ins_since_vacuum, srcstats);
+ RELFSTAT_ACC(blocks_fetched, srcstats);
+ RELFSTAT_ACC(blocks_hit, srcstats);
+ RELFSTAT_ACC(vacuum_count, srcstats);
+ RELFSTAT_ACC(autovacuum_count, srcstats);
+ RELFSTAT_ACC(analyze_count, srcstats);
+ RELFSTAT_ACC(autoanalyze_count, srcstats);
+ RELFSTAT_ACC(total_vacuum_time, srcstats);
+ RELFSTAT_ACC(total_autovacuum_time, srcstats);
+ RELFSTAT_ACC(total_analyze_time, srcstats);
+ RELFSTAT_ACC(total_autoanalyze_time, srcstats);
+#undef RELFSTAT_ACC
+
+ /* Replace those statistics */
+#define RELFSTAT_REP(fld, stats_to_rep) \
+ (dstshstats->stats.fld = stats_to_rep->fld)
+ RELFSTAT_REP(lastscan, srcstats);
+ RELFSTAT_REP(last_vacuum_time, srcstats);
+ RELFSTAT_REP(last_autovacuum_time, srcstats);
+ RELFSTAT_REP(last_analyze_time, srcstats);
+ RELFSTAT_REP(last_autoanalyze_time, srcstats);
+#undef RELFSTAT_REP
+ }
pgstat_unlock_entry(dst_ref);
}
@@ -136,6 +194,7 @@ void
pgstat_assoc_relation(Relation rel)
{
RelFileLocator locator;
+ PgStat_TableStatus *pgstat_info;
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
@@ -164,14 +223,54 @@ pgstat_assoc_relation(Relation rel)
locator.relNumber = rel->rd_id;
}
+ /*
+ * If this relation was rewritten during the current transaction we may be
+ * reopening it with its new RelFileLocator. In that case, continue using
+ * the stats entry associated with the old locator rather than creating a
+ * new one. This ensures all stats from before and after the rewrite are
+ * tracked in a single entry which will be properly copied to the new
+ * locator at transaction commit.
+ */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (locator.dbOid == rewrite->new_locator.dbOid &&
+ locator.spcOid == rewrite->new_locator.spcOid &&
+ locator.relNumber == rewrite->new_locator.relNumber)
+ {
+ pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator);
+ goto found_entry;
+ }
+ }
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(locator);
+ pgstat_info = pgstat_prep_relation_pending(locator);
+
+found_entry:
+ rel->pgstat_info = pgstat_info;
+
+ /*
+ * For relations stats, we key by physical file location, not by relation
+ * OID. This means during operations like ALTER TYPE it's possible that
+ * the relation OID changes but the relfilenode stays the same (no actual
+ * rewrite needed). Unlink the old relation first.
+ */
+ if (pgstat_info->relation != NULL &&
+ pgstat_info->relation != rel)
+ {
+ pgstat_info->relation->pgstat_info = NULL;
+ pgstat_info->relation = NULL;
+ }
/* don't allow link a stats to multiple relcache entries */
- Assert(rel->pgstat_info->relation == NULL);
+ Assert(pgstat_info->relation == NULL);
/* mark this relation as the owner */
- rel->pgstat_info->relation = rel;
+ pgstat_info->relation = rel;
}
/*
@@ -214,14 +313,37 @@ pgstat_drop_relation(Relation rel)
{
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ bool skip_transactional_drop = false;
/* don't track stats for relations without storage */
if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
return;
- pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_locator.dbOid,
- RelFileLocatorToPgStatObjid(rel->rd_locator));
+ /* Check if this drop is part of a pending rewrite */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid &&
+ rel->rd_locator.spcOid == rewrite->old_locator.spcOid &&
+ rel->rd_locator.relNumber == rewrite->old_locator.relNumber)
+ {
+ skip_transactional_drop = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * If it is part of a rewrite, drop its stats later, for example in
+ * AtEOXact_PgStat_Relations(), so skip it here.
+ */
+ if (!skip_transactional_drop)
+ pgstat_drop_transactional(PGSTAT_KIND_RELATION,
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -666,6 +788,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit)
}
tabstat->trans = NULL;
}
+
+ /* preserve the stats in case of rewrite */
+ if (isCommit && pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *prev = NULL;
+ PgStat_PendingRewrite *current = pending_rewrites;
+ PgStat_PendingRewrite *next;
+
+ /* reverse the rewrites list to process in chronological order */
+ while (current != NULL)
+ {
+ next = current->next;
+ current->next = prev;
+ prev = current;
+ current = next;
+ }
+
+ /* now process rewrites in chronological order */
+ for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ pgstat_copy_relation_stats(rewrite->new_locator,
+ rewrite->old_locator, true);
+
+ /* drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
+ }
+
+ pending_rewrites = NULL;
}
/*
@@ -681,6 +845,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in
PgStat_TableXactStatus *trans;
PgStat_TableXactStatus *next_trans;
+ /*
+ * If we don't commit then remove the associated rewrites if any, to keep
+ * the rewrite chain in sync with what will be eventually committed.
+ */
+ if (!isCommit)
+ {
+ PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites;
+
+ while (*rewrite_ptr != NULL)
+ {
+ if ((*rewrite_ptr)->nest_level >= nestDepth)
+ {
+ PgStat_PendingRewrite *to_remove = *rewrite_ptr;
+
+ *rewrite_ptr = (*rewrite_ptr)->next;
+ pfree(to_remove);
+ }
+ else
+ {
+ rewrite_ptr = &((*rewrite_ptr)->next);
+ }
+ }
+ }
+
for (trans = xact_state->first; trans != NULL; trans = next_trans)
{
PgStat_TableStatus *tabstat;
@@ -760,11 +948,19 @@ void
AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
{
PgStat_TableXactStatus *trans;
+ PgStat_PendingRewrite *rewrite;
+ /*
+ * For each tabstat, find its matching rewrite and remove it from the
+ * pending rewrites list. This way, after processing all tabstats, pending
+ * rewrites will only contain rewrite only transactions.
+ */
for (trans = xact_state->first; trans != NULL; trans = trans->next)
{
PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY;
TwoPhasePgStatRecord record;
+ PgStat_PendingRewrite **rewrite_ptr;
+ bool found_rewrite = false;
Assert(trans->nest_level == 1);
Assert(trans->upper == NULL);
@@ -784,10 +980,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.locator = tabstat->locator;
record.truncdropped = trans->truncdropped;
+ record.rewrite_nest_level = 0;
+
+ /*
+ * Look for a matching rewrite and remove it from pending rewrites. We
+ * check three possible matches:
+ *
+ * The new_locator when stats have been added after the rewrite. The
+ * old_locator when stats have been added before the rewrite but not
+ * after. The original_locator when this tabstat is part of a rewrite
+ * chain.
+ */
+ rewrite_ptr = &pending_rewrites;
+ while (*rewrite_ptr != NULL)
+ {
+ rewrite = *rewrite_ptr;
+
+ if ((record.locator.dbOid == rewrite->new_locator.dbOid &&
+ record.locator.spcOid == rewrite->new_locator.spcOid &&
+ record.locator.relNumber == rewrite->new_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->old_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->old_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->old_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->original_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->original_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->original_locator.relNumber))
+ {
+ /*
+ * Found matching rewrite. Record the rewrite information and
+ * remove this rewrite from the list since it's now handled.
+ */
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.locator = rewrite->new_locator;
+ found_rewrite = true;
+
+ /* Remove from pending_rewrites list */
+ *rewrite_ptr = rewrite->next;
+ pfree(rewrite);
+ break;
+ }
+ else
+ {
+ /* Move to next rewrite in the list */
+ rewrite_ptr = &(rewrite->next);
+ }
+ }
+
+ /* If no rewrite found, clear the rewrite fields */
+ if (!found_rewrite)
+ {
+ memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator));
+ }
+
+ RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
+ &record, sizeof(TwoPhasePgStatRecord));
+ }
+
+ /*
+ * Now process any rewrites still pending. These are rewrite only
+ * transactions. We need to preserve their stats even though there's no
+ * tabstat entry for them.
+ */
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ TwoPhasePgStatRecord record;
+
+ memset(&record, 0, sizeof(TwoPhasePgStatRecord));
+ record.locator = rewrite->new_locator;
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.truncdropped = false;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
&record, sizeof(TwoPhasePgStatRecord));
}
+
+ pending_rewrites = NULL;
}
/*
@@ -810,6 +1079,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
tabstat = trans->parent;
tabstat->trans = NULL;
}
+
+ pending_rewrites = NULL;
}
/*
@@ -845,6 +1116,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
pgstat_info->counts.changed_tuples +=
rec->tuples_inserted + rec->tuples_updated +
rec->tuples_deleted;
+
+ if (rec->rewrite_nest_level > 0)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ /* Flush any pending stats for old locator first */
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ /* Copy stats from old to new locator */
+ pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator,
+ true);
+
+ /* Drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
}
/*
@@ -859,9 +1153,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
{
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
+ RelFileLocator target_locator;
+
+ /*
+ * For aborted transactions with rewrites (like TRUNCATE), we need to
+ * restore stats to the old locator, not the new one. The new locator
+ * should be dropped since the rewrite is being rolled back.
+ */
+ if (rec->rewrite_nest_level > 0)
+ {
+ /* Use the old locator */
+ target_locator = rec->rewrite_old_locator;
+ }
+ else
+ {
+ /* No rewrite, use the original locator */
+ target_locator = rec->locator;
+ }
/* Find or create a tabstat entry for the target locator */
- pgstat_info = pgstat_prep_relation_pending(rec->locator);
+ pgstat_info = pgstat_prep_relation_pending(target_locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -916,7 +1227,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
+ TimestampTz t;
+
+ /*
+ * Checking the transaction state due to the flush call in
+ * pgstat_twophase_postcommit() that would break the assertion on the
+ * state in GetCurrentTransactionStopTimestamp().
+ */
+ if (!IsTransactionState())
+ t = GetCurrentTransactionStopTimestamp();
+ else
+ t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
@@ -1167,3 +1488,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
ReleaseSysCache(tuple);
return result;
}
+
+/*
+ * Mark that a relation rewrite has occurred, preserving the original locator
+ * so stats can be copied at transaction commit.
+ */
+void
+pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator)
+{
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *existing;
+ RelFileLocator original_locator = old_locator;
+
+ for (existing = pending_rewrites; existing != NULL; existing = existing->next)
+ {
+ if (old_locator.dbOid == existing->new_locator.dbOid &&
+ old_locator.spcOid == existing->new_locator.spcOid &&
+ old_locator.relNumber == existing->new_locator.relNumber)
+ {
+ original_locator = existing->original_locator;
+ break;
+ }
+ }
+
+ /* Allocate in TopTransactionContext memory context */
+ rewrite = MemoryContextAlloc(TopTransactionContext,
+ sizeof(PgStat_PendingRewrite));
+
+ rewrite->old_locator = old_locator;
+ rewrite->new_locator = new_locator;
+ rewrite->original_locator = original_locator;
+ rewrite->nest_level = GetCurrentTransactionNestLevel();
+
+ /* Add to the list */
+ rewrite->next = pending_rewrites;
+ pending_rewrites = rewrite;
+}
+
+void
+pgstat_clear_rewrite(void)
+{
+ pending_rewrites = NULL;
+}
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index bc9864bd8d9..f8cf3755ce2 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel)
}
pgStatXactStack = NULL;
+ pgstat_clear_rewrite();
+
/* Make sure any stats snapshot is thrown away */
pgstat_clear_snapshot();
}
@@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo
void
pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid)
{
- if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL))
+ PgStat_EntryRef *entry_ref;
+
+ entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL);
+
+ if (entry_ref)
{
+ /*
+ * For relations stats, we key by physical file location, not by
+ * relation OID. This means during operations like ALTER TYPE where
+ * the relation OID changes but the relfilenode stays the same (no
+ * actual rewrite needed), we'll find an existing entry.
+ *
+ * This is expected behavior, we want to preserve stats across the
+ * catalog change. Simply reset and recreate the entry for the new
+ * relation OID without warning.
+ */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ pgstat_reset(kind, dboid, objid);
+ create_drop_transactional_internal(kind, dboid, objid, true);
+ return;
+ }
+
ereport(WARNING,
errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64,
(pgstat_get_kind_info(kind))->name, dboid,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 2d0cb7bcfd4..c98e5c51d63 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -85,6 +85,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/relmapper.h"
#include "utils/resowner.h"
#include "utils/snapmgr.h"
@@ -3780,6 +3781,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
MultiXactId minmulti = InvalidMultiXactId;
TransactionId freezeXid = InvalidTransactionId;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator = relation->rd_locator;
if (!IsBinaryUpgrade)
{
@@ -3951,6 +3953,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
table_close(pg_class, RowExclusiveLock);
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Make the pg_class row change or relation map change visible. This will
* cause the relcache entry to get updated, too.
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 3102f86aa24..8750c025bbe 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -670,7 +670,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id);
extern void pgstat_create_relation(Relation rel);
extern void pgstat_drop_relation(Relation rel);
-extern void pgstat_copy_relation_stats(Relation dst, Relation src);
+extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment);
extern void pgstat_init_relation(Relation rel);
extern void pgstat_assoc_relation(Relation rel);
@@ -682,6 +682,9 @@ extern void pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_mark_rewrite(RelFileLocator old_locator,
+ RelFileLocator new_locator);
+extern void pgstat_clear_rewrite(void);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3451538565e..0ca3eae8026 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2268,6 +2268,7 @@ PgStat_KindInfo
PgStat_LocalState
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
+PgStat_PendingRewrite
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
--tV2Nti+lXhoeT6I5--
^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v12 2/2] handle relation statistics correctly during rewrites
@ 2025-11-04 13:52 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Bertrand Drouvot @ 2025-11-04 13:52 UTC (permalink / raw)
Now that PGSTAT_KIND_RELATION is keyed by refilenode, we need to handle rewrites.
To do so, this patch:
- Adds PgStat_PendingRewrite, a new struct to track rewrite operations within
a transaction, storing the old locator, new locator, and original locator (for
rewrite chains). This allows stats to be copied from the original location to
the final location at commit time.
- Adds a new function, pgstat_mark_rewrite(), called when a table rewrite begins.
It records the rewrite operation in a local list and detects rewrite chains by
checking if the old_locator matches any existing new_locator, preserving the
chain's original_locator.
- Modifies pgstat_copy_relation_stats(), to accept RelFileLocators instead of
Relations, with a new increment parameter to accumulate stats (needed for rewrite
chains with DML between rewrites).
- Ensures that AtEOXact_PgStat_Relations(), AtPrepare_PgStat_Relations(),
pgstat_twophase_postcommit()/postabort() pgstat_drop_relation() handle the
PgStat_PendingRewrite list correctly.
Note that due to the new flush call in pgstat_twophase_postcommit() we can not
call GetCurrentTransactionStopTimestamp() in pgstat_relation_flush_cb(). So,
adding a check to handle this special case and call GetCurrentTimestamp() instead.
Note that we'd call GetCurrentTimestamp() only if there is a rewrite, so that
the GetCurrentTimestamp() extra cost should be negligible. Another solution
could be to trigger the flush from FinishPreparedTransaction() but that's not
worth the extra complexity.
The new pending_rewrites list is traversed in multiple places. The overhead
should be negligible in comparison to a rewrite and the list should not contain
a lot of rewrites in practice.
The pending_rewrites list is traversed in multiple places. In typical usage,
the list will contain only a few entries so the traversal cost is negligible (
furthermore in comparison to a rewrite).
---
src/backend/catalog/index.c | 2 +-
src/backend/commands/cluster.c | 5 +
src/backend/commands/tablecmds.c | 6 +
src/backend/utils/activity/pgstat_relation.c | 391 ++++++++++++++++++-
src/backend/utils/activity/pgstat_xact.c | 25 +-
src/backend/utils/cache/relcache.c | 6 +
src/include/pgstat.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 424 insertions(+), 17 deletions(-)
92.8% src/backend/utils/activity/
4.9% src/backend/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 43de42ce39e..88ef00609db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1793,7 +1793,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId);
/* copy over statistics from old to new index */
- pgstat_copy_relation_stats(newClassRel, oldClassRel);
+ pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 60a4617a585..3a446f32921 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1196,6 +1196,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
rel1 = relation_open(r1, NoLock);
rel2 = relation_open(r2, NoLock);
+
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind))
+ pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator);
+
rel2->rd_createSubid = rel1->rd_createSubid;
rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid;
rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 85242dcc245..8d10d9d1e0f 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -16883,6 +16883,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
Oid reltoastrelid;
RelFileNumber newrelfilenumber;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator;
List *reltoastidxids = NIL;
ListCell *lc;
@@ -16921,6 +16922,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
newrlocator = rel->rd_locator;
newrlocator.relNumber = newrelfilenumber;
newrlocator.spcOid = newTableSpace;
+ oldrlocator = rel->rd_locator;
/* hand off to AM to actually create new rel storage and copy the data */
if (rel->rd_rel->relkind == RELKIND_INDEX)
@@ -16933,6 +16935,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
table_relation_copy_data(rel, &newrlocator);
}
+ /* mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Update the pg_class row.
*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 89bf0cbed56..4d929972fab 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -30,6 +30,19 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
+/* Pending rewrite operations for stats copying */
+typedef struct PgStat_PendingRewrite
+{
+ RelFileLocator old_locator;
+ RelFileLocator new_locator;
+ RelFileLocator original_locator;
+ int nest_level; /* Transaction nesting level where rewrite
+ * occurred */
+ struct PgStat_PendingRewrite *next;
+} PgStat_PendingRewrite;
+
+/* The pending rewrites list for current transaction */
+static PgStat_PendingRewrite *pending_rewrites = NULL;
/* Record that's written to 2PC state file when pgstat state is persisted */
typedef struct TwoPhasePgStatRecord
@@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter deleted_pre_truncdrop;
RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
+ RelFileLocator rewrite_old_locator;
+ int rewrite_nest_level;
} TwoPhasePgStatRecord;
@@ -54,27 +69,70 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
/*
- * Copy stats between relations. This is used for things like REINDEX
+ * Copy stats between RelFileLocator. This is used for things like REINDEX
* CONCURRENTLY.
*/
void
-pgstat_copy_relation_stats(Relation dst, Relation src)
+pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment)
{
PgStat_StatTabEntry *srcstats;
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src));
+ srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION,
+ src.dbOid,
+ RelFileLocatorToPgStatObjid(src));
if (!srcstats)
return;
dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
- dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(dst),
+ dst.dbOid,
+ RelFileLocatorToPgStatObjid(dst),
false);
dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats;
- dstshstats->stats = *srcstats;
+
+ if (!increment)
+ dstshstats->stats = *srcstats;
+ else
+ {
+ /* Increment those statistics */
+#define RELFSTAT_ACC(fld, stats_to_add) \
+ (dstshstats->stats.fld += stats_to_add->fld)
+ RELFSTAT_ACC(numscans, srcstats);
+ RELFSTAT_ACC(tuples_returned, srcstats);
+ RELFSTAT_ACC(tuples_fetched, srcstats);
+ RELFSTAT_ACC(tuples_inserted, srcstats);
+ RELFSTAT_ACC(tuples_updated, srcstats);
+ RELFSTAT_ACC(tuples_deleted, srcstats);
+ RELFSTAT_ACC(tuples_hot_updated, srcstats);
+ RELFSTAT_ACC(tuples_newpage_updated, srcstats);
+ RELFSTAT_ACC(live_tuples, srcstats);
+ RELFSTAT_ACC(dead_tuples, srcstats);
+ RELFSTAT_ACC(mod_since_analyze, srcstats);
+ RELFSTAT_ACC(ins_since_vacuum, srcstats);
+ RELFSTAT_ACC(blocks_fetched, srcstats);
+ RELFSTAT_ACC(blocks_hit, srcstats);
+ RELFSTAT_ACC(vacuum_count, srcstats);
+ RELFSTAT_ACC(autovacuum_count, srcstats);
+ RELFSTAT_ACC(analyze_count, srcstats);
+ RELFSTAT_ACC(autoanalyze_count, srcstats);
+ RELFSTAT_ACC(total_vacuum_time, srcstats);
+ RELFSTAT_ACC(total_autovacuum_time, srcstats);
+ RELFSTAT_ACC(total_analyze_time, srcstats);
+ RELFSTAT_ACC(total_autoanalyze_time, srcstats);
+#undef RELFSTAT_ACC
+
+ /* Replace those statistics */
+#define RELFSTAT_REP(fld, stats_to_rep) \
+ (dstshstats->stats.fld = stats_to_rep->fld)
+ RELFSTAT_REP(lastscan, srcstats);
+ RELFSTAT_REP(last_vacuum_time, srcstats);
+ RELFSTAT_REP(last_autovacuum_time, srcstats);
+ RELFSTAT_REP(last_analyze_time, srcstats);
+ RELFSTAT_REP(last_autoanalyze_time, srcstats);
+#undef RELFSTAT_REP
+ }
pgstat_unlock_entry(dst_ref);
}
@@ -136,6 +194,7 @@ void
pgstat_assoc_relation(Relation rel)
{
RelFileLocator locator;
+ PgStat_TableStatus *pgstat_info;
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
@@ -164,14 +223,54 @@ pgstat_assoc_relation(Relation rel)
locator.relNumber = rel->rd_id;
}
+ /*
+ * If this relation was rewritten during the current transaction we may be
+ * reopening it with its new RelFileLocator. In that case, continue using
+ * the stats entry associated with the old locator rather than creating a
+ * new one. This ensures all stats from before and after the rewrite are
+ * tracked in a single entry which will be properly copied to the new
+ * locator at transaction commit.
+ */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (locator.dbOid == rewrite->new_locator.dbOid &&
+ locator.spcOid == rewrite->new_locator.spcOid &&
+ locator.relNumber == rewrite->new_locator.relNumber)
+ {
+ pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator);
+ goto found_entry;
+ }
+ }
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(locator);
+ pgstat_info = pgstat_prep_relation_pending(locator);
+
+found_entry:
+ rel->pgstat_info = pgstat_info;
+
+ /*
+ * For relations stats, we key by physical file location, not by relation
+ * OID. This means during operations like ALTER TYPE it's possible that
+ * the relation OID changes but the relfilenode stays the same (no actual
+ * rewrite needed). Unlink the old relation first.
+ */
+ if (pgstat_info->relation != NULL &&
+ pgstat_info->relation != rel)
+ {
+ pgstat_info->relation->pgstat_info = NULL;
+ pgstat_info->relation = NULL;
+ }
/* don't allow link a stats to multiple relcache entries */
- Assert(rel->pgstat_info->relation == NULL);
+ Assert(pgstat_info->relation == NULL);
/* mark this relation as the owner */
- rel->pgstat_info->relation = rel;
+ pgstat_info->relation = rel;
}
/*
@@ -214,14 +313,37 @@ pgstat_drop_relation(Relation rel)
{
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ bool skip_transactional_drop = false;
/* don't track stats for relations without storage */
if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
return;
- pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_locator.dbOid,
- RelFileLocatorToPgStatObjid(rel->rd_locator));
+ /* Check if this drop is part of a pending rewrite */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid &&
+ rel->rd_locator.spcOid == rewrite->old_locator.spcOid &&
+ rel->rd_locator.relNumber == rewrite->old_locator.relNumber)
+ {
+ skip_transactional_drop = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * If it is part of a rewrite, drop its stats later, for example in
+ * AtEOXact_PgStat_Relations(), so skip it here.
+ */
+ if (!skip_transactional_drop)
+ pgstat_drop_transactional(PGSTAT_KIND_RELATION,
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -666,6 +788,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit)
}
tabstat->trans = NULL;
}
+
+ /* preserve the stats in case of rewrite */
+ if (isCommit && pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *prev = NULL;
+ PgStat_PendingRewrite *current = pending_rewrites;
+ PgStat_PendingRewrite *next;
+
+ /* reverse the rewrites list to process in chronological order */
+ while (current != NULL)
+ {
+ next = current->next;
+ current->next = prev;
+ prev = current;
+ current = next;
+ }
+
+ /* now process rewrites in chronological order */
+ for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ pgstat_copy_relation_stats(rewrite->new_locator,
+ rewrite->old_locator, true);
+
+ /* drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
+ }
+
+ pending_rewrites = NULL;
}
/*
@@ -681,6 +845,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in
PgStat_TableXactStatus *trans;
PgStat_TableXactStatus *next_trans;
+ /*
+ * If we don't commit then remove the associated rewrites if any, to keep
+ * the rewrite chain in sync with what will be eventually committed.
+ */
+ if (!isCommit)
+ {
+ PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites;
+
+ while (*rewrite_ptr != NULL)
+ {
+ if ((*rewrite_ptr)->nest_level >= nestDepth)
+ {
+ PgStat_PendingRewrite *to_remove = *rewrite_ptr;
+
+ *rewrite_ptr = (*rewrite_ptr)->next;
+ pfree(to_remove);
+ }
+ else
+ {
+ rewrite_ptr = &((*rewrite_ptr)->next);
+ }
+ }
+ }
+
for (trans = xact_state->first; trans != NULL; trans = next_trans)
{
PgStat_TableStatus *tabstat;
@@ -760,11 +948,19 @@ void
AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
{
PgStat_TableXactStatus *trans;
+ PgStat_PendingRewrite *rewrite;
+ /*
+ * For each tabstat, find its matching rewrite and remove it from the
+ * pending rewrites list. This way, after processing all tabstats, pending
+ * rewrites will only contain rewrite only transactions.
+ */
for (trans = xact_state->first; trans != NULL; trans = trans->next)
{
PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY;
TwoPhasePgStatRecord record;
+ PgStat_PendingRewrite **rewrite_ptr;
+ bool found_rewrite = false;
Assert(trans->nest_level == 1);
Assert(trans->upper == NULL);
@@ -784,10 +980,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.locator = tabstat->locator;
record.truncdropped = trans->truncdropped;
+ record.rewrite_nest_level = 0;
+
+ /*
+ * Look for a matching rewrite and remove it from pending rewrites. We
+ * check three possible matches:
+ *
+ * The new_locator when stats have been added after the rewrite. The
+ * old_locator when stats have been added before the rewrite but not
+ * after. The original_locator when this tabstat is part of a rewrite
+ * chain.
+ */
+ rewrite_ptr = &pending_rewrites;
+ while (*rewrite_ptr != NULL)
+ {
+ rewrite = *rewrite_ptr;
+
+ if ((record.locator.dbOid == rewrite->new_locator.dbOid &&
+ record.locator.spcOid == rewrite->new_locator.spcOid &&
+ record.locator.relNumber == rewrite->new_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->old_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->old_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->old_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->original_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->original_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->original_locator.relNumber))
+ {
+ /*
+ * Found matching rewrite. Record the rewrite information and
+ * remove this rewrite from the list since it's now handled.
+ */
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.locator = rewrite->new_locator;
+ found_rewrite = true;
+
+ /* Remove from pending_rewrites list */
+ *rewrite_ptr = rewrite->next;
+ pfree(rewrite);
+ break;
+ }
+ else
+ {
+ /* Move to next rewrite in the list */
+ rewrite_ptr = &(rewrite->next);
+ }
+ }
+
+ /* If no rewrite found, clear the rewrite fields */
+ if (!found_rewrite)
+ {
+ memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator));
+ }
+
+ RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
+ &record, sizeof(TwoPhasePgStatRecord));
+ }
+
+ /*
+ * Now process any rewrites still pending. These are rewrite only
+ * transactions. We need to preserve their stats even though there's no
+ * tabstat entry for them.
+ */
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ TwoPhasePgStatRecord record;
+
+ memset(&record, 0, sizeof(TwoPhasePgStatRecord));
+ record.locator = rewrite->new_locator;
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.truncdropped = false;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
&record, sizeof(TwoPhasePgStatRecord));
}
+
+ pending_rewrites = NULL;
}
/*
@@ -810,6 +1079,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
tabstat = trans->parent;
tabstat->trans = NULL;
}
+
+ pending_rewrites = NULL;
}
/*
@@ -845,6 +1116,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
pgstat_info->counts.changed_tuples +=
rec->tuples_inserted + rec->tuples_updated +
rec->tuples_deleted;
+
+ if (rec->rewrite_nest_level > 0)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ /* Flush any pending stats for old locator first */
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ /* Copy stats from old to new locator */
+ pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator,
+ true);
+
+ /* Drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
}
/*
@@ -859,9 +1153,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
{
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
+ RelFileLocator target_locator;
+
+ /*
+ * For aborted transactions with rewrites (like TRUNCATE), we need to
+ * restore stats to the old locator, not the new one. The new locator
+ * should be dropped since the rewrite is being rolled back.
+ */
+ if (rec->rewrite_nest_level > 0)
+ {
+ /* Use the old locator */
+ target_locator = rec->rewrite_old_locator;
+ }
+ else
+ {
+ /* No rewrite, use the original locator */
+ target_locator = rec->locator;
+ }
/* Find or create a tabstat entry for the target locator */
- pgstat_info = pgstat_prep_relation_pending(rec->locator);
+ pgstat_info = pgstat_prep_relation_pending(target_locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -916,7 +1227,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
+ TimestampTz t;
+
+ /*
+ * Checking the transaction state due to the flush call in
+ * pgstat_twophase_postcommit() that would break the assertion on the
+ * state in GetCurrentTransactionStopTimestamp().
+ */
+ if (!IsTransactionState())
+ t = GetCurrentTransactionStopTimestamp();
+ else
+ t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
@@ -1167,3 +1488,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
ReleaseSysCache(tuple);
return result;
}
+
+/*
+ * Mark that a relation rewrite has occurred, preserving the original locator
+ * so stats can be copied at transaction commit.
+ */
+void
+pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator)
+{
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *existing;
+ RelFileLocator original_locator = old_locator;
+
+ for (existing = pending_rewrites; existing != NULL; existing = existing->next)
+ {
+ if (old_locator.dbOid == existing->new_locator.dbOid &&
+ old_locator.spcOid == existing->new_locator.spcOid &&
+ old_locator.relNumber == existing->new_locator.relNumber)
+ {
+ original_locator = existing->original_locator;
+ break;
+ }
+ }
+
+ /* Allocate in TopTransactionContext memory context */
+ rewrite = MemoryContextAlloc(TopTransactionContext,
+ sizeof(PgStat_PendingRewrite));
+
+ rewrite->old_locator = old_locator;
+ rewrite->new_locator = new_locator;
+ rewrite->original_locator = original_locator;
+ rewrite->nest_level = GetCurrentTransactionNestLevel();
+
+ /* Add to the list */
+ rewrite->next = pending_rewrites;
+ pending_rewrites = rewrite;
+}
+
+void
+pgstat_clear_rewrite(void)
+{
+ pending_rewrites = NULL;
+}
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index 5e2d69e6297..8ed8f5317f3 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel)
}
pgStatXactStack = NULL;
+ pgstat_clear_rewrite();
+
/* Make sure any stats snapshot is thrown away */
pgstat_clear_snapshot();
}
@@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo
void
pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid)
{
- if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL))
+ PgStat_EntryRef *entry_ref;
+
+ entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL);
+
+ if (entry_ref)
{
+ /*
+ * For relations stats, we key by physical file location, not by
+ * relation OID. This means during operations like ALTER TYPE where
+ * the relation OID changes but the relfilenode stays the same (no
+ * actual rewrite needed), we'll find an existing entry.
+ *
+ * This is expected behavior, we want to preserve stats across the
+ * catalog change. Simply reset and recreate the entry for the new
+ * relation OID without warning.
+ */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ pgstat_reset(kind, dboid, objid);
+ create_drop_transactional_internal(kind, dboid, objid, true);
+ return;
+ }
+
ereport(WARNING,
errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64,
(pgstat_get_kind_info(kind))->name, dboid,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index a1c88c6b1b6..dc5948f446c 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -85,6 +85,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/relmapper.h"
#include "utils/resowner.h"
#include "utils/snapmgr.h"
@@ -3775,6 +3776,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
MultiXactId minmulti = InvalidMultiXactId;
TransactionId freezeXid = InvalidTransactionId;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator = relation->rd_locator;
if (!IsBinaryUpgrade)
{
@@ -3946,6 +3948,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
table_close(pg_class, RowExclusiveLock);
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Make the pg_class row change or relation map change visible. This will
* cause the relcache entry to get updated, too.
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 689c624f373..d0d5473f8a5 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -673,7 +673,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id);
extern void pgstat_create_relation(Relation rel);
extern void pgstat_drop_relation(Relation rel);
-extern void pgstat_copy_relation_stats(Relation dst, Relation src);
+extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment);
extern void pgstat_init_relation(Relation rel);
extern void pgstat_assoc_relation(Relation rel);
@@ -685,6 +685,9 @@ extern void pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_mark_rewrite(RelFileLocator old_locator,
+ RelFileLocator new_locator);
+extern void pgstat_clear_rewrite(void);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3250564d4ff..fbcbe5da26a 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2281,6 +2281,7 @@ PgStat_KindInfo
PgStat_LocalState
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
+PgStat_PendingRewrite
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
--sdA5zuaL94ts0rne--
^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v11 2/2] handle relation statistics correctly during rewrites
@ 2025-11-04 13:52 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Bertrand Drouvot @ 2025-11-04 13:52 UTC (permalink / raw)
Now that PGSTAT_KIND_RELATION is keyed by refilenode, we need to handle rewrites.
To do so, this patch:
- Adds PgStat_PendingRewrite, a new struct to track rewrite operations within
a transaction, storing the old locator, new locator, and original locator (for
rewrite chains). This allows stats to be copied from the original location to
the final location at commit time.
- Adds a new function, pgstat_mark_rewrite(), called when a table rewrite begins.
It records the rewrite operation in a local list and detects rewrite chains by
checking if the old_locator matches any existing new_locator, preserving the
chain's original_locator.
- Modifies pgstat_copy_relation_stats(), to accept RelFileLocators instead of
Relations, with a new increment parameter to accumulate stats (needed for rewrite
chains with DML between rewrites).
- Ensures that AtEOXact_PgStat_Relations(), AtPrepare_PgStat_Relations(),
pgstat_twophase_postcommit()/postabort() pgstat_drop_relation() handle the
PgStat_PendingRewrite list correctly.
Note that due to the new flush call in pgstat_twophase_postcommit() we can not
call GetCurrentTransactionStopTimestamp() in pgstat_relation_flush_cb(). So,
adding a check to handle this special case and call GetCurrentTimestamp() instead.
Note that we'd call GetCurrentTimestamp() only if there is a rewrite, so that
the GetCurrentTimestamp() extra cost should be negligible. Another solution
could be to trigger the flush from FinishPreparedTransaction() but that's not
worth the extra complexity.
The new pending_rewrites list is traversed in multiple places. The overhead
should be negligible in comparison to a rewrite and the list should not contain
a lot of rewrites in practice.
The pending_rewrites list is traversed in multiple places. In typical usage,
the list will contain only a few entries so the traversal cost is negligible (
furthermore in comparison to a rewrite).
---
src/backend/catalog/index.c | 2 +-
src/backend/commands/cluster.c | 5 +
src/backend/commands/tablecmds.c | 6 +
src/backend/utils/activity/pgstat_relation.c | 391 ++++++++++++++++++-
src/backend/utils/activity/pgstat_xact.c | 25 +-
src/backend/utils/cache/relcache.c | 6 +
src/include/pgstat.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 424 insertions(+), 17 deletions(-)
92.8% src/backend/utils/activity/
4.9% src/backend/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 43de42ce39e..88ef00609db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1793,7 +1793,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId);
/* copy over statistics from old to new index */
- pgstat_copy_relation_stats(newClassRel, oldClassRel);
+ pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 60a4617a585..3a446f32921 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1196,6 +1196,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
rel1 = relation_open(r1, NoLock);
rel2 = relation_open(r2, NoLock);
+
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind))
+ pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator);
+
rel2->rd_createSubid = rel1->rd_createSubid;
rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid;
rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index df1ba112b35..6244d04ab12 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -16896,6 +16896,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
Oid reltoastrelid;
RelFileNumber newrelfilenumber;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator;
List *reltoastidxids = NIL;
ListCell *lc;
@@ -16934,6 +16935,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
newrlocator = rel->rd_locator;
newrlocator.relNumber = newrelfilenumber;
newrlocator.spcOid = newTableSpace;
+ oldrlocator = rel->rd_locator;
/* hand off to AM to actually create new rel storage and copy the data */
if (rel->rd_rel->relkind == RELKIND_INDEX)
@@ -16946,6 +16948,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
table_relation_copy_data(rel, &newrlocator);
}
+ /* mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Update the pg_class row.
*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 89bf0cbed56..4d929972fab 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -30,6 +30,19 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
+/* Pending rewrite operations for stats copying */
+typedef struct PgStat_PendingRewrite
+{
+ RelFileLocator old_locator;
+ RelFileLocator new_locator;
+ RelFileLocator original_locator;
+ int nest_level; /* Transaction nesting level where rewrite
+ * occurred */
+ struct PgStat_PendingRewrite *next;
+} PgStat_PendingRewrite;
+
+/* The pending rewrites list for current transaction */
+static PgStat_PendingRewrite *pending_rewrites = NULL;
/* Record that's written to 2PC state file when pgstat state is persisted */
typedef struct TwoPhasePgStatRecord
@@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter deleted_pre_truncdrop;
RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
+ RelFileLocator rewrite_old_locator;
+ int rewrite_nest_level;
} TwoPhasePgStatRecord;
@@ -54,27 +69,70 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
/*
- * Copy stats between relations. This is used for things like REINDEX
+ * Copy stats between RelFileLocator. This is used for things like REINDEX
* CONCURRENTLY.
*/
void
-pgstat_copy_relation_stats(Relation dst, Relation src)
+pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment)
{
PgStat_StatTabEntry *srcstats;
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src));
+ srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION,
+ src.dbOid,
+ RelFileLocatorToPgStatObjid(src));
if (!srcstats)
return;
dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
- dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(dst),
+ dst.dbOid,
+ RelFileLocatorToPgStatObjid(dst),
false);
dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats;
- dstshstats->stats = *srcstats;
+
+ if (!increment)
+ dstshstats->stats = *srcstats;
+ else
+ {
+ /* Increment those statistics */
+#define RELFSTAT_ACC(fld, stats_to_add) \
+ (dstshstats->stats.fld += stats_to_add->fld)
+ RELFSTAT_ACC(numscans, srcstats);
+ RELFSTAT_ACC(tuples_returned, srcstats);
+ RELFSTAT_ACC(tuples_fetched, srcstats);
+ RELFSTAT_ACC(tuples_inserted, srcstats);
+ RELFSTAT_ACC(tuples_updated, srcstats);
+ RELFSTAT_ACC(tuples_deleted, srcstats);
+ RELFSTAT_ACC(tuples_hot_updated, srcstats);
+ RELFSTAT_ACC(tuples_newpage_updated, srcstats);
+ RELFSTAT_ACC(live_tuples, srcstats);
+ RELFSTAT_ACC(dead_tuples, srcstats);
+ RELFSTAT_ACC(mod_since_analyze, srcstats);
+ RELFSTAT_ACC(ins_since_vacuum, srcstats);
+ RELFSTAT_ACC(blocks_fetched, srcstats);
+ RELFSTAT_ACC(blocks_hit, srcstats);
+ RELFSTAT_ACC(vacuum_count, srcstats);
+ RELFSTAT_ACC(autovacuum_count, srcstats);
+ RELFSTAT_ACC(analyze_count, srcstats);
+ RELFSTAT_ACC(autoanalyze_count, srcstats);
+ RELFSTAT_ACC(total_vacuum_time, srcstats);
+ RELFSTAT_ACC(total_autovacuum_time, srcstats);
+ RELFSTAT_ACC(total_analyze_time, srcstats);
+ RELFSTAT_ACC(total_autoanalyze_time, srcstats);
+#undef RELFSTAT_ACC
+
+ /* Replace those statistics */
+#define RELFSTAT_REP(fld, stats_to_rep) \
+ (dstshstats->stats.fld = stats_to_rep->fld)
+ RELFSTAT_REP(lastscan, srcstats);
+ RELFSTAT_REP(last_vacuum_time, srcstats);
+ RELFSTAT_REP(last_autovacuum_time, srcstats);
+ RELFSTAT_REP(last_analyze_time, srcstats);
+ RELFSTAT_REP(last_autoanalyze_time, srcstats);
+#undef RELFSTAT_REP
+ }
pgstat_unlock_entry(dst_ref);
}
@@ -136,6 +194,7 @@ void
pgstat_assoc_relation(Relation rel)
{
RelFileLocator locator;
+ PgStat_TableStatus *pgstat_info;
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
@@ -164,14 +223,54 @@ pgstat_assoc_relation(Relation rel)
locator.relNumber = rel->rd_id;
}
+ /*
+ * If this relation was rewritten during the current transaction we may be
+ * reopening it with its new RelFileLocator. In that case, continue using
+ * the stats entry associated with the old locator rather than creating a
+ * new one. This ensures all stats from before and after the rewrite are
+ * tracked in a single entry which will be properly copied to the new
+ * locator at transaction commit.
+ */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (locator.dbOid == rewrite->new_locator.dbOid &&
+ locator.spcOid == rewrite->new_locator.spcOid &&
+ locator.relNumber == rewrite->new_locator.relNumber)
+ {
+ pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator);
+ goto found_entry;
+ }
+ }
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(locator);
+ pgstat_info = pgstat_prep_relation_pending(locator);
+
+found_entry:
+ rel->pgstat_info = pgstat_info;
+
+ /*
+ * For relations stats, we key by physical file location, not by relation
+ * OID. This means during operations like ALTER TYPE it's possible that
+ * the relation OID changes but the relfilenode stays the same (no actual
+ * rewrite needed). Unlink the old relation first.
+ */
+ if (pgstat_info->relation != NULL &&
+ pgstat_info->relation != rel)
+ {
+ pgstat_info->relation->pgstat_info = NULL;
+ pgstat_info->relation = NULL;
+ }
/* don't allow link a stats to multiple relcache entries */
- Assert(rel->pgstat_info->relation == NULL);
+ Assert(pgstat_info->relation == NULL);
/* mark this relation as the owner */
- rel->pgstat_info->relation = rel;
+ pgstat_info->relation = rel;
}
/*
@@ -214,14 +313,37 @@ pgstat_drop_relation(Relation rel)
{
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ bool skip_transactional_drop = false;
/* don't track stats for relations without storage */
if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
return;
- pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_locator.dbOid,
- RelFileLocatorToPgStatObjid(rel->rd_locator));
+ /* Check if this drop is part of a pending rewrite */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid &&
+ rel->rd_locator.spcOid == rewrite->old_locator.spcOid &&
+ rel->rd_locator.relNumber == rewrite->old_locator.relNumber)
+ {
+ skip_transactional_drop = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * If it is part of a rewrite, drop its stats later, for example in
+ * AtEOXact_PgStat_Relations(), so skip it here.
+ */
+ if (!skip_transactional_drop)
+ pgstat_drop_transactional(PGSTAT_KIND_RELATION,
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -666,6 +788,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit)
}
tabstat->trans = NULL;
}
+
+ /* preserve the stats in case of rewrite */
+ if (isCommit && pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *prev = NULL;
+ PgStat_PendingRewrite *current = pending_rewrites;
+ PgStat_PendingRewrite *next;
+
+ /* reverse the rewrites list to process in chronological order */
+ while (current != NULL)
+ {
+ next = current->next;
+ current->next = prev;
+ prev = current;
+ current = next;
+ }
+
+ /* now process rewrites in chronological order */
+ for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ pgstat_copy_relation_stats(rewrite->new_locator,
+ rewrite->old_locator, true);
+
+ /* drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
+ }
+
+ pending_rewrites = NULL;
}
/*
@@ -681,6 +845,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in
PgStat_TableXactStatus *trans;
PgStat_TableXactStatus *next_trans;
+ /*
+ * If we don't commit then remove the associated rewrites if any, to keep
+ * the rewrite chain in sync with what will be eventually committed.
+ */
+ if (!isCommit)
+ {
+ PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites;
+
+ while (*rewrite_ptr != NULL)
+ {
+ if ((*rewrite_ptr)->nest_level >= nestDepth)
+ {
+ PgStat_PendingRewrite *to_remove = *rewrite_ptr;
+
+ *rewrite_ptr = (*rewrite_ptr)->next;
+ pfree(to_remove);
+ }
+ else
+ {
+ rewrite_ptr = &((*rewrite_ptr)->next);
+ }
+ }
+ }
+
for (trans = xact_state->first; trans != NULL; trans = next_trans)
{
PgStat_TableStatus *tabstat;
@@ -760,11 +948,19 @@ void
AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
{
PgStat_TableXactStatus *trans;
+ PgStat_PendingRewrite *rewrite;
+ /*
+ * For each tabstat, find its matching rewrite and remove it from the
+ * pending rewrites list. This way, after processing all tabstats, pending
+ * rewrites will only contain rewrite only transactions.
+ */
for (trans = xact_state->first; trans != NULL; trans = trans->next)
{
PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY;
TwoPhasePgStatRecord record;
+ PgStat_PendingRewrite **rewrite_ptr;
+ bool found_rewrite = false;
Assert(trans->nest_level == 1);
Assert(trans->upper == NULL);
@@ -784,10 +980,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.locator = tabstat->locator;
record.truncdropped = trans->truncdropped;
+ record.rewrite_nest_level = 0;
+
+ /*
+ * Look for a matching rewrite and remove it from pending rewrites. We
+ * check three possible matches:
+ *
+ * The new_locator when stats have been added after the rewrite. The
+ * old_locator when stats have been added before the rewrite but not
+ * after. The original_locator when this tabstat is part of a rewrite
+ * chain.
+ */
+ rewrite_ptr = &pending_rewrites;
+ while (*rewrite_ptr != NULL)
+ {
+ rewrite = *rewrite_ptr;
+
+ if ((record.locator.dbOid == rewrite->new_locator.dbOid &&
+ record.locator.spcOid == rewrite->new_locator.spcOid &&
+ record.locator.relNumber == rewrite->new_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->old_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->old_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->old_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->original_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->original_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->original_locator.relNumber))
+ {
+ /*
+ * Found matching rewrite. Record the rewrite information and
+ * remove this rewrite from the list since it's now handled.
+ */
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.locator = rewrite->new_locator;
+ found_rewrite = true;
+
+ /* Remove from pending_rewrites list */
+ *rewrite_ptr = rewrite->next;
+ pfree(rewrite);
+ break;
+ }
+ else
+ {
+ /* Move to next rewrite in the list */
+ rewrite_ptr = &(rewrite->next);
+ }
+ }
+
+ /* If no rewrite found, clear the rewrite fields */
+ if (!found_rewrite)
+ {
+ memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator));
+ }
+
+ RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
+ &record, sizeof(TwoPhasePgStatRecord));
+ }
+
+ /*
+ * Now process any rewrites still pending. These are rewrite only
+ * transactions. We need to preserve their stats even though there's no
+ * tabstat entry for them.
+ */
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ TwoPhasePgStatRecord record;
+
+ memset(&record, 0, sizeof(TwoPhasePgStatRecord));
+ record.locator = rewrite->new_locator;
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.truncdropped = false;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
&record, sizeof(TwoPhasePgStatRecord));
}
+
+ pending_rewrites = NULL;
}
/*
@@ -810,6 +1079,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
tabstat = trans->parent;
tabstat->trans = NULL;
}
+
+ pending_rewrites = NULL;
}
/*
@@ -845,6 +1116,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
pgstat_info->counts.changed_tuples +=
rec->tuples_inserted + rec->tuples_updated +
rec->tuples_deleted;
+
+ if (rec->rewrite_nest_level > 0)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ /* Flush any pending stats for old locator first */
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ /* Copy stats from old to new locator */
+ pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator,
+ true);
+
+ /* Drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
}
/*
@@ -859,9 +1153,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
{
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
+ RelFileLocator target_locator;
+
+ /*
+ * For aborted transactions with rewrites (like TRUNCATE), we need to
+ * restore stats to the old locator, not the new one. The new locator
+ * should be dropped since the rewrite is being rolled back.
+ */
+ if (rec->rewrite_nest_level > 0)
+ {
+ /* Use the old locator */
+ target_locator = rec->rewrite_old_locator;
+ }
+ else
+ {
+ /* No rewrite, use the original locator */
+ target_locator = rec->locator;
+ }
/* Find or create a tabstat entry for the target locator */
- pgstat_info = pgstat_prep_relation_pending(rec->locator);
+ pgstat_info = pgstat_prep_relation_pending(target_locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -916,7 +1227,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
+ TimestampTz t;
+
+ /*
+ * Checking the transaction state due to the flush call in
+ * pgstat_twophase_postcommit() that would break the assertion on the
+ * state in GetCurrentTransactionStopTimestamp().
+ */
+ if (!IsTransactionState())
+ t = GetCurrentTransactionStopTimestamp();
+ else
+ t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
@@ -1167,3 +1488,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
ReleaseSysCache(tuple);
return result;
}
+
+/*
+ * Mark that a relation rewrite has occurred, preserving the original locator
+ * so stats can be copied at transaction commit.
+ */
+void
+pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator)
+{
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *existing;
+ RelFileLocator original_locator = old_locator;
+
+ for (existing = pending_rewrites; existing != NULL; existing = existing->next)
+ {
+ if (old_locator.dbOid == existing->new_locator.dbOid &&
+ old_locator.spcOid == existing->new_locator.spcOid &&
+ old_locator.relNumber == existing->new_locator.relNumber)
+ {
+ original_locator = existing->original_locator;
+ break;
+ }
+ }
+
+ /* Allocate in TopTransactionContext memory context */
+ rewrite = MemoryContextAlloc(TopTransactionContext,
+ sizeof(PgStat_PendingRewrite));
+
+ rewrite->old_locator = old_locator;
+ rewrite->new_locator = new_locator;
+ rewrite->original_locator = original_locator;
+ rewrite->nest_level = GetCurrentTransactionNestLevel();
+
+ /* Add to the list */
+ rewrite->next = pending_rewrites;
+ pending_rewrites = rewrite;
+}
+
+void
+pgstat_clear_rewrite(void)
+{
+ pending_rewrites = NULL;
+}
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index 5e2d69e6297..8ed8f5317f3 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel)
}
pgStatXactStack = NULL;
+ pgstat_clear_rewrite();
+
/* Make sure any stats snapshot is thrown away */
pgstat_clear_snapshot();
}
@@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo
void
pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid)
{
- if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL))
+ PgStat_EntryRef *entry_ref;
+
+ entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL);
+
+ if (entry_ref)
{
+ /*
+ * For relations stats, we key by physical file location, not by
+ * relation OID. This means during operations like ALTER TYPE where
+ * the relation OID changes but the relfilenode stays the same (no
+ * actual rewrite needed), we'll find an existing entry.
+ *
+ * This is expected behavior, we want to preserve stats across the
+ * catalog change. Simply reset and recreate the entry for the new
+ * relation OID without warning.
+ */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ pgstat_reset(kind, dboid, objid);
+ create_drop_transactional_internal(kind, dboid, objid, true);
+ return;
+ }
+
ereport(WARNING,
errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64,
(pgstat_get_kind_info(kind))->name, dboid,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 6b634c9fff1..692d084acb2 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -85,6 +85,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/relmapper.h"
#include "utils/resowner.h"
#include "utils/snapmgr.h"
@@ -3775,6 +3776,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
MultiXactId minmulti = InvalidMultiXactId;
TransactionId freezeXid = InvalidTransactionId;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator = relation->rd_locator;
if (!IsBinaryUpgrade)
{
@@ -3946,6 +3948,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
table_close(pg_class, RowExclusiveLock);
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Make the pg_class row change or relation map change visible. This will
* cause the relcache entry to get updated, too.
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index e9e4d32c3b8..a9a4fa9f8f2 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -669,7 +669,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id);
extern void pgstat_create_relation(Relation rel);
extern void pgstat_drop_relation(Relation rel);
-extern void pgstat_copy_relation_stats(Relation dst, Relation src);
+extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment);
extern void pgstat_init_relation(Relation rel);
extern void pgstat_assoc_relation(Relation rel);
@@ -681,6 +681,9 @@ extern void pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_mark_rewrite(RelFileLocator old_locator,
+ RelFileLocator new_locator);
+extern void pgstat_clear_rewrite(void);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 241945734ec..05b1380c08c 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2279,6 +2279,7 @@ PgStat_KindInfo
PgStat_LocalState
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
+PgStat_PendingRewrite
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
--zLA8DXafBEP889wv--
^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v10 2/2] handle relation statistics correctly during rewrites
@ 2025-11-04 13:52 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Bertrand Drouvot @ 2025-11-04 13:52 UTC (permalink / raw)
Now that PGSTAT_KIND_RELATION is keyed by refilenode, we need to handle rewrites.
To do so, this patch:
- Adds PgStat_PendingRewrite, a new struct to track rewrite operations within
a transaction, storing the old locator, new locator, and original locator (for
rewrite chains). This allows stats to be copied from the original location to
the final location at commit time.
- Adds a new function, pgstat_mark_rewrite(), called when a table rewrite begins.
It records the rewrite operation in a local list and detects rewrite chains by
checking if the old_locator matches any existing new_locator, preserving the
chain's original_locator.
- Modifies pgstat_copy_relation_stats(), to accept RelFileLocators instead of
Relations, with a new increment parameter to accumulate stats (needed for rewrite
chains with DML between rewrites).
- Ensures that AtEOXact_PgStat_Relations(), AtPrepare_PgStat_Relations(),
pgstat_twophase_postcommit()/postabort() pgstat_drop_relation() handle the
PgStat_PendingRewrite list correctly.
Note that due to the new flush call in pgstat_twophase_postcommit() we can not
call GetCurrentTransactionStopTimestamp() in pgstat_relation_flush_cb(). So,
adding a check to handle this special case and call GetCurrentTimestamp() instead.
Note that we'd call GetCurrentTimestamp() only if there is a rewrite, so that
the GetCurrentTimestamp() extra cost should be negligible. Another solution
could be to trigger the flush from FinishPreparedTransaction() but that's not
worth the extra complexity.
The new pending_rewrites list is traversed in multiple places. The overhead
should be negligible in comparison to a rewrite and the list should not contain
a lot of rewrites in practice.
The pending_rewrites list is traversed in multiple places. In typical usage,
the list will contain only a few entries so the traversal cost is negligible (
furthermore in comparison to a rewrite).
---
src/backend/catalog/index.c | 2 +-
src/backend/commands/cluster.c | 5 +
src/backend/commands/tablecmds.c | 6 +
src/backend/utils/activity/pgstat_relation.c | 391 ++++++++++++++++++-
src/backend/utils/activity/pgstat_xact.c | 25 +-
src/backend/utils/cache/relcache.c | 6 +
src/include/pgstat.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 424 insertions(+), 17 deletions(-)
92.8% src/backend/utils/activity/
4.9% src/backend/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 8dea58ad96b..b71925a22c3 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1795,7 +1795,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId);
/* copy over statistics from old to new index */
- pgstat_copy_relation_stats(newClassRel, oldClassRel);
+ pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 2120c85ccb4..6155b12afab 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1196,6 +1196,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
rel1 = relation_open(r1, NoLock);
rel2 = relation_open(r2, NoLock);
+
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind))
+ pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator);
+
rel2->rd_createSubid = rel1->rd_createSubid;
rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid;
rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 6b1a00ed477..9de70f321ed 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -16884,6 +16884,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
Oid reltoastrelid;
RelFileNumber newrelfilenumber;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator;
List *reltoastidxids = NIL;
ListCell *lc;
@@ -16922,6 +16923,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
newrlocator = rel->rd_locator;
newrlocator.relNumber = newrelfilenumber;
newrlocator.spcOid = newTableSpace;
+ oldrlocator = rel->rd_locator;
/* hand off to AM to actually create new rel storage and copy the data */
if (rel->rd_rel->relkind == RELKIND_INDEX)
@@ -16934,6 +16936,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
table_relation_copy_data(rel, &newrlocator);
}
+ /* mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Update the pg_class row.
*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index a267e93f8be..123fb50d98f 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -30,6 +30,19 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
+/* Pending rewrite operations for stats copying */
+typedef struct PgStat_PendingRewrite
+{
+ RelFileLocator old_locator;
+ RelFileLocator new_locator;
+ RelFileLocator original_locator;
+ int nest_level; /* Transaction nesting level where rewrite
+ * occurred */
+ struct PgStat_PendingRewrite *next;
+} PgStat_PendingRewrite;
+
+/* The pending rewrites list for current transaction */
+static PgStat_PendingRewrite *pending_rewrites = NULL;
/* Record that's written to 2PC state file when pgstat state is persisted */
typedef struct TwoPhasePgStatRecord
@@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter deleted_pre_truncdrop;
RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
+ RelFileLocator rewrite_old_locator;
+ int rewrite_nest_level;
} TwoPhasePgStatRecord;
@@ -54,27 +69,70 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
/*
- * Copy stats between relations. This is used for things like REINDEX
+ * Copy stats between RelFileLocator. This is used for things like REINDEX
* CONCURRENTLY.
*/
void
-pgstat_copy_relation_stats(Relation dst, Relation src)
+pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment)
{
PgStat_StatTabEntry *srcstats;
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src));
+ srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION,
+ src.dbOid,
+ RelFileLocatorToPgStatObjid(src));
if (!srcstats)
return;
dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
- dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(dst),
+ dst.dbOid,
+ RelFileLocatorToPgStatObjid(dst),
false);
dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats;
- dstshstats->stats = *srcstats;
+
+ if (!increment)
+ dstshstats->stats = *srcstats;
+ else
+ {
+ /* Increment those statistics */
+#define RELFSTAT_ACC(fld, stats_to_add) \
+ (dstshstats->stats.fld += stats_to_add->fld)
+ RELFSTAT_ACC(numscans, srcstats);
+ RELFSTAT_ACC(tuples_returned, srcstats);
+ RELFSTAT_ACC(tuples_fetched, srcstats);
+ RELFSTAT_ACC(tuples_inserted, srcstats);
+ RELFSTAT_ACC(tuples_updated, srcstats);
+ RELFSTAT_ACC(tuples_deleted, srcstats);
+ RELFSTAT_ACC(tuples_hot_updated, srcstats);
+ RELFSTAT_ACC(tuples_newpage_updated, srcstats);
+ RELFSTAT_ACC(live_tuples, srcstats);
+ RELFSTAT_ACC(dead_tuples, srcstats);
+ RELFSTAT_ACC(mod_since_analyze, srcstats);
+ RELFSTAT_ACC(ins_since_vacuum, srcstats);
+ RELFSTAT_ACC(blocks_fetched, srcstats);
+ RELFSTAT_ACC(blocks_hit, srcstats);
+ RELFSTAT_ACC(vacuum_count, srcstats);
+ RELFSTAT_ACC(autovacuum_count, srcstats);
+ RELFSTAT_ACC(analyze_count, srcstats);
+ RELFSTAT_ACC(autoanalyze_count, srcstats);
+ RELFSTAT_ACC(total_vacuum_time, srcstats);
+ RELFSTAT_ACC(total_autovacuum_time, srcstats);
+ RELFSTAT_ACC(total_analyze_time, srcstats);
+ RELFSTAT_ACC(total_autoanalyze_time, srcstats);
+#undef RELFSTAT_ACC
+
+ /* Replace those statistics */
+#define RELFSTAT_REP(fld, stats_to_rep) \
+ (dstshstats->stats.fld = stats_to_rep->fld)
+ RELFSTAT_REP(lastscan, srcstats);
+ RELFSTAT_REP(last_vacuum_time, srcstats);
+ RELFSTAT_REP(last_autovacuum_time, srcstats);
+ RELFSTAT_REP(last_analyze_time, srcstats);
+ RELFSTAT_REP(last_autoanalyze_time, srcstats);
+#undef RELFSTAT_REP
+ }
pgstat_unlock_entry(dst_ref);
}
@@ -136,6 +194,7 @@ void
pgstat_assoc_relation(Relation rel)
{
RelFileLocator locator;
+ PgStat_TableStatus *pgstat_info;
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
@@ -164,14 +223,54 @@ pgstat_assoc_relation(Relation rel)
locator.relNumber = rel->rd_id;
}
+ /*
+ * If this relation was rewritten during the current transaction we may be
+ * reopening it with its new RelFileLocator. In that case, continue using
+ * the stats entry associated with the old locator rather than creating a
+ * new one. This ensures all stats from before and after the rewrite are
+ * tracked in a single entry which will be properly copied to the new
+ * locator at transaction commit.
+ */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (locator.dbOid == rewrite->new_locator.dbOid &&
+ locator.spcOid == rewrite->new_locator.spcOid &&
+ locator.relNumber == rewrite->new_locator.relNumber)
+ {
+ pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator);
+ goto found_entry;
+ }
+ }
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(locator);
+ pgstat_info = pgstat_prep_relation_pending(locator);
+
+found_entry:
+ rel->pgstat_info = pgstat_info;
+
+ /*
+ * For relations stats, we key by physical file location, not by relation
+ * OID. This means during operations like ALTER TYPE it's possible that
+ * the relation OID changes but the relfilenode stays the same (no actual
+ * rewrite needed). Unlink the old relation first.
+ */
+ if (pgstat_info->relation != NULL &&
+ pgstat_info->relation != rel)
+ {
+ pgstat_info->relation->pgstat_info = NULL;
+ pgstat_info->relation = NULL;
+ }
/* don't allow link a stats to multiple relcache entries */
- Assert(rel->pgstat_info->relation == NULL);
+ Assert(pgstat_info->relation == NULL);
/* mark this relation as the owner */
- rel->pgstat_info->relation = rel;
+ pgstat_info->relation = rel;
}
/*
@@ -214,14 +313,37 @@ pgstat_drop_relation(Relation rel)
{
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ bool skip_transactional_drop = false;
/* don't track stats for relations without storage */
if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
return;
- pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_locator.dbOid,
- RelFileLocatorToPgStatObjid(rel->rd_locator));
+ /* Check if this drop is part of a pending rewrite */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid &&
+ rel->rd_locator.spcOid == rewrite->old_locator.spcOid &&
+ rel->rd_locator.relNumber == rewrite->old_locator.relNumber)
+ {
+ skip_transactional_drop = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * If it is part of a rewrite, drop its stats later, for example in
+ * AtEOXact_PgStat_Relations(), so skip it here.
+ */
+ if (!skip_transactional_drop)
+ pgstat_drop_transactional(PGSTAT_KIND_RELATION,
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -666,6 +788,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit)
}
tabstat->trans = NULL;
}
+
+ /* preserve the stats in case of rewrite */
+ if (isCommit && pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *prev = NULL;
+ PgStat_PendingRewrite *current = pending_rewrites;
+ PgStat_PendingRewrite *next;
+
+ /* reverse the rewrites list to process in chronological order */
+ while (current != NULL)
+ {
+ next = current->next;
+ current->next = prev;
+ prev = current;
+ current = next;
+ }
+
+ /* now process rewrites in chronological order */
+ for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ pgstat_copy_relation_stats(rewrite->new_locator,
+ rewrite->old_locator, true);
+
+ /* drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
+ }
+
+ pending_rewrites = NULL;
}
/*
@@ -681,6 +845,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in
PgStat_TableXactStatus *trans;
PgStat_TableXactStatus *next_trans;
+ /*
+ * If we don't commit then remove the associated rewrites if any, to keep
+ * the rewrite chain in sync with what will be eventually committed.
+ */
+ if (!isCommit)
+ {
+ PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites;
+
+ while (*rewrite_ptr != NULL)
+ {
+ if ((*rewrite_ptr)->nest_level >= nestDepth)
+ {
+ PgStat_PendingRewrite *to_remove = *rewrite_ptr;
+
+ *rewrite_ptr = (*rewrite_ptr)->next;
+ pfree(to_remove);
+ }
+ else
+ {
+ rewrite_ptr = &((*rewrite_ptr)->next);
+ }
+ }
+ }
+
for (trans = xact_state->first; trans != NULL; trans = next_trans)
{
PgStat_TableStatus *tabstat;
@@ -760,11 +948,19 @@ void
AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
{
PgStat_TableXactStatus *trans;
+ PgStat_PendingRewrite *rewrite;
+ /*
+ * For each tabstat, find its matching rewrite and remove it from the
+ * pending rewrites list. This way, after processing all tabstats, pending
+ * rewrites will only contain rewrite only transactions.
+ */
for (trans = xact_state->first; trans != NULL; trans = trans->next)
{
PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY;
TwoPhasePgStatRecord record;
+ PgStat_PendingRewrite **rewrite_ptr;
+ bool found_rewrite = false;
Assert(trans->nest_level == 1);
Assert(trans->upper == NULL);
@@ -784,10 +980,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.locator = tabstat->locator;
record.truncdropped = trans->truncdropped;
+ record.rewrite_nest_level = 0;
+
+ /*
+ * Look for a matching rewrite and remove it from pending rewrites. We
+ * check three possible matches:
+ *
+ * The new_locator when stats have been added after the rewrite. The
+ * old_locator when stats have been added before the rewrite but not
+ * after. The original_locator when this tabstat is part of a rewrite
+ * chain.
+ */
+ rewrite_ptr = &pending_rewrites;
+ while (*rewrite_ptr != NULL)
+ {
+ rewrite = *rewrite_ptr;
+
+ if ((record.locator.dbOid == rewrite->new_locator.dbOid &&
+ record.locator.spcOid == rewrite->new_locator.spcOid &&
+ record.locator.relNumber == rewrite->new_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->old_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->old_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->old_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->original_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->original_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->original_locator.relNumber))
+ {
+ /*
+ * Found matching rewrite. Record the rewrite information and
+ * remove this rewrite from the list since it's now handled.
+ */
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.locator = rewrite->new_locator;
+ found_rewrite = true;
+
+ /* Remove from pending_rewrites list */
+ *rewrite_ptr = rewrite->next;
+ pfree(rewrite);
+ break;
+ }
+ else
+ {
+ /* Move to next rewrite in the list */
+ rewrite_ptr = &(rewrite->next);
+ }
+ }
+
+ /* If no rewrite found, clear the rewrite fields */
+ if (!found_rewrite)
+ {
+ memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator));
+ }
+
+ RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
+ &record, sizeof(TwoPhasePgStatRecord));
+ }
+
+ /*
+ * Now process any rewrites still pending. These are rewrite only
+ * transactions. We need to preserve their stats even though there's no
+ * tabstat entry for them.
+ */
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ TwoPhasePgStatRecord record;
+
+ memset(&record, 0, sizeof(TwoPhasePgStatRecord));
+ record.locator = rewrite->new_locator;
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.truncdropped = false;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
&record, sizeof(TwoPhasePgStatRecord));
}
+
+ pending_rewrites = NULL;
}
/*
@@ -810,6 +1079,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
tabstat = trans->parent;
tabstat->trans = NULL;
}
+
+ pending_rewrites = NULL;
}
/*
@@ -845,6 +1116,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
pgstat_info->counts.changed_tuples +=
rec->tuples_inserted + rec->tuples_updated +
rec->tuples_deleted;
+
+ if (rec->rewrite_nest_level > 0)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ /* Flush any pending stats for old locator first */
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ /* Copy stats from old to new locator */
+ pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator,
+ true);
+
+ /* Drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
}
/*
@@ -859,9 +1153,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
{
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
+ RelFileLocator target_locator;
+
+ /*
+ * For aborted transactions with rewrites (like TRUNCATE), we need to
+ * restore stats to the old locator, not the new one. The new locator
+ * should be dropped since the rewrite is being rolled back.
+ */
+ if (rec->rewrite_nest_level > 0)
+ {
+ /* Use the old locator */
+ target_locator = rec->rewrite_old_locator;
+ }
+ else
+ {
+ /* No rewrite, use the original locator */
+ target_locator = rec->locator;
+ }
/* Find or create a tabstat entry for the target locator */
- pgstat_info = pgstat_prep_relation_pending(rec->locator);
+ pgstat_info = pgstat_prep_relation_pending(target_locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -916,7 +1227,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
+ TimestampTz t;
+
+ /*
+ * Checking the transaction state due to the flush call in
+ * pgstat_twophase_postcommit() that would break the assertion on the
+ * state in GetCurrentTransactionStopTimestamp().
+ */
+ if (!IsTransactionState())
+ t = GetCurrentTransactionStopTimestamp();
+ else
+ t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
@@ -1167,3 +1488,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
ReleaseSysCache(tuple);
return result;
}
+
+/*
+ * Mark that a relation rewrite has occurred, preserving the original locator
+ * so stats can be copied at transaction commit.
+ */
+void
+pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator)
+{
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *existing;
+ RelFileLocator original_locator = old_locator;
+
+ for (existing = pending_rewrites; existing != NULL; existing = existing->next)
+ {
+ if (old_locator.dbOid == existing->new_locator.dbOid &&
+ old_locator.spcOid == existing->new_locator.spcOid &&
+ old_locator.relNumber == existing->new_locator.relNumber)
+ {
+ original_locator = existing->original_locator;
+ break;
+ }
+ }
+
+ /* Allocate in TopTransactionContext memory context */
+ rewrite = MemoryContextAlloc(TopTransactionContext,
+ sizeof(PgStat_PendingRewrite));
+
+ rewrite->old_locator = old_locator;
+ rewrite->new_locator = new_locator;
+ rewrite->original_locator = original_locator;
+ rewrite->nest_level = GetCurrentTransactionNestLevel();
+
+ /* Add to the list */
+ rewrite->next = pending_rewrites;
+ pending_rewrites = rewrite;
+}
+
+void
+pgstat_clear_rewrite(void)
+{
+ pending_rewrites = NULL;
+}
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index bc9864bd8d9..f8cf3755ce2 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel)
}
pgStatXactStack = NULL;
+ pgstat_clear_rewrite();
+
/* Make sure any stats snapshot is thrown away */
pgstat_clear_snapshot();
}
@@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo
void
pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid)
{
- if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL))
+ PgStat_EntryRef *entry_ref;
+
+ entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL);
+
+ if (entry_ref)
{
+ /*
+ * For relations stats, we key by physical file location, not by
+ * relation OID. This means during operations like ALTER TYPE where
+ * the relation OID changes but the relfilenode stays the same (no
+ * actual rewrite needed), we'll find an existing entry.
+ *
+ * This is expected behavior, we want to preserve stats across the
+ * catalog change. Simply reset and recreate the entry for the new
+ * relation OID without warning.
+ */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ pgstat_reset(kind, dboid, objid);
+ create_drop_transactional_internal(kind, dboid, objid, true);
+ return;
+ }
+
ereport(WARNING,
errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64,
(pgstat_get_kind_info(kind))->name, dboid,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 2d0cb7bcfd4..c98e5c51d63 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -85,6 +85,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/relmapper.h"
#include "utils/resowner.h"
#include "utils/snapmgr.h"
@@ -3780,6 +3781,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
MultiXactId minmulti = InvalidMultiXactId;
TransactionId freezeXid = InvalidTransactionId;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator = relation->rd_locator;
if (!IsBinaryUpgrade)
{
@@ -3951,6 +3953,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
table_close(pg_class, RowExclusiveLock);
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Make the pg_class row change or relation map change visible. This will
* cause the relcache entry to get updated, too.
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 3102f86aa24..8750c025bbe 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -670,7 +670,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id);
extern void pgstat_create_relation(Relation rel);
extern void pgstat_drop_relation(Relation rel);
-extern void pgstat_copy_relation_stats(Relation dst, Relation src);
+extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment);
extern void pgstat_init_relation(Relation rel);
extern void pgstat_assoc_relation(Relation rel);
@@ -682,6 +682,9 @@ extern void pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_mark_rewrite(RelFileLocator old_locator,
+ RelFileLocator new_locator);
+extern void pgstat_clear_rewrite(void);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 04845d5e680..3184e9c464b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2269,6 +2269,7 @@ PgStat_KindInfo
PgStat_LocalState
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
+PgStat_PendingRewrite
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
--xJsrX25x3tm7Tn81--
^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v8 2/2] handle relation statistics correctly during rewrites
@ 2025-11-04 13:52 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Bertrand Drouvot @ 2025-11-04 13:52 UTC (permalink / raw)
Now that PGSTAT_KIND_RELATION is keyed by refilenode, we need to handle rewrites.
To do so, this patch:
- Adds PgStat_PendingRewrite, a new struct to track rewrite operations within
a transaction, storing the old locator, new locator, and original locator (for
rewrite chains). This allows stats to be copied from the original location to
the final location at commit time.
- Adds a new function, pgstat_mark_rewrite(), called when a table rewrite begins.
It records the rewrite operation in a local list and detects rewrite chains by
checking if the old_locator matches any existing new_locator, preserving the
chain's original_locator.
- Modifies pgstat_copy_relation_stats(), to accept RelFileLocators instead of
Relations, with a new increment parameter to accumulate stats (needed for rewrite
chains with DML between rewrites).
- Ensures that AtEOXact_PgStat_Relations(), AtPrepare_PgStat_Relations(),
pgstat_twophase_postcommit()/postabort() pgstat_drop_relation() handle the
PgStat_PendingRewrite list correctly.
Note that due to the new flush call in pgstat_twophase_postcommit() we can not
call GetCurrentTransactionStopTimestamp() in pgstat_relation_flush_cb(). So,
adding a check to handle this special case and call GetCurrentTimestamp() instead.
Note that we'd call GetCurrentTimestamp() only if there is a rewrite, so that
the GetCurrentTimestamp() extra cost should be negligible. Another solution
could be to trigger the flush from FinishPreparedTransaction() but that's not
worth the extra complexity.
The new pending_rewrites list is traversed in multiple places. The overhead
should be negligible in comparison to a rewrite and the list should not contain
a lot of rewrites in practice.
The pending_rewrites list is traversed in multiple places. In typical usage,
the list will contain only a few entries so the traversal cost is negligible (
furthermore in comparison to a rewrite).
---
src/backend/catalog/index.c | 2 +-
src/backend/commands/cluster.c | 5 +
src/backend/commands/tablecmds.c | 6 +
src/backend/utils/activity/pgstat_relation.c | 391 ++++++++++++++++++-
src/backend/utils/activity/pgstat_xact.c | 25 +-
src/backend/utils/cache/relcache.c | 6 +
src/include/pgstat.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 424 insertions(+), 17 deletions(-)
92.8% src/backend/utils/activity/
4.9% src/backend/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 8dea58ad96b..b71925a22c3 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1795,7 +1795,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId);
/* copy over statistics from old to new index */
- pgstat_copy_relation_stats(newClassRel, oldClassRel);
+ pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 2120c85ccb4..6155b12afab 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1196,6 +1196,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
rel1 = relation_open(r1, NoLock);
rel2 = relation_open(r2, NoLock);
+
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind))
+ pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator);
+
rel2->rd_createSubid = rel1->rd_createSubid;
rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid;
rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 6b1a00ed477..9de70f321ed 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -16884,6 +16884,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
Oid reltoastrelid;
RelFileNumber newrelfilenumber;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator;
List *reltoastidxids = NIL;
ListCell *lc;
@@ -16922,6 +16923,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
newrlocator = rel->rd_locator;
newrlocator.relNumber = newrelfilenumber;
newrlocator.spcOid = newTableSpace;
+ oldrlocator = rel->rd_locator;
/* hand off to AM to actually create new rel storage and copy the data */
if (rel->rd_rel->relkind == RELKIND_INDEX)
@@ -16934,6 +16936,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
table_relation_copy_data(rel, &newrlocator);
}
+ /* mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Update the pg_class row.
*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index ee6f2eb2bdb..83103189fb5 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -30,6 +30,19 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
+/* Pending rewrite operations for stats copying */
+typedef struct PgStat_PendingRewrite
+{
+ RelFileLocator old_locator;
+ RelFileLocator new_locator;
+ RelFileLocator original_locator;
+ int nest_level; /* Transaction nesting level where rewrite
+ * occurred */
+ struct PgStat_PendingRewrite *next;
+} PgStat_PendingRewrite;
+
+/* The pending rewrites list for current transaction */
+static PgStat_PendingRewrite *pending_rewrites = NULL;
/* Record that's written to 2PC state file when pgstat state is persisted */
typedef struct TwoPhasePgStatRecord
@@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter deleted_pre_truncdrop;
RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
+ RelFileLocator rewrite_old_locator;
+ int rewrite_nest_level;
} TwoPhasePgStatRecord;
@@ -54,27 +69,70 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
/*
- * Copy stats between relations. This is used for things like REINDEX
+ * Copy stats between RelFileLocator. This is used for things like REINDEX
* CONCURRENTLY.
*/
void
-pgstat_copy_relation_stats(Relation dst, Relation src)
+pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment)
{
PgStat_StatTabEntry *srcstats;
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src));
+ srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION,
+ src.dbOid,
+ RelFileLocatorToPgStatObjid(src));
if (!srcstats)
return;
dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
- dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(dst),
+ dst.dbOid,
+ RelFileLocatorToPgStatObjid(dst),
false);
dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats;
- dstshstats->stats = *srcstats;
+
+ if (!increment)
+ dstshstats->stats = *srcstats;
+ else
+ {
+ /* Increment those statistics */
+#define RELFSTAT_ACC(fld, stats_to_add) \
+ (dstshstats->stats.fld += stats_to_add->fld)
+ RELFSTAT_ACC(numscans, srcstats);
+ RELFSTAT_ACC(tuples_returned, srcstats);
+ RELFSTAT_ACC(tuples_fetched, srcstats);
+ RELFSTAT_ACC(tuples_inserted, srcstats);
+ RELFSTAT_ACC(tuples_updated, srcstats);
+ RELFSTAT_ACC(tuples_deleted, srcstats);
+ RELFSTAT_ACC(tuples_hot_updated, srcstats);
+ RELFSTAT_ACC(tuples_newpage_updated, srcstats);
+ RELFSTAT_ACC(live_tuples, srcstats);
+ RELFSTAT_ACC(dead_tuples, srcstats);
+ RELFSTAT_ACC(mod_since_analyze, srcstats);
+ RELFSTAT_ACC(ins_since_vacuum, srcstats);
+ RELFSTAT_ACC(blocks_fetched, srcstats);
+ RELFSTAT_ACC(blocks_hit, srcstats);
+ RELFSTAT_ACC(vacuum_count, srcstats);
+ RELFSTAT_ACC(autovacuum_count, srcstats);
+ RELFSTAT_ACC(analyze_count, srcstats);
+ RELFSTAT_ACC(autoanalyze_count, srcstats);
+ RELFSTAT_ACC(total_vacuum_time, srcstats);
+ RELFSTAT_ACC(total_autovacuum_time, srcstats);
+ RELFSTAT_ACC(total_analyze_time, srcstats);
+ RELFSTAT_ACC(total_autoanalyze_time, srcstats);
+#undef RELFSTAT_ACC
+
+ /* Replace those statistics */
+#define RELFSTAT_REP(fld, stats_to_rep) \
+ (dstshstats->stats.fld = stats_to_rep->fld)
+ RELFSTAT_REP(lastscan, srcstats);
+ RELFSTAT_REP(last_vacuum_time, srcstats);
+ RELFSTAT_REP(last_autovacuum_time, srcstats);
+ RELFSTAT_REP(last_analyze_time, srcstats);
+ RELFSTAT_REP(last_autoanalyze_time, srcstats);
+#undef RELFSTAT_REP
+ }
pgstat_unlock_entry(dst_ref);
}
@@ -136,6 +194,7 @@ void
pgstat_assoc_relation(Relation rel)
{
RelFileLocator locator;
+ PgStat_TableStatus *pgstat_info;
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
@@ -165,14 +224,54 @@ pgstat_assoc_relation(Relation rel)
locator.relNumber = rel->rd_id;
}
+ /*
+ * If this relation was rewritten during the current transaction we may be
+ * reopening it with its new RelFileLocator. In that case, continue using
+ * the stats entry associated with the old locator rather than creating a
+ * new one. This ensures all stats from before and after the rewrite are
+ * tracked in a single entry which will be properly copied to the new
+ * locator at transaction commit.
+ */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (locator.dbOid == rewrite->new_locator.dbOid &&
+ locator.spcOid == rewrite->new_locator.spcOid &&
+ locator.relNumber == rewrite->new_locator.relNumber)
+ {
+ pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator);
+ goto found_entry;
+ }
+ }
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(locator);
+ pgstat_info = pgstat_prep_relation_pending(locator);
+
+found_entry:
+ rel->pgstat_info = pgstat_info;
+
+ /*
+ * For relations stats, we key by physical file location, not by relation
+ * OID. This means during operations like ALTER TYPE it's possible that
+ * the relation OID changes but the relfilenode stays the same (no actual
+ * rewrite needed). Unlink the old relation first.
+ */
+ if (pgstat_info->relation != NULL &&
+ pgstat_info->relation != rel)
+ {
+ pgstat_info->relation->pgstat_info = NULL;
+ pgstat_info->relation = NULL;
+ }
/* don't allow link a stats to multiple relcache entries */
- Assert(rel->pgstat_info->relation == NULL);
+ Assert(pgstat_info->relation == NULL);
/* mark this relation as the owner */
- rel->pgstat_info->relation = rel;
+ pgstat_info->relation = rel;
}
/*
@@ -215,14 +314,37 @@ pgstat_drop_relation(Relation rel)
{
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ bool skip_transactional_drop = false;
/* don't track stats for relations without storage */
if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
return;
- pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_locator.dbOid,
- RelFileLocatorToPgStatObjid(rel->rd_locator));
+ /* Check if this drop is part of a pending rewrite */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid &&
+ rel->rd_locator.spcOid == rewrite->old_locator.spcOid &&
+ rel->rd_locator.relNumber == rewrite->old_locator.relNumber)
+ {
+ skip_transactional_drop = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * If it is part of a rewrite, drop its stats later, for example in
+ * AtEOXact_PgStat_Relations(), so skip it here.
+ */
+ if (!skip_transactional_drop)
+ pgstat_drop_transactional(PGSTAT_KIND_RELATION,
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -660,6 +782,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit)
}
tabstat->trans = NULL;
}
+
+ /* preserve the stats in case of rewrite */
+ if (isCommit && pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *prev = NULL;
+ PgStat_PendingRewrite *current = pending_rewrites;
+ PgStat_PendingRewrite *next;
+
+ /* reverse the rewrites list to process in chronological order */
+ while (current != NULL)
+ {
+ next = current->next;
+ current->next = prev;
+ prev = current;
+ current = next;
+ }
+
+ /* now process rewrites in chronological order */
+ for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ pgstat_copy_relation_stats(rewrite->new_locator,
+ rewrite->old_locator, true);
+
+ /* drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
+ }
+
+ pending_rewrites = NULL;
}
/*
@@ -675,6 +839,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in
PgStat_TableXactStatus *trans;
PgStat_TableXactStatus *next_trans;
+ /*
+ * If we don't commit then remove the associated rewrites if any, to keep
+ * the rewrite chain in sync with what will be eventually committed.
+ */
+ if (!isCommit)
+ {
+ PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites;
+
+ while (*rewrite_ptr != NULL)
+ {
+ if ((*rewrite_ptr)->nest_level >= nestDepth)
+ {
+ PgStat_PendingRewrite *to_remove = *rewrite_ptr;
+
+ *rewrite_ptr = (*rewrite_ptr)->next;
+ pfree(to_remove);
+ }
+ else
+ {
+ rewrite_ptr = &((*rewrite_ptr)->next);
+ }
+ }
+ }
+
for (trans = xact_state->first; trans != NULL; trans = next_trans)
{
PgStat_TableStatus *tabstat;
@@ -754,11 +942,19 @@ void
AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
{
PgStat_TableXactStatus *trans;
+ PgStat_PendingRewrite *rewrite;
+ /*
+ * For each tabstat, find its matching rewrite and remove it from the
+ * pending rewrites list. This way, after processing all tabstats, pending
+ * rewrites will only contain rewrite only transactions.
+ */
for (trans = xact_state->first; trans != NULL; trans = trans->next)
{
PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY;
TwoPhasePgStatRecord record;
+ PgStat_PendingRewrite **rewrite_ptr;
+ bool found_rewrite = false;
Assert(trans->nest_level == 1);
Assert(trans->upper == NULL);
@@ -778,10 +974,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.locator = tabstat->locator;
record.truncdropped = trans->truncdropped;
+ record.rewrite_nest_level = 0;
+
+ /*
+ * Look for a matching rewrite and remove it from pending rewrites. We
+ * check three possible matches:
+ *
+ * The new_locator when stats have been added after the rewrite. The
+ * old_locator when stats have been added before the rewrite but not
+ * after. The original_locator when this tabstat is part of a rewrite
+ * chain.
+ */
+ rewrite_ptr = &pending_rewrites;
+ while (*rewrite_ptr != NULL)
+ {
+ rewrite = *rewrite_ptr;
+
+ if ((record.locator.dbOid == rewrite->new_locator.dbOid &&
+ record.locator.spcOid == rewrite->new_locator.spcOid &&
+ record.locator.relNumber == rewrite->new_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->old_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->old_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->old_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->original_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->original_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->original_locator.relNumber))
+ {
+ /*
+ * Found matching rewrite. Record the rewrite information and
+ * remove this rewrite from the list since it's now handled.
+ */
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.locator = rewrite->new_locator;
+ found_rewrite = true;
+
+ /* Remove from pending_rewrites list */
+ *rewrite_ptr = rewrite->next;
+ pfree(rewrite);
+ break;
+ }
+ else
+ {
+ /* Move to next rewrite in the list */
+ rewrite_ptr = &(rewrite->next);
+ }
+ }
+
+ /* If no rewrite found, clear the rewrite fields */
+ if (!found_rewrite)
+ {
+ memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator));
+ }
+
+ RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
+ &record, sizeof(TwoPhasePgStatRecord));
+ }
+
+ /*
+ * Now process any rewrites still pending. These are rewrite only
+ * transactions. We need to preserve their stats even though there's no
+ * tabstat entry for them.
+ */
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ TwoPhasePgStatRecord record;
+
+ memset(&record, 0, sizeof(TwoPhasePgStatRecord));
+ record.locator = rewrite->new_locator;
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.truncdropped = false;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
&record, sizeof(TwoPhasePgStatRecord));
}
+
+ pending_rewrites = NULL;
}
/*
@@ -804,6 +1073,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
tabstat = trans->parent;
tabstat->trans = NULL;
}
+
+ pending_rewrites = NULL;
}
/*
@@ -839,6 +1110,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
pgstat_info->counts.changed_tuples +=
rec->tuples_inserted + rec->tuples_updated +
rec->tuples_deleted;
+
+ if (rec->rewrite_nest_level > 0)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ /* Flush any pending stats for old locator first */
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ /* Copy stats from old to new locator */
+ pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator,
+ true);
+
+ /* Drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
}
/*
@@ -853,9 +1147,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
{
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
+ RelFileLocator target_locator;
+
+ /*
+ * For aborted transactions with rewrites (like TRUNCATE), we need to
+ * restore stats to the old locator, not the new one. The new locator
+ * should be dropped since the rewrite is being rolled back.
+ */
+ if (rec->rewrite_nest_level > 0)
+ {
+ /* Use the old locator */
+ target_locator = rec->rewrite_old_locator;
+ }
+ else
+ {
+ /* No rewrite, use the original locator */
+ target_locator = rec->locator;
+ }
/* Find or create a tabstat entry for the target locator */
- pgstat_info = pgstat_prep_relation_pending(rec->locator);
+ pgstat_info = pgstat_prep_relation_pending(target_locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -910,7 +1221,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
+ TimestampTz t;
+
+ /*
+ * Checking the transaction state due to the flush call in
+ * pgstat_twophase_postcommit() that would break the assertion on the
+ * state in GetCurrentTransactionStopTimestamp().
+ */
+ if (!IsTransactionState())
+ t = GetCurrentTransactionStopTimestamp();
+ else
+ t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
@@ -1162,3 +1483,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
ReleaseSysCache(tuple);
return result;
}
+
+/*
+ * Mark that a relation rewrite has occurred, preserving the original locator
+ * so stats can be copied at transaction commit.
+ */
+void
+pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator)
+{
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *existing;
+ RelFileLocator original_locator = old_locator;
+
+ for (existing = pending_rewrites; existing != NULL; existing = existing->next)
+ {
+ if (old_locator.dbOid == existing->new_locator.dbOid &&
+ old_locator.spcOid == existing->new_locator.spcOid &&
+ old_locator.relNumber == existing->new_locator.relNumber)
+ {
+ original_locator = existing->original_locator;
+ break;
+ }
+ }
+
+ /* Allocate in TopTransactionContext memory context */
+ rewrite = MemoryContextAlloc(TopTransactionContext,
+ sizeof(PgStat_PendingRewrite));
+
+ rewrite->old_locator = old_locator;
+ rewrite->new_locator = new_locator;
+ rewrite->original_locator = original_locator;
+ rewrite->nest_level = GetCurrentTransactionNestLevel();
+
+ /* Add to the list */
+ rewrite->next = pending_rewrites;
+ pending_rewrites = rewrite;
+}
+
+void
+pgstat_clear_rewrite(void)
+{
+ pending_rewrites = NULL;
+}
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index bc9864bd8d9..f8cf3755ce2 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel)
}
pgStatXactStack = NULL;
+ pgstat_clear_rewrite();
+
/* Make sure any stats snapshot is thrown away */
pgstat_clear_snapshot();
}
@@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo
void
pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid)
{
- if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL))
+ PgStat_EntryRef *entry_ref;
+
+ entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL);
+
+ if (entry_ref)
{
+ /*
+ * For relations stats, we key by physical file location, not by
+ * relation OID. This means during operations like ALTER TYPE where
+ * the relation OID changes but the relfilenode stays the same (no
+ * actual rewrite needed), we'll find an existing entry.
+ *
+ * This is expected behavior, we want to preserve stats across the
+ * catalog change. Simply reset and recreate the entry for the new
+ * relation OID without warning.
+ */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ pgstat_reset(kind, dboid, objid);
+ create_drop_transactional_internal(kind, dboid, objid, true);
+ return;
+ }
+
ereport(WARNING,
errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64,
(pgstat_get_kind_info(kind))->name, dboid,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 2d0cb7bcfd4..c98e5c51d63 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -85,6 +85,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/relmapper.h"
#include "utils/resowner.h"
#include "utils/snapmgr.h"
@@ -3780,6 +3781,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
MultiXactId minmulti = InvalidMultiXactId;
TransactionId freezeXid = InvalidTransactionId;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator = relation->rd_locator;
if (!IsBinaryUpgrade)
{
@@ -3951,6 +3953,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
table_close(pg_class, RowExclusiveLock);
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Make the pg_class row change or relation map change visible. This will
* cause the relcache entry to get updated, too.
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index c9e451094f3..33d530740d0 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -670,7 +670,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id);
extern void pgstat_create_relation(Relation rel);
extern void pgstat_drop_relation(Relation rel);
-extern void pgstat_copy_relation_stats(Relation dst, Relation src);
+extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment);
extern void pgstat_init_relation(Relation rel);
extern void pgstat_assoc_relation(Relation rel);
@@ -682,6 +682,9 @@ extern void pgstat_report_vacuum(RelFileLocator locator, PgStat_Counter livetupl
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_mark_rewrite(RelFileLocator old_locator,
+ RelFileLocator new_locator);
+extern void pgstat_clear_rewrite(void);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3451538565e..0ca3eae8026 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2268,6 +2268,7 @@ PgStat_KindInfo
PgStat_LocalState
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
+PgStat_PendingRewrite
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
--BHD/2LBqtxMXWADJ--
^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v9 2/2] handle relation statistics correctly during rewrites
@ 2025-11-04 13:52 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Bertrand Drouvot @ 2025-11-04 13:52 UTC (permalink / raw)
Now that PGSTAT_KIND_RELATION is keyed by refilenode, we need to handle rewrites.
To do so, this patch:
- Adds PgStat_PendingRewrite, a new struct to track rewrite operations within
a transaction, storing the old locator, new locator, and original locator (for
rewrite chains). This allows stats to be copied from the original location to
the final location at commit time.
- Adds a new function, pgstat_mark_rewrite(), called when a table rewrite begins.
It records the rewrite operation in a local list and detects rewrite chains by
checking if the old_locator matches any existing new_locator, preserving the
chain's original_locator.
- Modifies pgstat_copy_relation_stats(), to accept RelFileLocators instead of
Relations, with a new increment parameter to accumulate stats (needed for rewrite
chains with DML between rewrites).
- Ensures that AtEOXact_PgStat_Relations(), AtPrepare_PgStat_Relations(),
pgstat_twophase_postcommit()/postabort() pgstat_drop_relation() handle the
PgStat_PendingRewrite list correctly.
Note that due to the new flush call in pgstat_twophase_postcommit() we can not
call GetCurrentTransactionStopTimestamp() in pgstat_relation_flush_cb(). So,
adding a check to handle this special case and call GetCurrentTimestamp() instead.
Note that we'd call GetCurrentTimestamp() only if there is a rewrite, so that
the GetCurrentTimestamp() extra cost should be negligible. Another solution
could be to trigger the flush from FinishPreparedTransaction() but that's not
worth the extra complexity.
The new pending_rewrites list is traversed in multiple places. The overhead
should be negligible in comparison to a rewrite and the list should not contain
a lot of rewrites in practice.
The pending_rewrites list is traversed in multiple places. In typical usage,
the list will contain only a few entries so the traversal cost is negligible (
furthermore in comparison to a rewrite).
---
src/backend/catalog/index.c | 2 +-
src/backend/commands/cluster.c | 5 +
src/backend/commands/tablecmds.c | 6 +
src/backend/utils/activity/pgstat_relation.c | 391 ++++++++++++++++++-
src/backend/utils/activity/pgstat_xact.c | 25 +-
src/backend/utils/cache/relcache.c | 6 +
src/include/pgstat.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 424 insertions(+), 17 deletions(-)
92.8% src/backend/utils/activity/
4.9% src/backend/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 8dea58ad96b..b71925a22c3 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1795,7 +1795,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId);
/* copy over statistics from old to new index */
- pgstat_copy_relation_stats(newClassRel, oldClassRel);
+ pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 2120c85ccb4..6155b12afab 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1196,6 +1196,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
rel1 = relation_open(r1, NoLock);
rel2 = relation_open(r2, NoLock);
+
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind))
+ pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator);
+
rel2->rd_createSubid = rel1->rd_createSubid;
rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid;
rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 6b1a00ed477..9de70f321ed 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -16884,6 +16884,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
Oid reltoastrelid;
RelFileNumber newrelfilenumber;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator;
List *reltoastidxids = NIL;
ListCell *lc;
@@ -16922,6 +16923,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
newrlocator = rel->rd_locator;
newrlocator.relNumber = newrelfilenumber;
newrlocator.spcOid = newTableSpace;
+ oldrlocator = rel->rd_locator;
/* hand off to AM to actually create new rel storage and copy the data */
if (rel->rd_rel->relkind == RELKIND_INDEX)
@@ -16934,6 +16936,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
table_relation_copy_data(rel, &newrlocator);
}
+ /* mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Update the pg_class row.
*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 48bf93cae6e..b6ba8fdf440 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -30,6 +30,19 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
+/* Pending rewrite operations for stats copying */
+typedef struct PgStat_PendingRewrite
+{
+ RelFileLocator old_locator;
+ RelFileLocator new_locator;
+ RelFileLocator original_locator;
+ int nest_level; /* Transaction nesting level where rewrite
+ * occurred */
+ struct PgStat_PendingRewrite *next;
+} PgStat_PendingRewrite;
+
+/* The pending rewrites list for current transaction */
+static PgStat_PendingRewrite *pending_rewrites = NULL;
/* Record that's written to 2PC state file when pgstat state is persisted */
typedef struct TwoPhasePgStatRecord
@@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter deleted_pre_truncdrop;
RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
+ RelFileLocator rewrite_old_locator;
+ int rewrite_nest_level;
} TwoPhasePgStatRecord;
@@ -54,27 +69,70 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
/*
- * Copy stats between relations. This is used for things like REINDEX
+ * Copy stats between RelFileLocator. This is used for things like REINDEX
* CONCURRENTLY.
*/
void
-pgstat_copy_relation_stats(Relation dst, Relation src)
+pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment)
{
PgStat_StatTabEntry *srcstats;
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src));
+ srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION,
+ src.dbOid,
+ RelFileLocatorToPgStatObjid(src));
if (!srcstats)
return;
dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
- dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(dst),
+ dst.dbOid,
+ RelFileLocatorToPgStatObjid(dst),
false);
dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats;
- dstshstats->stats = *srcstats;
+
+ if (!increment)
+ dstshstats->stats = *srcstats;
+ else
+ {
+ /* Increment those statistics */
+#define RELFSTAT_ACC(fld, stats_to_add) \
+ (dstshstats->stats.fld += stats_to_add->fld)
+ RELFSTAT_ACC(numscans, srcstats);
+ RELFSTAT_ACC(tuples_returned, srcstats);
+ RELFSTAT_ACC(tuples_fetched, srcstats);
+ RELFSTAT_ACC(tuples_inserted, srcstats);
+ RELFSTAT_ACC(tuples_updated, srcstats);
+ RELFSTAT_ACC(tuples_deleted, srcstats);
+ RELFSTAT_ACC(tuples_hot_updated, srcstats);
+ RELFSTAT_ACC(tuples_newpage_updated, srcstats);
+ RELFSTAT_ACC(live_tuples, srcstats);
+ RELFSTAT_ACC(dead_tuples, srcstats);
+ RELFSTAT_ACC(mod_since_analyze, srcstats);
+ RELFSTAT_ACC(ins_since_vacuum, srcstats);
+ RELFSTAT_ACC(blocks_fetched, srcstats);
+ RELFSTAT_ACC(blocks_hit, srcstats);
+ RELFSTAT_ACC(vacuum_count, srcstats);
+ RELFSTAT_ACC(autovacuum_count, srcstats);
+ RELFSTAT_ACC(analyze_count, srcstats);
+ RELFSTAT_ACC(autoanalyze_count, srcstats);
+ RELFSTAT_ACC(total_vacuum_time, srcstats);
+ RELFSTAT_ACC(total_autovacuum_time, srcstats);
+ RELFSTAT_ACC(total_analyze_time, srcstats);
+ RELFSTAT_ACC(total_autoanalyze_time, srcstats);
+#undef RELFSTAT_ACC
+
+ /* Replace those statistics */
+#define RELFSTAT_REP(fld, stats_to_rep) \
+ (dstshstats->stats.fld = stats_to_rep->fld)
+ RELFSTAT_REP(lastscan, srcstats);
+ RELFSTAT_REP(last_vacuum_time, srcstats);
+ RELFSTAT_REP(last_autovacuum_time, srcstats);
+ RELFSTAT_REP(last_analyze_time, srcstats);
+ RELFSTAT_REP(last_autoanalyze_time, srcstats);
+#undef RELFSTAT_REP
+ }
pgstat_unlock_entry(dst_ref);
}
@@ -136,6 +194,7 @@ void
pgstat_assoc_relation(Relation rel)
{
RelFileLocator locator;
+ PgStat_TableStatus *pgstat_info;
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
@@ -164,14 +223,54 @@ pgstat_assoc_relation(Relation rel)
locator.relNumber = rel->rd_id;
}
+ /*
+ * If this relation was rewritten during the current transaction we may be
+ * reopening it with its new RelFileLocator. In that case, continue using
+ * the stats entry associated with the old locator rather than creating a
+ * new one. This ensures all stats from before and after the rewrite are
+ * tracked in a single entry which will be properly copied to the new
+ * locator at transaction commit.
+ */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (locator.dbOid == rewrite->new_locator.dbOid &&
+ locator.spcOid == rewrite->new_locator.spcOid &&
+ locator.relNumber == rewrite->new_locator.relNumber)
+ {
+ pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator);
+ goto found_entry;
+ }
+ }
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(locator);
+ pgstat_info = pgstat_prep_relation_pending(locator);
+
+found_entry:
+ rel->pgstat_info = pgstat_info;
+
+ /*
+ * For relations stats, we key by physical file location, not by relation
+ * OID. This means during operations like ALTER TYPE it's possible that
+ * the relation OID changes but the relfilenode stays the same (no actual
+ * rewrite needed). Unlink the old relation first.
+ */
+ if (pgstat_info->relation != NULL &&
+ pgstat_info->relation != rel)
+ {
+ pgstat_info->relation->pgstat_info = NULL;
+ pgstat_info->relation = NULL;
+ }
/* don't allow link a stats to multiple relcache entries */
- Assert(rel->pgstat_info->relation == NULL);
+ Assert(pgstat_info->relation == NULL);
/* mark this relation as the owner */
- rel->pgstat_info->relation = rel;
+ pgstat_info->relation = rel;
}
/*
@@ -214,14 +313,37 @@ pgstat_drop_relation(Relation rel)
{
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ bool skip_transactional_drop = false;
/* don't track stats for relations without storage */
if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
return;
- pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_locator.dbOid,
- RelFileLocatorToPgStatObjid(rel->rd_locator));
+ /* Check if this drop is part of a pending rewrite */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid &&
+ rel->rd_locator.spcOid == rewrite->old_locator.spcOid &&
+ rel->rd_locator.relNumber == rewrite->old_locator.relNumber)
+ {
+ skip_transactional_drop = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * If it is part of a rewrite, drop its stats later, for example in
+ * AtEOXact_PgStat_Relations(), so skip it here.
+ */
+ if (!skip_transactional_drop)
+ pgstat_drop_transactional(PGSTAT_KIND_RELATION,
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -666,6 +788,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit)
}
tabstat->trans = NULL;
}
+
+ /* preserve the stats in case of rewrite */
+ if (isCommit && pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *prev = NULL;
+ PgStat_PendingRewrite *current = pending_rewrites;
+ PgStat_PendingRewrite *next;
+
+ /* reverse the rewrites list to process in chronological order */
+ while (current != NULL)
+ {
+ next = current->next;
+ current->next = prev;
+ prev = current;
+ current = next;
+ }
+
+ /* now process rewrites in chronological order */
+ for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ pgstat_copy_relation_stats(rewrite->new_locator,
+ rewrite->old_locator, true);
+
+ /* drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
+ }
+
+ pending_rewrites = NULL;
}
/*
@@ -681,6 +845,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in
PgStat_TableXactStatus *trans;
PgStat_TableXactStatus *next_trans;
+ /*
+ * If we don't commit then remove the associated rewrites if any, to keep
+ * the rewrite chain in sync with what will be eventually committed.
+ */
+ if (!isCommit)
+ {
+ PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites;
+
+ while (*rewrite_ptr != NULL)
+ {
+ if ((*rewrite_ptr)->nest_level >= nestDepth)
+ {
+ PgStat_PendingRewrite *to_remove = *rewrite_ptr;
+
+ *rewrite_ptr = (*rewrite_ptr)->next;
+ pfree(to_remove);
+ }
+ else
+ {
+ rewrite_ptr = &((*rewrite_ptr)->next);
+ }
+ }
+ }
+
for (trans = xact_state->first; trans != NULL; trans = next_trans)
{
PgStat_TableStatus *tabstat;
@@ -760,11 +948,19 @@ void
AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
{
PgStat_TableXactStatus *trans;
+ PgStat_PendingRewrite *rewrite;
+ /*
+ * For each tabstat, find its matching rewrite and remove it from the
+ * pending rewrites list. This way, after processing all tabstats, pending
+ * rewrites will only contain rewrite only transactions.
+ */
for (trans = xact_state->first; trans != NULL; trans = trans->next)
{
PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY;
TwoPhasePgStatRecord record;
+ PgStat_PendingRewrite **rewrite_ptr;
+ bool found_rewrite = false;
Assert(trans->nest_level == 1);
Assert(trans->upper == NULL);
@@ -784,10 +980,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.locator = tabstat->locator;
record.truncdropped = trans->truncdropped;
+ record.rewrite_nest_level = 0;
+
+ /*
+ * Look for a matching rewrite and remove it from pending rewrites. We
+ * check three possible matches:
+ *
+ * The new_locator when stats have been added after the rewrite. The
+ * old_locator when stats have been added before the rewrite but not
+ * after. The original_locator when this tabstat is part of a rewrite
+ * chain.
+ */
+ rewrite_ptr = &pending_rewrites;
+ while (*rewrite_ptr != NULL)
+ {
+ rewrite = *rewrite_ptr;
+
+ if ((record.locator.dbOid == rewrite->new_locator.dbOid &&
+ record.locator.spcOid == rewrite->new_locator.spcOid &&
+ record.locator.relNumber == rewrite->new_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->old_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->old_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->old_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->original_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->original_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->original_locator.relNumber))
+ {
+ /*
+ * Found matching rewrite. Record the rewrite information and
+ * remove this rewrite from the list since it's now handled.
+ */
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.locator = rewrite->new_locator;
+ found_rewrite = true;
+
+ /* Remove from pending_rewrites list */
+ *rewrite_ptr = rewrite->next;
+ pfree(rewrite);
+ break;
+ }
+ else
+ {
+ /* Move to next rewrite in the list */
+ rewrite_ptr = &(rewrite->next);
+ }
+ }
+
+ /* If no rewrite found, clear the rewrite fields */
+ if (!found_rewrite)
+ {
+ memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator));
+ }
+
+ RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
+ &record, sizeof(TwoPhasePgStatRecord));
+ }
+
+ /*
+ * Now process any rewrites still pending. These are rewrite only
+ * transactions. We need to preserve their stats even though there's no
+ * tabstat entry for them.
+ */
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ TwoPhasePgStatRecord record;
+
+ memset(&record, 0, sizeof(TwoPhasePgStatRecord));
+ record.locator = rewrite->new_locator;
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.truncdropped = false;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
&record, sizeof(TwoPhasePgStatRecord));
}
+
+ pending_rewrites = NULL;
}
/*
@@ -810,6 +1079,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
tabstat = trans->parent;
tabstat->trans = NULL;
}
+
+ pending_rewrites = NULL;
}
/*
@@ -845,6 +1116,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
pgstat_info->counts.changed_tuples +=
rec->tuples_inserted + rec->tuples_updated +
rec->tuples_deleted;
+
+ if (rec->rewrite_nest_level > 0)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ /* Flush any pending stats for old locator first */
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ /* Copy stats from old to new locator */
+ pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator,
+ true);
+
+ /* Drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
}
/*
@@ -859,9 +1153,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
{
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
+ RelFileLocator target_locator;
+
+ /*
+ * For aborted transactions with rewrites (like TRUNCATE), we need to
+ * restore stats to the old locator, not the new one. The new locator
+ * should be dropped since the rewrite is being rolled back.
+ */
+ if (rec->rewrite_nest_level > 0)
+ {
+ /* Use the old locator */
+ target_locator = rec->rewrite_old_locator;
+ }
+ else
+ {
+ /* No rewrite, use the original locator */
+ target_locator = rec->locator;
+ }
/* Find or create a tabstat entry for the target locator */
- pgstat_info = pgstat_prep_relation_pending(rec->locator);
+ pgstat_info = pgstat_prep_relation_pending(target_locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -916,7 +1227,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
+ TimestampTz t;
+
+ /*
+ * Checking the transaction state due to the flush call in
+ * pgstat_twophase_postcommit() that would break the assertion on the
+ * state in GetCurrentTransactionStopTimestamp().
+ */
+ if (!IsTransactionState())
+ t = GetCurrentTransactionStopTimestamp();
+ else
+ t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
@@ -1167,3 +1488,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
ReleaseSysCache(tuple);
return result;
}
+
+/*
+ * Mark that a relation rewrite has occurred, preserving the original locator
+ * so stats can be copied at transaction commit.
+ */
+void
+pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator)
+{
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *existing;
+ RelFileLocator original_locator = old_locator;
+
+ for (existing = pending_rewrites; existing != NULL; existing = existing->next)
+ {
+ if (old_locator.dbOid == existing->new_locator.dbOid &&
+ old_locator.spcOid == existing->new_locator.spcOid &&
+ old_locator.relNumber == existing->new_locator.relNumber)
+ {
+ original_locator = existing->original_locator;
+ break;
+ }
+ }
+
+ /* Allocate in TopTransactionContext memory context */
+ rewrite = MemoryContextAlloc(TopTransactionContext,
+ sizeof(PgStat_PendingRewrite));
+
+ rewrite->old_locator = old_locator;
+ rewrite->new_locator = new_locator;
+ rewrite->original_locator = original_locator;
+ rewrite->nest_level = GetCurrentTransactionNestLevel();
+
+ /* Add to the list */
+ rewrite->next = pending_rewrites;
+ pending_rewrites = rewrite;
+}
+
+void
+pgstat_clear_rewrite(void)
+{
+ pending_rewrites = NULL;
+}
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index bc9864bd8d9..f8cf3755ce2 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel)
}
pgStatXactStack = NULL;
+ pgstat_clear_rewrite();
+
/* Make sure any stats snapshot is thrown away */
pgstat_clear_snapshot();
}
@@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo
void
pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid)
{
- if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL))
+ PgStat_EntryRef *entry_ref;
+
+ entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL);
+
+ if (entry_ref)
{
+ /*
+ * For relations stats, we key by physical file location, not by
+ * relation OID. This means during operations like ALTER TYPE where
+ * the relation OID changes but the relfilenode stays the same (no
+ * actual rewrite needed), we'll find an existing entry.
+ *
+ * This is expected behavior, we want to preserve stats across the
+ * catalog change. Simply reset and recreate the entry for the new
+ * relation OID without warning.
+ */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ pgstat_reset(kind, dboid, objid);
+ create_drop_transactional_internal(kind, dboid, objid, true);
+ return;
+ }
+
ereport(WARNING,
errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64,
(pgstat_get_kind_info(kind))->name, dboid,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 2d0cb7bcfd4..c98e5c51d63 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -85,6 +85,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/relmapper.h"
#include "utils/resowner.h"
#include "utils/snapmgr.h"
@@ -3780,6 +3781,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
MultiXactId minmulti = InvalidMultiXactId;
TransactionId freezeXid = InvalidTransactionId;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator = relation->rd_locator;
if (!IsBinaryUpgrade)
{
@@ -3951,6 +3953,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
table_close(pg_class, RowExclusiveLock);
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Make the pg_class row change or relation map change visible. This will
* cause the relcache entry to get updated, too.
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 3102f86aa24..8750c025bbe 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -670,7 +670,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id);
extern void pgstat_create_relation(Relation rel);
extern void pgstat_drop_relation(Relation rel);
-extern void pgstat_copy_relation_stats(Relation dst, Relation src);
+extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment);
extern void pgstat_init_relation(Relation rel);
extern void pgstat_assoc_relation(Relation rel);
@@ -682,6 +682,9 @@ extern void pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_mark_rewrite(RelFileLocator old_locator,
+ RelFileLocator new_locator);
+extern void pgstat_clear_rewrite(void);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3451538565e..0ca3eae8026 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2268,6 +2268,7 @@ PgStat_KindInfo
PgStat_LocalState
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
+PgStat_PendingRewrite
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
--tV2Nti+lXhoeT6I5--
^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v12 2/2] handle relation statistics correctly during rewrites
@ 2025-11-04 13:52 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Bertrand Drouvot @ 2025-11-04 13:52 UTC (permalink / raw)
Now that PGSTAT_KIND_RELATION is keyed by refilenode, we need to handle rewrites.
To do so, this patch:
- Adds PgStat_PendingRewrite, a new struct to track rewrite operations within
a transaction, storing the old locator, new locator, and original locator (for
rewrite chains). This allows stats to be copied from the original location to
the final location at commit time.
- Adds a new function, pgstat_mark_rewrite(), called when a table rewrite begins.
It records the rewrite operation in a local list and detects rewrite chains by
checking if the old_locator matches any existing new_locator, preserving the
chain's original_locator.
- Modifies pgstat_copy_relation_stats(), to accept RelFileLocators instead of
Relations, with a new increment parameter to accumulate stats (needed for rewrite
chains with DML between rewrites).
- Ensures that AtEOXact_PgStat_Relations(), AtPrepare_PgStat_Relations(),
pgstat_twophase_postcommit()/postabort() pgstat_drop_relation() handle the
PgStat_PendingRewrite list correctly.
Note that due to the new flush call in pgstat_twophase_postcommit() we can not
call GetCurrentTransactionStopTimestamp() in pgstat_relation_flush_cb(). So,
adding a check to handle this special case and call GetCurrentTimestamp() instead.
Note that we'd call GetCurrentTimestamp() only if there is a rewrite, so that
the GetCurrentTimestamp() extra cost should be negligible. Another solution
could be to trigger the flush from FinishPreparedTransaction() but that's not
worth the extra complexity.
The new pending_rewrites list is traversed in multiple places. The overhead
should be negligible in comparison to a rewrite and the list should not contain
a lot of rewrites in practice.
The pending_rewrites list is traversed in multiple places. In typical usage,
the list will contain only a few entries so the traversal cost is negligible (
furthermore in comparison to a rewrite).
---
src/backend/catalog/index.c | 2 +-
src/backend/commands/cluster.c | 5 +
src/backend/commands/tablecmds.c | 6 +
src/backend/utils/activity/pgstat_relation.c | 391 ++++++++++++++++++-
src/backend/utils/activity/pgstat_xact.c | 25 +-
src/backend/utils/cache/relcache.c | 6 +
src/include/pgstat.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 424 insertions(+), 17 deletions(-)
92.8% src/backend/utils/activity/
4.9% src/backend/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 43de42ce39e..88ef00609db 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1793,7 +1793,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId);
/* copy over statistics from old to new index */
- pgstat_copy_relation_stats(newClassRel, oldClassRel);
+ pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 60a4617a585..3a446f32921 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -1196,6 +1196,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
rel1 = relation_open(r1, NoLock);
rel2 = relation_open(r2, NoLock);
+
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind))
+ pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator);
+
rel2->rd_createSubid = rel1->rd_createSubid;
rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid;
rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 85242dcc245..8d10d9d1e0f 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -16883,6 +16883,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
Oid reltoastrelid;
RelFileNumber newrelfilenumber;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator;
List *reltoastidxids = NIL;
ListCell *lc;
@@ -16921,6 +16922,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
newrlocator = rel->rd_locator;
newrlocator.relNumber = newrelfilenumber;
newrlocator.spcOid = newTableSpace;
+ oldrlocator = rel->rd_locator;
/* hand off to AM to actually create new rel storage and copy the data */
if (rel->rd_rel->relkind == RELKIND_INDEX)
@@ -16933,6 +16935,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
table_relation_copy_data(rel, &newrlocator);
}
+ /* mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Update the pg_class row.
*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 89bf0cbed56..4d929972fab 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -30,6 +30,19 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
+/* Pending rewrite operations for stats copying */
+typedef struct PgStat_PendingRewrite
+{
+ RelFileLocator old_locator;
+ RelFileLocator new_locator;
+ RelFileLocator original_locator;
+ int nest_level; /* Transaction nesting level where rewrite
+ * occurred */
+ struct PgStat_PendingRewrite *next;
+} PgStat_PendingRewrite;
+
+/* The pending rewrites list for current transaction */
+static PgStat_PendingRewrite *pending_rewrites = NULL;
/* Record that's written to 2PC state file when pgstat state is persisted */
typedef struct TwoPhasePgStatRecord
@@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter deleted_pre_truncdrop;
RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
+ RelFileLocator rewrite_old_locator;
+ int rewrite_nest_level;
} TwoPhasePgStatRecord;
@@ -54,27 +69,70 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
/*
- * Copy stats between relations. This is used for things like REINDEX
+ * Copy stats between RelFileLocator. This is used for things like REINDEX
* CONCURRENTLY.
*/
void
-pgstat_copy_relation_stats(Relation dst, Relation src)
+pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment)
{
PgStat_StatTabEntry *srcstats;
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src));
+ srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION,
+ src.dbOid,
+ RelFileLocatorToPgStatObjid(src));
if (!srcstats)
return;
dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
- dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(dst),
+ dst.dbOid,
+ RelFileLocatorToPgStatObjid(dst),
false);
dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats;
- dstshstats->stats = *srcstats;
+
+ if (!increment)
+ dstshstats->stats = *srcstats;
+ else
+ {
+ /* Increment those statistics */
+#define RELFSTAT_ACC(fld, stats_to_add) \
+ (dstshstats->stats.fld += stats_to_add->fld)
+ RELFSTAT_ACC(numscans, srcstats);
+ RELFSTAT_ACC(tuples_returned, srcstats);
+ RELFSTAT_ACC(tuples_fetched, srcstats);
+ RELFSTAT_ACC(tuples_inserted, srcstats);
+ RELFSTAT_ACC(tuples_updated, srcstats);
+ RELFSTAT_ACC(tuples_deleted, srcstats);
+ RELFSTAT_ACC(tuples_hot_updated, srcstats);
+ RELFSTAT_ACC(tuples_newpage_updated, srcstats);
+ RELFSTAT_ACC(live_tuples, srcstats);
+ RELFSTAT_ACC(dead_tuples, srcstats);
+ RELFSTAT_ACC(mod_since_analyze, srcstats);
+ RELFSTAT_ACC(ins_since_vacuum, srcstats);
+ RELFSTAT_ACC(blocks_fetched, srcstats);
+ RELFSTAT_ACC(blocks_hit, srcstats);
+ RELFSTAT_ACC(vacuum_count, srcstats);
+ RELFSTAT_ACC(autovacuum_count, srcstats);
+ RELFSTAT_ACC(analyze_count, srcstats);
+ RELFSTAT_ACC(autoanalyze_count, srcstats);
+ RELFSTAT_ACC(total_vacuum_time, srcstats);
+ RELFSTAT_ACC(total_autovacuum_time, srcstats);
+ RELFSTAT_ACC(total_analyze_time, srcstats);
+ RELFSTAT_ACC(total_autoanalyze_time, srcstats);
+#undef RELFSTAT_ACC
+
+ /* Replace those statistics */
+#define RELFSTAT_REP(fld, stats_to_rep) \
+ (dstshstats->stats.fld = stats_to_rep->fld)
+ RELFSTAT_REP(lastscan, srcstats);
+ RELFSTAT_REP(last_vacuum_time, srcstats);
+ RELFSTAT_REP(last_autovacuum_time, srcstats);
+ RELFSTAT_REP(last_analyze_time, srcstats);
+ RELFSTAT_REP(last_autoanalyze_time, srcstats);
+#undef RELFSTAT_REP
+ }
pgstat_unlock_entry(dst_ref);
}
@@ -136,6 +194,7 @@ void
pgstat_assoc_relation(Relation rel)
{
RelFileLocator locator;
+ PgStat_TableStatus *pgstat_info;
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
@@ -164,14 +223,54 @@ pgstat_assoc_relation(Relation rel)
locator.relNumber = rel->rd_id;
}
+ /*
+ * If this relation was rewritten during the current transaction we may be
+ * reopening it with its new RelFileLocator. In that case, continue using
+ * the stats entry associated with the old locator rather than creating a
+ * new one. This ensures all stats from before and after the rewrite are
+ * tracked in a single entry which will be properly copied to the new
+ * locator at transaction commit.
+ */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (locator.dbOid == rewrite->new_locator.dbOid &&
+ locator.spcOid == rewrite->new_locator.spcOid &&
+ locator.relNumber == rewrite->new_locator.relNumber)
+ {
+ pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator);
+ goto found_entry;
+ }
+ }
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(locator);
+ pgstat_info = pgstat_prep_relation_pending(locator);
+
+found_entry:
+ rel->pgstat_info = pgstat_info;
+
+ /*
+ * For relations stats, we key by physical file location, not by relation
+ * OID. This means during operations like ALTER TYPE it's possible that
+ * the relation OID changes but the relfilenode stays the same (no actual
+ * rewrite needed). Unlink the old relation first.
+ */
+ if (pgstat_info->relation != NULL &&
+ pgstat_info->relation != rel)
+ {
+ pgstat_info->relation->pgstat_info = NULL;
+ pgstat_info->relation = NULL;
+ }
/* don't allow link a stats to multiple relcache entries */
- Assert(rel->pgstat_info->relation == NULL);
+ Assert(pgstat_info->relation == NULL);
/* mark this relation as the owner */
- rel->pgstat_info->relation = rel;
+ pgstat_info->relation = rel;
}
/*
@@ -214,14 +313,37 @@ pgstat_drop_relation(Relation rel)
{
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ bool skip_transactional_drop = false;
/* don't track stats for relations without storage */
if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
return;
- pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_locator.dbOid,
- RelFileLocatorToPgStatObjid(rel->rd_locator));
+ /* Check if this drop is part of a pending rewrite */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid &&
+ rel->rd_locator.spcOid == rewrite->old_locator.spcOid &&
+ rel->rd_locator.relNumber == rewrite->old_locator.relNumber)
+ {
+ skip_transactional_drop = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * If it is part of a rewrite, drop its stats later, for example in
+ * AtEOXact_PgStat_Relations(), so skip it here.
+ */
+ if (!skip_transactional_drop)
+ pgstat_drop_transactional(PGSTAT_KIND_RELATION,
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -666,6 +788,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit)
}
tabstat->trans = NULL;
}
+
+ /* preserve the stats in case of rewrite */
+ if (isCommit && pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *prev = NULL;
+ PgStat_PendingRewrite *current = pending_rewrites;
+ PgStat_PendingRewrite *next;
+
+ /* reverse the rewrites list to process in chronological order */
+ while (current != NULL)
+ {
+ next = current->next;
+ current->next = prev;
+ prev = current;
+ current = next;
+ }
+
+ /* now process rewrites in chronological order */
+ for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ pgstat_copy_relation_stats(rewrite->new_locator,
+ rewrite->old_locator, true);
+
+ /* drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
+ }
+
+ pending_rewrites = NULL;
}
/*
@@ -681,6 +845,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in
PgStat_TableXactStatus *trans;
PgStat_TableXactStatus *next_trans;
+ /*
+ * If we don't commit then remove the associated rewrites if any, to keep
+ * the rewrite chain in sync with what will be eventually committed.
+ */
+ if (!isCommit)
+ {
+ PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites;
+
+ while (*rewrite_ptr != NULL)
+ {
+ if ((*rewrite_ptr)->nest_level >= nestDepth)
+ {
+ PgStat_PendingRewrite *to_remove = *rewrite_ptr;
+
+ *rewrite_ptr = (*rewrite_ptr)->next;
+ pfree(to_remove);
+ }
+ else
+ {
+ rewrite_ptr = &((*rewrite_ptr)->next);
+ }
+ }
+ }
+
for (trans = xact_state->first; trans != NULL; trans = next_trans)
{
PgStat_TableStatus *tabstat;
@@ -760,11 +948,19 @@ void
AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
{
PgStat_TableXactStatus *trans;
+ PgStat_PendingRewrite *rewrite;
+ /*
+ * For each tabstat, find its matching rewrite and remove it from the
+ * pending rewrites list. This way, after processing all tabstats, pending
+ * rewrites will only contain rewrite only transactions.
+ */
for (trans = xact_state->first; trans != NULL; trans = trans->next)
{
PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY;
TwoPhasePgStatRecord record;
+ PgStat_PendingRewrite **rewrite_ptr;
+ bool found_rewrite = false;
Assert(trans->nest_level == 1);
Assert(trans->upper == NULL);
@@ -784,10 +980,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.locator = tabstat->locator;
record.truncdropped = trans->truncdropped;
+ record.rewrite_nest_level = 0;
+
+ /*
+ * Look for a matching rewrite and remove it from pending rewrites. We
+ * check three possible matches:
+ *
+ * The new_locator when stats have been added after the rewrite. The
+ * old_locator when stats have been added before the rewrite but not
+ * after. The original_locator when this tabstat is part of a rewrite
+ * chain.
+ */
+ rewrite_ptr = &pending_rewrites;
+ while (*rewrite_ptr != NULL)
+ {
+ rewrite = *rewrite_ptr;
+
+ if ((record.locator.dbOid == rewrite->new_locator.dbOid &&
+ record.locator.spcOid == rewrite->new_locator.spcOid &&
+ record.locator.relNumber == rewrite->new_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->old_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->old_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->old_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->original_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->original_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->original_locator.relNumber))
+ {
+ /*
+ * Found matching rewrite. Record the rewrite information and
+ * remove this rewrite from the list since it's now handled.
+ */
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.locator = rewrite->new_locator;
+ found_rewrite = true;
+
+ /* Remove from pending_rewrites list */
+ *rewrite_ptr = rewrite->next;
+ pfree(rewrite);
+ break;
+ }
+ else
+ {
+ /* Move to next rewrite in the list */
+ rewrite_ptr = &(rewrite->next);
+ }
+ }
+
+ /* If no rewrite found, clear the rewrite fields */
+ if (!found_rewrite)
+ {
+ memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator));
+ }
+
+ RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
+ &record, sizeof(TwoPhasePgStatRecord));
+ }
+
+ /*
+ * Now process any rewrites still pending. These are rewrite only
+ * transactions. We need to preserve their stats even though there's no
+ * tabstat entry for them.
+ */
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ TwoPhasePgStatRecord record;
+
+ memset(&record, 0, sizeof(TwoPhasePgStatRecord));
+ record.locator = rewrite->new_locator;
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.truncdropped = false;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
&record, sizeof(TwoPhasePgStatRecord));
}
+
+ pending_rewrites = NULL;
}
/*
@@ -810,6 +1079,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
tabstat = trans->parent;
tabstat->trans = NULL;
}
+
+ pending_rewrites = NULL;
}
/*
@@ -845,6 +1116,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
pgstat_info->counts.changed_tuples +=
rec->tuples_inserted + rec->tuples_updated +
rec->tuples_deleted;
+
+ if (rec->rewrite_nest_level > 0)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ /* Flush any pending stats for old locator first */
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ /* Copy stats from old to new locator */
+ pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator,
+ true);
+
+ /* Drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
}
/*
@@ -859,9 +1153,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
{
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
+ RelFileLocator target_locator;
+
+ /*
+ * For aborted transactions with rewrites (like TRUNCATE), we need to
+ * restore stats to the old locator, not the new one. The new locator
+ * should be dropped since the rewrite is being rolled back.
+ */
+ if (rec->rewrite_nest_level > 0)
+ {
+ /* Use the old locator */
+ target_locator = rec->rewrite_old_locator;
+ }
+ else
+ {
+ /* No rewrite, use the original locator */
+ target_locator = rec->locator;
+ }
/* Find or create a tabstat entry for the target locator */
- pgstat_info = pgstat_prep_relation_pending(rec->locator);
+ pgstat_info = pgstat_prep_relation_pending(target_locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -916,7 +1227,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
+ TimestampTz t;
+
+ /*
+ * Checking the transaction state due to the flush call in
+ * pgstat_twophase_postcommit() that would break the assertion on the
+ * state in GetCurrentTransactionStopTimestamp().
+ */
+ if (!IsTransactionState())
+ t = GetCurrentTransactionStopTimestamp();
+ else
+ t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
@@ -1167,3 +1488,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
ReleaseSysCache(tuple);
return result;
}
+
+/*
+ * Mark that a relation rewrite has occurred, preserving the original locator
+ * so stats can be copied at transaction commit.
+ */
+void
+pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator)
+{
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *existing;
+ RelFileLocator original_locator = old_locator;
+
+ for (existing = pending_rewrites; existing != NULL; existing = existing->next)
+ {
+ if (old_locator.dbOid == existing->new_locator.dbOid &&
+ old_locator.spcOid == existing->new_locator.spcOid &&
+ old_locator.relNumber == existing->new_locator.relNumber)
+ {
+ original_locator = existing->original_locator;
+ break;
+ }
+ }
+
+ /* Allocate in TopTransactionContext memory context */
+ rewrite = MemoryContextAlloc(TopTransactionContext,
+ sizeof(PgStat_PendingRewrite));
+
+ rewrite->old_locator = old_locator;
+ rewrite->new_locator = new_locator;
+ rewrite->original_locator = original_locator;
+ rewrite->nest_level = GetCurrentTransactionNestLevel();
+
+ /* Add to the list */
+ rewrite->next = pending_rewrites;
+ pending_rewrites = rewrite;
+}
+
+void
+pgstat_clear_rewrite(void)
+{
+ pending_rewrites = NULL;
+}
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index 5e2d69e6297..8ed8f5317f3 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel)
}
pgStatXactStack = NULL;
+ pgstat_clear_rewrite();
+
/* Make sure any stats snapshot is thrown away */
pgstat_clear_snapshot();
}
@@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo
void
pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid)
{
- if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL))
+ PgStat_EntryRef *entry_ref;
+
+ entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL);
+
+ if (entry_ref)
{
+ /*
+ * For relations stats, we key by physical file location, not by
+ * relation OID. This means during operations like ALTER TYPE where
+ * the relation OID changes but the relfilenode stays the same (no
+ * actual rewrite needed), we'll find an existing entry.
+ *
+ * This is expected behavior, we want to preserve stats across the
+ * catalog change. Simply reset and recreate the entry for the new
+ * relation OID without warning.
+ */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ pgstat_reset(kind, dboid, objid);
+ create_drop_transactional_internal(kind, dboid, objid, true);
+ return;
+ }
+
ereport(WARNING,
errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64,
(pgstat_get_kind_info(kind))->name, dboid,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index a1c88c6b1b6..dc5948f446c 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -85,6 +85,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/relmapper.h"
#include "utils/resowner.h"
#include "utils/snapmgr.h"
@@ -3775,6 +3776,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
MultiXactId minmulti = InvalidMultiXactId;
TransactionId freezeXid = InvalidTransactionId;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator = relation->rd_locator;
if (!IsBinaryUpgrade)
{
@@ -3946,6 +3948,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
table_close(pg_class, RowExclusiveLock);
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Make the pg_class row change or relation map change visible. This will
* cause the relcache entry to get updated, too.
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 689c624f373..d0d5473f8a5 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -673,7 +673,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id);
extern void pgstat_create_relation(Relation rel);
extern void pgstat_drop_relation(Relation rel);
-extern void pgstat_copy_relation_stats(Relation dst, Relation src);
+extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment);
extern void pgstat_init_relation(Relation rel);
extern void pgstat_assoc_relation(Relation rel);
@@ -685,6 +685,9 @@ extern void pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_mark_rewrite(RelFileLocator old_locator,
+ RelFileLocator new_locator);
+extern void pgstat_clear_rewrite(void);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3250564d4ff..fbcbe5da26a 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2281,6 +2281,7 @@ PgStat_KindInfo
PgStat_LocalState
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
+PgStat_PendingRewrite
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
--sdA5zuaL94ts0rne--
^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v16 2/2] handle relation statistics correctly during rewrites
@ 2026-05-18 15:20 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Bertrand Drouvot @ 2026-05-18 15:20 UTC (permalink / raw)
This patch ensures that relation statistics are preserved correctly when a
relation is rewritten (e.g., during CLUSTER, VACUUM FULL, ALTER TABLE
operations that require a table rewrite, or TRUNCATE).
Since patch 0001 keys PGSTAT_KIND_RELATION by the physical RelFileLocator
rather than the relation OID, a rewrite (which assigns a new relfilenode)
would lose the accumulated statistics if not handled specially.
The approach is:
1. When a rewrite occurs (RelationSetNewRelfilenumber or swap_relation_files),
record the old and new locators via pgstat_mark_rewrite().
2. During pgstat_assoc_relation(), if the relation's new locator matches a
pending rewrite, continue using the stats entry associated with the old
locator. This ensures stats accumulated after the rewrite are still
tracked in the original entry.
3. At transaction commit (AtEOXact_PgStat_Relations), process pending
rewrites: flush any pending stats for the old locator, copy them to the
new locator's stats entry, and drop the old entry.
4. At subtransaction abort, remove rewrites from the aborted nesting level.
5. For two-phase transactions, the rewrite information is recorded in the
TwoPhasePgStatRecord and replayed at commit/abort time.
This also handles chained rewrites (multiple rewrites of the same relation
in a single transaction) by tracking the original locator through the chain.
---
src/backend/catalog/index.c | 2 +-
src/backend/commands/repack.c | 5 +
src/backend/commands/tablecmds.c | 6 +
src/backend/utils/activity/pgstat_relation.c | 392 ++++++++++++++++++-
src/backend/utils/activity/pgstat_xact.c | 25 +-
src/backend/utils/cache/relcache.c | 6 +
src/include/pgstat.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 425 insertions(+), 17 deletions(-)
92.8% src/backend/utils/activity/
4.8% src/backend/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 9407c357f27..335ff1e50f6 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1813,7 +1813,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId);
/* copy over statistics from old to new index */
- pgstat_copy_relation_stats(newClassRel, oldClassRel);
+ pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/repack.c b/src/backend/commands/repack.c
index 351a3cc32e8..eda4135bcd1 100644
--- a/src/backend/commands/repack.c
+++ b/src/backend/commands/repack.c
@@ -1633,6 +1633,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
rel1 = relation_open(r1, NoLock);
rel2 = relation_open(r2, NoLock);
+
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind))
+ pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator);
+
rel2->rd_createSubid = rel1->rd_createSubid;
rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid;
rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 92b0f38c353..b553b9a4970 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -17129,6 +17129,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
Oid reltoastrelid;
RelFileNumber newrelfilenumber;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator;
List *reltoastidxids = NIL;
ListCell *lc;
@@ -17167,6 +17168,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
newrlocator = rel->rd_locator;
newrlocator.relNumber = newrelfilenumber;
newrlocator.spcOid = newTableSpace;
+ oldrlocator = rel->rd_locator;
/* hand off to AM to actually create new rel storage and copy the data */
if (rel->rd_rel->relkind == RELKIND_INDEX)
@@ -17179,6 +17181,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
table_relation_copy_data(rel, &newrlocator);
}
+ /* mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Update the pg_class row.
*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 7fcda7f7518..18d1b9fc59d 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -30,6 +30,19 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
+/* Pending rewrite operations for stats copying */
+typedef struct PgStat_PendingRewrite
+{
+ RelFileLocator old_locator;
+ RelFileLocator new_locator;
+ RelFileLocator original_locator;
+ int nest_level; /* Transaction nesting level where rewrite
+ * occurred */
+ struct PgStat_PendingRewrite *next;
+} PgStat_PendingRewrite;
+
+/* The pending rewrites list for current transaction */
+static PgStat_PendingRewrite *pending_rewrites = NULL;
/* Record that's written to 2PC state file when pgstat state is persisted */
typedef struct TwoPhasePgStatRecord
@@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter deleted_pre_truncdrop;
RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
+ RelFileLocator rewrite_old_locator;
+ int rewrite_nest_level;
} TwoPhasePgStatRecord;
@@ -54,27 +69,71 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
/*
- * Copy stats between relations. This is used for things like REINDEX
+ * Copy stats between RelFileLocator. This is used for things like REINDEX
* CONCURRENTLY.
*/
void
-pgstat_copy_relation_stats(Relation dst, Relation src)
+pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment)
{
PgStat_StatTabEntry *srcstats;
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src), NULL);
+ srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION,
+ src.dbOid,
+ RelFileLocatorToPgStatObjid(src),
+ NULL);
if (!srcstats)
return;
dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
- dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(dst),
+ dst.dbOid,
+ RelFileLocatorToPgStatObjid(dst),
false);
dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats;
- dstshstats->stats = *srcstats;
+
+ if (!increment)
+ dstshstats->stats = *srcstats;
+ else
+ {
+ /* Increment those statistics */
+#define RELFSTAT_ACC(fld, stats_to_add) \
+ (dstshstats->stats.fld += stats_to_add->fld)
+ RELFSTAT_ACC(numscans, srcstats);
+ RELFSTAT_ACC(tuples_returned, srcstats);
+ RELFSTAT_ACC(tuples_fetched, srcstats);
+ RELFSTAT_ACC(tuples_inserted, srcstats);
+ RELFSTAT_ACC(tuples_updated, srcstats);
+ RELFSTAT_ACC(tuples_deleted, srcstats);
+ RELFSTAT_ACC(tuples_hot_updated, srcstats);
+ RELFSTAT_ACC(tuples_newpage_updated, srcstats);
+ RELFSTAT_ACC(live_tuples, srcstats);
+ RELFSTAT_ACC(dead_tuples, srcstats);
+ RELFSTAT_ACC(mod_since_analyze, srcstats);
+ RELFSTAT_ACC(ins_since_vacuum, srcstats);
+ RELFSTAT_ACC(blocks_fetched, srcstats);
+ RELFSTAT_ACC(blocks_hit, srcstats);
+ RELFSTAT_ACC(vacuum_count, srcstats);
+ RELFSTAT_ACC(autovacuum_count, srcstats);
+ RELFSTAT_ACC(analyze_count, srcstats);
+ RELFSTAT_ACC(autoanalyze_count, srcstats);
+ RELFSTAT_ACC(total_vacuum_time, srcstats);
+ RELFSTAT_ACC(total_autovacuum_time, srcstats);
+ RELFSTAT_ACC(total_analyze_time, srcstats);
+ RELFSTAT_ACC(total_autoanalyze_time, srcstats);
+#undef RELFSTAT_ACC
+
+ /* Replace those statistics */
+#define RELFSTAT_REP(fld, stats_to_rep) \
+ (dstshstats->stats.fld = stats_to_rep->fld)
+ RELFSTAT_REP(lastscan, srcstats);
+ RELFSTAT_REP(last_vacuum_time, srcstats);
+ RELFSTAT_REP(last_autovacuum_time, srcstats);
+ RELFSTAT_REP(last_analyze_time, srcstats);
+ RELFSTAT_REP(last_autoanalyze_time, srcstats);
+#undef RELFSTAT_REP
+ }
pgstat_unlock_entry(dst_ref);
}
@@ -136,6 +195,7 @@ void
pgstat_assoc_relation(Relation rel)
{
RelFileLocator locator;
+ PgStat_TableStatus *pgstat_info;
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
@@ -164,14 +224,54 @@ pgstat_assoc_relation(Relation rel)
locator.relNumber = rel->rd_id;
}
+ /*
+ * If this relation was rewritten during the current transaction we may be
+ * reopening it with its new RelFileLocator. In that case, continue using
+ * the stats entry associated with the old locator rather than creating a
+ * new one. This ensures all stats from before and after the rewrite are
+ * tracked in a single entry which will be properly copied to the new
+ * locator at transaction commit.
+ */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (locator.dbOid == rewrite->new_locator.dbOid &&
+ locator.spcOid == rewrite->new_locator.spcOid &&
+ locator.relNumber == rewrite->new_locator.relNumber)
+ {
+ pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator);
+ goto found_entry;
+ }
+ }
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(locator);
+ pgstat_info = pgstat_prep_relation_pending(locator);
+
+found_entry:
+ rel->pgstat_info = pgstat_info;
+
+ /*
+ * For relations stats, we key by physical file location, not by relation
+ * OID. This means during operations like ALTER TYPE it's possible that
+ * the relation OID changes but the relfilenode stays the same (no actual
+ * rewrite needed). Unlink the old relation first.
+ */
+ if (pgstat_info->relation != NULL &&
+ pgstat_info->relation != rel)
+ {
+ pgstat_info->relation->pgstat_info = NULL;
+ pgstat_info->relation = NULL;
+ }
/* don't allow link a stats to multiple relcache entries */
- Assert(rel->pgstat_info->relation == NULL);
+ Assert(pgstat_info->relation == NULL);
/* mark this relation as the owner */
- rel->pgstat_info->relation = rel;
+ pgstat_info->relation = rel;
}
/*
@@ -214,14 +314,37 @@ pgstat_drop_relation(Relation rel)
{
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ bool skip_transactional_drop = false;
/* don't track stats for relations without storage */
if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
return;
- pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_locator.dbOid,
- RelFileLocatorToPgStatObjid(rel->rd_locator));
+ /* Check if this drop is part of a pending rewrite */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid &&
+ rel->rd_locator.spcOid == rewrite->old_locator.spcOid &&
+ rel->rd_locator.relNumber == rewrite->old_locator.relNumber)
+ {
+ skip_transactional_drop = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * If it is part of a rewrite, drop its stats later, for example in
+ * AtEOXact_PgStat_Relations(), so skip it here.
+ */
+ if (!skip_transactional_drop)
+ pgstat_drop_transactional(PGSTAT_KIND_RELATION,
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -668,6 +791,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit)
}
tabstat->trans = NULL;
}
+
+ /* preserve the stats in case of rewrite */
+ if (isCommit && pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *prev = NULL;
+ PgStat_PendingRewrite *current = pending_rewrites;
+ PgStat_PendingRewrite *next;
+
+ /* reverse the rewrites list to process in chronological order */
+ while (current != NULL)
+ {
+ next = current->next;
+ current->next = prev;
+ prev = current;
+ current = next;
+ }
+
+ /* now process rewrites in chronological order */
+ for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ pgstat_copy_relation_stats(rewrite->new_locator,
+ rewrite->old_locator, true);
+
+ /* drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
+ }
+
+ pending_rewrites = NULL;
}
/*
@@ -683,6 +848,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in
PgStat_TableXactStatus *trans;
PgStat_TableXactStatus *next_trans;
+ /*
+ * If we don't commit then remove the associated rewrites if any, to keep
+ * the rewrite chain in sync with what will be eventually committed.
+ */
+ if (!isCommit)
+ {
+ PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites;
+
+ while (*rewrite_ptr != NULL)
+ {
+ if ((*rewrite_ptr)->nest_level >= nestDepth)
+ {
+ PgStat_PendingRewrite *to_remove = *rewrite_ptr;
+
+ *rewrite_ptr = (*rewrite_ptr)->next;
+ pfree(to_remove);
+ }
+ else
+ {
+ rewrite_ptr = &((*rewrite_ptr)->next);
+ }
+ }
+ }
+
for (trans = xact_state->first; trans != NULL; trans = next_trans)
{
PgStat_TableStatus *tabstat;
@@ -762,11 +951,19 @@ void
AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
{
PgStat_TableXactStatus *trans;
+ PgStat_PendingRewrite *rewrite;
+ /*
+ * For each tabstat, find its matching rewrite and remove it from the
+ * pending rewrites list. This way, after processing all tabstats, pending
+ * rewrites will only contain rewrite only transactions.
+ */
for (trans = xact_state->first; trans != NULL; trans = trans->next)
{
PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY;
TwoPhasePgStatRecord record;
+ PgStat_PendingRewrite **rewrite_ptr;
+ bool found_rewrite = false;
Assert(trans->nest_level == 1);
Assert(trans->upper == NULL);
@@ -786,10 +983,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.locator = tabstat->locator;
record.truncdropped = trans->truncdropped;
+ record.rewrite_nest_level = 0;
+
+ /*
+ * Look for a matching rewrite and remove it from pending rewrites. We
+ * check three possible matches:
+ *
+ * The new_locator when stats have been added after the rewrite. The
+ * old_locator when stats have been added before the rewrite but not
+ * after. The original_locator when this tabstat is part of a rewrite
+ * chain.
+ */
+ rewrite_ptr = &pending_rewrites;
+ while (*rewrite_ptr != NULL)
+ {
+ rewrite = *rewrite_ptr;
+
+ if ((record.locator.dbOid == rewrite->new_locator.dbOid &&
+ record.locator.spcOid == rewrite->new_locator.spcOid &&
+ record.locator.relNumber == rewrite->new_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->old_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->old_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->old_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->original_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->original_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->original_locator.relNumber))
+ {
+ /*
+ * Found matching rewrite. Record the rewrite information and
+ * remove this rewrite from the list since it's now handled.
+ */
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.locator = rewrite->new_locator;
+ found_rewrite = true;
+
+ /* Remove from pending_rewrites list */
+ *rewrite_ptr = rewrite->next;
+ pfree(rewrite);
+ break;
+ }
+ else
+ {
+ /* Move to next rewrite in the list */
+ rewrite_ptr = &(rewrite->next);
+ }
+ }
+
+ /* If no rewrite found, clear the rewrite fields */
+ if (!found_rewrite)
+ {
+ memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator));
+ }
+
+ RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
+ &record, sizeof(TwoPhasePgStatRecord));
+ }
+
+ /*
+ * Now process any rewrites still pending. These are rewrite only
+ * transactions. We need to preserve their stats even though there's no
+ * tabstat entry for them.
+ */
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ TwoPhasePgStatRecord record;
+
+ memset(&record, 0, sizeof(TwoPhasePgStatRecord));
+ record.locator = rewrite->new_locator;
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.truncdropped = false;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
&record, sizeof(TwoPhasePgStatRecord));
}
+
+ pending_rewrites = NULL;
}
/*
@@ -812,6 +1082,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
tabstat = trans->parent;
tabstat->trans = NULL;
}
+
+ pending_rewrites = NULL;
}
/*
@@ -847,6 +1119,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
pgstat_info->counts.changed_tuples +=
rec->tuples_inserted + rec->tuples_updated +
rec->tuples_deleted;
+
+ if (rec->rewrite_nest_level > 0)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ /* Flush any pending stats for old locator first */
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ /* Copy stats from old to new locator */
+ pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator,
+ true);
+
+ /* Drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
}
/*
@@ -861,9 +1156,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
{
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
+ RelFileLocator target_locator;
+
+ /*
+ * For aborted transactions with rewrites (like TRUNCATE), we need to
+ * restore stats to the old locator, not the new one. The new locator
+ * should be dropped since the rewrite is being rolled back.
+ */
+ if (rec->rewrite_nest_level > 0)
+ {
+ /* Use the old locator */
+ target_locator = rec->rewrite_old_locator;
+ }
+ else
+ {
+ /* No rewrite, use the original locator */
+ target_locator = rec->locator;
+ }
/* Find or create a tabstat entry for the target locator */
- pgstat_info = pgstat_prep_relation_pending(rec->locator);
+ pgstat_info = pgstat_prep_relation_pending(target_locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -918,7 +1230,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
+ TimestampTz t;
+
+ /*
+ * Checking the transaction state due to the flush call in
+ * pgstat_twophase_postcommit() that would break the assertion on the
+ * state in GetCurrentTransactionStopTimestamp().
+ */
+ if (!IsTransactionState())
+ t = GetCurrentTransactionStopTimestamp();
+ else
+ t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
@@ -1169,3 +1491,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
ReleaseSysCache(tuple);
return result;
}
+
+/*
+ * Mark that a relation rewrite has occurred, preserving the original locator
+ * so stats can be copied at transaction commit.
+ */
+void
+pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator)
+{
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *existing;
+ RelFileLocator original_locator = old_locator;
+
+ for (existing = pending_rewrites; existing != NULL; existing = existing->next)
+ {
+ if (old_locator.dbOid == existing->new_locator.dbOid &&
+ old_locator.spcOid == existing->new_locator.spcOid &&
+ old_locator.relNumber == existing->new_locator.relNumber)
+ {
+ original_locator = existing->original_locator;
+ break;
+ }
+ }
+
+ /* Allocate in TopTransactionContext memory context */
+ rewrite = MemoryContextAlloc(TopTransactionContext,
+ sizeof(PgStat_PendingRewrite));
+
+ rewrite->old_locator = old_locator;
+ rewrite->new_locator = new_locator;
+ rewrite->original_locator = original_locator;
+ rewrite->nest_level = GetCurrentTransactionNestLevel();
+
+ /* Add to the list */
+ rewrite->next = pending_rewrites;
+ pending_rewrites = rewrite;
+}
+
+void
+pgstat_clear_rewrite(void)
+{
+ pending_rewrites = NULL;
+}
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index 5e2d69e6297..8ed8f5317f3 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel)
}
pgStatXactStack = NULL;
+ pgstat_clear_rewrite();
+
/* Make sure any stats snapshot is thrown away */
pgstat_clear_snapshot();
}
@@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo
void
pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid)
{
- if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL))
+ PgStat_EntryRef *entry_ref;
+
+ entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL);
+
+ if (entry_ref)
{
+ /*
+ * For relations stats, we key by physical file location, not by
+ * relation OID. This means during operations like ALTER TYPE where
+ * the relation OID changes but the relfilenode stays the same (no
+ * actual rewrite needed), we'll find an existing entry.
+ *
+ * This is expected behavior, we want to preserve stats across the
+ * catalog change. Simply reset and recreate the entry for the new
+ * relation OID without warning.
+ */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ pgstat_reset(kind, dboid, objid);
+ create_drop_transactional_internal(kind, dboid, objid, true);
+ return;
+ }
+
ereport(WARNING,
errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64,
(pgstat_get_kind_info(kind))->name, dboid,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 0572ab424e7..d84058db102 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -87,6 +87,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/relmapper.h"
#include "utils/resowner.h"
#include "utils/snapmgr.h"
@@ -3782,6 +3783,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
MultiXactId minmulti = InvalidMultiXactId;
TransactionId freezeXid = InvalidTransactionId;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator = relation->rd_locator;
if (!IsBinaryUpgrade)
{
@@ -3953,6 +3955,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
table_close(pg_class, RowExclusiveLock);
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Make the pg_class row change or relation map change visible. This will
* cause the relcache entry to get updated, too.
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 941a39bd16e..5344ece9ecf 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -697,7 +697,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id);
extern void pgstat_create_relation(Relation rel);
extern void pgstat_drop_relation(Relation rel);
-extern void pgstat_copy_relation_stats(Relation dst, Relation src);
+extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment);
extern void pgstat_init_relation(Relation rel);
extern void pgstat_assoc_relation(Relation rel);
@@ -709,6 +709,9 @@ extern void pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_mark_rewrite(RelFileLocator old_locator,
+ RelFileLocator new_locator);
+extern void pgstat_clear_rewrite(void);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8cf40c87043..1544f54e918 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2349,6 +2349,7 @@ PgStat_LockEntry
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
PgStat_PendingLock
+PgStat_PendingRewrite
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
--Uc+t0mng6H5VV2zk--
^ permalink raw reply [nested|flat] 78+ messages in thread
* [PATCH v16 2/2] handle relation statistics correctly during rewrites
@ 2026-05-18 15:20 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 78+ messages in thread
From: Bertrand Drouvot @ 2026-05-18 15:20 UTC (permalink / raw)
This patch ensures that relation statistics are preserved correctly when a
relation is rewritten (e.g., during CLUSTER, VACUUM FULL, ALTER TABLE
operations that require a table rewrite, or TRUNCATE).
Since patch 0001 keys PGSTAT_KIND_RELATION by the physical RelFileLocator
rather than the relation OID, a rewrite (which assigns a new relfilenode)
would lose the accumulated statistics if not handled specially.
The approach is:
1. When a rewrite occurs (RelationSetNewRelfilenumber or swap_relation_files),
record the old and new locators via pgstat_mark_rewrite().
2. During pgstat_assoc_relation(), if the relation's new locator matches a
pending rewrite, continue using the stats entry associated with the old
locator. This ensures stats accumulated after the rewrite are still
tracked in the original entry.
3. At transaction commit (AtEOXact_PgStat_Relations), process pending
rewrites: flush any pending stats for the old locator, copy them to the
new locator's stats entry, and drop the old entry.
4. At subtransaction abort, remove rewrites from the aborted nesting level.
5. For two-phase transactions, the rewrite information is recorded in the
TwoPhasePgStatRecord and replayed at commit/abort time.
This also handles chained rewrites (multiple rewrites of the same relation
in a single transaction) by tracking the original locator through the chain.
---
src/backend/catalog/index.c | 2 +-
src/backend/commands/repack.c | 5 +
src/backend/commands/tablecmds.c | 6 +
src/backend/utils/activity/pgstat_relation.c | 392 ++++++++++++++++++-
src/backend/utils/activity/pgstat_xact.c | 25 +-
src/backend/utils/cache/relcache.c | 6 +
src/include/pgstat.h | 5 +-
src/tools/pgindent/typedefs.list | 1 +
8 files changed, 425 insertions(+), 17 deletions(-)
92.8% src/backend/utils/activity/
4.8% src/backend/
diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c
index 9407c357f27..335ff1e50f6 100644
--- a/src/backend/catalog/index.c
+++ b/src/backend/catalog/index.c
@@ -1813,7 +1813,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName)
changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId);
/* copy over statistics from old to new index */
- pgstat_copy_relation_stats(newClassRel, oldClassRel);
+ pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false);
/* Copy data of pg_statistic from the old index to the new one */
CopyStatistics(oldIndexId, newIndexId);
diff --git a/src/backend/commands/repack.c b/src/backend/commands/repack.c
index 351a3cc32e8..eda4135bcd1 100644
--- a/src/backend/commands/repack.c
+++ b/src/backend/commands/repack.c
@@ -1633,6 +1633,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class,
rel1 = relation_open(r1, NoLock);
rel2 = relation_open(r2, NoLock);
+
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind))
+ pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator);
+
rel2->rd_createSubid = rel1->rd_createSubid;
rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid;
rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid;
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 92b0f38c353..b553b9a4970 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -17129,6 +17129,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
Oid reltoastrelid;
RelFileNumber newrelfilenumber;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator;
List *reltoastidxids = NIL;
ListCell *lc;
@@ -17167,6 +17168,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
newrlocator = rel->rd_locator;
newrlocator.relNumber = newrelfilenumber;
newrlocator.spcOid = newTableSpace;
+ oldrlocator = rel->rd_locator;
/* hand off to AM to actually create new rel storage and copy the data */
if (rel->rd_rel->relkind == RELKIND_INDEX)
@@ -17179,6 +17181,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode)
table_relation_copy_data(rel, &newrlocator);
}
+ /* mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Update the pg_class row.
*
diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c
index 7fcda7f7518..18d1b9fc59d 100644
--- a/src/backend/utils/activity/pgstat_relation.c
+++ b/src/backend/utils/activity/pgstat_relation.c
@@ -30,6 +30,19 @@
#include "utils/syscache.h"
#include "utils/timestamp.h"
+/* Pending rewrite operations for stats copying */
+typedef struct PgStat_PendingRewrite
+{
+ RelFileLocator old_locator;
+ RelFileLocator new_locator;
+ RelFileLocator original_locator;
+ int nest_level; /* Transaction nesting level where rewrite
+ * occurred */
+ struct PgStat_PendingRewrite *next;
+} PgStat_PendingRewrite;
+
+/* The pending rewrites list for current transaction */
+static PgStat_PendingRewrite *pending_rewrites = NULL;
/* Record that's written to 2PC state file when pgstat state is persisted */
typedef struct TwoPhasePgStatRecord
@@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord
PgStat_Counter deleted_pre_truncdrop;
RelFileLocator locator; /* table's rd_locator */
bool truncdropped; /* was the relation truncated/dropped? */
+ RelFileLocator rewrite_old_locator;
+ int rewrite_nest_level;
} TwoPhasePgStatRecord;
@@ -54,27 +69,71 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans);
/*
- * Copy stats between relations. This is used for things like REINDEX
+ * Copy stats between RelFileLocator. This is used for things like REINDEX
* CONCURRENTLY.
*/
void
-pgstat_copy_relation_stats(Relation dst, Relation src)
+pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment)
{
PgStat_StatTabEntry *srcstats;
PgStatShared_Relation *dstshstats;
PgStat_EntryRef *dst_ref;
- srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src), NULL);
+ srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION,
+ src.dbOid,
+ RelFileLocatorToPgStatObjid(src),
+ NULL);
if (!srcstats)
return;
dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION,
- dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId,
- RelationGetRelid(dst),
+ dst.dbOid,
+ RelFileLocatorToPgStatObjid(dst),
false);
dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats;
- dstshstats->stats = *srcstats;
+
+ if (!increment)
+ dstshstats->stats = *srcstats;
+ else
+ {
+ /* Increment those statistics */
+#define RELFSTAT_ACC(fld, stats_to_add) \
+ (dstshstats->stats.fld += stats_to_add->fld)
+ RELFSTAT_ACC(numscans, srcstats);
+ RELFSTAT_ACC(tuples_returned, srcstats);
+ RELFSTAT_ACC(tuples_fetched, srcstats);
+ RELFSTAT_ACC(tuples_inserted, srcstats);
+ RELFSTAT_ACC(tuples_updated, srcstats);
+ RELFSTAT_ACC(tuples_deleted, srcstats);
+ RELFSTAT_ACC(tuples_hot_updated, srcstats);
+ RELFSTAT_ACC(tuples_newpage_updated, srcstats);
+ RELFSTAT_ACC(live_tuples, srcstats);
+ RELFSTAT_ACC(dead_tuples, srcstats);
+ RELFSTAT_ACC(mod_since_analyze, srcstats);
+ RELFSTAT_ACC(ins_since_vacuum, srcstats);
+ RELFSTAT_ACC(blocks_fetched, srcstats);
+ RELFSTAT_ACC(blocks_hit, srcstats);
+ RELFSTAT_ACC(vacuum_count, srcstats);
+ RELFSTAT_ACC(autovacuum_count, srcstats);
+ RELFSTAT_ACC(analyze_count, srcstats);
+ RELFSTAT_ACC(autoanalyze_count, srcstats);
+ RELFSTAT_ACC(total_vacuum_time, srcstats);
+ RELFSTAT_ACC(total_autovacuum_time, srcstats);
+ RELFSTAT_ACC(total_analyze_time, srcstats);
+ RELFSTAT_ACC(total_autoanalyze_time, srcstats);
+#undef RELFSTAT_ACC
+
+ /* Replace those statistics */
+#define RELFSTAT_REP(fld, stats_to_rep) \
+ (dstshstats->stats.fld = stats_to_rep->fld)
+ RELFSTAT_REP(lastscan, srcstats);
+ RELFSTAT_REP(last_vacuum_time, srcstats);
+ RELFSTAT_REP(last_autovacuum_time, srcstats);
+ RELFSTAT_REP(last_analyze_time, srcstats);
+ RELFSTAT_REP(last_autoanalyze_time, srcstats);
+#undef RELFSTAT_REP
+ }
pgstat_unlock_entry(dst_ref);
}
@@ -136,6 +195,7 @@ void
pgstat_assoc_relation(Relation rel)
{
RelFileLocator locator;
+ PgStat_TableStatus *pgstat_info;
Assert(rel->pgstat_enabled);
Assert(rel->pgstat_info == NULL);
@@ -164,14 +224,54 @@ pgstat_assoc_relation(Relation rel)
locator.relNumber = rel->rd_id;
}
+ /*
+ * If this relation was rewritten during the current transaction we may be
+ * reopening it with its new RelFileLocator. In that case, continue using
+ * the stats entry associated with the old locator rather than creating a
+ * new one. This ensures all stats from before and after the rewrite are
+ * tracked in a single entry which will be properly copied to the new
+ * locator at transaction commit.
+ */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (locator.dbOid == rewrite->new_locator.dbOid &&
+ locator.spcOid == rewrite->new_locator.spcOid &&
+ locator.relNumber == rewrite->new_locator.relNumber)
+ {
+ pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator);
+ goto found_entry;
+ }
+ }
+ }
+
/* Else find or make the PgStat_TableStatus entry, and update link */
- rel->pgstat_info = pgstat_prep_relation_pending(locator);
+ pgstat_info = pgstat_prep_relation_pending(locator);
+
+found_entry:
+ rel->pgstat_info = pgstat_info;
+
+ /*
+ * For relations stats, we key by physical file location, not by relation
+ * OID. This means during operations like ALTER TYPE it's possible that
+ * the relation OID changes but the relfilenode stays the same (no actual
+ * rewrite needed). Unlink the old relation first.
+ */
+ if (pgstat_info->relation != NULL &&
+ pgstat_info->relation != rel)
+ {
+ pgstat_info->relation->pgstat_info = NULL;
+ pgstat_info->relation = NULL;
+ }
/* don't allow link a stats to multiple relcache entries */
- Assert(rel->pgstat_info->relation == NULL);
+ Assert(pgstat_info->relation == NULL);
/* mark this relation as the owner */
- rel->pgstat_info->relation = rel;
+ pgstat_info->relation = rel;
}
/*
@@ -214,14 +314,37 @@ pgstat_drop_relation(Relation rel)
{
int nest_level = GetCurrentTransactionNestLevel();
PgStat_TableStatus *pgstat_info;
+ bool skip_transactional_drop = false;
/* don't track stats for relations without storage */
if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind))
return;
- pgstat_drop_transactional(PGSTAT_KIND_RELATION,
- rel->rd_locator.dbOid,
- RelFileLocatorToPgStatObjid(rel->rd_locator));
+ /* Check if this drop is part of a pending rewrite */
+ if (pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid &&
+ rel->rd_locator.spcOid == rewrite->old_locator.spcOid &&
+ rel->rd_locator.relNumber == rewrite->old_locator.relNumber)
+ {
+ skip_transactional_drop = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * If it is part of a rewrite, drop its stats later, for example in
+ * AtEOXact_PgStat_Relations(), so skip it here.
+ */
+ if (!skip_transactional_drop)
+ pgstat_drop_transactional(PGSTAT_KIND_RELATION,
+ rel->rd_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rel->rd_locator));
if (!pgstat_should_count_relation(rel))
return;
@@ -668,6 +791,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit)
}
tabstat->trans = NULL;
}
+
+ /* preserve the stats in case of rewrite */
+ if (isCommit && pending_rewrites != NULL)
+ {
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *prev = NULL;
+ PgStat_PendingRewrite *current = pending_rewrites;
+ PgStat_PendingRewrite *next;
+
+ /* reverse the rewrites list to process in chronological order */
+ while (current != NULL)
+ {
+ next = current->next;
+ current->next = prev;
+ prev = current;
+ current = next;
+ }
+
+ /* now process rewrites in chronological order */
+ for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ pgstat_copy_relation_stats(rewrite->new_locator,
+ rewrite->old_locator, true);
+
+ /* drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rewrite->old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rewrite->old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
+ }
+
+ pending_rewrites = NULL;
}
/*
@@ -683,6 +848,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in
PgStat_TableXactStatus *trans;
PgStat_TableXactStatus *next_trans;
+ /*
+ * If we don't commit then remove the associated rewrites if any, to keep
+ * the rewrite chain in sync with what will be eventually committed.
+ */
+ if (!isCommit)
+ {
+ PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites;
+
+ while (*rewrite_ptr != NULL)
+ {
+ if ((*rewrite_ptr)->nest_level >= nestDepth)
+ {
+ PgStat_PendingRewrite *to_remove = *rewrite_ptr;
+
+ *rewrite_ptr = (*rewrite_ptr)->next;
+ pfree(to_remove);
+ }
+ else
+ {
+ rewrite_ptr = &((*rewrite_ptr)->next);
+ }
+ }
+ }
+
for (trans = xact_state->first; trans != NULL; trans = next_trans)
{
PgStat_TableStatus *tabstat;
@@ -762,11 +951,19 @@ void
AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
{
PgStat_TableXactStatus *trans;
+ PgStat_PendingRewrite *rewrite;
+ /*
+ * For each tabstat, find its matching rewrite and remove it from the
+ * pending rewrites list. This way, after processing all tabstats, pending
+ * rewrites will only contain rewrite only transactions.
+ */
for (trans = xact_state->first; trans != NULL; trans = trans->next)
{
PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY;
TwoPhasePgStatRecord record;
+ PgStat_PendingRewrite **rewrite_ptr;
+ bool found_rewrite = false;
Assert(trans->nest_level == 1);
Assert(trans->upper == NULL);
@@ -786,10 +983,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
record.locator = tabstat->locator;
record.truncdropped = trans->truncdropped;
+ record.rewrite_nest_level = 0;
+
+ /*
+ * Look for a matching rewrite and remove it from pending rewrites. We
+ * check three possible matches:
+ *
+ * The new_locator when stats have been added after the rewrite. The
+ * old_locator when stats have been added before the rewrite but not
+ * after. The original_locator when this tabstat is part of a rewrite
+ * chain.
+ */
+ rewrite_ptr = &pending_rewrites;
+ while (*rewrite_ptr != NULL)
+ {
+ rewrite = *rewrite_ptr;
+
+ if ((record.locator.dbOid == rewrite->new_locator.dbOid &&
+ record.locator.spcOid == rewrite->new_locator.spcOid &&
+ record.locator.relNumber == rewrite->new_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->old_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->old_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->old_locator.relNumber) ||
+ (tabstat->locator.dbOid == rewrite->original_locator.dbOid &&
+ tabstat->locator.spcOid == rewrite->original_locator.spcOid &&
+ tabstat->locator.relNumber == rewrite->original_locator.relNumber))
+ {
+ /*
+ * Found matching rewrite. Record the rewrite information and
+ * remove this rewrite from the list since it's now handled.
+ */
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.locator = rewrite->new_locator;
+ found_rewrite = true;
+
+ /* Remove from pending_rewrites list */
+ *rewrite_ptr = rewrite->next;
+ pfree(rewrite);
+ break;
+ }
+ else
+ {
+ /* Move to next rewrite in the list */
+ rewrite_ptr = &(rewrite->next);
+ }
+ }
+
+ /* If no rewrite found, clear the rewrite fields */
+ if (!found_rewrite)
+ {
+ memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator));
+ }
+
+ RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
+ &record, sizeof(TwoPhasePgStatRecord));
+ }
+
+ /*
+ * Now process any rewrites still pending. These are rewrite only
+ * transactions. We need to preserve their stats even though there's no
+ * tabstat entry for them.
+ */
+ for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next)
+ {
+ TwoPhasePgStatRecord record;
+
+ memset(&record, 0, sizeof(TwoPhasePgStatRecord));
+ record.locator = rewrite->new_locator;
+ record.rewrite_old_locator = rewrite->original_locator;
+ record.rewrite_nest_level = rewrite->nest_level;
+ record.truncdropped = false;
RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0,
&record, sizeof(TwoPhasePgStatRecord));
}
+
+ pending_rewrites = NULL;
}
/*
@@ -812,6 +1082,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state)
tabstat = trans->parent;
tabstat->trans = NULL;
}
+
+ pending_rewrites = NULL;
}
/*
@@ -847,6 +1119,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info,
pgstat_info->counts.changed_tuples +=
rec->tuples_inserted + rec->tuples_updated +
rec->tuples_deleted;
+
+ if (rec->rewrite_nest_level > 0)
+ {
+ PgStat_EntryRef *old_entry_ref;
+
+ /* Flush any pending stats for old locator first */
+ old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator));
+
+ if (old_entry_ref && old_entry_ref->pending)
+ pgstat_relation_flush_cb(old_entry_ref, false);
+
+ /* Copy stats from old to new locator */
+ pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator,
+ true);
+
+ /* Drop old locator's stats */
+ if (!pgstat_drop_entry(PGSTAT_KIND_RELATION,
+ rec->rewrite_old_locator.dbOid,
+ RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)))
+ pgstat_request_entry_refs_gc();
+ }
}
/*
@@ -861,9 +1156,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info,
{
TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata;
PgStat_TableStatus *pgstat_info;
+ RelFileLocator target_locator;
+
+ /*
+ * For aborted transactions with rewrites (like TRUNCATE), we need to
+ * restore stats to the old locator, not the new one. The new locator
+ * should be dropped since the rewrite is being rolled back.
+ */
+ if (rec->rewrite_nest_level > 0)
+ {
+ /* Use the old locator */
+ target_locator = rec->rewrite_old_locator;
+ }
+ else
+ {
+ /* No rewrite, use the original locator */
+ target_locator = rec->locator;
+ }
/* Find or create a tabstat entry for the target locator */
- pgstat_info = pgstat_prep_relation_pending(rec->locator);
+ pgstat_info = pgstat_prep_relation_pending(target_locator);
/* Same math as in AtEOXact_PgStat, abort case */
if (rec->truncdropped)
@@ -918,7 +1230,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait)
tabentry->numscans += lstats->counts.numscans;
if (lstats->counts.numscans)
{
- TimestampTz t = GetCurrentTransactionStopTimestamp();
+ TimestampTz t;
+
+ /*
+ * Checking the transaction state due to the flush call in
+ * pgstat_twophase_postcommit() that would break the assertion on the
+ * state in GetCurrentTransactionStopTimestamp().
+ */
+ if (!IsTransactionState())
+ t = GetCurrentTransactionStopTimestamp();
+ else
+ t = GetCurrentTimestamp();
if (t > tabentry->lastscan)
tabentry->lastscan = t;
@@ -1169,3 +1491,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator)
ReleaseSysCache(tuple);
return result;
}
+
+/*
+ * Mark that a relation rewrite has occurred, preserving the original locator
+ * so stats can be copied at transaction commit.
+ */
+void
+pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator)
+{
+ PgStat_PendingRewrite *rewrite;
+ PgStat_PendingRewrite *existing;
+ RelFileLocator original_locator = old_locator;
+
+ for (existing = pending_rewrites; existing != NULL; existing = existing->next)
+ {
+ if (old_locator.dbOid == existing->new_locator.dbOid &&
+ old_locator.spcOid == existing->new_locator.spcOid &&
+ old_locator.relNumber == existing->new_locator.relNumber)
+ {
+ original_locator = existing->original_locator;
+ break;
+ }
+ }
+
+ /* Allocate in TopTransactionContext memory context */
+ rewrite = MemoryContextAlloc(TopTransactionContext,
+ sizeof(PgStat_PendingRewrite));
+
+ rewrite->old_locator = old_locator;
+ rewrite->new_locator = new_locator;
+ rewrite->original_locator = original_locator;
+ rewrite->nest_level = GetCurrentTransactionNestLevel();
+
+ /* Add to the list */
+ rewrite->next = pending_rewrites;
+ pending_rewrites = rewrite;
+}
+
+void
+pgstat_clear_rewrite(void)
+{
+ pending_rewrites = NULL;
+}
diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c
index 5e2d69e6297..8ed8f5317f3 100644
--- a/src/backend/utils/activity/pgstat_xact.c
+++ b/src/backend/utils/activity/pgstat_xact.c
@@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel)
}
pgStatXactStack = NULL;
+ pgstat_clear_rewrite();
+
/* Make sure any stats snapshot is thrown away */
pgstat_clear_snapshot();
}
@@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo
void
pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid)
{
- if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL))
+ PgStat_EntryRef *entry_ref;
+
+ entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL);
+
+ if (entry_ref)
{
+ /*
+ * For relations stats, we key by physical file location, not by
+ * relation OID. This means during operations like ALTER TYPE where
+ * the relation OID changes but the relfilenode stays the same (no
+ * actual rewrite needed), we'll find an existing entry.
+ *
+ * This is expected behavior, we want to preserve stats across the
+ * catalog change. Simply reset and recreate the entry for the new
+ * relation OID without warning.
+ */
+ if (kind == PGSTAT_KIND_RELATION)
+ {
+ pgstat_reset(kind, dboid, objid);
+ create_drop_transactional_internal(kind, dboid, objid, true);
+ return;
+ }
+
ereport(WARNING,
errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64,
(pgstat_get_kind_info(kind))->name, dboid,
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 0572ab424e7..d84058db102 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -87,6 +87,7 @@
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
+#include "utils/pgstat_internal.h"
#include "utils/relmapper.h"
#include "utils/resowner.h"
#include "utils/snapmgr.h"
@@ -3782,6 +3783,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
MultiXactId minmulti = InvalidMultiXactId;
TransactionId freezeXid = InvalidTransactionId;
RelFileLocator newrlocator;
+ RelFileLocator oldrlocator = relation->rd_locator;
if (!IsBinaryUpgrade)
{
@@ -3953,6 +3955,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence)
table_close(pg_class, RowExclusiveLock);
+ /* Mark that a rewrite happened */
+ if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind))
+ pgstat_mark_rewrite(oldrlocator, newrlocator);
+
/*
* Make the pg_class row change or relation map change visible. This will
* cause the relcache entry to get updated, too.
diff --git a/src/include/pgstat.h b/src/include/pgstat.h
index 941a39bd16e..5344ece9ecf 100644
--- a/src/include/pgstat.h
+++ b/src/include/pgstat.h
@@ -697,7 +697,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id);
extern void pgstat_create_relation(Relation rel);
extern void pgstat_drop_relation(Relation rel);
-extern void pgstat_copy_relation_stats(Relation dst, Relation src);
+extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment);
extern void pgstat_init_relation(Relation rel);
extern void pgstat_assoc_relation(Relation rel);
@@ -709,6 +709,9 @@ extern void pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples,
extern void pgstat_report_analyze(Relation rel,
PgStat_Counter livetuples, PgStat_Counter deadtuples,
bool resetcounter, TimestampTz starttime);
+extern void pgstat_mark_rewrite(RelFileLocator old_locator,
+ RelFileLocator new_locator);
+extern void pgstat_clear_rewrite(void);
/*
* If stats are enabled, but pending data hasn't been prepared yet, call
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 8cf40c87043..1544f54e918 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2349,6 +2349,7 @@ PgStat_LockEntry
PgStat_PendingDroppedStatsItem
PgStat_PendingIO
PgStat_PendingLock
+PgStat_PendingRewrite
PgStat_SLRUStats
PgStat_ShmemControl
PgStat_Snapshot
--
2.34.1
--Uc+t0mng6H5VV2zk--
^ permalink raw reply [nested|flat] 78+ messages in thread
end of thread, other threads:[~2026-05-18 15:20 UTC | newest]
Thread overview: 78+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-01-31 14:42 Re: [PoC] Improve dead tuple storage for lazy vacuum Masahiko Sawada <[email protected]>
2023-02-07 09:25 ` John Naylor <[email protected]>
2023-02-07 10:22 ` John Naylor <[email protected]>
2023-02-09 07:08 ` Masahiko Sawada <[email protected]>
2023-02-09 12:56 ` John Naylor <[email protected]>
2023-02-10 06:51 ` John Naylor <[email protected]>
2023-02-10 07:15 ` Masahiko Sawada <[email protected]>
2023-02-11 05:33 ` John Naylor <[email protected]>
2023-02-13 07:50 ` Masahiko Sawada <[email protected]>
2023-02-14 11:24 ` John Naylor <[email protected]>
2023-02-14 12:36 ` John Naylor <[email protected]>
2023-02-16 03:24 ` Masahiko Sawada <[email protected]>
2023-02-16 09:22 ` John Naylor <[email protected]>
2023-02-16 16:44 ` Andres Freund <[email protected]>
2023-02-17 08:00 ` John Naylor <[email protected]>
2023-02-20 05:56 ` Masahiko Sawada <[email protected]>
2023-02-22 06:15 ` Masahiko Sawada <[email protected]>
2023-02-22 07:35 ` John Naylor <[email protected]>
2023-02-22 08:29 ` Masahiko Sawada <[email protected]>
2023-02-22 09:55 ` John Naylor <[email protected]>
2023-02-23 09:40 ` John Naylor <[email protected]>
2023-02-24 08:40 ` Masahiko Sawada <[email protected]>
2023-02-27 17:07 ` John Naylor <[email protected]>
2023-02-24 05:50 ` Masahiko Sawada <[email protected]>
2023-02-28 06:42 ` John Naylor <[email protected]>
2023-02-28 13:20 ` Masahiko Sawada <[email protected]>
2023-02-28 15:09 ` Masahiko Sawada <[email protected]>
2023-03-01 06:37 ` John Naylor <[email protected]>
2023-03-01 11:58 ` Masahiko Sawada <[email protected]>
2023-03-03 11:03 ` John Naylor <[email protected]>
2023-03-06 06:27 ` Masahiko Sawada <[email protected]>
2023-03-06 16:00 ` John Naylor <[email protected]>
2023-03-07 01:24 ` Masahiko Sawada <[email protected]>
2023-03-08 04:40 ` John Naylor <[email protected]>
2023-03-09 06:51 ` Masahiko Sawada <[email protected]>
2023-03-10 06:42 ` John Naylor <[email protected]>
2023-03-10 14:30 ` Masahiko Sawada <[email protected]>
2023-03-10 15:26 ` Masahiko Sawada <[email protected]>
2023-04-17 15:20 ` Masahiko Sawada <[email protected]>
2023-03-11 15:54 ` John Naylor <[email protected]>
2023-03-13 01:41 ` Masahiko Sawada <[email protected]>
2023-03-13 13:28 ` John Naylor <[email protected]>
2023-03-13 14:55 ` Masahiko Sawada <[email protected]>
2023-03-14 11:27 ` John Naylor <[email protected]>
2023-03-15 02:32 ` Masahiko Sawada <[email protected]>
2023-03-17 07:02 ` John Naylor <[email protected]>
2023-03-17 07:49 ` Masahiko Sawada <[email protected]>
2023-03-20 05:24 ` Masahiko Sawada <[email protected]>
2023-04-07 09:55 ` John Naylor <[email protected]>
2023-04-17 13:48 ` Masahiko Sawada <[email protected]>
2023-04-19 07:02 ` John Naylor <[email protected]>
2023-04-24 05:45 ` Masahiko Sawada <[email protected]>
2023-05-23 23:16 ` John Naylor <[email protected]>
2023-06-05 10:31 ` Masahiko Sawada <[email protected]>
2023-06-06 05:13 ` John Naylor <[email protected]>
2023-06-13 05:46 ` Masahiko Sawada <[email protected]>
2023-06-14 06:23 ` John Naylor <[email protected]>
2023-05-08 10:23 ` John Naylor <[email protected]>
2025-11-04 13:52 [PATCH v15 2/2] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]>
2025-11-04 13:52 [PATCH v7 3/3] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]>
2025-11-04 13:52 [PATCH v11 2/2] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]>
2025-11-04 13:52 [PATCH v13 2/2] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]>
2025-11-04 13:52 [PATCH v8 2/2] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]>
2025-11-04 13:52 [PATCH v14 2/2] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]>
2025-11-04 13:52 [PATCH v14 2/2] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]>
2025-11-04 13:52 [PATCH v10 2/2] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]>
2025-11-04 13:52 [PATCH v7 3/3] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]>
2025-11-04 13:52 [PATCH v15 2/2] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]>
2025-11-04 13:52 [PATCH v13 2/2] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]>
2025-11-04 13:52 [PATCH v9 2/2] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]>
2025-11-04 13:52 [PATCH v12 2/2] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]>
2025-11-04 13:52 [PATCH v11 2/2] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]>
2025-11-04 13:52 [PATCH v10 2/2] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]>
2025-11-04 13:52 [PATCH v8 2/2] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]>
2025-11-04 13:52 [PATCH v9 2/2] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]>
2025-11-04 13:52 [PATCH v12 2/2] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]>
2026-05-18 15:20 [PATCH v16 2/2] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]>
2026-05-18 15:20 [PATCH v16 2/2] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox