Subject: Re: [PoC] Improve dead tuple storage for lazy vacuum

From: Masahiko Sawada <[email-masked]>
Date: Tue, 31 Jan 2023 23:42:32 +0900

On Mon, Jan 30, 2023 at 11:30 PM Masahiko Sawada <[email-masked]> wrote:
>
> On Mon, Jan 30, 2023 at 1:31 PM John Naylor
> <[email-masked]> wrote:
> >
> > On Sun, Jan 29, 2023 at 9:50 PM Masahiko Sawada <[email-masked]> wrote:
> > >
> > > On Sat, Jan 28, 2023 at 8:33 PM John Naylor
> > > <[email-masked]> wrote:
> >
> > > > The first implementation should be simple, easy to test/verify, easy to understand, and easy to replace. As much as possible anyway.
> > >
> > > Yes, but if a concurrent writer waits for another process to finish
> > > the iteration, it ends up waiting on a lwlock, which is not
> > > interruptible.
> > >
> > > >
> > > > > So the idea is that we set iter_active to true (with the
> > > > > lock in exclusive mode), and prevent concurrent updates when the flag
> > > > > is true.
> > > >
> > > > ...by throwing elog(ERROR)? I'm not so sure users of this API would prefer that to waiting.
> > >
> > > Right. I think if we want to wait rather than an ERROR, the waiter
> > > should wait in an interruptible way, for example, a condition
> > > variable. I did a simpler way in the v22 patch.
> > >
> > > ...but looking at dshash.c, dshash_seq_next() seems to return an entry
> > > while holding a lwlock on the partition. My assumption might be wrong.
> >
> > Using partitions there makes holding a lock less painful on average, I imagine, but I don't know the details there.
> >
> > If we make it clear that the first committed version is not (yet) designed for high concurrency with mixed read-write workloads, I think waiting (as a protocol) is fine. If waiting is a problem for some use case, at that point we should just go all the way and replace the locking entirely. In fact, it might be good to spell this out in the top-level comment and include a link to the second ART paper.
>
> Agreed. Will update the comments.
>
> >
> > > > [thinks some more...] Is there an API-level assumption that hasn't been spelled out? Would it help to have a parameter for whether the iteration function wants to reserve the privilege to perform writes? It could take the appropriate lock at the start, and there could then be multiple read-only iterators, but only one read/write iterator. Note, I'm just guessing here, and I don't want to make things more difficult for future improvements.
> > >
> > > Seems a good idea. Given the use case for parallel heap vacuum, it
> > > would be a good idea to support having multiple read-only writers. The
> > > iteration of the v22 is read-only, so if we want to support read-write
> > > iterator, we would need to support a function that modifies the
> > > current key-value returned by the iteration.
> >
> > Okay, so updating during iteration is not currently supported. It could in the future, but I'd say that can also wait for fine-grained concurrency support. Intermediate-term, we should at least make it straightforward to support:
> >
> > 1) parallel heap vacuum  -> multiple read-only iterators
> > 2) parallel heap pruning -> multiple writers
> >
> > It may or may not be worth it for someone to actually start either of those projects, and there are other ways to improve vacuum that may be more pressing. That said, it seems the tid store with global locking would certainly work fine for #1 and maybe "not too bad" for #2. #2 can also mitigate waiting by using larger batching, or the leader process could "pre-warm" the tid store with zero-values using block numbers from the visibility map.
>
> True. Using a larger batching method seems to be worth testing when we
> implement the parallel heap pruning.
>
> In the next version patch, I'm going to update the locking support
> part and incorporate other comments I got.
>

I've attached v24 patches. The locking support patch is separated
(0005 patch). Also I kept the updates for TidStore and the vacuum
integration from v23 separate.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

---

From: John Naylor <[email-masked]>
Date: Tue, 7 Feb 2023 16:25:44 +0700

> integration from v23 separate.

Okay, that's a lot more simple, and closer to what I imagined. For v25, I
squashed v24's additions and added a couple of my own. I've kept the CF
status at "needs review" because no specific action is required at the
moment.

I did start to review the TID store some more, but that's on hold because
something else came up: On a lark I decided to re-run some benchmarks to
see if anything got lost in converting to a template, and that led me down
a rabbit hole -- some good and bad news on that below.

0001:

I removed the uint64 case, as discussed. There is now a brief commit
message, but needs to be fleshed out a bit. I took another look at the Arm
optimization that Nathan found some month ago, for forming the highbit
mask, but that doesn't play nicely with how node32 uses it, so I decided
against it. I added a comment to describe the reasoning in case someone
else gets a similar idea.

I briefly looked into "separate-commit TODO: move non-SIMD fallbacks to
their own header to clean up the #ifdef maze.", but decided it wasn't such
a clear win to justify starting the work now. It's still in the back of my
mind, but I removed the reminder from the commit message.

0003:

The template now requires the value to be passed as a pointer. That was a
pretty trivial change, but affected multiple other patches, so not sent
separately. Also adds a forgotten RT_ prefix to the bitmap macros and adds
a top comment to the *_impl.h headers. There are some comment fixes. The
changes were either trivial or discussed earlier, so also not sent
separately.

0004/5: I wanted to measure the load time as well as search time in
bench_search_random_nodes(). That's kept separate to make it easier to test
other patch versions.

The bad news is that the speed of loading TIDs in
bench_seq/shuffle_search() has regressed noticeably. I can't reproduce this
in any other bench function and was the reason for writing 0005 to begin
with. More confusingly, my efforts to fix this improved *other* functions,
but the former didn't budge at all. First the patches:

0006 adds and removes some "inline" declarations (where it made sense), and
added some for "pg_noinline" based on Andres' advice some months ago.

0007 removes some dead code. RT_NODE_INSERT_INNER is only called during
RT_SET_EXTEND, so it can't possibly find an existing key. This kind of
change is much easier with the inner/node cases handled together in a
template, as far as being sure of how those cases are different. I thought
about trying the search in assert builds and verifying it doesn't exist,
but thought yet another #ifdef would be too messy.

v25-addendum-try-no-maintain-order.txt -- It makes optional keeping the key
chunks in order for the linear-search nodes. I believe the TID store no
longer cares about the ordering, but this is a text file for now because I
don't want to clutter the CI with a behavior change. Also, the second ART
paper (on concurrency) mentioned that some locking schemes don't allow
these arrays to be shifted. So it might make sense to give up entirely on
guaranteeing ordered iteration, or at least make it optional as in the
patch.

Now for some numbers:

========================================
psql -c "select * from bench_search_random_nodes(10*1000*1000)"
(min load time of three)

v15:
 mem_allocated | load_ms | search_ms
---------------+---------+-----------
     334182184 |    3352 |      2073

v25-0005:
 mem_allocated | load_ms | search_ms
---------------+---------+-----------
     331987008 |    3426 |      2126

v25-0006 (inlining or not):
 mem_allocated | load_ms | search_ms
---------------+---------+-----------
     331987008 |    3327 |      2035

v25-0007 (remove dead code):
 mem_allocated | load_ms | search_ms
---------------+---------+-----------
     331987008 |    3313 |      2037

v25-addendum...txt (no ordering):
 mem_allocated | load_ms | search_ms
---------------+---------+-----------
     331987008 |    2762 |      2042

Allowing unordered inserts helps a lot here in loading. That's expected
because there are a lot of inserts into the linear nodes. 0006 might help a
little.

========================================
psql -c "select avg(load_ms) from generate_series(1,30) x(x), lateral
(select * from bench_load_random_int(500 * 1000 * (1+x-x))) a"

v15:
         avg
----------------------
 207.3000000000000000

v25-0005:
         avg
----------------------
 190.6000000000000000

v25-0006 (inlining or not):
         avg
----------------------
 189.3333333333333333

v25-0007 (remove dead code):
         avg
----------------------
 186.4666666666666667

v25-addendum...txt (no ordering):
         avg
----------------------
 179.7000000000000000

Most of the improvement from v15 to v25 probably comes from the change from
node4 to node3, and this test stresses that node the most. That shows in
the total memory used: it goes from 152MB to 132MB. Allowing unordered
inserts helps some, the others are not convincing.

========================================
psql -c "select rt_load_ms, rt_search_ms from bench_seq_search(0, 1 * 1000
* 1000)"
(min load time of three)

v15:
 rt_load_ms | rt_search_ms
------------+--------------
        113 |          455

v25-0005:
 rt_load_ms | rt_search_ms
------------+--------------
        135 |          456

v25-0006 (inlining or not):
 rt_load_ms | rt_search_ms
------------+--------------
        136 |          455

v25-0007 (remove dead code):
 rt_load_ms | rt_search_ms
------------+--------------
        135 |          455

v25-addendum...txt (no ordering):
 rt_load_ms | rt_search_ms
------------+--------------
        134 |          455

Note: The regression seems to have started in v17, which is the first with
a full template.

Nothing so far has helped here, and previous experience has shown that
trying to profile 100ms will not be useful. Instead of putting more effort
into diving deeper, it seems a better use of time to write a benchmark that
calls the tid store itself. That's more realistic, since this function was
intended to test load and search of tids, but the tid store doesn't quite
operate so simply anymore. What do you think, Masahiko?

I'm inclined to keep 0006, because it might give a slight boost, and 0007
because it's never a bad idea to remove dead code.
--
John Naylor
EDB: http://www.enterprisedb.com
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
index 4e00b46d9b..3f831227c9 100644
--- a/src/include/lib/radixtree_insert_impl.h
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -80,9 +80,10 @@
 				}
 				else
 				{
-					int			insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
+					int			insertpos;// = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
 					int			count = n3->base.n.count;
-
+#ifdef RT_MAINTAIN_ORDERING
+					insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
 					/* shift chunks and children */
 					if (insertpos < count)
 					{
@@ -95,6 +96,9 @@
 												   count, insertpos);
 #endif
 					}
+#else
+					insertpos = count;
+#endif	/* order */
 
 					n3->base.chunks[insertpos] = chunk;
 #ifdef RT_NODE_LEVEL_LEAF
@@ -186,8 +190,10 @@
 				}
 				else
 				{
-					int	insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
+					int			insertpos;
 					int count = n32->base.n.count;
+#ifdef RT_MAINTAIN_ORDERING
+					insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
 
 					if (insertpos < count)
 					{
@@ -200,6 +206,9 @@
 												   count, insertpos);
 #endif
 					}
+#else
+					insertpos = count;
+#endif
 
 					n32->base.chunks[insertpos] = chunk;
 #ifdef RT_NODE_LEVEL_LEAF
From 86c2d232a0ea193a856cb0348e0825b5e4b7a4b7 Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Tue, 6 Dec 2022 13:39:41 +0700
Subject: [PATCH v25 2/9] Move some bitmap logic out of bitmapset.c

Add pg_rightmost_one32/64 functions. This functionality was previously
private to bitmapset.c as the RIGHTMOST_ONE macro. It has practical
use in other contexts, so move to pg_bitutils.h.

Also move the logic for selecting appropriate pg_bitutils functions
based on word size to bitmapset.h for wider visibility and add
appropriate selection for bmw_rightmost_one().

Since the previous macro relied on casting to signedbitmapword,
and the new functions do not, remove that typedef.

Design input and review by Tom Lane
Discussion: https://www.postgresql.org/message-id/CAFBsxsFW2JjTo58jtDB%2B3sZhxMx3t-3evew8%3DAcr%2BGGhC%2BkFaA%40mail.gmail.com
---
 src/backend/nodes/bitmapset.c    | 36 ++------------------------------
 src/include/nodes/bitmapset.h    | 16 ++++++++++++--
 src/include/port/pg_bitutils.h   | 31 +++++++++++++++++++++++++++
 src/tools/pgindent/typedefs.list |  1 -
 4 files changed, 47 insertions(+), 37 deletions(-)

diff --git a/src/backend/nodes/bitmapset.c b/src/backend/nodes/bitmapset.c
index 98660524ad..fcd8e2ccbc 100644
--- a/src/backend/nodes/bitmapset.c
+++ b/src/backend/nodes/bitmapset.c
@@ -32,39 +32,7 @@
 #define BITMAPSET_SIZE(nwords)	\
 	(offsetof(Bitmapset, words) + (nwords) * sizeof(bitmapword))
 
-/*----------
- * This is a well-known cute trick for isolating the rightmost one-bit
- * in a word.  It assumes two's complement arithmetic.  Consider any
- * nonzero value, and focus attention on the rightmost one.  The value is
- * then something like
- *				xxxxxx10000
- * where x's are unspecified bits.  The two's complement negative is formed
- * by inverting all the bits and adding one.  Inversion gives
- *				yyyyyy01111
- * where each y is the inverse of the corresponding x.  Incrementing gives
- *				yyyyyy10000
- * and then ANDing with the original value gives
- *				00000010000
- * This works for all cases except original value = zero, where of course
- * we get zero.
- *----------
- */
-#define RIGHTMOST_ONE(x) ((signedbitmapword) (x) & -((signedbitmapword) (x)))
-
-#define HAS_MULTIPLE_ONES(x)	((bitmapword) RIGHTMOST_ONE(x) != (x))
-
-/* Select appropriate bit-twiddling functions for bitmap word size */
-#if BITS_PER_BITMAPWORD == 32
-#define bmw_leftmost_one_pos(w)		pg_leftmost_one_pos32(w)
-#define bmw_rightmost_one_pos(w)	pg_rightmost_one_pos32(w)
-#define bmw_popcount(w)				pg_popcount32(w)
-#elif BITS_PER_BITMAPWORD == 64
-#define bmw_leftmost_one_pos(w)		pg_leftmost_one_pos64(w)
-#define bmw_rightmost_one_pos(w)	pg_rightmost_one_pos64(w)
-#define bmw_popcount(w)				pg_popcount64(w)
-#else
-#error "invalid BITS_PER_BITMAPWORD"
-#endif
+#define HAS_MULTIPLE_ONES(x)	(bmw_rightmost_one(x) != (x))

 /*
@@ -1013,7 +981,7 @@ bms_first_member(Bitmapset *a)
 		{
 			int			result;
 
-			w = RIGHTMOST_ONE(w);
+			w = bmw_rightmost_one(w);
 			a->words[wordnum] &= ~w;
 
 			result = wordnum * BITS_PER_BITMAPWORD;
diff --git a/src/include/nodes/bitmapset.h b/src/include/nodes/bitmapset.h
index 3d2225e1ae..5f9a511b4a 100644
--- a/src/include/nodes/bitmapset.h
+++ b/src/include/nodes/bitmapset.h
@@ -38,13 +38,11 @@ struct List;
 
 #define BITS_PER_BITMAPWORD 64
 typedef uint64 bitmapword;		/* must be an unsigned type */
-typedef int64 signedbitmapword; /* must be the matching signed type */
 
 #else
 
 #define BITS_PER_BITMAPWORD 32
 typedef uint32 bitmapword;		/* must be an unsigned type */
-typedef int32 signedbitmapword; /* must be the matching signed type */
 
 #endif
 
@@ -75,6 +73,20 @@ typedef enum
 	BMS_MULTIPLE				/* >1 member */
 } BMS_Membership;
 
+/* Select appropriate bit-twiddling functions for bitmap word size */
+#if BITS_PER_BITMAPWORD == 32
+#define bmw_rightmost_one(w)		pg_rightmost_one32(w)
+#define bmw_leftmost_one_pos(w)		pg_leftmost_one_pos32(w)
+#define bmw_rightmost_one_pos(w)	pg_rightmost_one_pos32(w)
+#define bmw_popcount(w)				pg_popcount32(w)
+#elif BITS_PER_BITMAPWORD == 64
+#define bmw_rightmost_one(w)		pg_rightmost_one64(w)
+#define bmw_leftmost_one_pos(w)		pg_leftmost_one_pos64(w)
+#define bmw_rightmost_one_pos(w)	pg_rightmost_one_pos64(w)
+#define bmw_popcount(w)				pg_popcount64(w)
+#else
+#error "invalid BITS_PER_BITMAPWORD"
+#endif
 
 /*
  * function prototypes in nodes/bitmapset.c
diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h
index a49a9c03d9..7235ad25ee 100644
--- a/src/include/port/pg_bitutils.h
+++ b/src/include/port/pg_bitutils.h
@@ -17,6 +17,37 @@ extern PGDLLIMPORT const uint8 pg_leftmost_one_pos[256];
 extern PGDLLIMPORT const uint8 pg_rightmost_one_pos[256];
 extern PGDLLIMPORT const uint8 pg_number_of_ones[256];
 
+/*----------
+ * This is a well-known cute trick for isolating the rightmost one-bit
+ * in a word.  It assumes two's complement arithmetic.  Consider any
+ * nonzero value, and focus attention on the rightmost one.  The value is
+ * then something like
+ *				xxxxxx10000
+ * where x's are unspecified bits.  The two's complement negative is formed
+ * by inverting all the bits and adding one.  Inversion gives
+ *				yyyyyy01111
+ * where each y is the inverse of the corresponding x.  Incrementing gives
+ *				yyyyyy10000
+ * and then ANDing with the original value gives
+ *				00000010000
+ * This works for all cases except original value = zero, where of course
+ * we get zero.
+ *----------
+ */
+static inline uint32
+pg_rightmost_one32(uint32 word)
+{
+	int32 result = (int32) word & -((int32) word);
+	return (uint32) result;
+}
+
+static inline uint64
+pg_rightmost_one64(uint64 word)
+{
+	int64 result = (int64) word & -((int64) word);
+	return (uint64) result;
+}
+
 /*
  * pg_leftmost_one_pos32
  *		Returns the position of the most significant set bit in "word",
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 07fbb7ccf6..f4d1d60cd2 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3662,7 +3662,6 @@ shmem_request_hook_type
 shmem_startup_hook_type
 sig_atomic_t
 sigjmp_buf
-signedbitmapword
 sigset_t
 size_t
 slist_head
-- 
2.39.1

From 949c6eef5ff7cc4f8ef2673f9aa63142a1d913ae Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Mon, 24 Oct 2022 14:07:09 +0900
Subject: [PATCH v25 1/9] Introduce helper SIMD functions for small byte arrays

vector8_min - helper for emulating ">=" semantics

vector8_highbit_mask - used to turn the result of a vector
comparison into a bitmask

Masahiko Sawada

Reviewed by Nathan Bossart, additional adjustments by me
Discussion: https://www.postgresql.org/message-id/CAD21AoDap240WDDdUDE0JMpCmuMMnGajrKrkCRxM7zn9Xk3JRA%40mail.gmail.com
---
 src/include/port/simd.h | 47 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)

diff --git a/src/include/port/simd.h b/src/include/port/simd.h
index c836360d4b..350e2caaea 100644
--- a/src/include/port/simd.h
+++ b/src/include/port/simd.h
@@ -79,6 +79,7 @@ static inline bool vector8_has_le(const Vector8 v, const uint8 c);
 static inline bool vector8_is_highbit_set(const Vector8 v);
 #ifndef USE_NO_SIMD
 static inline bool vector32_is_highbit_set(const Vector32 v);
+static inline uint32 vector8_highbit_mask(const Vector8 v);
 #endif
 
 /* arithmetic operations */
@@ -96,6 +97,7 @@ static inline Vector8 vector8_ssub(const Vector8 v1, const Vector8 v2);
  */
 #ifndef USE_NO_SIMD
 static inline Vector8 vector8_eq(const Vector8 v1, const Vector8 v2);
+static inline Vector8 vector8_min(const Vector8 v1, const Vector8 v2);
 static inline Vector32 vector32_eq(const Vector32 v1, const Vector32 v2);
 #endif
 
@@ -299,6 +301,36 @@ vector32_is_highbit_set(const Vector32 v)
 }
 #endif							/* ! USE_NO_SIMD */
 
+/*
+ * Return a bitmask formed from the high-bit of each element.
+ */
+#ifndef USE_NO_SIMD
+static inline uint32
+vector8_highbit_mask(const Vector8 v)
+{
+#ifdef USE_SSE2
+	return (uint32) _mm_movemask_epi8(v);
+#elif defined(USE_NEON)
+	/*
+	 * Note: There is a faster way to do this, but it returns a uint64 and
+	 * and if the caller wanted to extract the bit position using CTZ,
+	 * it would have to divide that result by 4.
+	 */
+	static const uint8 mask[16] = {
+		1 << 0, 1 << 1, 1 << 2, 1 << 3,
+		1 << 4, 1 << 5, 1 << 6, 1 << 7,
+		1 << 0, 1 << 1, 1 << 2, 1 << 3,
+		1 << 4, 1 << 5, 1 << 6, 1 << 7,
+	  };
+
+	uint8x16_t masked = vandq_u8(vld1q_u8(mask), (uint8x16_t) vshrq_n_s8(v, 7));
+	uint8x16_t maskedhi = vextq_u8(masked, masked, 8);
+
+	return (uint32) vaddvq_u16((uint16x8_t) vzip1q_u8(masked, maskedhi));
+#endif
+}
+#endif							/* ! USE_NO_SIMD */
+
 /*
  * Return the bitwise OR of the inputs
  */
@@ -372,4 +404,19 @@ vector32_eq(const Vector32 v1, const Vector32 v2)
 }
 #endif							/* ! USE_NO_SIMD */
 
+/*
+ * Given two vectors, return a vector with the minimum element of each.
+ */
+#ifndef USE_NO_SIMD
+static inline Vector8
+vector8_min(const Vector8 v1, const Vector8 v2)
+{
+#ifdef USE_SSE2
+	return _mm_min_epu8(v1, v2);
+#elif defined(USE_NEON)
+	return vminq_u8(v1, v2);
+#endif
+}
+#endif							/* ! USE_NO_SIMD */
+
 #endif							/* SIMD_H */
-- 
2.39.1

From 8edd5b4c0fcbf7681c5388faaf85a96ae451c99e Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Tue, 7 Feb 2023 13:06:00 +0700
Subject: [PATCH v25 5/9] Measure load time of bench_search_random_nodes

---
 .../bench_radix_tree/bench_radix_tree--1.0.sql  |  1 +
 contrib/bench_radix_tree/bench_radix_tree.c     | 17 ++++++++++++-----
 2 files changed, 13 insertions(+), 5 deletions(-)

diff --git a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
index 2fd689aa91..95eedbbe10 100644
--- a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
+++ b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
@@ -47,6 +47,7 @@ create function bench_search_random_nodes(
 cnt int8,
 filter_str text DEFAULT NULL,
 OUT mem_allocated int8,
+OUT load_ms int8,
 OUT search_ms int8)
 returns record
 as 'MODULE_PATHNAME'
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
index 73ddee32de..7d1e2eee57 100644
--- a/contrib/bench_radix_tree/bench_radix_tree.c
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -395,9 +395,10 @@ bench_search_random_nodes(PG_FUNCTION_ARGS)
 				end_time;
 	long		secs;
 	int			usecs;
+	int64		load_time_ms;
 	int64		search_time_ms;
-	Datum		values[2] = {0};
-	bool		nulls[2] = {0};
+	Datum		values[3] = {0};
+	bool		nulls[3] = {0};
 	/* from trial and error */
 	uint64 filter = (((uint64) 0x7F<<32) | (0x07<<24) | (0xFF<<16) | 0xFF);
 
@@ -416,13 +417,18 @@ bench_search_random_nodes(PG_FUNCTION_ARGS)
 
 	rt = rt_create(CurrentMemoryContext);
 
+	start_time = GetCurrentTimestamp();
 	for (uint64 i = 0; i < cnt; i++)
 	{
-		const uint64 hash = hash64(i);
-		const uint64 key = hash & filter;
+		uint64 hash = hash64(i);
+		uint64 key = hash & filter;
 
 		rt_set(rt, key, &key);
 	}
+	end_time = GetCurrentTimestamp();
+
+	TimestampDifference(start_time, end_time, &secs, &usecs);
+	load_time_ms = secs * 1000 + usecs / 1000;
 
 	elog(NOTICE, "sleeping for 2 seconds...");
 	pg_usleep(2 * 1000000L);
@@ -449,7 +455,8 @@ bench_search_random_nodes(PG_FUNCTION_ARGS)
 	rt_stats(rt);
 
 	values[0] = Int64GetDatum(rt_memory_usage(rt));
-	values[1] = Int64GetDatum(search_time_ms);
+	values[1] = Int64GetDatum(load_time_ms);
+	values[2] = Int64GetDatum(search_time_ms);
 
 	rt_free(rt);
 	PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
-- 
2.39.1

From 6fb21eb0b44b5923c0b736d82e86b1d4a40a71d6 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Fri, 16 Sep 2022 11:57:03 +0900
Subject: [PATCH v25 4/9] Tool for measuring radix tree performance

Includes Meson support, but commented out to avoid warnings

XXX: Not for commit
---
 contrib/bench_radix_tree/Makefile             |  21 +
 .../bench_radix_tree--1.0.sql                 |  76 ++
 contrib/bench_radix_tree/bench_radix_tree.c   | 656 ++++++++++++++++++
 .../bench_radix_tree/bench_radix_tree.control |   6 +
 contrib/bench_radix_tree/expected/bench.out   |  13 +
 contrib/bench_radix_tree/meson.build          |  33 +
 contrib/bench_radix_tree/sql/bench.sql        |  16 +
 contrib/meson.build                           |   1 +
 8 files changed, 822 insertions(+)
 create mode 100644 contrib/bench_radix_tree/Makefile
 create mode 100644 contrib/bench_radix_tree/bench_radix_tree--1.0.sql
 create mode 100644 contrib/bench_radix_tree/bench_radix_tree.c
 create mode 100644 contrib/bench_radix_tree/bench_radix_tree.control
 create mode 100644 contrib/bench_radix_tree/expected/bench.out
 create mode 100644 contrib/bench_radix_tree/meson.build
 create mode 100644 contrib/bench_radix_tree/sql/bench.sql

diff --git a/contrib/bench_radix_tree/Makefile b/contrib/bench_radix_tree/Makefile
new file mode 100644
index 0000000000..952bb0ceae
--- /dev/null
+++ b/contrib/bench_radix_tree/Makefile
@@ -0,0 +1,21 @@
+# contrib/bench_radix_tree/Makefile
+
+MODULE_big = bench_radix_tree
+OBJS = \
+	bench_radix_tree.o
+
+EXTENSION = bench_radix_tree
+DATA = bench_radix_tree--1.0.sql
+
+REGRESS = bench_fixed_height
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/bench_radix_tree
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
new file mode 100644
index 0000000000..2fd689aa91
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
@@ -0,0 +1,76 @@
+/* contrib/bench_radix_tree/bench_radix_tree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION bench_radix_tree" to load this file. \quit
+
+create function bench_shuffle_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_seq_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_load_random_int(
+cnt int8,
+OUT mem_allocated int8,
+OUT load_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_search_random_nodes(
+cnt int8,
+filter_str text DEFAULT NULL,
+OUT mem_allocated int8,
+OUT search_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C VOLATILE PARALLEL UNSAFE;
+
+create function bench_fixed_height_search(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT rt_search_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_node128_load(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_sparseload_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
new file mode 100644
index 0000000000..73ddee32de
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -0,0 +1,656 @@
+/*-------------------------------------------------------------------------
+ *
+ * bench_radix_tree.c
+ *
+ * Copyright (c) 2016-2022, PostgreSQL Global Development Group
+ *
+ *	  contrib/bench_radix_tree/bench_radix_tree.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "funcapi.h"
+#include "lib/radixtree.h"
+#include <math.h>
+#include "miscadmin.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/timestamp.h"
+
+PG_MODULE_MAGIC;
+
+/* run benchmark also for binary-search case? */
+/* #define MEASURE_BINARY_SEARCH 1 */
+
+#define TIDS_PER_BLOCK_FOR_LOAD		30
+#define TIDS_PER_BLOCK_FOR_LOOKUP	50
+
+#define RT_DEBUG
+#define RT_PREFIX rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE uint64
+// WIP: compiles with warnings because rt_attach is defined but not used
+// #define RT_SHMEM
+#include "lib/radixtree.h"
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+	return tree->ctl->num_keys;
+}
+
+
+PG_FUNCTION_INFO_V1(bench_seq_search);
+PG_FUNCTION_INFO_V1(bench_shuffle_search);
+PG_FUNCTION_INFO_V1(bench_load_random_int);
+PG_FUNCTION_INFO_V1(bench_fixed_height_search);
+PG_FUNCTION_INFO_V1(bench_search_random_nodes);
+PG_FUNCTION_INFO_V1(bench_node128_load);
+
+static uint64
+tid_to_key_off(ItemPointer tid, uint32 *off)
+{
+	uint64		upper;
+	uint32		shift = pg_ceil_log2_32(MaxHeapTuplesPerPage);
+	int64		tid_i;
+
+	Assert(ItemPointerGetOffsetNumber(tid) < MaxHeapTuplesPerPage);
+
+	tid_i = ItemPointerGetOffsetNumber(tid);
+	tid_i |= (uint64) ItemPointerGetBlockNumber(tid) << shift;
+
+	/* log(sizeof(uint64) * BITS_PER_BYTE, 2) = log(64, 2) = 6 */
+	*off = tid_i & ((1 << 6) - 1);
+	upper = tid_i >> 6;
+	Assert(*off < (sizeof(uint64) * BITS_PER_BYTE));
+
+	Assert(*off < 64);
+
+	return upper;
+}
+
+static int
+shuffle_randrange(pg_prng_state *state, int lower, int upper)
+{
+	return (int) floor(pg_prng_double(state) * ((upper - lower) + 0.999999)) + lower;
+}
+
+/* Naive Fisher-Yates implementation*/
+static void
+shuffle_itemptrs(ItemPointer itemptr, uint64 nitems)
+{
+	/* reproducability */
+	pg_prng_state state;
+
+	pg_prng_seed(&state, 0);
+
+	for (int i = 0; i < nitems - 1; i++)
+	{
+		int			j = shuffle_randrange(&state, i, nitems - 1);
+		ItemPointerData t = itemptr[j];
+
+		itemptr[j] = itemptr[i];
+		itemptr[i] = t;
+	}
+}
+
+static ItemPointer
+generate_tids(BlockNumber minblk, BlockNumber maxblk, int ntids_per_blk, uint64 *ntids_p,
+			  bool random_block)
+{
+	ItemPointer tids;
+	uint64		maxitems;
+	uint64		ntids = 0;
+	pg_prng_state state;
+
+	maxitems = (maxblk - minblk + 1) * ntids_per_blk;
+	tids = MemoryContextAllocHuge(TopTransactionContext,
+								  sizeof(ItemPointerData) * maxitems);
+
+	if (random_block)
+		pg_prng_seed(&state, 0x9E3779B185EBCA87);
+
+	for (BlockNumber blk = minblk; blk < maxblk; blk++)
+	{
+		if (random_block && !pg_prng_bool(&state))
+			continue;
+
+		for (OffsetNumber off = FirstOffsetNumber;
+			 off <= ntids_per_blk; off++)
+		{
+			CHECK_FOR_INTERRUPTS();
+
+			ItemPointerSetBlockNumber(&(tids[ntids]), blk);
+			ItemPointerSetOffsetNumber(&(tids[ntids]), off);
+
+			ntids++;
+		}
+	}
+
+	*ntids_p = ntids;
+	return tids;
+}
+
+#ifdef MEASURE_BINARY_SEARCH
+static int
+vac_cmp_itemptr(const void *left, const void *right)
+{
+	BlockNumber lblk,
+				rblk;
+	OffsetNumber loff,
+				roff;
+
+	lblk = ItemPointerGetBlockNumber((ItemPointer) left);
+	rblk = ItemPointerGetBlockNumber((ItemPointer) right);
+
+	if (lblk < rblk)
+		return -1;
+	if (lblk > rblk)
+		return 1;
+
+	loff = ItemPointerGetOffsetNumber((ItemPointer) left);
+	roff = ItemPointerGetOffsetNumber((ItemPointer) right);
+
+	if (loff < roff)
+		return -1;
+	if (loff > roff)
+		return 1;
+
+	return 0;
+}
+#endif
+
+static Datum
+bench_search(FunctionCallInfo fcinfo, bool shuffle)
+{
+	BlockNumber minblk = PG_GETARG_INT32(0);
+	BlockNumber maxblk = PG_GETARG_INT32(1);
+	bool		random_block = PG_GETARG_BOOL(2);
+	rt_radix_tree *rt = NULL;
+	uint64		ntids;
+	uint64		key;
+	uint64		last_key = PG_UINT64_MAX;
+	uint64		val = 0;
+	ItemPointer tids;
+	TupleDesc	tupdesc;
+	TimestampTz start_time,
+				end_time;
+	long		secs;
+	int			usecs;
+	int64		rt_load_ms,
+				rt_search_ms;
+	Datum		values[7];
+	bool		nulls[7];
+
+	/* Build a tuple descriptor for our result type */
+	if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+		elog(ERROR, "return type must be a row type");
+
+	tids = generate_tids(minblk, maxblk, TIDS_PER_BLOCK_FOR_LOAD, &ntids, random_block);
+
+	/* measure the load time of the radix tree */
+	rt = rt_create(CurrentMemoryContext);
+	start_time = GetCurrentTimestamp();
+	for (int i = 0; i < ntids; i++)
+	{
+		ItemPointer tid = &(tids[i]);
+		uint32		off;
+
+		CHECK_FOR_INTERRUPTS();
+
+		key = tid_to_key_off(tid, &off);
+
+		if (last_key != PG_UINT64_MAX && last_key != key)
+		{
+			rt_set(rt, last_key, &val);
+			val = 0;
+		}
+
+		last_key = key;
+		val |= (uint64) 1 << off;
+	}
+	if (last_key != PG_UINT64_MAX)
+		rt_set(rt, last_key, &val);
+
+	end_time = GetCurrentTimestamp();
+	TimestampDifference(start_time, end_time, &secs, &usecs);
+	rt_load_ms = secs * 1000 + usecs / 1000;
+
+	rt_stats(rt);
+
+	if (shuffle)
+		shuffle_itemptrs(tids, ntids);
+
+	elog(NOTICE, "sleeping for 2 seconds...");
+	pg_usleep(2 * 1000000L);
+
+	/* meaure the serach time of the radix tree */
+	start_time = GetCurrentTimestamp();
+	for (int i = 0; i < ntids; i++)
+	{
+		ItemPointer tid = &(tids[i]);
+		uint32		off;
+		volatile bool ret;		/* prevent calling rt_search from being
+								 * optimized out */
+
+		CHECK_FOR_INTERRUPTS();
+
+		key = tid_to_key_off(tid, &off);
+
+		ret = rt_search(rt, key, &val);
+		(void) ret;
+	}
+	end_time = GetCurrentTimestamp();
+	TimestampDifference(start_time, end_time, &secs, &usecs);
+	rt_search_ms = secs * 1000 + usecs / 1000;
+
+	MemSet(nulls, false, sizeof(nulls));
+	values[0] = Int64GetDatum(rt_num_entries(rt));
+	values[1] = Int64GetDatum(rt_memory_usage(rt));
+	values[2] = Int64GetDatum(sizeof(ItemPointerData) * ntids);
+	values[3] = Int64GetDatum(rt_load_ms);
+	nulls[4] = true;			/* ar_load_ms */
+	values[5] = Int64GetDatum(rt_search_ms);
+	nulls[6] = true;			/* ar_search_ms */
+
+#ifdef MEASURE_BINARY_SEARCH
+	{
+		ItemPointer itemptrs = NULL;
+
+		int64		ar_load_ms,
+					ar_search_ms;
+
+		/* measure the load time of the array */
+		itemptrs = MemoryContextAllocHuge(CurrentMemoryContext,
+										  sizeof(ItemPointerData) * ntids);
+		start_time = GetCurrentTimestamp();
+		for (int i = 0; i < ntids; i++)
+		{
+			ItemPointerSetBlockNumber(&(itemptrs[i]),
+									  ItemPointerGetBlockNumber(&(tids[i])));
+			ItemPointerSetOffsetNumber(&(itemptrs[i]),
+									   ItemPointerGetOffsetNumber(&(tids[i])));
+		}
+		end_time = GetCurrentTimestamp();
+		TimestampDifference(start_time, end_time, &secs, &usecs);
+		ar_load_ms = secs * 1000 + usecs / 1000;
+
+		/* next, measure the serach time of the array */
+		start_time = GetCurrentTimestamp();
+		for (int i = 0; i < ntids; i++)
+		{
+			ItemPointer tid = &(tids[i]);
+			volatile bool ret;	/* prevent calling bsearch from being
+								 * optimized out */
+
+			CHECK_FOR_INTERRUPTS();
+
+			ret = bsearch((void *) tid,
+						  (void *) itemptrs,
+						  ntids,
+						  sizeof(ItemPointerData),
+						  vac_cmp_itemptr);
+			(void) ret;
+		}
+		end_time = GetCurrentTimestamp();
+		TimestampDifference(start_time, end_time, &secs, &usecs);
+		ar_search_ms = secs * 1000 + usecs / 1000;
+
+		/* set the result */
+		nulls[4] = false;
+		values[4] = Int64GetDatum(ar_load_ms);
+		nulls[6] = false;
+		values[6] = Int64GetDatum(ar_search_ms);
+	}
+#endif
+
+	rt_free(rt);
+	PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_seq_search(PG_FUNCTION_ARGS)
+{
+	return bench_search(fcinfo, false);
+}
+
+Datum
+bench_shuffle_search(PG_FUNCTION_ARGS)
+{
+	return bench_search(fcinfo, true);
+}
+
+Datum
+bench_load_random_int(PG_FUNCTION_ARGS)
+{
+	uint64		cnt = (uint64) PG_GETARG_INT64(0);
+	rt_radix_tree *rt;
+	pg_prng_state state;
+	TupleDesc	tupdesc;
+	TimestampTz start_time,
+				end_time;
+	long		secs;
+	int			usecs;
+	int64		load_time_ms;
+	Datum		values[2];
+	bool		nulls[2];
+
+	/* Build a tuple descriptor for our result type */
+	if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+		elog(ERROR, "return type must be a row type");
+
+	pg_prng_seed(&state, 0);
+	rt = rt_create(CurrentMemoryContext);
+
+	start_time = GetCurrentTimestamp();
+	for (uint64 i = 0; i < cnt; i++)
+	{
+		uint64		key = pg_prng_uint64(&state);
+
+		rt_set(rt, key, &key);
+	}
+	end_time = GetCurrentTimestamp();
+
+	TimestampDifference(start_time, end_time, &secs, &usecs);
+	load_time_ms = secs * 1000 + usecs / 1000;
+
+	rt_stats(rt);
+
+	MemSet(nulls, false, sizeof(nulls));
+	values[0] = Int64GetDatum(rt_memory_usage(rt));
+	values[1] = Int64GetDatum(load_time_ms);
+
+	rt_free(rt);
+	PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+/* copy of splitmix64() */
+static uint64
+hash64(uint64 x)
+{
+	x ^= x >> 30;
+	x *= UINT64CONST(0xbf58476d1ce4e5b9);
+	x ^= x >> 27;
+	x *= UINT64CONST(0x94d049bb133111eb);
+	x ^= x >> 31;
+	return x;
+}
+
+/* attempts to have a relatively even population of node kinds */
+Datum
+bench_search_random_nodes(PG_FUNCTION_ARGS)
+{
+	uint64		cnt = (uint64) PG_GETARG_INT64(0);
+	rt_radix_tree *rt;
+	TupleDesc	tupdesc;
+	TimestampTz start_time,
+				end_time;
+	long		secs;
+	int			usecs;
+	int64		search_time_ms;
+	Datum		values[2] = {0};
+	bool		nulls[2] = {0};
+	/* from trial and error */
+	uint64 filter = (((uint64) 0x7F<<32) | (0x07<<24) | (0xFF<<16) | 0xFF);
+
+	/* Build a tuple descriptor for our result type */
+	if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+		elog(ERROR, "return type must be a row type");
+
+	if (!PG_ARGISNULL(1))
+	{
+		char		*filter_str = text_to_cstring(PG_GETARG_TEXT_P(1));
+
+		if (sscanf(filter_str, "0x%lX", &filter) == 0)
+			elog(ERROR, "invalid filter string %s", filter_str);
+	}
+	elog(NOTICE, "bench with filter 0x%lX", filter);
+
+	rt = rt_create(CurrentMemoryContext);
+
+	for (uint64 i = 0; i < cnt; i++)
+	{
+		const uint64 hash = hash64(i);
+		const uint64 key = hash & filter;
+
+		rt_set(rt, key, &key);
+	}
+
+	elog(NOTICE, "sleeping for 2 seconds...");
+	pg_usleep(2 * 1000000L);
+
+	start_time = GetCurrentTimestamp();
+	for (uint64 i = 0; i < cnt; i++)
+	{
+		uint64 hash = hash64(i);
+		uint64 key = hash & filter;
+		uint64		val;
+		volatile bool ret;		/* prevent calling rt_search from being
+								 * optimized out */
+
+		CHECK_FOR_INTERRUPTS();
+
+		ret = rt_search(rt, key, &val);
+		(void) ret;
+	}
+	end_time = GetCurrentTimestamp();
+
+	TimestampDifference(start_time, end_time, &secs, &usecs);
+	search_time_ms = secs * 1000 + usecs / 1000;
+
+	rt_stats(rt);
+
+	values[0] = Int64GetDatum(rt_memory_usage(rt));
+	values[1] = Int64GetDatum(search_time_ms);
+
+	rt_free(rt);
+	PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_fixed_height_search(PG_FUNCTION_ARGS)
+{
+	int			fanout = PG_GETARG_INT32(0);
+	rt_radix_tree *rt;
+	TupleDesc	tupdesc;
+	TimestampTz start_time,
+				end_time;
+	long		secs;
+	int			usecs;
+	int64		rt_load_ms,
+				rt_search_ms;
+	Datum		values[5];
+	bool		nulls[5];
+
+	/* test boundary between vector and iteration */
+	const int	n_keys = 5 * 16 * 16 * 16 * 16;
+	uint64		r,
+				h,
+				i,
+				j,
+				k;
+	uint64		key_id;
+
+	/* Build a tuple descriptor for our result type */
+	if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+		elog(ERROR, "return type must be a row type");
+
+	rt = rt_create(CurrentMemoryContext);
+
+	start_time = GetCurrentTimestamp();
+
+	key_id = 0;
+
+	/*
+	 * lower nodes have limited fanout, the top is only limited by
+	 * bits-per-byte
+	 */
+	for (r = 1;; r++)
+	{
+		for (h = 1; h <= fanout; h++)
+		{
+			for (i = 1; i <= fanout; i++)
+			{
+				for (j = 1; j <= fanout; j++)
+				{
+					for (k = 1; k <= fanout; k++)
+					{
+						uint64		key;
+
+						key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+						CHECK_FOR_INTERRUPTS();
+
+						key_id++;
+						if (key_id > n_keys)
+							goto finish_set;
+
+						rt_set(rt, key, &key_id);
+					}
+				}
+			}
+		}
+	}
+finish_set:
+	end_time = GetCurrentTimestamp();
+	TimestampDifference(start_time, end_time, &secs, &usecs);
+	rt_load_ms = secs * 1000 + usecs / 1000;
+
+	rt_stats(rt);
+
+	/* meaure the search time of the radix tree */
+	start_time = GetCurrentTimestamp();
+
+	key_id = 0;
+	for (r = 1;; r++)
+	{
+		for (h = 1; h <= fanout; h++)
+		{
+			for (i = 1; i <= fanout; i++)
+			{
+				for (j = 1; j <= fanout; j++)
+				{
+					for (k = 1; k <= fanout; k++)
+					{
+						uint64		key,
+									val;
+
+						key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+						CHECK_FOR_INTERRUPTS();
+
+						key_id++;
+						if (key_id > n_keys)
+							goto finish_search;
+
+						rt_search(rt, key, &val);
+					}
+				}
+			}
+		}
+	}
+finish_search:
+	end_time = GetCurrentTimestamp();
+	TimestampDifference(start_time, end_time, &secs, &usecs);
+	rt_search_ms = secs * 1000 + usecs / 1000;
+
+	MemSet(nulls, false, sizeof(nulls));
+	values[0] = Int32GetDatum(fanout);
+	values[1] = Int64GetDatum(rt_num_entries(rt));
+	values[2] = Int64GetDatum(rt_memory_usage(rt));
+	values[3] = Int64GetDatum(rt_load_ms);
+	values[4] = Int64GetDatum(rt_search_ms);
+
+	rt_free(rt);
+	PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_node128_load(PG_FUNCTION_ARGS)
+{
+	int			fanout = PG_GETARG_INT32(0);
+	rt_radix_tree *rt;
+	TupleDesc	tupdesc;
+	TimestampTz start_time,
+				end_time;
+	long		secs;
+	int			usecs;
+	int64		rt_sparseload_ms;
+	Datum		values[5];
+	bool		nulls[5];
+
+	uint64		r,
+				h;
+	uint64		key_id;
+
+	/* Build a tuple descriptor for our result type */
+	if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+		elog(ERROR, "return type must be a row type");
+
+	rt = rt_create(CurrentMemoryContext);
+
+	key_id = 0;
+
+	for (r = 1; r <= fanout; r++)
+	{
+		for (h = 1; h <= fanout; h++)
+		{
+			uint64		key;
+
+			key = (r << 8) | (h);
+
+			key_id++;
+			rt_set(rt, key, &key_id);
+		}
+	}
+
+	rt_stats(rt);
+
+	/* measure sparse deletion and re-loading */
+	start_time = GetCurrentTimestamp();
+
+	for (int t = 0; t<10000; t++)
+	{
+		/* delete one key in each leaf */
+		for (r = 1; r <= fanout; r++)
+		{
+			uint64		key;
+
+			key = (r << 8) | (fanout);
+
+			rt_delete(rt, key);
+		}
+
+		/* add them all back */
+		key_id = 0;
+		for (r = 1; r <= fanout; r++)
+		{
+			uint64		key;
+
+			key = (r << 8) | (fanout);
+
+			key_id++;
+			rt_set(rt, key, &key_id);
+		}
+	}
+	end_time = GetCurrentTimestamp();
+	TimestampDifference(start_time, end_time, &secs, &usecs);
+	rt_sparseload_ms = secs * 1000 + usecs / 1000;
+
+	MemSet(nulls, false, sizeof(nulls));
+	values[0] = Int32GetDatum(fanout);
+	values[1] = Int64GetDatum(rt_num_entries(rt));
+	values[2] = Int64GetDatum(rt_memory_usage(rt));
+	values[3] = Int64GetDatum(rt_sparseload_ms);
+
+	rt_free(rt);
+	PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
diff --git a/contrib/bench_radix_tree/bench_radix_tree.control b/contrib/bench_radix_tree/bench_radix_tree.control
new file mode 100644
index 0000000000..1d988e6c9a
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.control
@@ -0,0 +1,6 @@
+# bench_radix_tree extension
+comment = 'benchmark suits for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/bench_radix_tree'
+relocatable = true
+trusted = true
diff --git a/contrib/bench_radix_tree/expected/bench.out b/contrib/bench_radix_tree/expected/bench.out
new file mode 100644
index 0000000000..60c303892e
--- /dev/null
+++ b/contrib/bench_radix_tree/expected/bench.out
@@ -0,0 +1,13 @@
+create extension bench_radix_tree;
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/bench_radix_tree/meson.build b/contrib/bench_radix_tree/meson.build
new file mode 100644
index 0000000000..332c1ae7df
--- /dev/null
+++ b/contrib/bench_radix_tree/meson.build
@@ -0,0 +1,33 @@
+bench_radix_tree_sources = files(
+  'bench_radix_tree.c',
+)
+
+if host_system == 'windows'
+  bench_radix_tree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+    '--NAME', 'bench_radix_tree',
+    '--FILEDESC', 'bench_radix_tree - performance test code for radix tree',])
+endif
+
+bench_radix_tree = shared_module('bench_radix_tree',
+  bench_radix_tree_sources,
+  link_with: pgport_srv,
+  kwargs: pg_mod_args,
+)
+testprep_targets += bench_radix_tree
+
+install_data(
+  'bench_radix_tree.control',
+  'bench_radix_tree--1.0.sql',
+  kwargs: contrib_data_args,
+)
+
+tests += {
+  'name': 'bench_radix_tree',
+  'sd': meson.current_source_dir(),
+  'bd': meson.current_build_dir(),
+  'regress': {
+    'sql': [
+      'bench_radix_tree',
+    ],
+  },
+}
diff --git a/contrib/bench_radix_tree/sql/bench.sql b/contrib/bench_radix_tree/sql/bench.sql
new file mode 100644
index 0000000000..a46018c9d4
--- /dev/null
+++ b/contrib/bench_radix_tree/sql/bench.sql
@@ -0,0 +1,16 @@
+create extension bench_radix_tree;
+
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/meson.build b/contrib/meson.build
index bd4a57c43c..52253de793 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,6 +12,7 @@ subdir('amcheck')
 subdir('auth_delay')
 subdir('auto_explain')
 subdir('basic_archive')
+#subdir('bench_radix_tree')
 subdir('bloom')
 subdir('basebackup_to_shell')
 subdir('bool_plperl')
-- 
2.39.1

From f421579a2e04baa04b258399e01f01485ce6f358 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Wed, 14 Sep 2022 12:38:51 +0000
Subject: [PATCH v25 3/9] Add radixtree template

WIP: commit message based on template comments
---
 src/backend/utils/mmgr/dsa.c                  |   12 +
 src/include/lib/radixtree.h                   | 2516 +++++++++++++++++
 src/include/lib/radixtree_delete_impl.h       |  122 +
 src/include/lib/radixtree_insert_impl.h       |  332 +++
 src/include/lib/radixtree_iter_impl.h         |  153 +
 src/include/lib/radixtree_search_impl.h       |  138 +
 src/include/utils/dsa.h                       |    1 +
 src/test/modules/Makefile                     |    1 +
 src/test/modules/meson.build                  |    1 +
 src/test/modules/test_radixtree/.gitignore    |    4 +
 src/test/modules/test_radixtree/Makefile      |   23 +
 src/test/modules/test_radixtree/README        |    7 +
 .../expected/test_radixtree.out               |   36 +
 src/test/modules/test_radixtree/meson.build   |   35 +
 .../test_radixtree/sql/test_radixtree.sql     |    7 +
 .../test_radixtree/test_radixtree--1.0.sql    |    8 +
 .../modules/test_radixtree/test_radixtree.c   |  674 +++++
 .../test_radixtree/test_radixtree.control     |    4 +
 src/tools/pginclude/cpluspluscheck            |    6 +
 src/tools/pginclude/headerscheck              |    6 +
 20 files changed, 4086 insertions(+)
 create mode 100644 src/include/lib/radixtree.h
 create mode 100644 src/include/lib/radixtree_delete_impl.h
 create mode 100644 src/include/lib/radixtree_insert_impl.h
 create mode 100644 src/include/lib/radixtree_iter_impl.h
 create mode 100644 src/include/lib/radixtree_search_impl.h
 create mode 100644 src/test/modules/test_radixtree/.gitignore
 create mode 100644 src/test/modules/test_radixtree/Makefile
 create mode 100644 src/test/modules/test_radixtree/README
 create mode 100644 src/test/modules/test_radixtree/expected/test_radixtree.out
 create mode 100644 src/test/modules/test_radixtree/meson.build
 create mode 100644 src/test/modules/test_radixtree/sql/test_radixtree.sql
 create mode 100644 src/test/modules/test_radixtree/test_radixtree--1.0.sql
 create mode 100644 src/test/modules/test_radixtree/test_radixtree.c
 create mode 100644 src/test/modules/test_radixtree/test_radixtree.control

diff --git a/src/backend/utils/mmgr/dsa.c b/src/backend/utils/mmgr/dsa.c
index f5a62061a3..80555aefff 100644
--- a/src/backend/utils/mmgr/dsa.c
+++ b/src/backend/utils/mmgr/dsa.c
@@ -1024,6 +1024,18 @@ dsa_set_size_limit(dsa_area *area, size_t limit)
 	LWLockRelease(DSA_AREA_LOCK(area));
 }
 
+size_t
+dsa_get_total_size(dsa_area *area)
+{
+	size_t		size;
+
+	LWLockAcquire(DSA_AREA_LOCK(area), LW_SHARED);
+	size = area->control->total_segment_size;
+	LWLockRelease(DSA_AREA_LOCK(area));
+
+	return size;
+}
+
 /*
  * Aggressively free all spare memory in the hope of returning DSM segments to
  * the operating system.
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
new file mode 100644
index 0000000000..d6919aef08
--- /dev/null
+++ b/src/include/lib/radixtree.h
@@ -0,0 +1,2516 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree.h
+ *		Template for adaptive radix tree.
+ *
+ * This module employs the idea from the paper "The Adaptive Radix Tree: ARTful
+ * Indexing for Main-Memory Databases" by Viktor Leis, Alfons Kemper, and Thomas
+ * Neumann, 2013. The radix tree uses adaptive node sizes, a small number of node
+ * types, each with a different numbers of elements. Depending on the number of
+ * children, the appropriate node type is used.
+ *
+ * WIP: notes about traditional radix tree trading off span vs height...
+ *
+ * There are two kinds of nodes, inner nodes and leaves. Inner nodes
+ * map partial keys to child pointers.
+ *
+ * The ART paper mentions three ways to implement leaves:
+ *
+ * "- Single-value leaves: The values are stored using an addi-
+ *  tional leaf node type which stores one value.
+ *  - Multi-value leaves: The values are stored in one of four
+ *  different leaf node types, which mirror the structure of
+ *  inner nodes, but contain values instead of pointers.
+ *  - Combined pointer/value slots: If values fit into point-
+ *  ers, no separate node types are necessary. Instead, each
+ *  pointer storage location in an inner node can either
+ *  store a pointer or a value."
+ *
+ * We chose "multi-value leaves" to avoid the additional pointer traversal
+ * required by "single-value leaves"
+ *
+ * For simplicity, the key is assumed to be 64-bit unsigned integer. The
+ * tree doesn't need to contain paths where the highest bytes of all keys
+ * are zero. That way, the tree's height adapts to the distribution of keys.
+ *
+ * TODO: In the future it might be worthwhile to offer configurability of
+ * leaf implementation for different use cases. Single-values leaves would
+ * give more flexibility in key type, including variable-length keys.
+ *
+ * There are some optimizations not yet implemented, particularly path
+ * compression and lazy path expansion.
+ *
+ * To handle concurrency, we use a single reader-writer lock for the radix
+ * tree. The radix tree is exclusively locked during write operations such
+ * as RT_SET() and RT_DELETE(), and shared locked during read operations
+ * such as RT_SEARCH(). An iteration also holds the shared lock on the radix
+ * tree until it is completed.
+ *
+ * TODO: The current locking mechanism is not optimized for high concurrency
+ * with mixed read-write workloads. In the future it might be worthwhile
+ * to replace it with the Optimistic Lock Coupling or ROWEX mentioned in
+ * the paper "The ART of Practical Synchronization" by the same authors as
+ * the ART paper, 2016.
+ *
+ * WIP: the radix tree nodes don't shrink.
+ *
+ * To generate a radix tree and associated functions for a use case several
+ * macros have to be #define'ed before this file is included.  Including
+ * the file #undef's all those, so a new radix tree can be generated
+ * afterwards.
+ * The relevant parameters are:
+ * - RT_PREFIX - prefix for all symbol names generated. A prefix of 'foo'
+ * 	 will result in radix tree type 'foo_radix_tree' and functions like
+ *	 'foo_create'/'foo_free' and so forth.
+ * - RT_DECLARE - if defined function prototypes and type declarations are
+ *	 generated
+ * - RT_DEFINE - if defined function definitions are generated
+ * - RT_SCOPE - in which scope (e.g. extern, static inline) do function
+ *	 declarations reside
+ * - RT_VALUE_TYPE - the type of the value.
+ *
+ * Optional parameters:
+ * - RT_SHMEM - if defined, the radix tree is created in the DSA area
+ *	 so that multiple processes can access it simultaneously.
+ * - RT_DEBUG - if defined add stats tracking and debugging functions
+ *
+ * Interface
+ * ---------
+ *
+ * RT_CREATE		- Create a new, empty radix tree
+ * RT_FREE			- Free the radix tree
+ * RT_SEARCH		- Search a key-value pair
+ * RT_SET			- Set a key-value pair
+ * RT_BEGIN_ITERATE	- Begin iterating through all key-value pairs
+ * RT_ITERATE_NEXT	- Return next key-value pair, if any
+ * RT_END_ITER		- End iteration
+ * RT_MEMORY_USAGE	- Get the memory usage
+ *
+ * Interface for Shared Memory
+ * ---------
+ *
+ * RT_ATTACH		- Attach to the radix tree
+ * RT_DETACH		- Detach from the radix tree
+ * RT_GET_HANDLE	- Return the handle of the radix tree
+ *
+ * Optional Interface
+ * ---------
+ *
+ * RT_DELETE		- Delete a key-value pair. Declared/define if RT_USE_DELETE is defined
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ *	  src/include/lib/radixtree.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/stringinfo.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "port/pg_bitutils.h"
+#include "port/simd.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/* helpers */
+#define RT_MAKE_PREFIX(a) CppConcat(a,_)
+#define RT_MAKE_NAME(name) RT_MAKE_NAME_(RT_MAKE_PREFIX(RT_PREFIX),name)
+#define RT_MAKE_NAME_(a,b) CppConcat(a,b)
+
+/* function declarations */
+#define RT_CREATE RT_MAKE_NAME(create)
+#define RT_FREE RT_MAKE_NAME(free)
+#define RT_SEARCH RT_MAKE_NAME(search)
+#ifdef RT_SHMEM
+#define RT_ATTACH RT_MAKE_NAME(attach)
+#define RT_DETACH RT_MAKE_NAME(detach)
+#define RT_GET_HANDLE RT_MAKE_NAME(get_handle)
+#endif
+#define RT_SET RT_MAKE_NAME(set)
+#define RT_BEGIN_ITERATE RT_MAKE_NAME(begin_iterate)
+#define RT_ITERATE_NEXT RT_MAKE_NAME(iterate_next)
+#define RT_END_ITERATE RT_MAKE_NAME(end_iterate)
+#ifdef RT_USE_DELETE
+#define RT_DELETE RT_MAKE_NAME(delete)
+#endif
+#define RT_MEMORY_USAGE RT_MAKE_NAME(memory_usage)
+#ifdef RT_DEBUG
+#define RT_DUMP RT_MAKE_NAME(dump)
+#define RT_DUMP_NODE RT_MAKE_NAME(dump_node)
+#define RT_DUMP_SEARCH RT_MAKE_NAME(dump_search)
+#define RT_STATS RT_MAKE_NAME(stats)
+#endif
+
+/* internal helper functions (no externally visible prototypes) */
+#define RT_NEW_ROOT RT_MAKE_NAME(new_root)
+#define RT_ALLOC_NODE RT_MAKE_NAME(alloc_node)
+#define RT_INIT_NODE RT_MAKE_NAME(init_node)
+#define RT_FREE_NODE RT_MAKE_NAME(free_node)
+#define RT_FREE_RECURSE RT_MAKE_NAME(free_recurse)
+#define RT_EXTEND RT_MAKE_NAME(extend)
+#define RT_SET_EXTEND RT_MAKE_NAME(set_extend)
+#define RT_SWITCH_NODE_KIND RT_MAKE_NAME(grow_node_kind)
+#define RT_COPY_NODE RT_MAKE_NAME(copy_node)
+#define RT_REPLACE_NODE RT_MAKE_NAME(replace_node)
+#define RT_PTR_GET_LOCAL RT_MAKE_NAME(ptr_get_local)
+#define RT_PTR_ALLOC_IS_VALID RT_MAKE_NAME(ptr_stored_is_valid)
+#define RT_NODE_3_SEARCH_EQ RT_MAKE_NAME(node_3_search_eq)
+#define RT_NODE_32_SEARCH_EQ RT_MAKE_NAME(node_32_search_eq)
+#define RT_NODE_3_GET_INSERTPOS RT_MAKE_NAME(node_3_get_insertpos)
+#define RT_NODE_32_GET_INSERTPOS RT_MAKE_NAME(node_32_get_insertpos)
+#define RT_CHUNK_CHILDREN_ARRAY_SHIFT RT_MAKE_NAME(chunk_children_array_shift)
+#define RT_CHUNK_VALUES_ARRAY_SHIFT RT_MAKE_NAME(chunk_values_array_shift)
+#define RT_CHUNK_CHILDREN_ARRAY_DELETE RT_MAKE_NAME(chunk_children_array_delete)
+#define RT_CHUNK_VALUES_ARRAY_DELETE RT_MAKE_NAME(chunk_values_array_delete)
+#define RT_CHUNK_CHILDREN_ARRAY_COPY RT_MAKE_NAME(chunk_children_array_copy)
+#define RT_CHUNK_VALUES_ARRAY_COPY RT_MAKE_NAME(chunk_values_array_copy)
+#define RT_NODE_125_IS_CHUNK_USED RT_MAKE_NAME(node_125_is_chunk_used)
+#define RT_NODE_INNER_125_GET_CHILD RT_MAKE_NAME(node_inner_125_get_child)
+#define RT_NODE_LEAF_125_GET_VALUE RT_MAKE_NAME(node_leaf_125_get_value)
+#define RT_NODE_INNER_256_IS_CHUNK_USED RT_MAKE_NAME(node_inner_256_is_chunk_used)
+#define RT_NODE_LEAF_256_IS_CHUNK_USED RT_MAKE_NAME(node_leaf_256_is_chunk_used)
+#define RT_NODE_INNER_256_GET_CHILD RT_MAKE_NAME(node_inner_256_get_child)
+#define RT_NODE_LEAF_256_GET_VALUE RT_MAKE_NAME(node_leaf_256_get_value)
+#define RT_NODE_INNER_256_SET RT_MAKE_NAME(node_inner_256_set)
+#define RT_NODE_LEAF_256_SET RT_MAKE_NAME(node_leaf_256_set)
+#define RT_NODE_INNER_256_DELETE RT_MAKE_NAME(node_inner_256_delete)
+#define RT_NODE_LEAF_256_DELETE RT_MAKE_NAME(node_leaf_256_delete)
+#define RT_KEY_GET_SHIFT RT_MAKE_NAME(key_get_shift)
+#define RT_SHIFT_GET_MAX_VAL RT_MAKE_NAME(shift_get_max_val)
+#define RT_NODE_SEARCH_INNER RT_MAKE_NAME(node_search_inner)
+#define RT_NODE_SEARCH_LEAF RT_MAKE_NAME(node_search_leaf)
+#define RT_NODE_UPDATE_INNER RT_MAKE_NAME(node_update_inner)
+#define RT_NODE_DELETE_INNER RT_MAKE_NAME(node_delete_inner)
+#define RT_NODE_DELETE_LEAF RT_MAKE_NAME(node_delete_leaf)
+#define RT_NODE_INSERT_INNER RT_MAKE_NAME(node_insert_inner)
+#define RT_NODE_INSERT_LEAF RT_MAKE_NAME(node_insert_leaf)
+#define RT_NODE_INNER_ITERATE_NEXT RT_MAKE_NAME(node_inner_iterate_next)
+#define RT_NODE_LEAF_ITERATE_NEXT RT_MAKE_NAME(node_leaf_iterate_next)
+#define RT_UPDATE_ITER_STACK RT_MAKE_NAME(update_iter_stack)
+#define RT_ITER_UPDATE_KEY RT_MAKE_NAME(iter_update_key)
+#define RT_VERIFY_NODE RT_MAKE_NAME(verify_node)
+
+/* type declarations */
+#define RT_RADIX_TREE RT_MAKE_NAME(radix_tree)
+#define RT_RADIX_TREE_CONTROL RT_MAKE_NAME(radix_tree_control)
+#define RT_ITER RT_MAKE_NAME(iter)
+#ifdef RT_SHMEM
+#define RT_HANDLE RT_MAKE_NAME(handle)
+#endif
+#define RT_NODE RT_MAKE_NAME(node)
+#define RT_NODE_ITER RT_MAKE_NAME(node_iter)
+#define RT_NODE_BASE_3 RT_MAKE_NAME(node_base_3)
+#define RT_NODE_BASE_32 RT_MAKE_NAME(node_base_32)
+#define RT_NODE_BASE_125 RT_MAKE_NAME(node_base_125)
+#define RT_NODE_BASE_256 RT_MAKE_NAME(node_base_256)
+#define RT_NODE_INNER_3 RT_MAKE_NAME(node_inner_3)
+#define RT_NODE_INNER_32 RT_MAKE_NAME(node_inner_32)
+#define RT_NODE_INNER_125 RT_MAKE_NAME(node_inner_125)
+#define RT_NODE_INNER_256 RT_MAKE_NAME(node_inner_256)
+#define RT_NODE_LEAF_3 RT_MAKE_NAME(node_leaf_3)
+#define RT_NODE_LEAF_32 RT_MAKE_NAME(node_leaf_32)
+#define RT_NODE_LEAF_125 RT_MAKE_NAME(node_leaf_125)
+#define RT_NODE_LEAF_256 RT_MAKE_NAME(node_leaf_256)
+#define RT_SIZE_CLASS RT_MAKE_NAME(size_class)
+#define RT_SIZE_CLASS_ELEM RT_MAKE_NAME(size_class_elem)
+#define RT_SIZE_CLASS_INFO RT_MAKE_NAME(size_class_info)
+#define RT_CLASS_3 RT_MAKE_NAME(class_3)
+#define RT_CLASS_32_MIN RT_MAKE_NAME(class_32_min)
+#define RT_CLASS_32_MAX RT_MAKE_NAME(class_32_max)
+#define RT_CLASS_125 RT_MAKE_NAME(class_125)
+#define RT_CLASS_256 RT_MAKE_NAME(class_256)
+
+/* generate forward declarations necessary to use the radix tree */
+#ifdef RT_DECLARE
+
+typedef struct RT_RADIX_TREE RT_RADIX_TREE;
+typedef struct RT_ITER RT_ITER;
+
+#ifdef RT_SHMEM
+typedef dsa_pointer RT_HANDLE;
+#endif
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id);
+RT_SCOPE RT_RADIX_TREE * RT_ATTACH(dsa_area *dsa, dsa_pointer dp);
+RT_SCOPE void RT_DETACH(RT_RADIX_TREE *tree);
+RT_SCOPE RT_HANDLE RT_GET_HANDLE(RT_RADIX_TREE *tree);
+#else
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx);
+#endif
+RT_SCOPE void RT_FREE(RT_RADIX_TREE *tree);
+
+RT_SCOPE bool RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p);
+RT_SCOPE bool RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p);
+#ifdef RT_USE_DELETE
+RT_SCOPE bool RT_DELETE(RT_RADIX_TREE *tree, uint64 key);
+#endif
+
+RT_SCOPE RT_ITER * RT_BEGIN_ITERATE(RT_RADIX_TREE *tree);
+RT_SCOPE bool RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p);
+RT_SCOPE void RT_END_ITERATE(RT_ITER *iter);
+
+RT_SCOPE uint64 RT_MEMORY_USAGE(RT_RADIX_TREE *tree);
+
+#ifdef RT_DEBUG
+RT_SCOPE void RT_DUMP(RT_RADIX_TREE *tree);
+RT_SCOPE void RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key);
+RT_SCOPE void RT_STATS(RT_RADIX_TREE *tree);
+#endif
+
+#endif							/* RT_DECLARE */
+
+
+/* generate implementation of the radix tree */
+#ifdef RT_DEFINE
+
+/* The number of bits encoded in one tree level */
+#define RT_NODE_SPAN	BITS_PER_BYTE
+
+/* The number of maximum slots in the node */
+#define RT_NODE_MAX_SLOTS (1 << RT_NODE_SPAN)
+
+/* Mask for extracting a chunk from the key */
+#define RT_CHUNK_MASK ((1 << RT_NODE_SPAN) - 1)
+
+/* Maximum shift the radix tree uses */
+#define RT_MAX_SHIFT	RT_KEY_GET_SHIFT(UINT64_MAX)
+
+/* Tree level the radix tree uses */
+#define RT_MAX_LEVEL	((sizeof(uint64) * BITS_PER_BYTE) / RT_NODE_SPAN)
+
+/*
+ * Number of bits necessary for isset array in the slot-index node.
+ * Since bitmapword can be 64 bits, the only values that make sense
+ * here are 64 and 128.
+ */
+#define RT_SLOT_IDX_LIMIT (RT_NODE_MAX_SLOTS / 2)
+
+/* Invalid index used in node-125 */
+#define RT_INVALID_SLOT_IDX	0xFF
+
+/* Get a chunk from the key */
+#define RT_GET_KEY_CHUNK(key, shift) ((uint8) (((key) >> (shift)) & RT_CHUNK_MASK))
+
+/* For accessing bitmaps */
+#define RT_BM_IDX(x)	((x) / BITS_PER_BITMAPWORD)
+#define RT_BM_BIT(x)	((x) % BITS_PER_BITMAPWORD)
+
+/*
+ * Node kinds
+ *
+ * The different node kinds are what make the tree "adaptive".
+ *
+ * Each node kind is associated with a different datatype and different
+ * search/set/delete/iterate algorithms adapted for its size. The largest
+ * kind, node256 is basically the same as a traditional radix tree,
+ * and would be most wasteful of memory when sparsely populated. The
+ * smaller nodes expend some additional CPU time to enable a smaller
+ * memory footprint.
+ *
+ * XXX There are 4 node kinds, and this should never be increased,
+ * for several reasons:
+ * 1. With 5 or more kinds, gcc tends to use a jump table for switch
+ *    statements.
+ * 2. The 4 kinds can be represented with 2 bits, so we have the option
+ *    in the future to tag the node pointer with the kind, even on
+ *    platforms with 32-bit pointers. This might speed up node traversal
+ *    in trees with highly random node kinds.
+ * 3. We can have multiple size classes per node kind.
+ */
+#define RT_NODE_KIND_3			0x00
+#define RT_NODE_KIND_32			0x01
+#define RT_NODE_KIND_125		0x02
+#define RT_NODE_KIND_256		0x03
+#define RT_NODE_KIND_COUNT		4
+
+/*
+ * Calculate the slab blocksize so that we can allocate at least 32 chunks
+ * from the block.
+ */
+#define RT_SLAB_BLOCK_SIZE(size)	\
+	Max((SLAB_DEFAULT_BLOCK_SIZE / (size)) * (size), (size) * 32)
+
+/* Common type for all nodes types */
+typedef struct RT_NODE
+{
+	/*
+	 * Number of children.  We use uint16 to be able to indicate 256 children
+	 * at the fanout of 8.
+	 */
+	uint16		count;
+
+	/*
+	 * Max capacity for the current size class. Storing this in the
+	 * node enables multiple size classes per node kind.
+	 * Technically, kinds with a single size class don't need this, so we could
+	 * keep this in the individual base types, but the code is simpler this way.
+	 * Note: node256 is unique in that it cannot possibly have more than a
+	 * single size class, so for that kind we store zero, and uint8 is
+	 * sufficient for other kinds.
+	 */
+	uint8		fanout;
+
+	/*
+	 * Shift indicates which part of the key space is represented by this
+	 * node. That is, the key is shifted by 'shift' and the lowest
+	 * RT_NODE_SPAN bits are then represented in chunk.
+	 */
+	uint8		shift;
+
+	/* Node kind, one per search/set algorithm */
+	uint8		kind;
+} RT_NODE;
+
+
+#define RT_PTR_LOCAL RT_NODE *
+
+#ifdef RT_SHMEM
+#define RT_PTR_ALLOC dsa_pointer
+#else
+#define RT_PTR_ALLOC RT_PTR_LOCAL
+#endif
+
+
+#ifdef RT_SHMEM
+#define RT_INVALID_PTR_ALLOC InvalidDsaPointer
+#else
+#define RT_INVALID_PTR_ALLOC NULL
+#endif
+
+#ifdef RT_SHMEM
+#define RT_LOCK_EXCLUSIVE(tree)	LWLockAcquire(&tree->ctl->lock, LW_EXCLUSIVE)
+#define RT_LOCK_SHARED(tree)	LWLockAcquire(&tree->ctl->lock, LW_SHARED)
+#define RT_UNLOCK(tree)			LWLockRelease(&tree->ctl->lock);
+#else
+#define RT_LOCK_EXCLUSIVE(tree)	((void) 0)
+#define RT_LOCK_SHARED(tree)	((void) 0)
+#define RT_UNLOCK(tree)			((void) 0)
+#endif
+
+/*
+ * Inner nodes and leaf nodes have analogous structure. To distinguish
+ * them at runtime, we take advantage of the fact that the key chunk
+ * is accessed by shifting: Inner tree nodes (shift > 0), store the
+ * pointer to its child node in the slot. In leaf nodes (shift == 0),
+ * the slot contains the value corresponding to the key.
+ */
+#define RT_NODE_IS_LEAF(n)			(((RT_PTR_LOCAL) (n))->shift == 0)
+
+#define RT_NODE_MUST_GROW(node) \
+	((node)->base.n.count == (node)->base.n.fanout)
+
+/*
+ * Base type of each node kinds for leaf and inner nodes.
+ * The base types must be a be able to accommodate the largest size
+ * class for variable-sized node kinds.
+ */
+typedef struct RT_NODE_BASE_3
+{
+	RT_NODE		n;
+
+	/* 3 children, for key chunks */
+	uint8		chunks[3];
+} RT_NODE_BASE_3;
+
+typedef struct RT_NODE_BASE_32
+{
+	RT_NODE		n;
+
+	/* 32 children, for key chunks */
+	uint8		chunks[32];
+} RT_NODE_BASE_32;
+
+/*
+ * node-125 uses slot_idx array, an array of RT_NODE_MAX_SLOTS length
+ * to store indexes into a second array that contains the values (or
+ * child pointers).
+ */
+typedef struct RT_NODE_BASE_125
+{
+	RT_NODE		n;
+
+	/* The index of slots for each fanout */
+	uint8		slot_idxs[RT_NODE_MAX_SLOTS];
+
+	/* bitmap to track which slots are in use */
+	bitmapword		isset[RT_BM_IDX(RT_SLOT_IDX_LIMIT)];
+} RT_NODE_BASE_125;
+
+typedef struct RT_NODE_BASE_256
+{
+	RT_NODE		n;
+} RT_NODE_BASE_256;
+
+/*
+ * Inner and leaf nodes.
+ *
+ * These are separate because the value type might be different than
+ * something fitting into a pointer-width type.
+ */
+typedef struct RT_NODE_INNER_3
+{
+	RT_NODE_BASE_3 base;
+
+	/* number of children depends on size class */
+	RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_3;
+
+typedef struct RT_NODE_LEAF_3
+{
+	RT_NODE_BASE_3 base;
+
+	/* number of values depends on size class */
+	RT_VALUE_TYPE	values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_3;
+
+typedef struct RT_NODE_INNER_32
+{
+	RT_NODE_BASE_32 base;
+
+	/* number of children depends on size class */
+	RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_32;
+
+typedef struct RT_NODE_LEAF_32
+{
+	RT_NODE_BASE_32 base;
+
+	/* number of values depends on size class */
+	RT_VALUE_TYPE	values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_32;
+
+typedef struct RT_NODE_INNER_125
+{
+	RT_NODE_BASE_125 base;
+
+	/* number of children depends on size class */
+	RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_125;
+
+typedef struct RT_NODE_LEAF_125
+{
+	RT_NODE_BASE_125 base;
+
+	/* number of values depends on size class */
+	RT_VALUE_TYPE	values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_125;
+
+/*
+ * node-256 is the largest node type. This node has an array
+ * for directly storing values (or child pointers in inner nodes).
+ * Unlike other node kinds, it's array size is by definition
+ * fixed.
+ */
+typedef struct RT_NODE_INNER_256
+{
+	RT_NODE_BASE_256 base;
+
+	/* Slots for 256 children */
+	RT_PTR_ALLOC children[RT_NODE_MAX_SLOTS];
+} RT_NODE_INNER_256;
+
+typedef struct RT_NODE_LEAF_256
+{
+	RT_NODE_BASE_256 base;
+
+	/*
+	 * Unlike with inner256, zero is a valid value here, so we use a
+	 * bitmap to track which slots are in use.
+	 */
+	bitmapword	isset[RT_BM_IDX(RT_NODE_MAX_SLOTS)];
+
+	/* Slots for 256 values */
+	RT_VALUE_TYPE	values[RT_NODE_MAX_SLOTS];
+} RT_NODE_LEAF_256;
+
+/*
+ * Node size classes
+ *
+ * Nodes of different kinds necessarily belong to different size classes.
+ * The main innovation in our implementation compared to the ART paper
+ * is decoupling the notion of size class from kind.
+ *
+ * The size classes within a given node kind have the same underlying
+ * type, but a variable number of children/values. This is possible
+ * because the base type contains small fixed data structures that
+ * work the same way regardless of how full the node is. We store the
+ * node's allocated capacity in the "fanout" member of RT_NODE, to allow
+ * runtime introspection.
+ *
+ * Growing from one node kind to another requires special code for each
+ * case, but growing from one size class to another within the same kind
+ * is basically just allocate + memcpy.
+ *
+ * The size classes have been chosen so that inner nodes on platforms
+ * with 64-bit pointers (and leaf nodes when using a 64-bit key) are
+ * equal to or slightly smaller than some DSA size class.
+ */
+typedef enum RT_SIZE_CLASS
+{
+	RT_CLASS_3 = 0,
+	RT_CLASS_32_MIN,
+	RT_CLASS_32_MAX,
+	RT_CLASS_125,
+	RT_CLASS_256
+} RT_SIZE_CLASS;
+
+/* Information for each size class */
+typedef struct RT_SIZE_CLASS_ELEM
+{
+	const char *name;
+	int			fanout;
+
+	/* slab chunk size */
+	Size		inner_size;
+	Size		leaf_size;
+} RT_SIZE_CLASS_ELEM;
+
+static const RT_SIZE_CLASS_ELEM RT_SIZE_CLASS_INFO[] = {
+	[RT_CLASS_3] = {
+		.name = "radix tree node 3",
+		.fanout = 3,
+		.inner_size = sizeof(RT_NODE_INNER_3) + 3 * sizeof(RT_PTR_ALLOC),
+		.leaf_size = sizeof(RT_NODE_LEAF_3) + 3 * sizeof(RT_VALUE_TYPE),
+	},
+	[RT_CLASS_32_MIN] = {
+		.name = "radix tree node 15",
+		.fanout = 15,
+		.inner_size = sizeof(RT_NODE_INNER_32) + 15 * sizeof(RT_PTR_ALLOC),
+		.leaf_size = sizeof(RT_NODE_LEAF_32) + 15 * sizeof(RT_VALUE_TYPE),
+	},
+	[RT_CLASS_32_MAX] = {
+		.name = "radix tree node 32",
+		.fanout = 32,
+		.inner_size = sizeof(RT_NODE_INNER_32) + 32 * sizeof(RT_PTR_ALLOC),
+		.leaf_size = sizeof(RT_NODE_LEAF_32) + 32 * sizeof(RT_VALUE_TYPE),
+	},
+	[RT_CLASS_125] = {
+		.name = "radix tree node 125",
+		.fanout = 125,
+		.inner_size = sizeof(RT_NODE_INNER_125) + 125 * sizeof(RT_PTR_ALLOC),
+		.leaf_size = sizeof(RT_NODE_LEAF_125) + 125 * sizeof(RT_VALUE_TYPE),
+	},
+	[RT_CLASS_256] = {
+		.name = "radix tree node 256",
+		.fanout = 256,
+		.inner_size = sizeof(RT_NODE_INNER_256),
+		.leaf_size = sizeof(RT_NODE_LEAF_256),
+	},
+};
+
+#define RT_SIZE_CLASS_COUNT lengthof(RT_SIZE_CLASS_INFO)
+
+#ifdef RT_SHMEM
+/* A magic value used to identify our radix tree */
+#define RT_RADIX_TREE_MAGIC 0x54A48167
+#endif
+
+/* Contains the actual tree and ancillary info */
+// WIP: this name is a bit strange
+typedef struct RT_RADIX_TREE_CONTROL
+{
+#ifdef RT_SHMEM
+	RT_HANDLE	handle;
+	uint32		magic;
+	LWLock		lock;
+#endif
+
+	RT_PTR_ALLOC root;
+	uint64		max_val;
+	uint64		num_keys;
+
+	/* statistics */
+#ifdef RT_DEBUG
+	int32		cnt[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE_CONTROL;
+
+/* Entry point for allocating and accessing the tree */
+typedef struct RT_RADIX_TREE
+{
+	MemoryContext context;
+
+	/* pointing to either local memory or DSA */
+	RT_RADIX_TREE_CONTROL *ctl;
+
+#ifdef RT_SHMEM
+	dsa_area   *dsa;
+#else
+	MemoryContextData *inner_slabs[RT_SIZE_CLASS_COUNT];
+	MemoryContextData *leaf_slabs[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE;
+
+/*
+ * Iteration support.
+ *
+ * Iterating the radix tree returns each pair of key and value in the ascending
+ * order of the key. To support this, the we iterate nodes of each level.
+ *
+ * RT_NODE_ITER struct is used to track the iteration within a node.
+ *
+ * RT_ITER is the struct for iteration of the radix tree, and uses RT_NODE_ITER
+ * in order to track the iteration of each level. During iteration, we also
+ * construct the key whenever updating the node iteration information, e.g., when
+ * advancing the current index within the node or when moving to the next node
+ * at the same level.
+ *
+ * XXX: Currently we allow only one process to do iteration. Therefore, rt_node_iter
+ * has the local pointers to nodes, rather than RT_PTR_ALLOC.
+ * We need either a safeguard to disallow other processes to begin the iteration
+ * while one process is doing or to allow multiple processes to do the iteration.
+ */
+typedef struct RT_NODE_ITER
+{
+	RT_PTR_LOCAL node;			/* current node being iterated */
+	int			current_idx;	/* current position. -1 for initial value */
+} RT_NODE_ITER;
+
+typedef struct RT_ITER
+{
+	RT_RADIX_TREE *tree;
+
+	/* Track the iteration on nodes of each level */
+	RT_NODE_ITER stack[RT_MAX_LEVEL];
+	int			stack_len;
+
+	/* The key is constructed during iteration */
+	uint64		key;
+} RT_ITER;
+
+
+static bool RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+								 uint64 key, RT_PTR_ALLOC child);
+static bool RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+								uint64 key, RT_VALUE_TYPE *value_p);
+
+/* verification (available only with assertion) */
+static void RT_VERIFY_NODE(RT_PTR_LOCAL node);
+
+/* Get the local address of an allocated node */
+static inline RT_PTR_LOCAL
+RT_PTR_GET_LOCAL(RT_RADIX_TREE *tree, RT_PTR_ALLOC node)
+{
+#ifdef RT_SHMEM
+	return dsa_get_address(tree->dsa, (dsa_pointer) node);
+#else
+	return node;
+#endif
+}
+
+static inline bool
+RT_PTR_ALLOC_IS_VALID(RT_PTR_ALLOC ptr)
+{
+#ifdef RT_SHMEM
+	return DsaPointerIsValid(ptr);
+#else
+	return PointerIsValid(ptr);
+#endif
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_3_SEARCH_EQ(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+	int			idx = -1;
+
+	for (int i = 0; i < node->n.count; i++)
+	{
+		if (node->chunks[i] == chunk)
+		{
+			idx = i;
+			break;
+		}
+	}
+
+	return idx;
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_3_GET_INSERTPOS(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+	int			idx;
+
+	for (idx = 0; idx < node->n.count; idx++)
+	{
+		if (node->chunks[idx] >= chunk)
+			break;
+	}
+
+	return idx;
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_32_SEARCH_EQ(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+	int			count = node->n.count;
+#ifndef USE_NO_SIMD
+	Vector8		spread_chunk;
+	Vector8		haystack1;
+	Vector8		haystack2;
+	Vector8		cmp1;
+	Vector8		cmp2;
+	uint32		bitfield;
+	int			index_simd = -1;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+	int			index = -1;
+
+	for (int i = 0; i < count; i++)
+	{
+		if (node->chunks[i] == chunk)
+		{
+			index = i;
+			break;
+		}
+	}
+#endif
+
+#ifndef USE_NO_SIMD
+	/* replicate the search key */
+	spread_chunk = vector8_broadcast(chunk);
+
+	/* compare to all 32 keys stored in the node */
+	vector8_load(&haystack1, &node->chunks[0]);
+	vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+	cmp1 = vector8_eq(spread_chunk, haystack1);
+	cmp2 = vector8_eq(spread_chunk, haystack2);
+
+	/* convert comparison to a bitfield */
+	bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+
+	/* mask off invalid entries */
+	bitfield &= ((UINT64CONST(1) << count) - 1);
+
+	/* convert bitfield to index by counting trailing zeros */
+	if (bitfield)
+		index_simd = pg_rightmost_one_pos32(bitfield);
+
+	Assert(index_simd == index);
+	return index_simd;
+#else
+	return index;
+#endif
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_32_GET_INSERTPOS(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+	int			count = node->n.count;
+#ifndef USE_NO_SIMD
+	Vector8		spread_chunk;
+	Vector8		haystack1;
+	Vector8		haystack2;
+	Vector8		cmp1;
+	Vector8		cmp2;
+	Vector8		min1;
+	Vector8		min2;
+	uint32		bitfield;
+	int			index_simd;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+	int			index;
+
+	for (index = 0; index < count; index++)
+	{
+		/*
+		 * This is coded with '>=' to match what we can do with SIMD,
+		 * with an assert to keep us honest.
+		 */
+		if (node->chunks[index] >= chunk)
+		{
+			Assert(node->chunks[index] != chunk);
+			break;
+		}
+	}
+#endif
+
+#ifndef USE_NO_SIMD
+	/*
+	 * This is a bit more complicated than RT_NODE_32_SEARCH_EQ(), because
+	 * no unsigned uint8 comparison instruction exists, at least for SSE2. So
+	 * we need to play some trickery using vector8_min() to effectively get
+	 * >=. There'll never be any equal elements in current uses, but that's
+	 * what we get here...
+	 */
+	spread_chunk = vector8_broadcast(chunk);
+	vector8_load(&haystack1, &node->chunks[0]);
+	vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+	min1 = vector8_min(spread_chunk, haystack1);
+	min2 = vector8_min(spread_chunk, haystack2);
+	cmp1 = vector8_eq(spread_chunk, min1);
+	cmp2 = vector8_eq(spread_chunk, min2);
+	bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+	bitfield &= ((UINT64CONST(1) << count) - 1);
+
+	if (bitfield)
+		index_simd = pg_rightmost_one_pos32(bitfield);
+	else
+		index_simd = count;
+
+	Assert(index_simd == index);
+	return index_simd;
+#else
+	return index;
+#endif
+}
+
+
+/*
+ * Functions to manipulate both chunks array and children/values array.
+ * These are used for node-3 and node-32.
+ */
+
+/* Shift the elements right at 'idx' by one */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_SHIFT(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+	memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+	memmove(&(children[idx + 1]), &(children[idx]), sizeof(RT_PTR_ALLOC) * (count - idx));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_SHIFT(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+	memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+	memmove(&(values[idx + 1]), &(values[idx]), sizeof(RT_VALUE_TYPE) * (count - idx));
+}
+
+/* Delete the element at 'idx' */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_DELETE(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+	memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+	memmove(&(children[idx]), &(children[idx + 1]), sizeof(RT_PTR_ALLOC) * (count - idx - 1));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_DELETE(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+	memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+	memmove(&(values[idx]), &(values[idx + 1]), sizeof(RT_VALUE_TYPE) * (count - idx - 1));
+}
+
+/* Copy both chunks and children/values arrays */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_COPY(uint8 *src_chunks, RT_PTR_ALLOC *src_children,
+						  uint8 *dst_chunks, RT_PTR_ALLOC *dst_children)
+{
+	const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+	const Size chunk_size = sizeof(uint8) * fanout;
+	const Size children_size = sizeof(RT_PTR_ALLOC) * fanout;
+
+	memcpy(dst_chunks, src_chunks, chunk_size);
+	memcpy(dst_children, src_children, children_size);
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_COPY(uint8 *src_chunks, RT_VALUE_TYPE *src_values,
+						uint8 *dst_chunks, RT_VALUE_TYPE *dst_values)
+{
+	const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+	const Size chunk_size = sizeof(uint8) * fanout;
+	const Size values_size = sizeof(RT_VALUE_TYPE) * fanout;
+
+	memcpy(dst_chunks, src_chunks, chunk_size);
+	memcpy(dst_values, src_values, values_size);
+}
+
+/* Functions to manipulate inner and leaf node-125 */
+
+/* Does the given chunk in the node has the value? */
+static inline bool
+RT_NODE_125_IS_CHUNK_USED(RT_NODE_BASE_125 *node, uint8 chunk)
+{
+	return node->slot_idxs[chunk] != RT_INVALID_SLOT_IDX;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_125_GET_CHILD(RT_NODE_INNER_125 *node, uint8 chunk)
+{
+	Assert(!RT_NODE_IS_LEAF(node));
+	return node->children[node->base.slot_idxs[chunk]];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_125_GET_VALUE(RT_NODE_LEAF_125 *node, uint8 chunk)
+{
+	Assert(RT_NODE_IS_LEAF(node));
+	Assert(((RT_NODE_BASE_125 *) node)->slot_idxs[chunk] != RT_INVALID_SLOT_IDX);
+	return node->values[node->base.slot_idxs[chunk]];
+}
+
+/* Functions to manipulate inner and leaf node-256 */
+
+/* Return true if the slot corresponding to the given chunk is in use */
+static inline bool
+RT_NODE_INNER_256_IS_CHUNK_USED(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+	Assert(!RT_NODE_IS_LEAF(node));
+	return node->children[chunk] != RT_INVALID_PTR_ALLOC;
+}
+
+static inline bool
+RT_NODE_LEAF_256_IS_CHUNK_USED(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+	int			idx = RT_BM_IDX(chunk);
+	int			bitnum = RT_BM_BIT(chunk);
+
+	Assert(RT_NODE_IS_LEAF(node));
+	return (node->isset[idx] & ((bitmapword) 1 << bitnum)) != 0;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_256_GET_CHILD(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+	Assert(!RT_NODE_IS_LEAF(node));
+	Assert(RT_NODE_INNER_256_IS_CHUNK_USED(node, chunk));
+	return node->children[chunk];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_256_GET_VALUE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+	Assert(RT_NODE_IS_LEAF(node));
+	Assert(RT_NODE_LEAF_256_IS_CHUNK_USED(node, chunk));
+	return node->values[chunk];
+}
+
+/* Set the child in the node-256 */
+static inline void
+RT_NODE_INNER_256_SET(RT_NODE_INNER_256 *node, uint8 chunk, RT_PTR_ALLOC child)
+{
+	Assert(!RT_NODE_IS_LEAF(node));
+	node->children[chunk] = child;
+}
+
+/* Set the value in the node-256 */
+static inline void
+RT_NODE_LEAF_256_SET(RT_NODE_LEAF_256 *node, uint8 chunk, RT_VALUE_TYPE value)
+{
+	int			idx = RT_BM_IDX(chunk);
+	int			bitnum = RT_BM_BIT(chunk);
+
+	Assert(RT_NODE_IS_LEAF(node));
+	node->isset[idx] |= ((bitmapword) 1 << bitnum);
+	node->values[chunk] = value;
+}
+
+/* Set the slot at the given chunk position */
+static inline void
+RT_NODE_INNER_256_DELETE(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+	Assert(!RT_NODE_IS_LEAF(node));
+	node->children[chunk] = RT_INVALID_PTR_ALLOC;
+}
+
+static inline void
+RT_NODE_LEAF_256_DELETE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+	int			idx = RT_BM_IDX(chunk);
+	int			bitnum = RT_BM_BIT(chunk);
+
+	Assert(RT_NODE_IS_LEAF(node));
+	node->isset[idx] &= ~((bitmapword) 1 << bitnum);
+}
+
+/*
+ * Return the largest shift that will allowing storing the given key.
+ */
+static inline int
+RT_KEY_GET_SHIFT(uint64 key)
+{
+	if (key == 0)
+		return 0;
+	else
+		return (pg_leftmost_one_pos64(key) / RT_NODE_SPAN) * RT_NODE_SPAN;
+}
+
+/*
+ * Return the max value that can be stored in the tree with the given shift.
+ */
+static uint64
+RT_SHIFT_GET_MAX_VAL(int shift)
+{
+	if (shift == RT_MAX_SHIFT)
+		return UINT64_MAX;
+
+	return (UINT64CONST(1) << (shift + RT_NODE_SPAN)) - 1;
+}
+
+/*
+ * Allocate a new node with the given node kind.
+ */
+static RT_PTR_ALLOC
+RT_ALLOC_NODE(RT_RADIX_TREE *tree, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+	RT_PTR_ALLOC allocnode;
+	size_t allocsize;
+
+	if (is_leaf)
+		allocsize = RT_SIZE_CLASS_INFO[size_class].leaf_size;
+	else
+		allocsize = RT_SIZE_CLASS_INFO[size_class].inner_size;
+
+#ifdef RT_SHMEM
+	allocnode = dsa_allocate(tree->dsa, allocsize);
+#else
+	if (is_leaf)
+		allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->leaf_slabs[size_class],
+													  allocsize);
+	else
+		allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->inner_slabs[size_class],
+													  allocsize);
+#endif
+
+#ifdef RT_DEBUG
+	/* update the statistics */
+	tree->ctl->cnt[size_class]++;
+#endif
+
+	return allocnode;
+}
+
+/* Initialize the node contents */
+static inline void
+RT_INIT_NODE(RT_PTR_LOCAL node, uint8 kind, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+	if (is_leaf)
+		MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].leaf_size);
+	else
+		MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].inner_size);
+
+	node->kind = kind;
+
+	if (kind == RT_NODE_KIND_256)
+		/* See comment for the RT_NODE type */
+		Assert(node->fanout == 0);
+	else
+		node->fanout = RT_SIZE_CLASS_INFO[size_class].fanout;
+
+	/* Initialize slot_idxs to invalid values */
+	if (kind == RT_NODE_KIND_125)
+	{
+		RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+
+		memset(n125->slot_idxs, RT_INVALID_SLOT_IDX, sizeof(n125->slot_idxs));
+	}
+}
+
+/*
+ * Create a new node as the root. Subordinate nodes will be created during
+ * the insertion.
+ */
+static void
+RT_NEW_ROOT(RT_RADIX_TREE *tree, uint64 key)
+{
+	int			shift = RT_KEY_GET_SHIFT(key);
+	bool		is_leaf = shift == 0;
+	RT_PTR_ALLOC allocnode;
+	RT_PTR_LOCAL newnode;
+
+	allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+	newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+	RT_INIT_NODE(newnode, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+	newnode->shift = shift;
+	tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(shift);
+	tree->ctl->root = allocnode;
+}
+
+static inline void
+RT_COPY_NODE(RT_PTR_LOCAL newnode, RT_PTR_LOCAL oldnode)
+{
+	newnode->shift = oldnode->shift;
+	newnode->count = oldnode->count;
+}
+
+/*
+ * Given a new allocated node and an old node, initalize the new
+ * node with the necessary fields and return its local pointer.
+ */
+static inline RT_PTR_LOCAL
+RT_SWITCH_NODE_KIND(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, RT_PTR_LOCAL node,
+				  uint8 new_kind, uint8 new_class, bool is_leaf)
+{
+	RT_PTR_LOCAL newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+	RT_INIT_NODE(newnode, new_kind, new_class, is_leaf);
+	RT_COPY_NODE(newnode, node);
+
+	return newnode;
+}
+
+/* Free the given node */
+static void
+RT_FREE_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode)
+{
+	/* If we're deleting the root node, make the tree empty */
+	if (tree->ctl->root == allocnode)
+	{
+		tree->ctl->root = RT_INVALID_PTR_ALLOC;
+		tree->ctl->max_val = 0;
+	}
+
+#ifdef RT_DEBUG
+	{
+		int i;
+		RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+		/* update the statistics */
+		for (i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+		{
+			if (node->fanout == RT_SIZE_CLASS_INFO[i].fanout)
+				break;
+		}
+
+		/* fanout of node256 is intentionally 0 */
+		if (i == RT_SIZE_CLASS_COUNT)
+			i = RT_CLASS_256;
+
+		tree->ctl->cnt[i]--;
+		Assert(tree->ctl->cnt[i] >= 0);
+	}
+#endif
+
+#ifdef RT_SHMEM
+	dsa_free(tree->dsa, allocnode);
+#else
+	pfree(allocnode);
+#endif
+}
+
+/* Update the parent's pointer when growing a node */
+static inline void
+RT_NODE_UPDATE_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC new_child)
+{
+#define RT_ACTION_UPDATE
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+#undef RT_ACTION_UPDATE
+}
+
+/*
+ * Replace old_child with new_child, and free the old one.
+ */
+static void
+RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
+				RT_PTR_ALLOC stored_old_child, RT_PTR_LOCAL old_child,
+				RT_PTR_ALLOC new_child, uint64 key)
+{
+#ifdef USE_ASSERT_CHECKING
+	RT_PTR_LOCAL new = RT_PTR_GET_LOCAL(tree, new_child);
+
+	Assert(old_child->shift == new->shift);
+	Assert(old_child->count == new->count);
+#endif
+
+	if (parent == old_child)
+	{
+		/* Replace the root node with the new larger node */
+		tree->ctl->root = new_child;
+	}
+	else
+		RT_NODE_UPDATE_INNER(parent, key, new_child);
+
+	RT_FREE_NODE(tree, stored_old_child);
+}
+
+/*
+ * The radix tree doesn't have sufficient height. Extend the radix tree so
+ * it can store the key.
+ */
+static void
+RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
+{
+	int			target_shift;
+	RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+	int			shift = root->shift + RT_NODE_SPAN;
+
+	target_shift = RT_KEY_GET_SHIFT(key);
+
+	/* Grow tree from 'shift' to 'target_shift' */
+	while (shift <= target_shift)
+	{
+		RT_PTR_ALLOC	allocnode;
+		RT_PTR_LOCAL	node;
+		RT_NODE_INNER_3 *n3;
+
+		allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, true);
+		node = RT_PTR_GET_LOCAL(tree, allocnode);
+		RT_INIT_NODE(node, RT_NODE_KIND_3, RT_CLASS_3, true);
+		node->shift = shift;
+		node->count = 1;
+
+		n3 = (RT_NODE_INNER_3 *) node;
+		n3->base.chunks[0] = 0;
+		n3->children[0] = tree->ctl->root;
+
+		/* Update the root */
+		tree->ctl->root = allocnode;
+
+		shift += RT_NODE_SPAN;
+	}
+
+	tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(target_shift);
+}
+
+/*
+ * The radix tree doesn't have inner and leaf nodes for given key-value pair.
+ * Insert inner and leaf nodes from 'node' to bottom.
+ */
+static inline void
+RT_SET_EXTEND(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
+			  RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node)
+{
+	int			shift = node->shift;
+
+	Assert(RT_PTR_GET_LOCAL(tree, stored_node) == node);
+
+	while (shift >= RT_NODE_SPAN)
+	{
+		RT_PTR_ALLOC allocchild;
+		RT_PTR_LOCAL newchild;
+		int			newshift = shift - RT_NODE_SPAN;
+		bool		is_leaf = newshift == 0;
+
+		allocchild = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+		newchild = RT_PTR_GET_LOCAL(tree, allocchild);
+		RT_INIT_NODE(newchild, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+		newchild->shift = newshift;
+		RT_NODE_INSERT_INNER(tree, parent, stored_node, node, key, allocchild);
+
+		parent = node;
+		node = newchild;
+		stored_node = allocchild;
+		shift -= RT_NODE_SPAN;
+	}
+
+	RT_NODE_INSERT_LEAF(tree, parent, stored_node, node, key, value_p);
+	tree->ctl->num_keys++;
+}
+
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the child
+ * pointer is set to child_p.
+ */
+static inline bool
+RT_NODE_SEARCH_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC *child_p)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the pointer
+ * to the value is set to value_p.
+ */
+static inline bool
+RT_NODE_SEARCH_LEAF(RT_PTR_LOCAL node, uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_INNER(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_LEAF(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+#endif
+
+/*
+ * Insert "child" into "node".
+ *
+ * "parent" is the parent of "node", so the grandparent of the child.
+ * If the node we're inserting into needs to grow, we update the parent's
+ * child pointer with the pointer to the new larger node.
+ */
+static bool
+RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+					uint64 key, RT_PTR_ALLOC child)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/* Like RT_NODE_INSERT_INNER, but for leaf nodes */
+static bool
+RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+					uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Create the radix tree in the given memory context and return it.
+ */
+RT_SCOPE RT_RADIX_TREE *
+#ifdef RT_SHMEM
+RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id)
+#else
+RT_CREATE(MemoryContext ctx)
+#endif
+{
+	RT_RADIX_TREE *tree;
+	MemoryContext old_ctx;
+#ifdef RT_SHMEM
+	dsa_pointer dp;
+#endif
+
+	old_ctx = MemoryContextSwitchTo(ctx);
+
+	tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+	tree->context = ctx;
+
+#ifdef RT_SHMEM
+	tree->dsa = dsa;
+	dp = dsa_allocate0(dsa, sizeof(RT_RADIX_TREE_CONTROL));
+	tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, dp);
+	tree->ctl->handle = dp;
+	tree->ctl->magic = RT_RADIX_TREE_MAGIC;
+	LWLockInitialize(&tree->ctl->lock, tranche_id);
+#else
+	tree->ctl = (RT_RADIX_TREE_CONTROL *) palloc0(sizeof(RT_RADIX_TREE_CONTROL));
+
+	/* Create a slab context for each size class */
+	for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+	{
+		RT_SIZE_CLASS_ELEM size_class = RT_SIZE_CLASS_INFO[i];
+		size_t inner_blocksize = RT_SLAB_BLOCK_SIZE(size_class.inner_size);
+		size_t leaf_blocksize = RT_SLAB_BLOCK_SIZE(size_class.leaf_size);
+
+		tree->inner_slabs[i] = SlabContextCreate(ctx,
+												 size_class.name,
+												 inner_blocksize,
+												 size_class.inner_size);
+		tree->leaf_slabs[i] = SlabContextCreate(ctx,
+												size_class.name,
+												leaf_blocksize,
+												size_class.leaf_size);
+	}
+#endif
+
+	tree->ctl->root = RT_INVALID_PTR_ALLOC;
+
+	MemoryContextSwitchTo(old_ctx);
+
+	return tree;
+}
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE *
+RT_ATTACH(dsa_area *dsa, RT_HANDLE handle)
+{
+	RT_RADIX_TREE *tree;
+	dsa_pointer	control;
+
+	tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+
+	/* Find the control object in shard memory */
+	control = handle;
+
+	tree->dsa = dsa;
+	tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, control);
+	Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+	return tree;
+}
+
+RT_SCOPE void
+RT_DETACH(RT_RADIX_TREE *tree)
+{
+	Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+	pfree(tree);
+}
+
+RT_SCOPE RT_HANDLE
+RT_GET_HANDLE(RT_RADIX_TREE *tree)
+{
+	Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+	return tree->ctl->handle;
+}
+
+/*
+ * Recursively free all nodes allocated to the DSA area.
+ */
+static inline void
+RT_FREE_RECURSE(RT_RADIX_TREE *tree, RT_PTR_ALLOC ptr)
+{
+	RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, ptr);
+
+	check_stack_depth();
+	CHECK_FOR_INTERRUPTS();
+
+	/* The leaf node doesn't have child pointers */
+	if (RT_NODE_IS_LEAF(node))
+	{
+		dsa_free(tree->dsa, ptr);
+		return;
+	}
+
+	switch (node->kind)
+	{
+		case RT_NODE_KIND_3:
+			{
+				RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+				for (int i = 0; i < n3->base.n.count; i++)
+					RT_FREE_RECURSE(tree, n3->children[i]);
+
+				break;
+			}
+		case RT_NODE_KIND_32:
+			{
+				RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+				for (int i = 0; i < n32->base.n.count; i++)
+					RT_FREE_RECURSE(tree, n32->children[i]);
+
+				break;
+			}
+		case RT_NODE_KIND_125:
+			{
+				RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) node;
+
+				for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+				{
+					if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+						continue;
+
+					RT_FREE_RECURSE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+				}
+
+				break;
+			}
+		case RT_NODE_KIND_256:
+			{
+				RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+				for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+				{
+					if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+						continue;
+
+					RT_FREE_RECURSE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+				}
+
+				break;
+			}
+	}
+
+	/* Free the inner node */
+	dsa_free(tree->dsa, ptr);
+}
+#endif
+
+/*
+ * Free the given radix tree.
+ */
+RT_SCOPE void
+RT_FREE(RT_RADIX_TREE *tree)
+{
+#ifdef RT_SHMEM
+	Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+	/* Free all memory used for radix tree nodes */
+	if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+		RT_FREE_RECURSE(tree, tree->ctl->root);
+
+	/*
+	 * Vandalize the control block to help catch programming error where
+	 * other backends access the memory formerly occupied by this radix tree.
+	 */
+	tree->ctl->magic = 0;
+	dsa_free(tree->dsa, tree->ctl->handle);
+#else
+	pfree(tree->ctl);
+
+	for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+	{
+		MemoryContextDelete(tree->inner_slabs[i]);
+		MemoryContextDelete(tree->leaf_slabs[i]);
+	}
+#endif
+
+	pfree(tree);
+}
+
+/*
+ * Set key to value. If the entry already exists, we update its value to 'value'
+ * and return true. Returns false if entry doesn't yet exist.
+ */
+RT_SCOPE bool
+RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+	int			shift;
+	bool		updated;
+	RT_PTR_LOCAL parent;
+	RT_PTR_ALLOC stored_child;
+	RT_PTR_LOCAL  child;
+
+#ifdef RT_SHMEM
+	Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+	RT_LOCK_EXCLUSIVE(tree);
+
+	/* Empty tree, create the root */
+	if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+		RT_NEW_ROOT(tree, key);
+
+	/* Extend the tree if necessary */
+	if (key > tree->ctl->max_val)
+		RT_EXTEND(tree, key);
+
+	stored_child = tree->ctl->root;
+	parent = RT_PTR_GET_LOCAL(tree, stored_child);
+	shift = parent->shift;
+
+	/* Descend the tree until we reach a leaf node */
+	while (shift >= 0)
+	{
+		RT_PTR_ALLOC new_child = RT_INVALID_PTR_ALLOC;;
+
+		child = RT_PTR_GET_LOCAL(tree, stored_child);
+
+		if (RT_NODE_IS_LEAF(child))
+			break;
+
+		if (!RT_NODE_SEARCH_INNER(child, key, &new_child))
+		{
+			RT_SET_EXTEND(tree, key, value_p, parent, stored_child, child);
+			RT_UNLOCK(tree);
+			return false;
+		}
+
+		parent = child;
+		stored_child = new_child;
+		shift -= RT_NODE_SPAN;
+	}
+
+	updated = RT_NODE_INSERT_LEAF(tree, parent, stored_child, child, key, value_p);
+
+	/* Update the statistics */
+	if (!updated)
+		tree->ctl->num_keys++;
+
+	RT_UNLOCK(tree);
+	return updated;
+}
+
+/*
+ * Search the given key in the radix tree. Return true if there is the key,
+ * otherwise return false. On success, we set the value to *value_p so it must
+ * not be NULL.
+ */
+RT_SCOPE bool
+RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+	RT_PTR_LOCAL node;
+	int			shift;
+	bool		found;
+
+#ifdef RT_SHMEM
+	Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+	Assert(value_p != NULL);
+
+	RT_LOCK_SHARED(tree);
+
+	if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+	{
+		RT_UNLOCK(tree);
+		return false;
+	}
+
+	node = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+	shift = node->shift;
+
+	/* Descend the tree until a leaf node */
+	while (shift >= 0)
+	{
+		RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+		if (RT_NODE_IS_LEAF(node))
+			break;
+
+		if (!RT_NODE_SEARCH_INNER(node, key, &child))
+		{
+			RT_UNLOCK(tree);
+			return false;
+		}
+
+		node = RT_PTR_GET_LOCAL(tree, child);
+		shift -= RT_NODE_SPAN;
+	}
+
+	found = RT_NODE_SEARCH_LEAF(node, key, value_p);
+
+	RT_UNLOCK(tree);
+	return found;
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Delete the given key from the radix tree. Return true if the key is found (and
+ * deleted), otherwise do nothing and return false.
+ */
+RT_SCOPE bool
+RT_DELETE(RT_RADIX_TREE *tree, uint64 key)
+{
+	RT_PTR_LOCAL node;
+	RT_PTR_ALLOC allocnode;
+	RT_PTR_ALLOC stack[RT_MAX_LEVEL] = {0};
+	int			shift;
+	int			level;
+	bool		deleted;
+
+#ifdef RT_SHMEM
+	Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+	RT_LOCK_EXCLUSIVE(tree);
+
+	if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+	{
+		RT_UNLOCK(tree);
+		return false;
+	}
+
+	/*
+	 * Descend the tree to search the key while building a stack of nodes we
+	 * visited.
+	 */
+	allocnode = tree->ctl->root;
+	node = RT_PTR_GET_LOCAL(tree, allocnode);
+	shift = node->shift;
+	level = -1;
+	while (shift > 0)
+	{
+		RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+		/* Push the current node to the stack */
+		stack[++level] = allocnode;
+		node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+		if (!RT_NODE_SEARCH_INNER(node, key, &child))
+		{
+			RT_UNLOCK(tree);
+			return false;
+		}
+
+		allocnode = child;
+		shift -= RT_NODE_SPAN;
+	}
+
+	/* Delete the key from the leaf node if exists */
+	node = RT_PTR_GET_LOCAL(tree, allocnode);
+	deleted = RT_NODE_DELETE_LEAF(node, key);
+
+	if (!deleted)
+	{
+		/* no key is found in the leaf node */
+		RT_UNLOCK(tree);
+		return false;
+	}
+
+	/* Found the key to delete. Update the statistics */
+	tree->ctl->num_keys--;
+
+	/*
+	 * Return if the leaf node still has keys and we don't need to delete the
+	 * node.
+	 */
+	if (node->count > 0)
+	{
+		RT_UNLOCK(tree);
+		return true;
+	}
+
+	/* Free the empty leaf node */
+	RT_FREE_NODE(tree, allocnode);
+
+	/* Delete the key in inner nodes recursively */
+	while (level >= 0)
+	{
+		allocnode = stack[level--];
+
+		node = RT_PTR_GET_LOCAL(tree, allocnode);
+		deleted = RT_NODE_DELETE_INNER(node, key);
+		Assert(deleted);
+
+		/* If the node didn't become empty, we stop deleting the key */
+		if (node->count > 0)
+			break;
+
+		/* The node became empty */
+		RT_FREE_NODE(tree, allocnode);
+	}
+
+	RT_UNLOCK(tree);
+	return true;
+}
+#endif
+
+static inline void
+RT_ITER_UPDATE_KEY(RT_ITER *iter, uint8 chunk, uint8 shift)
+{
+	iter->key &= ~(((uint64) RT_CHUNK_MASK) << shift);
+	iter->key |= (((uint64) chunk) << shift);
+}
+
+/*
+ * Advance the slot in the inner node. Return the child if exists, otherwise
+ * null.
+ */
+static inline RT_PTR_LOCAL
+RT_NODE_INNER_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Advance the slot in the leaf node. On success, return true and the value
+ * is set to value_p, otherwise return false.
+ */
+static inline bool
+RT_NODE_LEAF_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter,
+						  RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Update each node_iter for inner nodes in the iterator node stack.
+ */
+static void
+RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
+{
+	int			level = from;
+	RT_PTR_LOCAL node = from_node;
+
+	for (;;)
+	{
+		RT_NODE_ITER *node_iter = &(iter->stack[level--]);
+
+		node_iter->node = node;
+		node_iter->current_idx = -1;
+
+		/* We don't advance the leaf node iterator here */
+		if (RT_NODE_IS_LEAF(node))
+			return;
+
+		/* Advance to the next slot in the inner node */
+		node = RT_NODE_INNER_ITERATE_NEXT(iter, node_iter);
+
+		/* We must find the first children in the node */
+		Assert(node);
+	}
+}
+
+/*
+ * Create and return the iterator for the given radix tree.
+ *
+ * The radix tree is locked in shared mode during the iteration, so
+ * RT_END_ITERATE needs to be called when finished to release the lock.
+ */
+RT_SCOPE RT_ITER *
+RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
+{
+	MemoryContext old_ctx;
+	RT_ITER    *iter;
+	RT_PTR_LOCAL root;
+	int			top_level;
+
+	old_ctx = MemoryContextSwitchTo(tree->context);
+
+	iter = (RT_ITER *) palloc0(sizeof(RT_ITER));
+	iter->tree = tree;
+
+	RT_LOCK_SHARED(tree);
+
+	/* empty tree */
+	if (!iter->tree->ctl->root)
+		return iter;
+
+	root = RT_PTR_GET_LOCAL(tree, iter->tree->ctl->root);
+	top_level = root->shift / RT_NODE_SPAN;
+	iter->stack_len = top_level;
+
+	/*
+	 * Descend to the left most leaf node from the root. The key is being
+	 * constructed while descending to the leaf.
+	 */
+	RT_UPDATE_ITER_STACK(iter, root, top_level);
+
+	MemoryContextSwitchTo(old_ctx);
+
+	return iter;
+}
+
+/*
+ * Return true with setting key_p and value_p if there is next key.  Otherwise
+ * return false.
+ */
+RT_SCOPE bool
+RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p)
+{
+	/* Empty tree */
+	if (!iter->tree->ctl->root)
+		return false;
+
+	for (;;)
+	{
+		RT_PTR_LOCAL child = NULL;
+		RT_VALUE_TYPE value;
+		int			level;
+		bool		found;
+
+		/* Advance the leaf node iterator to get next key-value pair */
+		found = RT_NODE_LEAF_ITERATE_NEXT(iter, &(iter->stack[0]), &value);
+
+		if (found)
+		{
+			*key_p = iter->key;
+			*value_p = value;
+			return true;
+		}
+
+		/*
+		 * We've visited all values in the leaf node, so advance inner node
+		 * iterators from the level=1 until we find the next child node.
+		 */
+		for (level = 1; level <= iter->stack_len; level++)
+		{
+			child = RT_NODE_INNER_ITERATE_NEXT(iter, &(iter->stack[level]));
+
+			if (child)
+				break;
+		}
+
+		/* the iteration finished */
+		if (!child)
+			return false;
+
+		/*
+		 * Set the node to the node iterator and update the iterator stack
+		 * from this node.
+		 */
+		RT_UPDATE_ITER_STACK(iter, child, level - 1);
+
+		/* Node iterators are updated, so try again from the leaf */
+	}
+
+	return false;
+}
+
+/*
+ * Terminate the iteration and release the lock.
+ *
+ * This function needs to be called after finishing or when exiting an
+ * iteration.
+ */
+RT_SCOPE void
+RT_END_ITERATE(RT_ITER *iter)
+{
+#ifdef RT_SHMEM
+	Assert(LWLockHeldByMe(&iter->tree->ctl->lock));
+#endif
+
+	RT_UNLOCK(iter->tree);
+	pfree(iter);
+}
+
+/*
+ * Return the statistics of the amount of memory used by the radix tree.
+ */
+RT_SCOPE uint64
+RT_MEMORY_USAGE(RT_RADIX_TREE *tree)
+{
+	Size		total = 0;
+
+	RT_LOCK_SHARED(tree);
+
+#ifdef RT_SHMEM
+	Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+	total = dsa_get_total_size(tree->dsa);
+#else
+	for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+	{
+		total += MemoryContextMemAllocated(tree->inner_slabs[i], true);
+		total += MemoryContextMemAllocated(tree->leaf_slabs[i], true);
+	}
+#endif
+
+	RT_UNLOCK(tree);
+	return total;
+}
+
+/*
+ * Verify the radix tree node.
+ */
+static void
+RT_VERIFY_NODE(RT_PTR_LOCAL node)
+{
+#ifdef USE_ASSERT_CHECKING
+	Assert(node->count >= 0);
+
+	switch (node->kind)
+	{
+		case RT_NODE_KIND_3:
+			{
+				RT_NODE_BASE_3 *n3 = (RT_NODE_BASE_3 *) node;
+
+				for (int i = 1; i < n3->n.count; i++)
+					Assert(n3->chunks[i - 1] < n3->chunks[i]);
+
+				break;
+			}
+		case RT_NODE_KIND_32:
+			{
+				RT_NODE_BASE_32 *n32 = (RT_NODE_BASE_32 *) node;
+
+				for (int i = 1; i < n32->n.count; i++)
+					Assert(n32->chunks[i - 1] < n32->chunks[i]);
+
+				break;
+			}
+		case RT_NODE_KIND_125:
+			{
+				RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+				int			cnt = 0;
+
+				for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+				{
+					uint8		slot = n125->slot_idxs[i];
+					int			idx = RT_BM_IDX(slot);
+					int			bitnum = RT_BM_BIT(slot);
+
+					if (!RT_NODE_125_IS_CHUNK_USED(n125, i))
+						continue;
+
+					/* Check if the corresponding slot is used */
+					Assert(slot < node->fanout);
+					Assert((n125->isset[idx] & ((bitmapword) 1 << bitnum)) != 0);
+
+					cnt++;
+				}
+
+				Assert(n125->n.count == cnt);
+				break;
+			}
+		case RT_NODE_KIND_256:
+			{
+				if (RT_NODE_IS_LEAF(node))
+				{
+					RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+					int			cnt = 0;
+
+					for (int i = 0; i < RT_BM_IDX(RT_NODE_MAX_SLOTS); i++)
+						cnt += bmw_popcount(n256->isset[i]);
+
+					/* Check if the number of used chunk matches */
+					Assert(n256->base.n.count == cnt);
+
+					break;
+				}
+			}
+	}
+#endif
+}
+
+/***************** DEBUG FUNCTIONS *****************/
+#ifdef RT_DEBUG
+
+#define RT_UINT64_FORMAT_HEX "%" INT64_MODIFIER "X"
+
+RT_SCOPE void
+RT_STATS(RT_RADIX_TREE *tree)
+{
+	RT_LOCK_SHARED(tree);
+
+	fprintf(stderr, "max_val = " UINT64_FORMAT "\n", tree->ctl->max_val);
+	fprintf(stderr, "num_keys = " UINT64_FORMAT "\n", tree->ctl->num_keys);
+
+#ifdef RT_SHMEM
+	fprintf(stderr, "handle = " UINT64_FORMAT "\n", tree->ctl->handle);
+#endif
+
+	if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+	{
+		RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+
+		fprintf(stderr, "height = %d, n3 = %u, n15 = %u, n32 = %u, n125 = %u, n256 = %u\n",
+				root->shift / RT_NODE_SPAN,
+				tree->ctl->cnt[RT_CLASS_3],
+				tree->ctl->cnt[RT_CLASS_32_MIN],
+				tree->ctl->cnt[RT_CLASS_32_MAX],
+				tree->ctl->cnt[RT_CLASS_125],
+				tree->ctl->cnt[RT_CLASS_256]);
+	}
+
+	RT_UNLOCK(tree);
+}
+
+static void
+RT_DUMP_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, int level,
+			 bool recurse, StringInfo buf)
+{
+	RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+	StringInfoData spaces;
+
+	initStringInfo(&spaces);
+	appendStringInfoSpaces(&spaces, (level * 4) + 1);
+
+	appendStringInfo(buf, "%s%s[%s] kind %d, fanout %d, count %u, shift %u:\n",
+					 spaces.data,
+					 level == 0 ? "" : "-> ",
+					 RT_NODE_IS_LEAF(node) ? "LEAF" : "INNR",
+					 (node->kind == RT_NODE_KIND_3) ? 3 :
+					 (node->kind == RT_NODE_KIND_32) ? 32 :
+					 (node->kind == RT_NODE_KIND_125) ? 125 : 256,
+					 node->fanout == 0 ? 256 : node->fanout,
+					 node->count, node->shift);
+
+	switch (node->kind)
+	{
+		case RT_NODE_KIND_3:
+			{
+				for (int i = 0; i < node->count; i++)
+				{
+					if (RT_NODE_IS_LEAF(node))
+					{
+						RT_NODE_LEAF_3 *n3 = (RT_NODE_LEAF_3 *) node;
+
+						appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+										 spaces.data, i, n3->base.chunks[i]);
+					}
+					else
+					{
+						RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+						appendStringInfo(buf, "%schunk[%d] 0x%X",
+										 spaces.data, i, n3->base.chunks[i]);
+
+						if (recurse)
+						{
+							appendStringInfo(buf, "\n");
+							RT_DUMP_NODE(tree, n3->children[i], level + 1,
+										 recurse, buf);
+						}
+						else
+							appendStringInfo(buf, " (skipped)\n");
+					}
+				}
+				break;
+			}
+		case RT_NODE_KIND_32:
+			{
+				for (int i = 0; i < node->count; i++)
+				{
+					if (RT_NODE_IS_LEAF(node))
+					{
+						RT_NODE_LEAF_32 *n32 = (RT_NODE_LEAF_32 *) node;
+
+						appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+										 spaces.data, i, n32->base.chunks[i]);
+					}
+					else
+					{
+						RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+						appendStringInfo(buf, "%schunk[%d] 0x%X",
+										 spaces.data, i, n32->base.chunks[i]);
+
+						if (recurse)
+						{
+							appendStringInfo(buf, "\n");
+							RT_DUMP_NODE(tree, n32->children[i], level + 1,
+										 recurse, buf);
+						}
+						else
+							appendStringInfo(buf, " (skipped)\n");
+
+					}
+				}
+				break;
+			}
+		case RT_NODE_KIND_125:
+			{
+				RT_NODE_BASE_125 *b125 = (RT_NODE_BASE_125 *) node;
+				char *sep = "";
+
+				appendStringInfo(buf, "%sslot_idxs: ", spaces.data);
+				for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+				{
+					if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+						continue;
+
+					appendStringInfo(buf, "%s[%d]=%d ",
+									 sep, i, b125->slot_idxs[i]);
+					sep = ",";
+				}
+
+				appendStringInfo(buf, "\n%sisset-bitmap: ", spaces.data);
+				for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+					appendStringInfo(buf, "%X ", ((uint8 *) b125->isset)[i]);
+				appendStringInfo(buf, "\n");
+
+				for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+				{
+					if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+						continue;
+
+					if (RT_NODE_IS_LEAF(node))
+						appendStringInfo(buf, "%schunk 0x%X\n",
+										 spaces.data, i);
+					else
+					{
+						RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) b125;
+
+						appendStringInfo(buf, "%schunk 0x%X",
+										 spaces.data, i);
+
+						if (recurse)
+						{
+							appendStringInfo(buf, "\n");
+							RT_DUMP_NODE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i),
+										 level + 1, recurse, buf);
+						}
+						else
+							appendStringInfo(buf, " (skipped)\n");
+					}
+				}
+				break;
+			}
+		case RT_NODE_KIND_256:
+			{
+				if (RT_NODE_IS_LEAF(node))
+				{
+					RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+					appendStringInfo(buf, "%sisset-bitmap: ", spaces.data);
+					for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+						appendStringInfo(buf, "%X ", ((uint8 *) n256->isset)[i]);
+					appendStringInfo(buf, "\n");
+				}
+
+				for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+				{
+					if (RT_NODE_IS_LEAF(node))
+					{
+						RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+						if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+							continue;
+
+						appendStringInfo(buf, "%schunk 0x%X\n",
+										 spaces.data, i);
+					}
+					else
+					{
+						RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+						if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+							continue;
+
+						appendStringInfo(buf, "%schunk 0x%X",
+										 spaces.data, i);
+
+						if (recurse)
+						{
+							appendStringInfo(buf, "\n");
+							RT_DUMP_NODE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i),
+										 level + 1, recurse, buf);
+						}
+						else
+							appendStringInfo(buf, " (skipped)\n");
+					}
+				}
+				break;
+			}
+	}
+}
+
+RT_SCOPE void
+RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key)
+{
+	RT_PTR_ALLOC allocnode;
+	RT_PTR_LOCAL node;
+	StringInfoData buf;
+	int			shift;
+	int			level = 0;
+
+	RT_STATS(tree);
+
+	RT_LOCK_SHARED(tree);
+
+	if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+	{
+		RT_UNLOCK(tree);
+		fprintf(stderr, "empty tree\n");
+		return;
+	}
+
+	if (key > tree->ctl->max_val)
+	{
+		RT_UNLOCK(tree);
+		fprintf(stderr, "key " UINT64_FORMAT "(0x" RT_UINT64_FORMAT_HEX ") is larger than max val\n",
+				key, key);
+		return;
+	}
+
+	initStringInfo(&buf);
+	allocnode = tree->ctl->root;
+	node = RT_PTR_GET_LOCAL(tree, allocnode);
+	shift = node->shift;
+	while (shift >= 0)
+	{
+		RT_PTR_ALLOC child;
+
+		RT_DUMP_NODE(tree, allocnode, level, false, &buf);
+
+		if (RT_NODE_IS_LEAF(node))
+		{
+			RT_VALUE_TYPE	dummy;
+
+			/* We reached at a leaf node, find the corresponding slot */
+			RT_NODE_SEARCH_LEAF(node, key, &dummy);
+
+			break;
+		}
+
+		if (!RT_NODE_SEARCH_INNER(node, key, &child))
+			break;
+
+		allocnode = child;
+		node = RT_PTR_GET_LOCAL(tree, allocnode);
+		shift -= RT_NODE_SPAN;
+		level++;
+	}
+	RT_UNLOCK(tree);
+
+	fprintf(stderr, "%s", buf.data);
+}
+
+RT_SCOPE void
+RT_DUMP(RT_RADIX_TREE *tree)
+{
+	StringInfoData buf;
+
+	RT_STATS(tree);
+
+	RT_LOCK_SHARED(tree);
+
+	if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+	{
+		RT_UNLOCK(tree);
+		fprintf(stderr, "empty tree\n");
+		return;
+	}
+
+	initStringInfo(&buf);
+
+	RT_DUMP_NODE(tree, tree->ctl->root, 0, true, &buf);
+	RT_UNLOCK(tree);
+
+	fprintf(stderr, "%s",buf.data);
+}
+#endif
+
+#endif							/* RT_DEFINE */
+
+
+/* undefine external parameters, so next radix tree can be defined */
+#undef RT_PREFIX
+#undef RT_SCOPE
+#undef RT_DECLARE
+#undef RT_DEFINE
+#undef RT_VALUE_TYPE
+
+/* locally declared macros */
+#undef RT_MAKE_PREFIX
+#undef RT_MAKE_NAME
+#undef RT_MAKE_NAME_
+#undef RT_NODE_SPAN
+#undef RT_NODE_MAX_SLOTS
+#undef RT_CHUNK_MASK
+#undef RT_MAX_SHIFT
+#undef RT_MAX_LEVEL
+#undef RT_GET_KEY_CHUNK
+#undef RT_BM_IDX
+#undef RT_BM_BIT
+#undef RT_LOCK_EXCLUSIVE
+#undef RT_LOCK_SHARED
+#undef RT_UNLOCK
+#undef RT_NODE_IS_LEAF
+#undef RT_NODE_MUST_GROW
+#undef RT_NODE_KIND_COUNT
+#undef RT_SIZE_CLASS_COUNT
+#undef RT_SLOT_IDX_LIMIT
+#undef RT_INVALID_SLOT_IDX
+#undef RT_SLAB_BLOCK_SIZE
+#undef RT_RADIX_TREE_MAGIC
+#undef RT_UINT64_FORMAT_HEX
+
+/* type declarations */
+#undef RT_RADIX_TREE
+#undef RT_RADIX_TREE_CONTROL
+#undef RT_PTR_LOCAL
+#undef RT_PTR_ALLOC
+#undef RT_INVALID_PTR_ALLOC
+#undef RT_HANDLE
+#undef RT_ITER
+#undef RT_NODE
+#undef RT_NODE_ITER
+#undef RT_NODE_KIND_3
+#undef RT_NODE_KIND_32
+#undef RT_NODE_KIND_125
+#undef RT_NODE_KIND_256
+#undef RT_NODE_BASE_3
+#undef RT_NODE_BASE_32
+#undef RT_NODE_BASE_125
+#undef RT_NODE_BASE_256
+#undef RT_NODE_INNER_3
+#undef RT_NODE_INNER_32
+#undef RT_NODE_INNER_125
+#undef RT_NODE_INNER_256
+#undef RT_NODE_LEAF_3
+#undef RT_NODE_LEAF_32
+#undef RT_NODE_LEAF_125
+#undef RT_NODE_LEAF_256
+#undef RT_SIZE_CLASS
+#undef RT_SIZE_CLASS_ELEM
+#undef RT_SIZE_CLASS_INFO
+#undef RT_CLASS_3
+#undef RT_CLASS_32_MIN
+#undef RT_CLASS_32_MAX
+#undef RT_CLASS_125
+#undef RT_CLASS_256
+
+/* function declarations */
+#undef RT_CREATE
+#undef RT_FREE
+#undef RT_ATTACH
+#undef RT_DETACH
+#undef RT_GET_HANDLE
+#undef RT_SEARCH
+#undef RT_SET
+#undef RT_BEGIN_ITERATE
+#undef RT_ITERATE_NEXT
+#undef RT_END_ITERATE
+#undef RT_USE_DELETE
+#undef RT_DELETE
+#undef RT_MEMORY_USAGE
+#undef RT_DUMP
+#undef RT_DUMP_NODE
+#undef RT_DUMP_SEARCH
+#undef RT_STATS
+
+/* internal helper functions */
+#undef RT_NEW_ROOT
+#undef RT_ALLOC_NODE
+#undef RT_INIT_NODE
+#undef RT_FREE_NODE
+#undef RT_FREE_RECURSE
+#undef RT_EXTEND
+#undef RT_SET_EXTEND
+#undef RT_SWITCH_NODE_KIND
+#undef RT_COPY_NODE
+#undef RT_REPLACE_NODE
+#undef RT_PTR_GET_LOCAL
+#undef RT_PTR_ALLOC_IS_VALID
+#undef RT_NODE_3_SEARCH_EQ
+#undef RT_NODE_32_SEARCH_EQ
+#undef RT_NODE_3_GET_INSERTPOS
+#undef RT_NODE_32_GET_INSERTPOS
+#undef RT_CHUNK_CHILDREN_ARRAY_SHIFT
+#undef RT_CHUNK_VALUES_ARRAY_SHIFT
+#undef RT_CHUNK_CHILDREN_ARRAY_DELETE
+#undef RT_CHUNK_VALUES_ARRAY_DELETE
+#undef RT_CHUNK_CHILDREN_ARRAY_COPY
+#undef RT_CHUNK_VALUES_ARRAY_COPY
+#undef RT_NODE_125_IS_CHUNK_USED
+#undef RT_NODE_INNER_125_GET_CHILD
+#undef RT_NODE_LEAF_125_GET_VALUE
+#undef RT_NODE_INNER_256_IS_CHUNK_USED
+#undef RT_NODE_LEAF_256_IS_CHUNK_USED
+#undef RT_NODE_INNER_256_GET_CHILD
+#undef RT_NODE_LEAF_256_GET_VALUE
+#undef RT_NODE_INNER_256_SET
+#undef RT_NODE_LEAF_256_SET
+#undef RT_NODE_INNER_256_DELETE
+#undef RT_NODE_LEAF_256_DELETE
+#undef RT_KEY_GET_SHIFT
+#undef RT_SHIFT_GET_MAX_VAL
+#undef RT_NODE_SEARCH_INNER
+#undef RT_NODE_SEARCH_LEAF
+#undef RT_NODE_UPDATE_INNER
+#undef RT_NODE_DELETE_INNER
+#undef RT_NODE_DELETE_LEAF
+#undef RT_NODE_INSERT_INNER
+#undef RT_NODE_INSERT_LEAF
+#undef RT_NODE_INNER_ITERATE_NEXT
+#undef RT_NODE_LEAF_ITERATE_NEXT
+#undef RT_UPDATE_ITER_STACK
+#undef RT_ITER_UPDATE_KEY
+#undef RT_VERIFY_NODE
+
+#undef RT_DEBUG
diff --git a/src/include/lib/radixtree_delete_impl.h b/src/include/lib/radixtree_delete_impl.h
new file mode 100644
index 0000000000..5f6dda1f12
--- /dev/null
+++ b/src/include/lib/radixtree_delete_impl.h
@@ -0,0 +1,122 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_delete_impl.h
+ *	  Common implementation for deletion in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ * TODO: Shrink nodes when deletion would allow them to fit in a smaller
+ * size class.
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_delete_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+	uint8		chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+	Assert(RT_NODE_IS_LEAF(node));
+#else
+	Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+	switch (node->kind)
+	{
+		case RT_NODE_KIND_3:
+			{
+				RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+				int			idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+				if (idx < 0)
+					return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+				RT_CHUNK_VALUES_ARRAY_DELETE(n3->base.chunks, n3->values,
+										  n3->base.n.count, idx);
+#else
+				RT_CHUNK_CHILDREN_ARRAY_DELETE(n3->base.chunks, n3->children,
+											n3->base.n.count, idx);
+#endif
+				break;
+			}
+		case RT_NODE_KIND_32:
+			{
+				RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+				int			idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+				if (idx < 0)
+					return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+				RT_CHUNK_VALUES_ARRAY_DELETE(n32->base.chunks, n32->values,
+										  n32->base.n.count, idx);
+#else
+				RT_CHUNK_CHILDREN_ARRAY_DELETE(n32->base.chunks, n32->children,
+											n32->base.n.count, idx);
+#endif
+				break;
+			}
+		case RT_NODE_KIND_125:
+			{
+				RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+				int			slotpos = n125->base.slot_idxs[chunk];
+				int			idx;
+				int			bitnum;
+
+				if (slotpos == RT_INVALID_SLOT_IDX)
+					return false;
+
+				idx = RT_BM_IDX(slotpos);
+				bitnum = RT_BM_BIT(slotpos);
+				n125->base.isset[idx] &= ~((bitmapword) 1 << bitnum);
+				n125->base.slot_idxs[chunk] = RT_INVALID_SLOT_IDX;
+
+				break;
+			}
+		case RT_NODE_KIND_256:
+			{
+				RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+				if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+				if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+					return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+				RT_NODE_LEAF_256_DELETE(n256, chunk);
+#else
+				RT_NODE_INNER_256_DELETE(n256, chunk);
+#endif
+				break;
+			}
+	}
+
+	/* update statistics */
+	node->count--;
+
+	return true;
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
new file mode 100644
index 0000000000..c18e26b537
--- /dev/null
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -0,0 +1,332 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_insert_impl.h
+ *	  Common implementation for insertion in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_insert_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+	uint8		chunk = RT_GET_KEY_CHUNK(key, node->shift);
+	bool		chunk_exists = false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+	const bool is_leaf = true;
+	Assert(RT_NODE_IS_LEAF(node));
+#else
+	const bool is_leaf = false;
+	Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+	switch (node->kind)
+	{
+		case RT_NODE_KIND_3:
+			{
+				RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+				int			idx;
+
+				idx = RT_NODE_3_SEARCH_EQ(&n3->base, chunk);
+				if (idx != -1)
+				{
+					/* found the existing chunk */
+					chunk_exists = true;
+#ifdef RT_NODE_LEVEL_LEAF
+					n3->values[idx] = *value_p;
+#else
+					n3->children[idx] = child;
+#endif
+					break;
+				}
+
+				if (unlikely(RT_NODE_MUST_GROW(n3)))
+				{
+					RT_PTR_ALLOC allocnode;
+					RT_PTR_LOCAL newnode;
+					RT_NODE32_TYPE *new32;
+					const uint8 new_kind = RT_NODE_KIND_32;
+					const RT_SIZE_CLASS new_class = RT_CLASS_32_MIN;
+
+					/* grow node from 3 to 32 */
+					allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+					newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+					new32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+					RT_CHUNK_VALUES_ARRAY_COPY(n3->base.chunks, n3->values,
+											  new32->base.chunks, new32->values);
+#else
+					RT_CHUNK_CHILDREN_ARRAY_COPY(n3->base.chunks, n3->children,
+											  new32->base.chunks, new32->children);
+#endif
+					RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+					node = newnode;
+				}
+				else
+				{
+					int			insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
+					int			count = n3->base.n.count;
+
+					/* shift chunks and children */
+					if (insertpos < count)
+					{
+						Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+						RT_CHUNK_VALUES_ARRAY_SHIFT(n3->base.chunks, n3->values,
+												   count, insertpos);
+#else
+						RT_CHUNK_CHILDREN_ARRAY_SHIFT(n3->base.chunks, n3->children,
+												   count, insertpos);
+#endif
+					}
+
+					n3->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+					n3->values[insertpos] = *value_p;
+#else
+					n3->children[insertpos] = child;
+#endif
+					break;
+				}
+			}
+			/* FALLTHROUGH */
+		case RT_NODE_KIND_32:
+			{
+				const RT_SIZE_CLASS_ELEM class32_max = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX];
+				RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+				int			idx;
+
+				idx = RT_NODE_32_SEARCH_EQ(&n32->base, chunk);
+				if (idx != -1)
+				{
+					/* found the existing chunk */
+					chunk_exists = true;
+#ifdef RT_NODE_LEVEL_LEAF
+					n32->values[idx] = *value_p;
+#else
+					n32->children[idx] = child;
+#endif
+					break;
+				}
+
+				if (unlikely(RT_NODE_MUST_GROW(n32)) &&
+					n32->base.n.fanout < class32_max.fanout)
+				{
+					RT_PTR_ALLOC allocnode;
+					RT_PTR_LOCAL newnode;
+					const RT_SIZE_CLASS_ELEM class32_min = RT_SIZE_CLASS_INFO[RT_CLASS_32_MIN];
+					const RT_SIZE_CLASS new_class = RT_CLASS_32_MAX;
+
+					Assert(n32->base.n.fanout == class32_min.fanout);
+
+					/* grow to the next size class of this kind */
+					allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+					newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+					n32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+					memcpy(newnode, node, class32_min.leaf_size);
+#else
+					memcpy(newnode, node, class32_min.inner_size);
+#endif
+					newnode->fanout = class32_max.fanout;
+
+					RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+					node = newnode;
+				}
+
+				if (unlikely(RT_NODE_MUST_GROW(n32)))
+				{
+					RT_PTR_ALLOC allocnode;
+					RT_PTR_LOCAL newnode;
+					RT_NODE125_TYPE *new125;
+					const uint8 new_kind = RT_NODE_KIND_125;
+					const RT_SIZE_CLASS new_class = RT_CLASS_125;
+
+					Assert(n32->base.n.fanout == class32_max.fanout);
+
+					/* grow node from 32 to 125 */
+					allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+					newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+					new125 = (RT_NODE125_TYPE *) newnode;
+
+					for (int i = 0; i < class32_max.fanout; i++)
+					{
+						new125->base.slot_idxs[n32->base.chunks[i]] = i;
+#ifdef RT_NODE_LEVEL_LEAF
+						new125->values[i] = n32->values[i];
+#else
+						new125->children[i] = n32->children[i];
+#endif
+					}
+
+					/*
+					 * Since we just copied a dense array, we can set the bits
+					 * using a single store, provided the length of that array
+					 * is at most the number of bits in a bitmapword.
+					 */
+					Assert(class32_max.fanout <= sizeof(bitmapword) * BITS_PER_BYTE);
+					new125->base.isset[0] = (bitmapword) (((uint64) 1 << class32_max.fanout) - 1);
+
+					RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+					node = newnode;
+				}
+				else
+				{
+					int	insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
+					int count = n32->base.n.count;
+
+					if (insertpos < count)
+					{
+						Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+						RT_CHUNK_VALUES_ARRAY_SHIFT(n32->base.chunks, n32->values,
+												   count, insertpos);
+#else
+						RT_CHUNK_CHILDREN_ARRAY_SHIFT(n32->base.chunks, n32->children,
+												   count, insertpos);
+#endif
+					}
+
+					n32->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+					n32->values[insertpos] = *value_p;
+#else
+					n32->children[insertpos] = child;
+#endif
+					break;
+				}
+			}
+			/* FALLTHROUGH */
+		case RT_NODE_KIND_125:
+			{
+				RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+				int			slotpos = n125->base.slot_idxs[chunk];
+				int			cnt = 0;
+
+				if (slotpos != RT_INVALID_SLOT_IDX)
+				{
+					/* found the existing chunk */
+					chunk_exists = true;
+#ifdef RT_NODE_LEVEL_LEAF
+					n125->values[slotpos] = *value_p;
+#else
+					n125->children[slotpos] = child;
+#endif
+					break;
+				}
+
+				if (unlikely(RT_NODE_MUST_GROW(n125)))
+				{
+					RT_PTR_ALLOC allocnode;
+					RT_PTR_LOCAL newnode;
+					RT_NODE256_TYPE *new256;
+					const uint8 new_kind = RT_NODE_KIND_256;
+					const RT_SIZE_CLASS new_class = RT_CLASS_256;
+
+					/* grow node from 125 to 256 */
+					allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+					newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+					new256 = (RT_NODE256_TYPE *) newnode;
+
+					for (int i = 0; i < RT_NODE_MAX_SLOTS && cnt < n125->base.n.count; i++)
+					{
+						if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+							continue;
+#ifdef RT_NODE_LEVEL_LEAF
+						RT_NODE_LEAF_256_SET(new256, i, RT_NODE_LEAF_125_GET_VALUE(n125, i));
+#else
+						RT_NODE_INNER_256_SET(new256, i, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+						cnt++;
+					}
+
+					RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+					node = newnode;
+				}
+				else
+				{
+					int			idx;
+					bitmapword	inverse;
+
+					/* get the first word with at least one bit not set */
+					for (idx = 0; idx < RT_BM_IDX(RT_SLOT_IDX_LIMIT); idx++)
+					{
+						if (n125->base.isset[idx] < ~((bitmapword) 0))
+							break;
+					}
+
+					/* To get the first unset bit in X, get the first set bit in ~X */
+					inverse = ~(n125->base.isset[idx]);
+					slotpos = idx * BITS_PER_BITMAPWORD;
+					slotpos += bmw_rightmost_one_pos(inverse);
+					Assert(slotpos < node->fanout);
+
+					/* mark the slot used */
+					n125->base.isset[idx] |= bmw_rightmost_one(inverse);
+					n125->base.slot_idxs[chunk] = slotpos;
+
+#ifdef RT_NODE_LEVEL_LEAF
+					n125->values[slotpos] = *value_p;
+#else
+					n125->children[slotpos] = child;
+#endif
+					break;
+				}
+			}
+			/* FALLTHROUGH */
+		case RT_NODE_KIND_256:
+			{
+				RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+				chunk_exists = RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk);
+#else
+				chunk_exists = RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk);
+#endif
+				Assert(chunk_exists || node->count < RT_NODE_MAX_SLOTS);
+
+#ifdef RT_NODE_LEVEL_LEAF
+				RT_NODE_LEAF_256_SET(n256, chunk, *value_p);
+#else
+				RT_NODE_INNER_256_SET(n256, chunk, child);
+#endif
+				break;
+			}
+	}
+
+	/* Update statistics */
+	if (!chunk_exists)
+		node->count++;
+
+	/*
+	 * Done. Finally, verify the chunk and value is inserted or replaced
+	 * properly in the node.
+	 */
+	RT_VERIFY_NODE(node);
+
+	return chunk_exists;
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_iter_impl.h b/src/include/lib/radixtree_iter_impl.h
new file mode 100644
index 0000000000..98c78eb237
--- /dev/null
+++ b/src/include/lib/radixtree_iter_impl.h
@@ -0,0 +1,153 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_iter_impl.h
+ *	  Common implementation for iteration in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_iter_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+	bool		found = false;
+	uint8		key_chunk;
+
+#ifdef RT_NODE_LEVEL_LEAF
+	RT_VALUE_TYPE		value;
+
+	Assert(RT_NODE_IS_LEAF(node_iter->node));
+#else
+	RT_PTR_LOCAL child = NULL;
+
+	Assert(!RT_NODE_IS_LEAF(node_iter->node));
+#endif
+
+#ifdef RT_SHMEM
+	Assert(iter->tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+	switch (node_iter->node->kind)
+	{
+		case RT_NODE_KIND_3:
+			{
+				RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node_iter->node;
+
+				node_iter->current_idx++;
+				if (node_iter->current_idx >= n3->base.n.count)
+					break;
+#ifdef RT_NODE_LEVEL_LEAF
+				value = n3->values[node_iter->current_idx];
+#else
+				child = RT_PTR_GET_LOCAL(iter->tree, n3->children[node_iter->current_idx]);
+#endif
+				key_chunk = n3->base.chunks[node_iter->current_idx];
+				found = true;
+				break;
+			}
+		case RT_NODE_KIND_32:
+			{
+				RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node_iter->node;
+
+				node_iter->current_idx++;
+				if (node_iter->current_idx >= n32->base.n.count)
+					break;
+
+#ifdef RT_NODE_LEVEL_LEAF
+				value = n32->values[node_iter->current_idx];
+#else
+				child = RT_PTR_GET_LOCAL(iter->tree, n32->children[node_iter->current_idx]);
+#endif
+				key_chunk = n32->base.chunks[node_iter->current_idx];
+				found = true;
+				break;
+			}
+		case RT_NODE_KIND_125:
+			{
+				RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node_iter->node;
+				int			i;
+
+				for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+				{
+					if (RT_NODE_125_IS_CHUNK_USED((RT_NODE_BASE_125 *) n125, i))
+						break;
+				}
+
+				if (i >= RT_NODE_MAX_SLOTS)
+					break;
+
+				node_iter->current_idx = i;
+#ifdef RT_NODE_LEVEL_LEAF
+				value = RT_NODE_LEAF_125_GET_VALUE(n125, i);
+#else
+				child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+				key_chunk = i;
+				found = true;
+				break;
+			}
+		case RT_NODE_KIND_256:
+			{
+				RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node_iter->node;
+				int			i;
+
+				for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+				{
+#ifdef RT_NODE_LEVEL_LEAF
+					if (RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+#else
+					if (RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+#endif
+						break;
+				}
+
+				if (i >= RT_NODE_MAX_SLOTS)
+					break;
+
+				node_iter->current_idx = i;
+#ifdef RT_NODE_LEVEL_LEAF
+				value = RT_NODE_LEAF_256_GET_VALUE(n256, i);
+#else
+				child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+#endif
+				key_chunk = i;
+				found = true;
+				break;
+			}
+	}
+
+	if (found)
+	{
+		RT_ITER_UPDATE_KEY(iter, key_chunk, node_iter->node->shift);
+#ifdef RT_NODE_LEVEL_LEAF
+		*value_p = value;
+#endif
+	}
+
+#ifdef RT_NODE_LEVEL_LEAF
+	return found;
+#else
+	return child;
+#endif
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_search_impl.h b/src/include/lib/radixtree_search_impl.h
new file mode 100644
index 0000000000..a8925c75d0
--- /dev/null
+++ b/src/include/lib/radixtree_search_impl.h
@@ -0,0 +1,138 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_search_impl.h
+ *	  Common implementation for search in leaf and inner nodes, plus
+ *	  update for inner nodes only.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_search_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+	uint8		chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+	Assert(value_p != NULL);
+	Assert(RT_NODE_IS_LEAF(node));
+#else
+#ifndef RT_ACTION_UPDATE
+	Assert(child_p != NULL);
+#endif
+	Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+	switch (node->kind)
+	{
+		case RT_NODE_KIND_3:
+			{
+				RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+				int			idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+#ifdef RT_ACTION_UPDATE
+				Assert(idx >= 0);
+				n3->children[idx] = new_child;
+#else
+				if (idx < 0)
+					return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+				*value_p = n3->values[idx];
+#else
+				*child_p = n3->children[idx];
+#endif
+#endif							/* RT_ACTION_UPDATE */
+				break;
+			}
+		case RT_NODE_KIND_32:
+			{
+				RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+				int			idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+#ifdef RT_ACTION_UPDATE
+				Assert(idx >= 0);
+				n32->children[idx] = new_child;
+#else
+				if (idx < 0)
+					return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+				*value_p = n32->values[idx];
+#else
+				*child_p = n32->children[idx];
+#endif
+#endif							/* RT_ACTION_UPDATE */
+				break;
+			}
+		case RT_NODE_KIND_125:
+			{
+				RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+				int			slotpos = n125->base.slot_idxs[chunk];
+
+#ifdef RT_ACTION_UPDATE
+				Assert(slotpos != RT_INVALID_SLOT_IDX);
+				n125->children[slotpos] = new_child;
+#else
+				if (slotpos == RT_INVALID_SLOT_IDX)
+					return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+				*value_p = RT_NODE_LEAF_125_GET_VALUE(n125, chunk);
+#else
+				*child_p = RT_NODE_INNER_125_GET_CHILD(n125, chunk);
+#endif
+#endif							/* RT_ACTION_UPDATE */
+				break;
+			}
+		case RT_NODE_KIND_256:
+			{
+				RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_ACTION_UPDATE
+				RT_NODE_INNER_256_SET(n256, chunk, new_child);
+#else
+#ifdef RT_NODE_LEVEL_LEAF
+				if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+				if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+					return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+				*value_p = RT_NODE_LEAF_256_GET_VALUE(n256, chunk);
+#else
+				*child_p = RT_NODE_INNER_256_GET_CHILD(n256, chunk);
+#endif
+#endif							/* RT_ACTION_UPDATE */
+				break;
+			}
+	}
+
+#ifdef RT_ACTION_UPDATE
+	return;
+#else
+	return true;
+#endif							/* RT_ACTION_UPDATE */
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/utils/dsa.h b/src/include/utils/dsa.h
index 3ce4ee300a..2af215484f 100644
--- a/src/include/utils/dsa.h
+++ b/src/include/utils/dsa.h
@@ -121,6 +121,7 @@ extern dsa_handle dsa_get_handle(dsa_area *area);
 extern dsa_pointer dsa_allocate_extended(dsa_area *area, size_t size, int flags);
 extern void dsa_free(dsa_area *area, dsa_pointer dp);
 extern void *dsa_get_address(dsa_area *area, dsa_pointer dp);
+extern size_t dsa_get_total_size(dsa_area *area);
 extern void dsa_trim(dsa_area *area);
 extern void dsa_dump(dsa_area *area);
 
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index c629cbe383..9659eb85d7 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -28,6 +28,7 @@ SUBDIRS = \
 		  test_pg_db_role_setting \
 		  test_pg_dump \
 		  test_predtest \
+		  test_radixtree \
 		  test_rbtree \
 		  test_regex \
 		  test_rls_hooks \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 1baa6b558d..232cbdac80 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -24,6 +24,7 @@ subdir('test_parser')
 subdir('test_pg_db_role_setting')
 subdir('test_pg_dump')
 subdir('test_predtest')
+subdir('test_radixtree')
 subdir('test_rbtree')
 subdir('test_regex')
 subdir('test_rls_hooks')
diff --git a/src/test/modules/test_radixtree/.gitignore b/src/test/modules/test_radixtree/.gitignore
new file mode 100644
index 0000000000..5dcb3ff972
--- /dev/null
+++ b/src/test/modules/test_radixtree/.gitignore
@@ -0,0 +1,4 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
diff --git a/src/test/modules/test_radixtree/Makefile b/src/test/modules/test_radixtree/Makefile
new file mode 100644
index 0000000000..da06b93da3
--- /dev/null
+++ b/src/test/modules/test_radixtree/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_radixtree/Makefile
+
+MODULE_big = test_radixtree
+OBJS = \
+	$(WIN32RES) \
+	test_radixtree.o
+PGFILEDESC = "test_radixtree - test code for src/backend/lib/radixtree.c"
+
+EXTENSION = test_radixtree
+DATA = test_radixtree--1.0.sql
+
+REGRESS = test_radixtree
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_radixtree
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_radixtree/README b/src/test/modules/test_radixtree/README
new file mode 100644
index 0000000000..a8b271869a
--- /dev/null
+++ b/src/test/modules/test_radixtree/README
@@ -0,0 +1,7 @@
+test_integerset contains unit tests for testing the integer set implementation
+in src/backend/lib/integerset.c.
+
+The tests verify the correctness of the implementation, but they can also be
+used as a micro-benchmark.  If you set the 'intset_test_stats' flag in
+test_integerset.c, the tests will print extra information about execution time
+and memory usage.
diff --git a/src/test/modules/test_radixtree/expected/test_radixtree.out b/src/test/modules/test_radixtree/expected/test_radixtree.out
new file mode 100644
index 0000000000..ce645cb8b5
--- /dev/null
+++ b/src/test/modules/test_radixtree/expected/test_radixtree.out
@@ -0,0 +1,36 @@
+CREATE EXTENSION test_radixtree;
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
+NOTICE:  testing basic operations with leaf node 4
+NOTICE:  testing basic operations with inner node 4
+NOTICE:  testing basic operations with leaf node 32
+NOTICE:  testing basic operations with inner node 32
+NOTICE:  testing basic operations with leaf node 125
+NOTICE:  testing basic operations with inner node 125
+NOTICE:  testing basic operations with leaf node 256
+NOTICE:  testing basic operations with inner node 256
+NOTICE:  testing radix tree node types with shift "0"
+NOTICE:  testing radix tree node types with shift "8"
+NOTICE:  testing radix tree node types with shift "16"
+NOTICE:  testing radix tree node types with shift "24"
+NOTICE:  testing radix tree node types with shift "32"
+NOTICE:  testing radix tree node types with shift "40"
+NOTICE:  testing radix tree node types with shift "48"
+NOTICE:  testing radix tree node types with shift "56"
+NOTICE:  testing radix tree with pattern "all ones"
+NOTICE:  testing radix tree with pattern "alternating bits"
+NOTICE:  testing radix tree with pattern "clusters of ten"
+NOTICE:  testing radix tree with pattern "clusters of hundred"
+NOTICE:  testing radix tree with pattern "one-every-64k"
+NOTICE:  testing radix tree with pattern "sparse"
+NOTICE:  testing radix tree with pattern "single values, distance > 2^32"
+NOTICE:  testing radix tree with pattern "clusters, distance > 2^32"
+NOTICE:  testing radix tree with pattern "clusters, distance > 2^60"
+ test_radixtree 
+----------------
+ 
+(1 row)
+
diff --git a/src/test/modules/test_radixtree/meson.build b/src/test/modules/test_radixtree/meson.build
new file mode 100644
index 0000000000..6add06bbdb
--- /dev/null
+++ b/src/test/modules/test_radixtree/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_radixtree_sources = files(
+  'test_radixtree.c',
+)
+
+if host_system == 'windows'
+  test_radixtree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+    '--NAME', 'test_radixtree',
+    '--FILEDESC', 'test_radixtree - test code for src/include//lib/radixtree.h',])
+endif
+
+test_radixtree = shared_module('test_radixtree',
+  test_radixtree_sources,
+  link_with: pgport_srv,
+  kwargs: pg_mod_args,
+)
+testprep_targets += test_radixtree
+
+install_data(
+  'test_radixtree.control',
+  'test_radixtree--1.0.sql',
+  kwargs: contrib_data_args,
+)
+
+tests += {
+  'name': 'test_radixtree',
+  'sd': meson.current_source_dir(),
+  'bd': meson.current_build_dir(),
+  'regress': {
+    'sql': [
+      'test_radixtree',
+    ],
+  },
+}
diff --git a/src/test/modules/test_radixtree/sql/test_radixtree.sql b/src/test/modules/test_radixtree/sql/test_radixtree.sql
new file mode 100644
index 0000000000..41ece5e9f5
--- /dev/null
+++ b/src/test/modules/test_radixtree/sql/test_radixtree.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_radixtree;
+
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
diff --git a/src/test/modules/test_radixtree/test_radixtree--1.0.sql b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
new file mode 100644
index 0000000000..074a5a7ea7
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_radixtree/test_radixtree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_radixtree" to load this file. \quit
+
+CREATE FUNCTION test_radixtree()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
new file mode 100644
index 0000000000..f944945db9
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -0,0 +1,674 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_radixtree.c
+ *		Test radixtree set data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ *		src/test/modules/test_radixtree/test_radixtree.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "storage/lwlock.h"
+#include "utils/memutils.h"
+#include "utils/timestamp.h"
+
+#define UINT64_HEX_FORMAT "%" INT64_MODIFIER "X"
+
+/*
+ * The tests pass with uint32, but build with warnings because the string
+ * format expects uint64.
+ */
+typedef uint64 TestValueType;
+
+/*
+ * If you enable this, the "pattern" tests will print information about
+ * how long populating, probing, and iterating the test set takes, and
+ * how much memory the test set consumed.  That can be used as
+ * micro-benchmark of various operations and input patterns (you might
+ * want to increase the number of values used in each of the test, if
+ * you do that, to reduce noise).
+ *
+ * The information is printed to the server's stderr, mostly because
+ * that's where MemoryContextStats() output goes.
+ */
+static const bool rt_test_stats = false;
+
+static int	rt_node_kind_fanouts[] = {
+	0,
+	4,							/* RT_NODE_KIND_4 */
+	32,							/* RT_NODE_KIND_32 */
+	125,						/* RT_NODE_KIND_125 */
+	256							/* RT_NODE_KIND_256 */
+};
+/*
+ * A struct to define a pattern of integers, for use with the test_pattern()
+ * function.
+ */
+typedef struct
+{
+	char	   *test_name;		/* short name of the test, for humans */
+	char	   *pattern_str;	/* a bit pattern */
+	uint64		spacing;		/* pattern repeats at this interval */
+	uint64		num_values;		/* number of integers to set in total */
+}			test_spec;
+
+/* Test patterns borrowed from test_integerset.c */
+static const test_spec test_specs[] = {
+	{
+		"all ones", "1111111111",
+		10, 1000000
+	},
+	{
+		"alternating bits", "0101010101",
+		10, 1000000
+	},
+	{
+		"clusters of ten", "1111111111",
+		10000, 1000000
+	},
+	{
+		"clusters of hundred",
+		"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111",
+		10000, 1000000
+	},
+	{
+		"one-every-64k", "1",
+		65536, 1000000
+	},
+	{
+		"sparse", "100000000000000000000000000000001",
+		10000000, 1000000
+	},
+	{
+		"single values, distance > 2^32", "1",
+		UINT64CONST(10000000000), 100000
+	},
+	{
+		"clusters, distance > 2^32", "10101010",
+		UINT64CONST(10000000000), 1000000
+	},
+	{
+		"clusters, distance > 2^60", "10101010",
+		UINT64CONST(2000000000000000000),
+		23						/* can't be much higher than this, or we
+								 * overflow uint64 */
+	}
+};
+
+/* define the radix tree implementation to test */
+#define RT_PREFIX rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE TestValueType
+// WIP: compiles with warnings because rt_attach is defined but not used
+// #define RT_SHMEM
+#include "lib/radixtree.h"
+
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+	return tree->ctl->num_keys;
+}
+
+PG_MODULE_MAGIC;
+
+PG_FUNCTION_INFO_V1(test_radixtree);
+
+static void
+test_empty(void)
+{
+	rt_radix_tree *radixtree;
+	rt_iter		*iter;
+	TestValueType		dummy;
+	uint64		key;
+	TestValueType		val;
+
+#ifdef RT_SHMEM
+	int			tranche_id = LWLockNewTrancheId();
+	dsa_area   *dsa;
+	dsa = dsa_create(tranche_id);
+
+	radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+	radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+	if (rt_search(radixtree, 0, &dummy))
+		elog(ERROR, "rt_search on empty tree returned true");
+
+	if (rt_search(radixtree, 1, &dummy))
+		elog(ERROR, "rt_search on empty tree returned true");
+
+	if (rt_search(radixtree, PG_UINT64_MAX, &dummy))
+		elog(ERROR, "rt_search on empty tree returned true");
+
+	if (rt_delete(radixtree, 0))
+		elog(ERROR, "rt_delete on empty tree returned true");
+
+	if (rt_num_entries(radixtree) != 0)
+		elog(ERROR, "rt_num_entries on empty tree return non-zero");
+
+	iter = rt_begin_iterate(radixtree);
+
+	if (rt_iterate_next(iter, &key, &val))
+		elog(ERROR, "rt_itereate_next on empty tree returned true");
+
+	rt_end_iterate(iter);
+
+	rt_free(radixtree);
+
+#ifdef RT_SHMEM
+	dsa_detach(dsa);
+#endif
+}
+
+static void
+test_basic(int children, bool test_inner)
+{
+	rt_radix_tree	*radixtree;
+	uint64 *keys;
+	int	shift = test_inner ? 8 : 0;
+
+#ifdef RT_SHMEM
+	int			tranche_id = LWLockNewTrancheId();
+	dsa_area   *dsa;
+	dsa = dsa_create(tranche_id);
+#endif
+
+	elog(NOTICE, "testing basic operations with %s node %d",
+		 test_inner ? "inner" : "leaf", children);
+
+#ifdef RT_SHMEM
+	radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+	radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+	/* prepare keys in order like 1, 32, 2, 31, 2, ... */
+	keys = palloc(sizeof(uint64) * children);
+	for (int i = 0; i < children; i++)
+	{
+		if (i % 2 == 0)
+			keys[i] = (uint64) ((i / 2) + 1) << shift;
+		else
+			keys[i] = (uint64) (children - (i / 2)) << shift;
+	}
+
+	/* insert keys */
+	for (int i = 0; i < children; i++)
+	{
+		if (rt_set(radixtree, keys[i], (TestValueType*) &keys[i]))
+			elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+	}
+
+	/* look up keys */
+	for (int i = 0; i < children; i++)
+	{
+		TestValueType value;
+
+		if (!rt_search(radixtree, keys[i], &value))
+			elog(ERROR, "could not find key 0x" UINT64_HEX_FORMAT, keys[i]);
+		if (value != (TestValueType) keys[i])
+			elog(ERROR, "rt_search returned 0x" UINT64_HEX_FORMAT ", expected " UINT64_HEX_FORMAT,
+				 value, (TestValueType) keys[i]);
+	}
+
+	/* update keys */
+	for (int i = 0; i < children; i++)
+	{
+		TestValueType update = keys[i] + 1;
+		if (!rt_set(radixtree, keys[i], (TestValueType*) &update))
+			elog(ERROR, "could not update key 0x" UINT64_HEX_FORMAT, keys[i]);
+	}
+
+	/* repeat deleting and inserting keys */
+	for (int i = 0; i < children; i++)
+	{
+		if (!rt_delete(radixtree, keys[i]))
+			elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, keys[i]);
+		if (rt_set(radixtree, keys[i], (TestValueType*) &keys[i]))
+			elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+	}
+
+	pfree(keys);
+	rt_free(radixtree);
+#ifdef RT_SHMEM
+	dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Check if keys from start to end with the shift exist in the tree.
+ */
+static void
+check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end,
+					 int incr)
+{
+	for (int i = start; i < end; i++)
+	{
+		uint64		key = ((uint64) i << shift);
+		TestValueType		val;
+
+		if (!rt_search(radixtree, key, &val))
+			elog(ERROR, "key 0x" UINT64_HEX_FORMAT " is not found on node-%d",
+				 key, end);
+		if (val != (TestValueType) key)
+			elog(ERROR, "rt_search with key 0x" UINT64_HEX_FORMAT " returns 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+				 key, val, key);
+	}
+}
+
+static void
+test_node_types_insert(rt_radix_tree *radixtree, uint8 shift, bool insert_asc)
+{
+	uint64		num_entries;
+	int		ninserted = 0;
+	int		start = insert_asc ? 0 : 256;
+	int 	incr = insert_asc ? 1 : -1;
+	int		end = insert_asc ? 256 : 0;
+	int		node_kind_idx = 1;
+
+	for (int i = start; i != end; i += incr)
+	{
+		uint64		key = ((uint64) i << shift);
+		bool		found;
+
+		found = rt_set(radixtree, key, (TestValueType*) &key);
+		if (found)
+			elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " is found", key);
+
+		/*
+		 * After filling all slots in each node type, check if the values
+		 * are stored properly.
+		 */
+		if (ninserted == rt_node_kind_fanouts[node_kind_idx] - 1)
+		{
+			int check_start = insert_asc
+				? rt_node_kind_fanouts[node_kind_idx - 1]
+				: rt_node_kind_fanouts[node_kind_idx];
+			int check_end = insert_asc
+				? rt_node_kind_fanouts[node_kind_idx]
+				: rt_node_kind_fanouts[node_kind_idx - 1];
+
+			check_search_on_node(radixtree, shift, check_start, check_end, incr);
+			node_kind_idx++;
+		}
+
+		ninserted++;
+	}
+
+	num_entries = rt_num_entries(radixtree);
+
+	if (num_entries != 256)
+		elog(ERROR,
+			 "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+			 num_entries, UINT64CONST(256));
+}
+
+static void
+test_node_types_delete(rt_radix_tree *radixtree, uint8 shift)
+{
+	uint64		num_entries;
+
+	for (int i = 0; i < 256; i++)
+	{
+		uint64		key = ((uint64) i << shift);
+		bool		found;
+
+		found = rt_delete(radixtree, key);
+
+		if (!found)
+			elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, key);
+	}
+
+	num_entries = rt_num_entries(radixtree);
+
+	/* The tree must be empty */
+	if (num_entries != 0)
+		elog(ERROR,
+			 "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+			 num_entries, UINT64CONST(256));
+}
+
+/*
+ * Test for inserting and deleting key-value pairs to each node type at the given shift
+ * level.
+ */
+static void
+test_node_types(uint8 shift)
+{
+	rt_radix_tree *radixtree;
+
+#ifdef RT_SHMEM
+	int			tranche_id = LWLockNewTrancheId();
+	dsa_area   *dsa;
+	dsa = dsa_create(tranche_id);
+#endif
+
+	elog(NOTICE, "testing radix tree node types with shift \"%d\"", shift);
+
+#ifdef RT_SHMEM
+	radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+	radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+	/*
+	 * Insert and search entries for every node type at the 'shift' level,
+	 * then delete all entries to make it empty, and insert and search entries
+	 * again.
+	 */
+	test_node_types_insert(radixtree, shift, true);
+	test_node_types_delete(radixtree, shift);
+	test_node_types_insert(radixtree, shift, false);
+
+	rt_free(radixtree);
+#ifdef RT_SHMEM
+	dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Test with a repeating pattern, defined by the 'spec'.
+ */
+static void
+test_pattern(const test_spec * spec)
+{
+	rt_radix_tree *radixtree;
+	rt_iter    *iter;
+	MemoryContext radixtree_ctx;
+	TimestampTz starttime;
+	TimestampTz endtime;
+	uint64		n;
+	uint64		last_int;
+	uint64		ndeleted;
+	uint64		nbefore;
+	uint64		nafter;
+	int			patternlen;
+	uint64	   *pattern_values;
+	uint64		pattern_num_values;
+#ifdef RT_SHMEM
+	int			tranche_id = LWLockNewTrancheId();
+	dsa_area   *dsa;
+	dsa = dsa_create(tranche_id);
+#endif
+
+	elog(NOTICE, "testing radix tree with pattern \"%s\"", spec->test_name);
+	if (rt_test_stats)
+		fprintf(stderr, "-----\ntesting radix tree with pattern \"%s\"\n", spec->test_name);
+
+	/* Pre-process the pattern, creating an array of integers from it. */
+	patternlen = strlen(spec->pattern_str);
+	pattern_values = palloc(patternlen * sizeof(uint64));
+	pattern_num_values = 0;
+	for (int i = 0; i < patternlen; i++)
+	{
+		if (spec->pattern_str[i] == '1')
+			pattern_values[pattern_num_values++] = i;
+	}
+
+	/*
+	 * Allocate the radix tree.
+	 *
+	 * Allocate it in a separate memory context, so that we can print its
+	 * memory usage easily.
+	 */
+	radixtree_ctx = AllocSetContextCreate(CurrentMemoryContext,
+										  "radixtree test",
+										  ALLOCSET_SMALL_SIZES);
+	MemoryContextSetIdentifier(radixtree_ctx, spec->test_name);
+
+#ifdef RT_SHMEM
+	radixtree = rt_create(radixtree_ctx, dsa, tranche_id);
+#else
+	radixtree = rt_create(radixtree_ctx);
+#endif
+
+
+	/*
+	 * Add values to the set.
+	 */
+	starttime = GetCurrentTimestamp();
+
+	n = 0;
+	last_int = 0;
+	while (n < spec->num_values)
+	{
+		uint64		x = 0;
+
+		for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+		{
+			bool		found;
+
+			x = last_int + pattern_values[i];
+
+			found = rt_set(radixtree, x, (TestValueType*) &x);
+
+			if (found)
+				elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " found", x);
+
+			n++;
+		}
+		last_int += spec->spacing;
+	}
+
+	endtime = GetCurrentTimestamp();
+
+	if (rt_test_stats)
+		fprintf(stderr, "added " UINT64_FORMAT " values in %d ms\n",
+				spec->num_values, (int) (endtime - starttime) / 1000);
+
+	/*
+	 * Print stats on the amount of memory used.
+	 *
+	 * We print the usage reported by rt_memory_usage(), as well as the stats
+	 * from the memory context.  They should be in the same ballpark, but it's
+	 * hard to automate testing that, so if you're making changes to the
+	 * implementation, just observe that manually.
+	 */
+	if (rt_test_stats)
+	{
+		uint64		mem_usage;
+
+		/*
+		 * Also print memory usage as reported by rt_memory_usage().  It
+		 * should be in the same ballpark as the usage reported by
+		 * MemoryContextStats().
+		 */
+		mem_usage = rt_memory_usage(radixtree);
+		fprintf(stderr, "rt_memory_usage() reported " UINT64_FORMAT " (%0.2f bytes / integer)\n",
+				mem_usage, (double) mem_usage / spec->num_values);
+
+		MemoryContextStats(radixtree_ctx);
+	}
+
+	/* Check that rt_num_entries works */
+	n = rt_num_entries(radixtree);
+	if (n != spec->num_values)
+		elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT, n, spec->num_values);
+
+	/*
+	 * Test random-access probes with rt_search()
+	 */
+	starttime = GetCurrentTimestamp();
+
+	for (n = 0; n < 100000; n++)
+	{
+		bool		found;
+		bool		expected;
+		uint64		x;
+		TestValueType		v;
+
+		/*
+		 * Pick next value to probe at random.  We limit the probes to the
+		 * last integer that we added to the set, plus an arbitrary constant
+		 * (1000).  There's no point in probing the whole 0 - 2^64 range, if
+		 * only a small part of the integer space is used.  We would very
+		 * rarely hit values that are actually in the set.
+		 */
+		x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+		/* Do we expect this value to be present in the set? */
+		if (x >= last_int)
+			expected = false;
+		else
+		{
+			uint64		idx = x % spec->spacing;
+
+			if (idx >= patternlen)
+				expected = false;
+			else if (spec->pattern_str[idx] == '1')
+				expected = true;
+			else
+				expected = false;
+		}
+
+		/* Is it present according to rt_search() ? */
+		found = rt_search(radixtree, x, &v);
+
+		if (found != expected)
+			elog(ERROR, "mismatch at 0x" UINT64_HEX_FORMAT ": %d vs %d", x, found, expected);
+		if (found && (v != (TestValueType) x))
+			elog(ERROR, "found 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+				 v, x);
+	}
+	endtime = GetCurrentTimestamp();
+	if (rt_test_stats)
+		fprintf(stderr, "probed " UINT64_FORMAT " values in %d ms\n",
+				n, (int) (endtime - starttime) / 1000);
+
+	/*
+	 * Test iterator
+	 */
+	starttime = GetCurrentTimestamp();
+
+	iter = rt_begin_iterate(radixtree);
+	n = 0;
+	last_int = 0;
+	while (n < spec->num_values)
+	{
+		for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+		{
+			uint64		expected = last_int + pattern_values[i];
+			uint64		x;
+			TestValueType		val;
+
+			if (!rt_iterate_next(iter, &x, &val))
+				break;
+
+			if (x != expected)
+				elog(ERROR,
+					 "iterate returned wrong key; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d",
+					 x, expected, i);
+			if (val != (TestValueType) expected)
+				elog(ERROR,
+					 "iterate returned wrong value; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d", x, expected, i);
+			n++;
+		}
+		last_int += spec->spacing;
+	}
+	endtime = GetCurrentTimestamp();
+	if (rt_test_stats)
+		fprintf(stderr, "iterated " UINT64_FORMAT " values in %d ms\n",
+				n, (int) (endtime - starttime) / 1000);
+
+	rt_end_iterate(iter);
+
+	if (n < spec->num_values)
+		elog(ERROR, "iterator stopped short after " UINT64_FORMAT " entries, expected " UINT64_FORMAT, n, spec->num_values);
+	if (n > spec->num_values)
+		elog(ERROR, "iterator returned " UINT64_FORMAT " entries, " UINT64_FORMAT " was expected", n, spec->num_values);
+
+	/*
+	 * Test random-access probes with rt_delete()
+	 */
+	starttime = GetCurrentTimestamp();
+
+	nbefore = rt_num_entries(radixtree);
+	ndeleted = 0;
+	for (n = 0; n < 1; n++)
+	{
+		bool		found;
+		uint64		x;
+		TestValueType		v;
+
+		/*
+		 * Pick next value to probe at random.  We limit the probes to the
+		 * last integer that we added to the set, plus an arbitrary constant
+		 * (1000).  There's no point in probing the whole 0 - 2^64 range, if
+		 * only a small part of the integer space is used.  We would very
+		 * rarely hit values that are actually in the set.
+		 */
+		x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+		/* Is it present according to rt_search() ? */
+		found = rt_search(radixtree, x, &v);
+
+		if (!found)
+			continue;
+
+		/* If the key is found, delete it and check again */
+		if (!rt_delete(radixtree, x))
+			elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, x);
+		if (rt_search(radixtree, x, &v))
+			elog(ERROR, "found deleted key 0x" UINT64_HEX_FORMAT, x);
+		if (rt_delete(radixtree, x))
+			elog(ERROR, "deleted already-deleted key 0x" UINT64_HEX_FORMAT, x);
+
+		ndeleted++;
+	}
+	endtime = GetCurrentTimestamp();
+	if (rt_test_stats)
+		fprintf(stderr, "deleted " UINT64_FORMAT " values in %d ms\n",
+				ndeleted, (int) (endtime - starttime) / 1000);
+
+	nafter = rt_num_entries(radixtree);
+
+	/* Check that rt_num_entries works */
+	if ((nbefore - ndeleted) != nafter)
+		elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT "after " UINT64_FORMAT " deletion",
+			 nafter, (nbefore - ndeleted), ndeleted);
+
+	rt_free(radixtree);
+	MemoryContextDelete(radixtree_ctx);
+#ifdef RT_SHMEM
+	dsa_detach(dsa);
+#endif
+}
+
+Datum
+test_radixtree(PG_FUNCTION_ARGS)
+{
+	test_empty();
+
+	for (int i = 1; i < lengthof(rt_node_kind_fanouts); i++)
+	{
+		test_basic(rt_node_kind_fanouts[i], false);
+		test_basic(rt_node_kind_fanouts[i], true);
+	}
+
+	for (int shift = 0; shift <= (64 - 8); shift += 8)
+		test_node_types(shift);
+
+	/* Test different test patterns, with lots of entries */
+	for (int i = 0; i < lengthof(test_specs); i++)
+		test_pattern(&test_specs[i]);
+
+	PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_radixtree/test_radixtree.control b/src/test/modules/test_radixtree/test_radixtree.control
new file mode 100644
index 0000000000..e53f2a3e0c
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.control
@@ -0,0 +1,4 @@
+comment = 'Test code for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/test_radixtree'
+relocatable = true
diff --git a/src/tools/pginclude/cpluspluscheck b/src/tools/pginclude/cpluspluscheck
index e52fe9f509..09fa6e7432 100755
--- a/src/tools/pginclude/cpluspluscheck
+++ b/src/tools/pginclude/cpluspluscheck
@@ -101,6 +101,12 @@ do
 	test "$f" = src/include/nodes/nodetags.h && continue
 	test "$f" = src/backend/nodes/nodetags.h && continue
 
+	# radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+	test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+	test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+	test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+	test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
 	# These files are not meant to be included standalone, because
 	# they contain lists that might have multiple use-cases.
 	test "$f" = src/include/access/rmgrlist.h && continue
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index abbba7aa63..d4d2f1da03 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -96,6 +96,12 @@ do
 	test "$f" = src/include/nodes/nodetags.h && continue
 	test "$f" = src/backend/nodes/nodetags.h && continue
 
+	# radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+	test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+	test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+	test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+	test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
 	# These files are not meant to be included standalone, because
 	# they contain lists that might have multiple use-cases.
 	test "$f" = src/include/access/rmgrlist.h && continue
-- 
2.39.1

From 77541d3f48e6fef39645df5b3c535ac431e12194 Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Mon, 6 Feb 2023 21:04:14 +0700
Subject: [PATCH v25 6/9] Adjust some inlining declarations

---
 src/include/lib/radixtree.h | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index d6919aef08..4bd0aaa810 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -1124,7 +1124,7 @@ RT_INIT_NODE(RT_PTR_LOCAL node, uint8 kind, RT_SIZE_CLASS size_class, bool is_le
  * Create a new node as the root. Subordinate nodes will be created during
  * the insertion.
  */
-static void
+static pg_noinline void
 RT_NEW_ROOT(RT_RADIX_TREE *tree, uint64 key)
 {
 	int			shift = RT_KEY_GET_SHIFT(key);
@@ -1215,7 +1215,7 @@ RT_NODE_UPDATE_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC new_child)
 /*
  * Replace old_child with new_child, and free the old one.
  */
-static void
+static inline void
 RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
 				RT_PTR_ALLOC stored_old_child, RT_PTR_LOCAL old_child,
 				RT_PTR_ALLOC new_child, uint64 key)
@@ -1242,7 +1242,7 @@ RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
  * The radix tree doesn't have sufficient height. Extend the radix tree so
  * it can store the key.
  */
-static void
+static pg_noinline void
 RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
 {
 	int			target_shift;
@@ -1281,7 +1281,7 @@ RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
  * The radix tree doesn't have inner and leaf nodes for given key-value pair.
  * Insert inner and leaf nodes from 'node' to bottom.
  */
-static inline void
+static pg_noinline void
 RT_SET_EXTEND(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
 			  RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node)
 {
@@ -1486,7 +1486,7 @@ RT_GET_HANDLE(RT_RADIX_TREE *tree)
 /*
  * Recursively free all nodes allocated to the DSA area.
  */
-static inline void
+static void
 RT_FREE_RECURSE(RT_RADIX_TREE *tree, RT_PTR_ALLOC ptr)
 {
 	RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, ptr);
-- 
2.39.1

From f2a3340200ea26c17de5c5261adbeaada64ae4b6 Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Mon, 6 Feb 2023 22:04:50 +0700
Subject: [PATCH v25 7/9] Skip unnecessary searches in RT_NODE_INSERT_INNER

For inner nodes, we know the key chunk doesn't exist already,
otherwise we would have found it while descending the tree.

To reinforce this fact, declare this function to return void.
---
 src/include/lib/radixtree.h             |  4 +--
 src/include/lib/radixtree_insert_impl.h | 48 ++++++++++++-------------
 2 files changed, 24 insertions(+), 28 deletions(-)

diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index 4bd0aaa810..1cdb995e54 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -685,7 +685,7 @@ typedef struct RT_ITER
 } RT_ITER;

-static bool RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+static void RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
 								 uint64 key, RT_PTR_ALLOC child);
 static bool RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
 								uint64 key, RT_VALUE_TYPE *value_p);
@@ -1375,7 +1375,7 @@ RT_NODE_DELETE_LEAF(RT_PTR_LOCAL node, uint64 key)
  * If the node we're inserting into needs to grow, we update the parent's
  * child pointer with the pointer to the new larger node.
  */
-static bool
+static void
 RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
 					uint64 key, RT_PTR_ALLOC child)
 {
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
index c18e26b537..d56e58dcac 100644
--- a/src/include/lib/radixtree_insert_impl.h
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -28,10 +28,10 @@
 #endif
 
 	uint8		chunk = RT_GET_KEY_CHUNK(key, node->shift);
-	bool		chunk_exists = false;
 
 #ifdef RT_NODE_LEVEL_LEAF
 	const bool is_leaf = true;
+	bool		chunk_exists = false;
 	Assert(RT_NODE_IS_LEAF(node));
 #else
 	const bool is_leaf = false;
@@ -43,21 +43,18 @@
 		case RT_NODE_KIND_3:
 			{
 				RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
-				int			idx;
 
-				idx = RT_NODE_3_SEARCH_EQ(&n3->base, chunk);
+#ifdef RT_NODE_LEVEL_LEAF
+				int idx = RT_NODE_3_SEARCH_EQ(&n3->base, chunk);
+
 				if (idx != -1)
 				{
 					/* found the existing chunk */
 					chunk_exists = true;
-#ifdef RT_NODE_LEVEL_LEAF
 					n3->values[idx] = *value_p;
-#else
-					n3->children[idx] = child;
-#endif
 					break;
 				}
-
+#endif
 				if (unlikely(RT_NODE_MUST_GROW(n3)))
 				{
 					RT_PTR_ALLOC allocnode;
@@ -113,21 +110,18 @@
 			{
 				const RT_SIZE_CLASS_ELEM class32_max = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX];
 				RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
-				int			idx;
 
-				idx = RT_NODE_32_SEARCH_EQ(&n32->base, chunk);
+#ifdef RT_NODE_LEVEL_LEAF
+				int idx = RT_NODE_32_SEARCH_EQ(&n32->base, chunk);
+
 				if (idx != -1)
 				{
 					/* found the existing chunk */
 					chunk_exists = true;
-#ifdef RT_NODE_LEVEL_LEAF
 					n32->values[idx] = *value_p;
-#else
-					n32->children[idx] = child;
-#endif
 					break;
 				}
-
+#endif
 				if (unlikely(RT_NODE_MUST_GROW(n32)) &&
 					n32->base.n.fanout < class32_max.fanout)
 				{
@@ -220,21 +214,19 @@
 		case RT_NODE_KIND_125:
 			{
 				RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
-				int			slotpos = n125->base.slot_idxs[chunk];
+				int			slotpos;
 				int			cnt = 0;
 
+#ifdef RT_NODE_LEVEL_LEAF
+				slotpos = n125->base.slot_idxs[chunk];
 				if (slotpos != RT_INVALID_SLOT_IDX)
 				{
 					/* found the existing chunk */
 					chunk_exists = true;
-#ifdef RT_NODE_LEVEL_LEAF
 					n125->values[slotpos] = *value_p;
-#else
-					n125->children[slotpos] = child;
-#endif
 					break;
 				}
-
+#endif
 				if (unlikely(RT_NODE_MUST_GROW(n125)))
 				{
 					RT_PTR_ALLOC allocnode;
@@ -300,14 +292,10 @@
 
 #ifdef RT_NODE_LEVEL_LEAF
 				chunk_exists = RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk);
-#else
-				chunk_exists = RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk);
-#endif
 				Assert(chunk_exists || node->count < RT_NODE_MAX_SLOTS);
-
-#ifdef RT_NODE_LEVEL_LEAF
 				RT_NODE_LEAF_256_SET(n256, chunk, *value_p);
 #else
+				Assert(node->count < RT_NODE_MAX_SLOTS);
 				RT_NODE_INNER_256_SET(n256, chunk, child);
 #endif
 				break;
@@ -315,8 +303,12 @@
 	}
 
 	/* Update statistics */
+#ifdef RT_NODE_LEVEL_LEAF
 	if (!chunk_exists)
 		node->count++;
+#else
+		node->count++;
+#endif
 
 	/*
 	 * Done. Finally, verify the chunk and value is inserted or replaced
@@ -324,7 +316,11 @@
 	 */
 	RT_VERIFY_NODE(node);
 
+#ifdef RT_NODE_LEVEL_LEAF
 	return chunk_exists;
+#else
+	return;
+#endif
 
 #undef RT_NODE3_TYPE
 #undef RT_NODE32_TYPE
-- 
2.39.1

From 693e335f77211e9947cd356d9287c9af96e78815 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Fri, 23 Dec 2022 23:50:39 +0900
Subject: [PATCH v25 8/9] Add TIDStore, to store sets of TIDs (ItemPointerData)
 efficiently.

The TIDStore is designed to store large sets of TIDs efficiently, and
is backed by the radix tree. A TID is encoded into 64-bit key and
value and inserted to the radix tree.

The TIDStore is not used for anything yet, aside from the test code,
but the follow up patch integrates the TIDStore with lazy vacuum,
reducing lazy vacuum memory usage and lifting the 1GB limit on its
size, by storing the list of dead TIDs more efficiently.

This includes a unit test module, in src/test/modules/test_tidstore.
---
 doc/src/sgml/monitoring.sgml                  |   4 +
 src/backend/access/common/Makefile            |   1 +
 src/backend/access/common/meson.build         |   1 +
 src/backend/access/common/tidstore.c          | 688 ++++++++++++++++++
 src/backend/storage/lmgr/lwlock.c             |   2 +
 src/include/access/tidstore.h                 |  49 ++
 src/include/storage/lwlock.h                  |   1 +
 src/test/modules/Makefile                     |   1 +
 src/test/modules/meson.build                  |   1 +
 src/test/modules/test_tidstore/Makefile       |  23 +
 .../test_tidstore/expected/test_tidstore.out  |  13 +
 src/test/modules/test_tidstore/meson.build    |  35 +
 .../test_tidstore/sql/test_tidstore.sql       |   7 +
 .../test_tidstore/test_tidstore--1.0.sql      |   8 +
 .../modules/test_tidstore/test_tidstore.c     | 195 +++++
 .../test_tidstore/test_tidstore.control       |   4 +
 16 files changed, 1033 insertions(+)
 create mode 100644 src/backend/access/common/tidstore.c
 create mode 100644 src/include/access/tidstore.h
 create mode 100644 src/test/modules/test_tidstore/Makefile
 create mode 100644 src/test/modules/test_tidstore/expected/test_tidstore.out
 create mode 100644 src/test/modules/test_tidstore/meson.build
 create mode 100644 src/test/modules/test_tidstore/sql/test_tidstore.sql
 create mode 100644 src/test/modules/test_tidstore/test_tidstore--1.0.sql
 create mode 100644 src/test/modules/test_tidstore/test_tidstore.c
 create mode 100644 src/test/modules/test_tidstore/test_tidstore.control

diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 1756f1a4b6..d936aa3da3 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2192,6 +2192,10 @@ postgres   27093  0.0  0.0  30096  2752 ?        Ss   11:34   0:00 postgres: ser
       <entry>Waiting to access a shared TID bitmap during a parallel bitmap
        index scan.</entry>
      </row>
+     <row>
+      <entry><literal>SharedTidStore</literal></entry>
+      <entry>Waiting to access a shared TID store.</entry>
+     </row>
      <row>
       <entry><literal>SharedTupleStore</literal></entry>
       <entry>Waiting to access a shared tuple store during parallel
diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile
index b9aff0ccfd..67b8cc6108 100644
--- a/src/backend/access/common/Makefile
+++ b/src/backend/access/common/Makefile
@@ -27,6 +27,7 @@ OBJS = \
 	syncscan.o \
 	toast_compression.o \
 	toast_internals.o \
+	tidstore.o \
 	tupconvert.o \
 	tupdesc.o
 
diff --git a/src/backend/access/common/meson.build b/src/backend/access/common/meson.build
index f5ac17b498..fce19c09ce 100644
--- a/src/backend/access/common/meson.build
+++ b/src/backend/access/common/meson.build
@@ -15,6 +15,7 @@ backend_sources += files(
   'syncscan.c',
   'toast_compression.c',
   'toast_internals.c',
+  'tidstore.c',
   'tupconvert.c',
   'tupdesc.c',
 )
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
new file mode 100644
index 0000000000..4c72673ce9
--- /dev/null
+++ b/src/backend/access/common/tidstore.c
@@ -0,0 +1,688 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.c
+ *		Tid (ItemPointerData) storage implementation.
+ *
+ * This module provides a in-memory data structure to store Tids (ItemPointer).
+ * Internally, a tid is encoded as a pair of 64-bit key and 64-bit value, and
+ * stored in the radix tree.
+ *
+ * A TidStore can be shared among parallel worker processes by passing DSA area
+ * to tidstore_create(). Other backends can attach to the shared TidStore by
+ * tidstore_attach().
+ *
+ * Regarding the concurrency, it basically relies on the concurrency support in
+ * the radix tree, but we acquires the lock on a TidStore in some cases, for
+ * example, when to reset the store and when to access the number tids in the
+ * store (num_tids).
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ *	  src/backend/access/common/tidstore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/tidstore.h"
+#include "miscadmin.h"
+#include "port/pg_bitutils.h"
+#include "storage/lwlock.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/*
+ * For encoding purposes, tids are represented as a pair of 64-bit key and
+ * 64-bit value. First, we construct 64-bit unsigned integer by combining
+ * the block number and the offset number. The number of bits used for the
+ * offset number is specified by max_offsets in tidstore_create(). We are
+ * frugal with the bits, because smaller keys could help keeping the radix
+ * tree shallow.
+ *
+ * For example, a tid of heap with 8kB blocks uses the lowest 9 bits for
+ * the offset number and uses the next 32 bits for the block number. That
+ * is, only 41 bits are used:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ *
+ * X = bits used for offset number
+ * Y = bits used for block number
+ * u = unused bit
+ * (high on the left, low on the right)
+ *
+ * 9 bits are enough for the offset number, because MaxHeapTuplesPerPage < 2^9
+ * on 8kB blocks.
+ *
+ * The 64-bit value is the bitmap representation of the lowest 6 bits
+ * (TIDSTORE_VALUE_NBITS) of the integer, and the rest 35 bits are used
+ * as the key:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ *                                                |----| value
+ * |---------------------------------------------| key
+ *
+ * The maximum height of the radix tree is 5 in this case.
+ *
+ * If the number of bits for offset number fits in a 64-bit value, we don't
+ * encode tids but directly use the block number and the offset number as key
+ * and value, respectively.
+ */
+#define TIDSTORE_VALUE_NBITS	6	/* log(64, 2) */
+
+/* A magic value used to identify our TidStores. */
+#define TIDSTORE_MAGIC 0x826f6a10
+
+#define RT_PREFIX local_rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE uint64
+#include "lib/radixtree.h"
+
+#define RT_PREFIX shared_rt
+#define RT_SHMEM
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE uint64
+#include "lib/radixtree.h"
+
+/* The control object for a TidStore */
+typedef struct TidStoreControl
+{
+	/* the number of tids in the store */
+	int64	num_tids;
+
+	/* These values are never changed after creation */
+	size_t	max_bytes;		/* the maximum bytes a TidStore can use */
+	int		max_offset;		/* the maximum offset number */
+	int		offset_nbits;	/* the number of bits required for max_offset */
+	bool	encode_tids;	/* do we use tid encoding? */
+	int		offset_key_nbits;	/* the number of bits of a offset number
+								 * used for the key */
+
+	/* The below fields are used only in shared case */
+
+	uint32	magic;
+	LWLock	lock;
+
+	/* handles for TidStore and radix tree */
+	tidstore_handle		handle;
+	shared_rt_handle	tree_handle;
+} TidStoreControl;
+
+/* Per-backend state for a TidStore */
+struct TidStore
+{
+	/*
+	 * Control object. This is allocated in DSA area 'area' in the shared
+	 * case, otherwise in backend-local memory.
+	 */
+	TidStoreControl *control;
+
+	/* Storage for Tids. Use either one depending on TidStoreIsShared() */
+	union
+	{
+		local_rt_radix_tree *local;
+		shared_rt_radix_tree *shared;
+	} tree;
+
+	/* DSA area for TidStore if used */
+	dsa_area	*area;
+};
+#define TidStoreIsShared(ts) ((ts)->area != NULL)
+
+/* Iterator for TidStore */
+typedef struct TidStoreIter
+{
+	TidStore	*ts;
+
+	/* iterator of radix tree. Use either one depending on TidStoreIsShared() */
+	union
+	{
+		shared_rt_iter	*shared;
+		local_rt_iter	*local;
+	} tree_iter;
+
+	/* we returned all tids? */
+	bool		finished;
+
+	/* save for the next iteration */
+	uint64		next_key;
+	uint64		next_val;
+
+	/* output for the caller */
+	TidStoreIterResult result;
+} TidStoreIter;
+
+static void tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val);
+static inline BlockNumber key_get_blkno(TidStore *ts, uint64 key);
+static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off);
+
+/*
+ * Create a TidStore. The returned object is allocated in backend-local memory.
+ * The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
+ */
+TidStore *
+tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
+{
+	TidStore	*ts;
+
+	ts = palloc0(sizeof(TidStore));
+
+	/*
+	 * Create the radix tree for the main storage.
+	 *
+	 * Memory consumption depends on the number of stored tids, but also on the
+	 * distribution of them, how the radix tree stores, and the memory management
+	 * that backed the radix tree. The maximum bytes that a TidStore can
+	 * use is specified by the max_bytes in tidstore_create(). We want the total
+	 * amount of memory consumption by a TidStore not to exceed the max_bytes.
+	 *
+	 * In local TidStore cases, the radix tree uses slab allocators for each kind
+	 * of node class. The most memory consuming case while adding Tids associated
+	 * with one page (i.e. during tidstore_add_tids()) is that we allocate a new
+	 * slab block for a new radix tree node, which is approximately 70kB. Therefore,
+	 * we deduct 70kB from the max_bytes.
+	 *
+	 * In shared cases, DSA allocates the memory segments big enough to follow
+	 * a geometric series that approximately doubles the total DSA size (see
+	 * make_new_segment() in dsa.c). We simulated the how DSA increases segment
+	 * size and the simulation revealed, the 75% threshold for the maximum bytes
+	 * perfectly works in case where the max_bytes is a power-of-2, and the 60%
+	 * threshold works for other cases.
+	 */
+	if (area != NULL)
+	{
+		dsa_pointer dp;
+		float ratio = ((max_bytes & (max_bytes - 1)) == 0) ? 0.75 : 0.6;
+
+		ts->tree.shared = shared_rt_create(CurrentMemoryContext, area,
+										   LWTRANCHE_SHARED_TIDSTORE);
+
+		dp = dsa_allocate0(area, sizeof(TidStoreControl));
+		ts->control = (TidStoreControl *) dsa_get_address(area, dp);
+		ts->control->max_bytes = (uint64) (max_bytes * ratio);
+		ts->area = area;
+
+		ts->control->magic = TIDSTORE_MAGIC;
+		LWLockInitialize(&ts->control->lock, LWTRANCHE_SHARED_TIDSTORE);
+		ts->control->handle = dp;
+		ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+	}
+	else
+	{
+		ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+		ts->control = (TidStoreControl *) palloc0(sizeof(TidStoreControl));
+		ts->control->max_bytes = max_bytes - (70 * 1024);
+	}
+
+	ts->control->max_offset = max_offset;
+	ts->control->offset_nbits = pg_ceil_log2_32(max_offset);
+
+	/*
+	 * We use tid encoding if the number of bits for the offset number doesn't
+	 * fix in a value, uint64.
+	 */
+	if (ts->control->offset_nbits > TIDSTORE_VALUE_NBITS)
+	{
+		ts->control->encode_tids = true;
+		ts->control->offset_key_nbits =
+			ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
+	}
+	else
+	{
+		ts->control->encode_tids = false;
+		ts->control->offset_key_nbits = 0;
+	}
+
+	return ts;
+}
+
+/*
+ * Attach to the shared TidStore using a handle. The returned object is
+ * allocated in backend-local memory using the CurrentMemoryContext.
+ */
+TidStore *
+tidstore_attach(dsa_area *area, tidstore_handle handle)
+{
+	TidStore *ts;
+	dsa_pointer control;
+
+	Assert(area != NULL);
+	Assert(DsaPointerIsValid(handle));
+
+	/* create per-backend state */
+	ts = palloc0(sizeof(TidStore));
+
+	/* Find the control object in shared memory */
+	control = handle;
+
+	/* Set up the TidStore */
+	ts->control = (TidStoreControl *) dsa_get_address(area, control);
+	Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+	ts->tree.shared = shared_rt_attach(area, ts->control->tree_handle);
+	ts->area = area;
+
+	return ts;
+}
+
+/*
+ * Detach from a TidStore. This detaches from radix tree and frees the
+ * backend-local resources. The radix tree will continue to exist until
+ * it is either explicitly destroyed, or the area that backs it is returned
+ * to the operating system.
+ */
+void
+tidstore_detach(TidStore *ts)
+{
+	Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+	shared_rt_detach(ts->tree.shared);
+	pfree(ts);
+}
+
+/*
+ * Destroy a TidStore, returning all memory.
+ *
+ * TODO: The caller must be certain that no other backend will attempt to
+ * access the TidStore before calling this function. Other backend must
+ * explicitly call tidstore_detach to free up backend-local memory associated
+ * with the TidStore. The backend that calls tidstore_destroy must not call
+ * tidstore_detach.
+ */
+void
+tidstore_destroy(TidStore *ts)
+{
+	if (TidStoreIsShared(ts))
+	{
+		Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+		/*
+		 * Vandalize the control block to help catch programming error where
+		 * other backends access the memory formerly occupied by this radix
+		 * tree.
+		 */
+		ts->control->magic = 0;
+		dsa_free(ts->area, ts->control->handle);
+		shared_rt_free(ts->tree.shared);
+	}
+	else
+	{
+		pfree(ts->control);
+		local_rt_free(ts->tree.local);
+	}
+
+	pfree(ts);
+}
+
+/*
+ * Forget all collected Tids. It's similar to tidstore_destroy but we don't free
+ * entire TidStore but recreate only the radix tree storage.
+ */
+void
+tidstore_reset(TidStore *ts)
+{
+	if (TidStoreIsShared(ts))
+	{
+		Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+		LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+		/*
+		 * Free the radix tree and return allocated DSA segments to
+		 * the operating system.
+		 */
+		shared_rt_free(ts->tree.shared);
+		dsa_trim(ts->area);
+
+		/* Recreate the radix tree */
+		ts->tree.shared = shared_rt_create(CurrentMemoryContext, ts->area,
+										   LWTRANCHE_SHARED_TIDSTORE);
+
+		/* update the radix tree handle as we recreated it */
+		ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+
+		/* Reset the statistics */
+		ts->control->num_tids = 0;
+
+		LWLockRelease(&ts->control->lock);
+	}
+	else
+	{
+		local_rt_free(ts->tree.local);
+		ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+		/* Reset the statistics */
+		ts->control->num_tids = 0;
+	}
+}
+
+/* Add Tids on a block to TidStore */
+void
+tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+				  int num_offsets)
+{
+	ItemPointerData tid;
+	uint64	key_base;
+	uint64	*values;
+	int	nkeys;
+
+	Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+	if (ts->control->encode_tids)
+	{
+		key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
+		nkeys = UINT64CONST(1) << ts->control->offset_key_nbits;
+	}
+	else
+	{
+		key_base = (uint64) blkno;
+		nkeys = 1;
+	}
+	values = palloc0(sizeof(uint64) * nkeys);
+
+	ItemPointerSetBlockNumber(&tid, blkno);
+	for (int i = 0; i < num_offsets; i++)
+	{
+		uint64	key;
+		uint32	off;
+		int idx;
+
+		ItemPointerSetOffsetNumber(&tid, offsets[i]);
+
+		/* encode the tid to key and val */
+		key = tid_to_key_off(ts, &tid, &off);
+
+		idx = key - key_base;
+		Assert(idx >= 0 && idx < nkeys);
+
+		values[idx] |= UINT64CONST(1) << off;
+	}
+
+	if (TidStoreIsShared(ts))
+		LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+	/* insert the calculated key-values to the tree */
+	for (int i = 0; i < nkeys; i++)
+	{
+		if (values[i])
+		{
+			uint64 key = key_base + i;
+
+			if (TidStoreIsShared(ts))
+				shared_rt_set(ts->tree.shared, key, &values[i]);
+			else
+				local_rt_set(ts->tree.local, key, &values[i]);
+		}
+	}
+
+	/* update statistics */
+	ts->control->num_tids += num_offsets;
+
+	if (TidStoreIsShared(ts))
+		LWLockRelease(&ts->control->lock);
+
+	pfree(values);
+}
+
+/* Return true if the given tid is present in the TidStore */
+bool
+tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
+{
+	uint64 key;
+	uint64 val = 0;
+	uint32 off;
+	bool found;
+
+	key = tid_to_key_off(ts, tid, &off);
+
+	if (TidStoreIsShared(ts))
+		found = shared_rt_search(ts->tree.shared, key, &val);
+	else
+		found = local_rt_search(ts->tree.local, key, &val);
+
+	if (!found)
+		return false;
+
+	return (val & (UINT64CONST(1) << off)) != 0;
+}
+
+/*
+ * Prepare to iterate through a TidStore. Since the radix tree is locked during the
+ * iteration, so tidstore_end_iterate() needs to called when finished.
+ *
+ * Concurrent updates during the iteration will be blocked when inserting a
+ * key-value to the radix tree.
+ */
+TidStoreIter *
+tidstore_begin_iterate(TidStore *ts)
+{
+	TidStoreIter *iter;
+
+	Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+	iter = palloc0(sizeof(TidStoreIter));
+	iter->ts = ts;
+
+	iter->result.blkno = InvalidBlockNumber;
+	iter->result.offsets = palloc(sizeof(OffsetNumber) * ts->control->max_offset);
+
+	if (TidStoreIsShared(ts))
+		iter->tree_iter.shared = shared_rt_begin_iterate(ts->tree.shared);
+	else
+		iter->tree_iter.local = local_rt_begin_iterate(ts->tree.local);
+
+	/* If the TidStore is empty, there is no business */
+	if (tidstore_num_tids(ts) == 0)
+		iter->finished = true;
+
+	return iter;
+}
+
+static inline bool
+tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
+{
+	if (TidStoreIsShared(iter->ts))
+		return shared_rt_iterate_next(iter->tree_iter.shared, key, val);
+
+	return local_rt_iterate_next(iter->tree_iter.local, key, val);
+}
+
+/*
+ * Scan the TidStore and return a pointer to TidStoreIterResult that has tids
+ * in one block. We return the block numbers in ascending order and the offset
+ * numbers in each result is also sorted in ascending order.
+ */
+TidStoreIterResult *
+tidstore_iterate_next(TidStoreIter *iter)
+{
+	uint64 key;
+	uint64 val;
+	TidStoreIterResult *result = &(iter->result);
+
+	if (iter->finished)
+		return NULL;
+
+	if (BlockNumberIsValid(result->blkno))
+	{
+		/* Process the previously collected key-value */
+		result->num_offsets = 0;
+		tidstore_iter_extract_tids(iter, iter->next_key, iter->next_val);
+	}
+
+	while (tidstore_iter_kv(iter, &key, &val))
+	{
+		BlockNumber blkno;
+
+		blkno = key_get_blkno(iter->ts, key);
+
+		if (BlockNumberIsValid(result->blkno) && result->blkno != blkno)
+		{
+			/*
+			 * We got a key-value pair for a different block. So return the
+			 * collected tids, and remember the key-value for the next iteration.
+			 */
+			iter->next_key = key;
+			iter->next_val = val;
+			return result;
+		}
+
+		/* Collect tids extracted from the key-value pair */
+		tidstore_iter_extract_tids(iter, key, val);
+	}
+
+	iter->finished = true;
+	return result;
+}
+
+/*
+ * Finish an iteration over TidStore. This needs to be called after finishing
+ * or when existing an iteration.
+ */
+void
+tidstore_end_iterate(TidStoreIter *iter)
+{
+	if (TidStoreIsShared(iter->ts))
+		shared_rt_end_iterate(iter->tree_iter.shared);
+	else
+		local_rt_end_iterate(iter->tree_iter.local);
+
+	pfree(iter->result.offsets);
+	pfree(iter);
+}
+
+/* Return the number of tids we collected so far */
+int64
+tidstore_num_tids(TidStore *ts)
+{
+	uint64 num_tids;
+
+	Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+	if (!TidStoreIsShared(ts))
+		return ts->control->num_tids;
+
+	LWLockAcquire(&ts->control->lock, LW_SHARED);
+	num_tids = ts->control->num_tids;
+	LWLockRelease(&ts->control->lock);
+
+	return num_tids;
+}
+
+/* Return true if the current memory usage of TidStore exceeds the limit */
+bool
+tidstore_is_full(TidStore *ts)
+{
+	Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+	return (tidstore_memory_usage(ts) > ts->control->max_bytes);
+}
+
+/* Return the maximum memory TidStore can use */
+size_t
+tidstore_max_memory(TidStore *ts)
+{
+	Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+	return ts->control->max_bytes;
+}
+
+/* Return the memory usage of TidStore */
+size_t
+tidstore_memory_usage(TidStore *ts)
+{
+	Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+	/*
+	 * In the shared case, TidStoreControl and radix_tree are backed by the
+	 * same DSA area and rt_memory_usage() returns the value including both.
+	 * So we don't need to add the size of TidStoreControl separately.
+	 */
+	if (TidStoreIsShared(ts))
+		return sizeof(TidStore) + shared_rt_memory_usage(ts->tree.shared);
+
+	return sizeof(TidStore) + sizeof(TidStore) + local_rt_memory_usage(ts->tree.local);
+}
+
+/*
+ * Get a handle that can be used by other processes to attach to this TidStore
+ */
+tidstore_handle
+tidstore_get_handle(TidStore *ts)
+{
+	Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+	return ts->control->handle;
+}
+
+/* Extract tids from the given key-value pair */
+static void
+tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
+{
+	TidStoreIterResult *result = (&iter->result);
+
+	for (int i = 0; i < sizeof(uint64) * BITS_PER_BYTE; i++)
+	{
+		uint64	tid_i;
+		OffsetNumber	off;
+
+		if (i > iter->ts->control->max_offset)
+		{
+			Assert(!iter->ts->control->encode_tids);
+			break;
+		}
+
+		if ((val & (UINT64CONST(1) << i)) == 0)
+			continue;
+
+		tid_i = key << TIDSTORE_VALUE_NBITS;
+		tid_i |= i;
+
+		off = tid_i & ((UINT64CONST(1) << iter->ts->control->offset_nbits) - 1);
+
+		Assert(result->num_offsets < iter->ts->control->max_offset);
+		result->offsets[result->num_offsets++] = off;
+	}
+
+	result->blkno = key_get_blkno(iter->ts, key);
+}
+
+/* Get block number from the given key */
+static inline BlockNumber
+key_get_blkno(TidStore *ts, uint64 key)
+{
+	if (ts->control->encode_tids)
+		return (BlockNumber) (key >> ts->control->offset_key_nbits);
+
+	return (BlockNumber) key;
+}
+
+/* Encode a tid to key and offset */
+static inline uint64
+tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off)
+{
+	uint64 key;
+	uint64 tid_i;
+
+	if (!ts->control->encode_tids)
+	{
+		*off = ItemPointerGetOffsetNumber(tid);
+
+		/* Use the block number as the key */
+		return (int64) ItemPointerGetBlockNumber(tid);
+	}
+
+	tid_i = ItemPointerGetOffsetNumber(tid);
+	tid_i |= (uint64) ItemPointerGetBlockNumber(tid) << ts->control->offset_nbits;
+
+	*off = tid_i & ((UINT64CONST(1) << TIDSTORE_VALUE_NBITS) - 1);
+	key = tid_i >> TIDSTORE_VALUE_NBITS;
+	Assert(*off < (sizeof(uint64) * BITS_PER_BYTE));
+
+	return key;
+}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index d2ec396045..55b3a04097 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -176,6 +176,8 @@ static const char *const BuiltinTrancheNames[] = {
 	"SharedTupleStore",
 	/* LWTRANCHE_SHARED_TIDBITMAP: */
 	"SharedTidBitmap",
+	/* LWTRANCHE_SHARED_TIDSTORE: */
+	"SharedTidStore",
 	/* LWTRANCHE_PARALLEL_APPEND: */
 	"ParallelAppend",
 	/* LWTRANCHE_PER_XACT_PREDICATE_LIST: */
diff --git a/src/include/access/tidstore.h b/src/include/access/tidstore.h
new file mode 100644
index 0000000000..a35a52124a
--- /dev/null
+++ b/src/include/access/tidstore.h
@@ -0,0 +1,49 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.h
+ *	  Tid storage.
+ *
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/tidstore.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef TIDSTORE_H
+#define TIDSTORE_H
+
+#include "storage/itemptr.h"
+#include "utils/dsa.h"
+
+typedef dsa_pointer tidstore_handle;
+
+typedef struct TidStore TidStore;
+typedef struct TidStoreIter TidStoreIter;
+
+typedef struct TidStoreIterResult
+{
+	BlockNumber		blkno;
+	OffsetNumber	*offsets;
+	int				num_offsets;
+} TidStoreIterResult;
+
+extern TidStore *tidstore_create(size_t max_bytes, int max_offset, dsa_area *dsa);
+extern TidStore *tidstore_attach(dsa_area *dsa, dsa_pointer handle);
+extern void tidstore_detach(TidStore *ts);
+extern void tidstore_destroy(TidStore *ts);
+extern void tidstore_reset(TidStore *ts);
+extern void tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+							  int num_offsets);
+extern bool tidstore_lookup_tid(TidStore *ts, ItemPointer tid);
+extern TidStoreIter * tidstore_begin_iterate(TidStore *ts);
+extern TidStoreIterResult *tidstore_iterate_next(TidStoreIter *iter);
+extern void tidstore_end_iterate(TidStoreIter *iter);
+extern int64 tidstore_num_tids(TidStore *ts);
+extern bool tidstore_is_full(TidStore *ts);
+extern size_t tidstore_max_memory(TidStore *ts);
+extern size_t tidstore_memory_usage(TidStore *ts);
+extern tidstore_handle tidstore_get_handle(TidStore *ts);
+
+#endif		/* TIDSTORE_H */
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index d2c7afb8f4..07002fdfbe 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -199,6 +199,7 @@ typedef enum BuiltinTrancheIds
 	LWTRANCHE_PER_SESSION_RECORD_TYPMOD,
 	LWTRANCHE_SHARED_TUPLESTORE,
 	LWTRANCHE_SHARED_TIDBITMAP,
+	LWTRANCHE_SHARED_TIDSTORE,
 	LWTRANCHE_PARALLEL_APPEND,
 	LWTRANCHE_PER_XACT_PREDICATE_LIST,
 	LWTRANCHE_PGSTATS_DSA,
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 9659eb85d7..bddc16ada7 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
 		  test_rls_hooks \
 		  test_shm_mq \
 		  test_slru \
+		  test_tidstore \
 		  unsafe_tests \
 		  worker_spi
 
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index 232cbdac80..c0d5645ad8 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -30,5 +30,6 @@ subdir('test_regex')
 subdir('test_rls_hooks')
 subdir('test_shm_mq')
 subdir('test_slru')
+subdir('test_tidstore')
 subdir('unsafe_tests')
 subdir('worker_spi')
diff --git a/src/test/modules/test_tidstore/Makefile b/src/test/modules/test_tidstore/Makefile
new file mode 100644
index 0000000000..dab107d70c
--- /dev/null
+++ b/src/test/modules/test_tidstore/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_tidstore/Makefile
+
+MODULE_big = test_tidstore
+OBJS = \
+	$(WIN32RES) \
+	test_tidstore.o
+PGFILEDESC = "test_tidstore - test code for src/backend/access/common/tidstore.c"
+
+EXTENSION = test_tidstore
+DATA = test_tidstore--1.0.sql
+
+REGRESS = test_tidstore
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_tidstore
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_tidstore/expected/test_tidstore.out b/src/test/modules/test_tidstore/expected/test_tidstore.out
new file mode 100644
index 0000000000..7ff2f9af87
--- /dev/null
+++ b/src/test/modules/test_tidstore/expected/test_tidstore.out
@@ -0,0 +1,13 @@
+CREATE EXTENSION test_tidstore;
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
+NOTICE:  testing empty tidstore
+NOTICE:  testing basic operations
+ test_tidstore 
+---------------
+ 
+(1 row)
+
diff --git a/src/test/modules/test_tidstore/meson.build b/src/test/modules/test_tidstore/meson.build
new file mode 100644
index 0000000000..31f2da7b61
--- /dev/null
+++ b/src/test/modules/test_tidstore/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_tidstore_sources = files(
+  'test_tidstore.c',
+)
+
+if host_system == 'windows'
+  test_tidstore_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+    '--NAME', 'test_tidstore',
+    '--FILEDESC', 'test_tidstore - test code for src/backend/access/common/tidstore.c',])
+endif
+
+test_tidstore = shared_module('test_tidstore',
+  test_tidstore_sources,
+  link_with: pgport_srv,
+  kwargs: pg_mod_args,
+)
+testprep_targets += test_tidstore
+
+install_data(
+  'test_tidstore.control',
+  'test_tidstore--1.0.sql',
+  kwargs: contrib_data_args,
+)
+
+tests += {
+  'name': 'test_tidstore',
+  'sd': meson.current_source_dir(),
+  'bd': meson.current_build_dir(),
+  'regress': {
+    'sql': [
+      'test_tidstore',
+    ],
+  },
+}
diff --git a/src/test/modules/test_tidstore/sql/test_tidstore.sql b/src/test/modules/test_tidstore/sql/test_tidstore.sql
new file mode 100644
index 0000000000..03aea31815
--- /dev/null
+++ b/src/test/modules/test_tidstore/sql/test_tidstore.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_tidstore;
+
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
diff --git a/src/test/modules/test_tidstore/test_tidstore--1.0.sql b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
new file mode 100644
index 0000000000..47e9149900
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_tidstore/test_tidstore--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_tidstore" to load this file. \quit
+
+CREATE FUNCTION test_tidstore()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_tidstore/test_tidstore.c b/src/test/modules/test_tidstore/test_tidstore.c
new file mode 100644
index 0000000000..9b849ae8e8
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.c
@@ -0,0 +1,195 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_tidstore.c
+ *		Test TidStore data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ *		src/test/modules/test_tidstore/test_tidstore.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/tidstore.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "utils/memutils.h"
+
+PG_MODULE_MAGIC;
+
+#define TEST_TIDSTORE_MAX_BYTES (2 * 1024 * 1024L) /* 2MB */
+
+PG_FUNCTION_INFO_V1(test_tidstore);
+
+static void
+check_tid(TidStore *ts, BlockNumber blkno, OffsetNumber off, bool expect)
+{
+	ItemPointerData tid;
+	bool found;
+
+	ItemPointerSet(&tid, blkno, off);
+
+	found = tidstore_lookup_tid(ts, &tid);
+
+	if (found != expect)
+		elog(ERROR, "lookup TID (%u, %u) returned %d, expected %d",
+			 blkno, off, found, expect);
+}
+
+static void
+test_basic(int max_offset)
+{
+#define TEST_TIDSTORE_NUM_BLOCKS	5
+#define TEST_TIDSTORE_NUM_OFFSETS	5
+
+	TidStore *ts;
+	TidStoreIter *iter;
+	TidStoreIterResult *iter_result;
+	BlockNumber	blks[TEST_TIDSTORE_NUM_BLOCKS] = {
+		0, MaxBlockNumber, MaxBlockNumber - 1, 1, MaxBlockNumber / 2,
+	};
+	BlockNumber	blks_sorted[TEST_TIDSTORE_NUM_BLOCKS] = {
+		0, 1, MaxBlockNumber / 2, MaxBlockNumber - 1, MaxBlockNumber
+	};
+	OffsetNumber offs[TEST_TIDSTORE_NUM_OFFSETS];
+	int blk_idx;
+
+	/* prepare the offset array */
+	offs[0] = FirstOffsetNumber;
+	offs[1] = FirstOffsetNumber + 1;
+	offs[2] = max_offset / 2;
+	offs[3] = max_offset - 1;
+	offs[4] = max_offset;
+
+	ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
+
+	/* add tids */
+	for (int i = 0; i < TEST_TIDSTORE_NUM_BLOCKS; i++)
+		tidstore_add_tids(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
+
+	/* lookup test */
+	for (OffsetNumber off = FirstOffsetNumber ; off < max_offset; off++)
+	{
+		bool expect = false;
+		for (int i = 0; i < TEST_TIDSTORE_NUM_OFFSETS; i++)
+		{
+			if (offs[i] == off)
+			{
+				expect = true;
+				break;
+			}
+		}
+
+		check_tid(ts, 0, off, expect);
+		check_tid(ts, 2, off, false);
+		check_tid(ts, MaxBlockNumber - 2, off, false);
+		check_tid(ts, MaxBlockNumber, off, expect);
+	}
+
+	/* test the number of tids */
+	if (tidstore_num_tids(ts) != (TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS))
+		elog(ERROR, "tidstore_num_tids returned " UINT64_FORMAT ", expected %d",
+			 tidstore_num_tids(ts),
+			 TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS);
+
+	/* iteration test */
+	iter = tidstore_begin_iterate(ts);
+	blk_idx = 0;
+	while ((iter_result = tidstore_iterate_next(iter)) != NULL)
+	{
+		/* check the returned block number */
+		if (blks_sorted[blk_idx] != iter_result->blkno)
+			elog(ERROR, "tidstore_iterate_next returned block number %u, expected %u",
+				 iter_result->blkno, blks_sorted[blk_idx]);
+
+		/* check the returned offset numbers */
+		if (TEST_TIDSTORE_NUM_OFFSETS != iter_result->num_offsets)
+			elog(ERROR, "tidstore_iterate_next returned %u offsets, expected %u",
+				 iter_result->num_offsets, TEST_TIDSTORE_NUM_OFFSETS);
+
+		for (int i = 0; i < iter_result->num_offsets; i++)
+		{
+			if (offs[i] != iter_result->offsets[i])
+				elog(ERROR, "tidstore_iterate_next returned offset number %u on block %u, expected %u",
+					 iter_result->offsets[i], iter_result->blkno, offs[i]);
+		}
+
+		blk_idx++;
+	}
+
+	if (blk_idx != TEST_TIDSTORE_NUM_BLOCKS)
+		elog(ERROR, "tidstore_iterate_next returned %d blocks, expected %d",
+			 blk_idx, TEST_TIDSTORE_NUM_BLOCKS);
+
+	/* remove all tids */
+	tidstore_reset(ts);
+
+	/* test the number of tids */
+	if (tidstore_num_tids(ts) != 0)
+		elog(ERROR, "tidstore_num_tids on empty store returned non-zero");
+
+	/* lookup test for empty store */
+	for (OffsetNumber off = FirstOffsetNumber ; off < MaxHeapTuplesPerPage;
+		 off++)
+	{
+		check_tid(ts, 0, off, false);
+		check_tid(ts, 2, off, false);
+		check_tid(ts, MaxBlockNumber - 2, off, false);
+		check_tid(ts, MaxBlockNumber, off, false);
+	}
+
+	tidstore_destroy(ts);
+}
+
+static void
+test_empty(void)
+{
+	TidStore *ts;
+	TidStoreIter *iter;
+	ItemPointerData tid;
+
+	elog(NOTICE, "testing empty tidstore");
+
+	ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
+
+	ItemPointerSet(&tid, 0, FirstOffsetNumber);
+	if (tidstore_lookup_tid(ts, &tid))
+		elog(ERROR, "tidstore_lookup_tid for (0,1) on empty store returned true");
+
+	ItemPointerSet(&tid, MaxBlockNumber, MaxOffsetNumber);
+	if (tidstore_lookup_tid(ts, &tid))
+		elog(ERROR, "tidstore_lookup_tid for (%u,%u) on empty store returned true",
+			 MaxBlockNumber, MaxOffsetNumber);
+
+	if (tidstore_num_tids(ts) != 0)
+		elog(ERROR, "tidstore_num_entries on empty store returned non-zero");
+
+	if (tidstore_is_full(ts))
+		elog(ERROR, "tidstore_is_full on empty store returned true");
+
+	iter = tidstore_begin_iterate(ts);
+
+	if (tidstore_iterate_next(iter) != NULL)
+		elog(ERROR, "tidstore_iterate_next on empty store returned TIDs");
+
+	tidstore_end_iterate(iter);
+
+	tidstore_destroy(ts);
+}
+
+Datum
+test_tidstore(PG_FUNCTION_ARGS)
+{
+	test_empty();
+
+	elog(NOTICE, "testing basic operations");
+	test_basic(MaxHeapTuplesPerPage);
+	test_basic(10);
+
+	PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_tidstore/test_tidstore.control b/src/test/modules/test_tidstore/test_tidstore.control
new file mode 100644
index 0000000000..9b6bd4638f
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.control
@@ -0,0 +1,4 @@
+comment = 'Test code for tidstore'
+default_version = '1.0'
+module_pathname = '$libdir/test_tidstore'
+relocatable = true
-- 
2.39.1

From dcbcf6cdd786f9debf1536ac73093107debfafe8 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Tue, 17 Jan 2023 17:20:37 +0700
Subject: [PATCH v25 9/9] Use TIDStore for storing dead tuple TID during lazy
 vacuum

Previously, we used an array of ItemPointerData to store dead tuple
TIDs, which was not space efficient and slow to lookup. Also, we had
the 1GB limit on its size.

Now we use TIDStore to store dead tuple TIDs. Since the TIDStore,
backed by the radix tree, incrementaly allocates the memory, we get
rid of the 1GB limit.

Since we are no longer able to exactly estimate the maximum number of
TIDs can be stored the pg_stat_progress_vacuum shows the progress
information based on the amount of memory in bytes. The column names
are also changed to max_dead_tuple_bytes and num_dead_tuple_bytes.

In addition, since the TIDStore use the radix tree internally, the
minimum amount of memory required by TIDStore is 1MB, the inital DSA
segment size. Due to that, we increase the minimum value of
maintenance_work_mem (also autovacuum_work_mem) from 1MB to 2MB.

XXX: needs to bump catalog version
---
 doc/src/sgml/monitoring.sgml               |   8 +-
 src/backend/access/heap/vacuumlazy.c       | 278 ++++++++-------------
 src/backend/catalog/system_views.sql       |   2 +-
 src/backend/commands/vacuum.c              |  78 +-----
 src/backend/commands/vacuumparallel.c      |  73 +++---
 src/backend/postmaster/autovacuum.c        |   6 +-
 src/backend/storage/lmgr/lwlock.c          |   2 +
 src/backend/utils/misc/guc_tables.c        |   2 +-
 src/include/commands/progress.h            |   4 +-
 src/include/commands/vacuum.h              |  25 +-
 src/include/storage/lwlock.h               |   1 +
 src/test/regress/expected/cluster.out      |   2 +-
 src/test/regress/expected/create_index.out |   2 +-
 src/test/regress/expected/rules.out        |   4 +-
 src/test/regress/sql/cluster.sql           |   2 +-
 src/test/regress/sql/create_index.sql      |   2 +-
 16 files changed, 177 insertions(+), 314 deletions(-)

diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index d936aa3da3..0230c74e3d 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -6870,10 +6870,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
 
      <row>
       <entry role="catalog_table_entry"><para role="column_definition">
-       <structfield>max_dead_tuples</structfield> <type>bigint</type>
+       <structfield>max_dead_tuple_bytes</structfield> <type>bigint</type>
       </para>
       <para>
-       Number of dead tuples that we can store before needing to perform
+       Amount of dead tuple data that we can store before needing to perform
        an index vacuum cycle, based on
        <xref linkend="guc-maintenance-work-mem"/>.
       </para></entry>
@@ -6881,10 +6881,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
 
      <row>
       <entry role="catalog_table_entry"><para role="column_definition">
-       <structfield>num_dead_tuples</structfield> <type>bigint</type>
+       <structfield>num_dead_tuple_bytes</structfield> <type>bigint</type>
       </para>
       <para>
-       Number of dead tuples collected since the last index vacuum cycle.
+       Amount of dead tuple data collected since the last index vacuum cycle.
       </para></entry>
      </row>
     </tbody>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 8f14cf85f3..b4e40423a8 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3,18 +3,18 @@
  * vacuumlazy.c
  *	  Concurrent ("lazy") vacuuming.
  *
- * The major space usage for vacuuming is storage for the array of dead TIDs
+ * The major space usage for vacuuming is TidStore, a storage for dead TIDs
  * that are to be removed from indexes.  We want to ensure we can vacuum even
  * the very largest relations with finite memory space usage.  To do that, we
- * set upper bounds on the number of TIDs we can keep track of at once.
+ * set upper bounds on the maximum memory that can be used for keeping track
+ * of dead TIDs at once.
  *
  * We are willing to use at most maintenance_work_mem (or perhaps
  * autovacuum_work_mem) memory space to keep track of dead TIDs.  We initially
- * allocate an array of TIDs of that size, with an upper limit that depends on
- * table size (this limit ensures we don't allocate a huge area uselessly for
- * vacuuming small tables).  If the array threatens to overflow, we must call
- * lazy_vacuum to vacuum indexes (and to vacuum the pages that we've pruned).
- * This frees up the memory space dedicated to storing dead TIDs.
+ * create a TidStore with the maximum bytes that can be used by the TidStore.
+ * If the TidStore is full, we must call lazy_vacuum to vacuum indexes (and to
+ * vacuum the pages that we've pruned). This frees up the memory space dedicated
+ * to storing dead TIDs.
  *
  * In practice VACUUM will often complete its initial pass over the target
  * heap relation without ever running out of space to store TIDs.  This means
@@ -40,6 +40,7 @@
 #include "access/heapam_xlog.h"
 #include "access/htup_details.h"
 #include "access/multixact.h"
+#include "access/tidstore.h"
 #include "access/transam.h"
 #include "access/visibilitymap.h"
 #include "access/xact.h"
@@ -188,7 +189,7 @@ typedef struct LVRelState
 	 * lazy_vacuum_heap_rel, which marks the same LP_DEAD line pointers as
 	 * LP_UNUSED during second heap pass.
 	 */
-	VacDeadItems *dead_items;	/* TIDs whose index tuples we'll delete */
+	TidStore *dead_items;	/* TIDs whose index tuples we'll delete */
 	BlockNumber rel_pages;		/* total number of pages */
 	BlockNumber scanned_pages;	/* # pages examined (not skipped via VM) */
 	BlockNumber removed_pages;	/* # pages removed by relation truncation */
@@ -220,11 +221,14 @@ typedef struct LVRelState
 typedef struct LVPagePruneState
 {
 	bool		hastup;			/* Page prevents rel truncation? */
-	bool		has_lpdead_items;	/* includes existing LP_DEAD items */
+
+	/* collected offsets of LP_DEAD items including existing ones */
+	OffsetNumber	deadoffsets[MaxHeapTuplesPerPage];
+	int				num_offsets;
 
 	/*
 	 * State describes the proper VM bit states to set for the page following
-	 * pruning and freezing.  all_visible implies !has_lpdead_items, but don't
+	 * pruning and freezing.  all_visible implies num_offsets == 0, but don't
 	 * trust all_frozen result unless all_visible is also set to true.
 	 */
 	bool		all_visible;	/* Every item visible to all? */
@@ -259,8 +263,9 @@ static bool lazy_scan_noprune(LVRelState *vacrel, Buffer buf,
 static void lazy_vacuum(LVRelState *vacrel);
 static bool lazy_vacuum_all_indexes(LVRelState *vacrel);
 static void lazy_vacuum_heap_rel(LVRelState *vacrel);
-static int	lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
-								  Buffer buffer, int index, Buffer vmbuffer);
+static void	lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
+								  OffsetNumber *offsets, int num_offsets,
+								  Buffer buffer, Buffer vmbuffer);
 static bool lazy_check_wraparound_failsafe(LVRelState *vacrel);
 static void lazy_cleanup_all_indexes(LVRelState *vacrel);
 static IndexBulkDeleteResult *lazy_vacuum_one_index(Relation indrel,
@@ -487,11 +492,11 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
 	}
 
 	/*
-	 * Allocate dead_items array memory using dead_items_alloc.  This handles
-	 * parallel VACUUM initialization as part of allocating shared memory
-	 * space used for dead_items.  (But do a failsafe precheck first, to
-	 * ensure that parallel VACUUM won't be attempted at all when relfrozenxid
-	 * is already dangerously old.)
+	 * Allocate dead_items memory using dead_items_alloc.  This handles parallel
+	 * VACUUM initialization as part of allocating shared memory space used for
+	 * dead_items.  (But do a failsafe precheck first, to ensure that parallel
+	 * VACUUM won't be attempted at all when relfrozenxid is already dangerously
+	 * old.)
 	 */
 	lazy_check_wraparound_failsafe(vacrel);
 	dead_items_alloc(vacrel, params->nworkers);
@@ -797,7 +802,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
  *		have collected the TIDs whose index tuples need to be removed.
  *
  *		Finally, invokes lazy_vacuum_heap_rel to vacuum heap pages, which
- *		largely consists of marking LP_DEAD items (from collected TID array)
+ *		largely consists of marking LP_DEAD items (from vacrel->dead_items)
  *		as LP_UNUSED.  This has to happen in a second, final pass over the
  *		heap, to preserve a basic invariant that all index AMs rely on: no
  *		extant index tuple can ever be allowed to contain a TID that points to
@@ -825,21 +830,21 @@ lazy_scan_heap(LVRelState *vacrel)
 				blkno,
 				next_unskippable_block,
 				next_fsm_block_to_vacuum = 0;
-	VacDeadItems *dead_items = vacrel->dead_items;
+	TidStore *dead_items = vacrel->dead_items;
 	Buffer		vmbuffer = InvalidBuffer;
 	bool		next_unskippable_allvis,
 				skipping_current_range;
 	const int	initprog_index[] = {
 		PROGRESS_VACUUM_PHASE,
 		PROGRESS_VACUUM_TOTAL_HEAP_BLKS,
-		PROGRESS_VACUUM_MAX_DEAD_TUPLES
+		PROGRESS_VACUUM_MAX_DEAD_TUPLE_BYTES
 	};
 	int64		initprog_val[3];
 
 	/* Report that we're scanning the heap, advertising total # of blocks */
 	initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
 	initprog_val[1] = rel_pages;
-	initprog_val[2] = dead_items->max_items;
+	initprog_val[2] = tidstore_max_memory(vacrel->dead_items);
 	pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
 
 	/* Set up an initial range of skippable blocks using the visibility map */
@@ -906,8 +911,7 @@ lazy_scan_heap(LVRelState *vacrel)
 		 * dead_items TIDs, pause and do a cycle of vacuuming before we tackle
 		 * this page.
 		 */
-		Assert(dead_items->max_items >= MaxHeapTuplesPerPage);
-		if (dead_items->max_items - dead_items->num_items < MaxHeapTuplesPerPage)
+		if (tidstore_is_full(vacrel->dead_items))
 		{
 			/*
 			 * Before beginning index vacuuming, we release any pin we may
@@ -969,7 +973,7 @@ lazy_scan_heap(LVRelState *vacrel)
 				continue;
 			}
 
-			/* Collect LP_DEAD items in dead_items array, count tuples */
+			/* Collect LP_DEAD items in dead_items, count tuples */
 			if (lazy_scan_noprune(vacrel, buf, blkno, page, &hastup,
 								  &recordfreespace))
 			{
@@ -1011,14 +1015,14 @@ lazy_scan_heap(LVRelState *vacrel)
 		 * Prune, freeze, and count tuples.
 		 *
 		 * Accumulates details of remaining LP_DEAD line pointers on page in
-		 * dead_items array.  This includes LP_DEAD line pointers that we
-		 * pruned ourselves, as well as existing LP_DEAD line pointers that
-		 * were pruned some time earlier.  Also considers freezing XIDs in the
-		 * tuple headers of remaining items with storage.
+		 * dead_items.  This includes LP_DEAD line pointers that we pruned
+		 * ourselves, as well as existing LP_DEAD line pointers that were pruned
+		 * some time earlier.  Also considers freezing XIDs in the tuple headers
+		 * of remaining items with storage.
 		 */
 		lazy_scan_prune(vacrel, buf, blkno, page, &prunestate);
 
-		Assert(!prunestate.all_visible || !prunestate.has_lpdead_items);
+		Assert(!prunestate.all_visible || (prunestate.num_offsets == 0));
 
 		/* Remember the location of the last page with nonremovable tuples */
 		if (prunestate.hastup)
@@ -1034,14 +1038,12 @@ lazy_scan_heap(LVRelState *vacrel)
 			 * performed here can be thought of as the one-pass equivalent of
 			 * a call to lazy_vacuum().
 			 */
-			if (prunestate.has_lpdead_items)
+			if (prunestate.num_offsets > 0)
 			{
 				Size		freespace;
 
-				lazy_vacuum_heap_page(vacrel, blkno, buf, 0, vmbuffer);
-
-				/* Forget the LP_DEAD items that we just vacuumed */
-				dead_items->num_items = 0;
+				lazy_vacuum_heap_page(vacrel, blkno, prunestate.deadoffsets,
+									  prunestate.num_offsets, buf, vmbuffer);
 
 				/*
 				 * Periodically perform FSM vacuuming to make newly-freed
@@ -1078,7 +1080,16 @@ lazy_scan_heap(LVRelState *vacrel)
 			 * with prunestate-driven visibility map and FSM steps (just like
 			 * the two-pass strategy).
 			 */
-			Assert(dead_items->num_items == 0);
+			Assert(tidstore_num_tids(dead_items) == 0);
+		}
+		else if (prunestate.num_offsets > 0)
+		{
+			/* Save details of the LP_DEAD items from the page in dead_items */
+			tidstore_add_tids(dead_items, blkno, prunestate.deadoffsets,
+							  prunestate.num_offsets);
+
+			pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
+										 tidstore_memory_usage(dead_items));
 		}
 
 		/*
@@ -1145,7 +1156,7 @@ lazy_scan_heap(LVRelState *vacrel)
 		 * There should never be LP_DEAD items on a page with PD_ALL_VISIBLE
 		 * set, however.
 		 */
-		else if (prunestate.has_lpdead_items && PageIsAllVisible(page))
+		else if ((prunestate.num_offsets > 0) && PageIsAllVisible(page))
 		{
 			elog(WARNING, "page containing LP_DEAD items is marked as all-visible in relation \"%s\" page %u",
 				 vacrel->relname, blkno);
@@ -1193,7 +1204,7 @@ lazy_scan_heap(LVRelState *vacrel)
 		 * Final steps for block: drop cleanup lock, record free space in the
 		 * FSM
 		 */
-		if (prunestate.has_lpdead_items && vacrel->do_index_vacuuming)
+		if ((prunestate.num_offsets > 0) && vacrel->do_index_vacuuming)
 		{
 			/*
 			 * Wait until lazy_vacuum_heap_rel() to save free space.  This
@@ -1249,7 +1260,7 @@ lazy_scan_heap(LVRelState *vacrel)
 	 * Do index vacuuming (call each index's ambulkdelete routine), then do
 	 * related heap vacuuming
 	 */
-	if (dead_items->num_items > 0)
+	if (tidstore_num_tids(dead_items) > 0)
 		lazy_vacuum(vacrel);
 
 	/*
@@ -1524,9 +1535,9 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
  * The approach we take now is to restart pruning when the race condition is
  * detected.  This allows heap_page_prune() to prune the tuples inserted by
  * the now-aborted transaction.  This is a little crude, but it guarantees
- * that any items that make it into the dead_items array are simple LP_DEAD
- * line pointers, and that every remaining item with tuple storage is
- * considered as a candidate for freezing.
+ * that any items that make it into the dead_items are simple LP_DEAD line
+ * pointers, and that every remaining item with tuple storage is considered
+ * as a candidate for freezing.
  */
 static void
 lazy_scan_prune(LVRelState *vacrel,
@@ -1543,13 +1554,11 @@ lazy_scan_prune(LVRelState *vacrel,
 	HTSV_Result res;
 	int			tuples_deleted,
 				tuples_frozen,
-				lpdead_items,
 				live_tuples,
 				recently_dead_tuples;
 	int			nnewlpdead;
 	HeapPageFreeze pagefrz;
 	int64		fpi_before = pgWalUsage.wal_fpi;
-	OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
 	HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
 
 	Assert(BufferGetBlockNumber(buf) == blkno);
@@ -1571,7 +1580,6 @@ retry:
 	pagefrz.NoFreezePageRelminMxid = vacrel->NewRelminMxid;
 	tuples_deleted = 0;
 	tuples_frozen = 0;
-	lpdead_items = 0;
 	live_tuples = 0;
 	recently_dead_tuples = 0;
 
@@ -1580,9 +1588,9 @@ retry:
 	 *
 	 * We count tuples removed by the pruning step as tuples_deleted.  Its
 	 * final value can be thought of as the number of tuples that have been
-	 * deleted from the table.  It should not be confused with lpdead_items;
-	 * lpdead_items's final value can be thought of as the number of tuples
-	 * that were deleted from indexes.
+	 * deleted from the table.  It should not be confused with
+	 * prunestate->deadoffsets; prunestate->deadoffsets's final value can
+	 * be thought of as the number of tuples that were deleted from indexes.
 	 */
 	tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest,
 									 InvalidTransactionId, 0, &nnewlpdead,
@@ -1593,7 +1601,7 @@ retry:
 	 * requiring freezing among remaining tuples with storage
 	 */
 	prunestate->hastup = false;
-	prunestate->has_lpdead_items = false;
+	prunestate->num_offsets = 0;
 	prunestate->all_visible = true;
 	prunestate->all_frozen = true;
 	prunestate->visibility_cutoff_xid = InvalidTransactionId;
@@ -1638,7 +1646,7 @@ retry:
 			 * (This is another case where it's useful to anticipate that any
 			 * LP_DEAD items will become LP_UNUSED during the ongoing VACUUM.)
 			 */
-			deadoffsets[lpdead_items++] = offnum;
+			prunestate->deadoffsets[prunestate->num_offsets++] = offnum;
 			continue;
 		}
 
@@ -1875,7 +1883,7 @@ retry:
 	 */
 #ifdef USE_ASSERT_CHECKING
 	/* Note that all_frozen value does not matter when !all_visible */
-	if (prunestate->all_visible && lpdead_items == 0)
+	if (prunestate->all_visible && prunestate->num_offsets == 0)
 	{
 		TransactionId cutoff;
 		bool		all_frozen;
@@ -1888,28 +1896,9 @@ retry:
 	}
 #endif
 
-	/*
-	 * Now save details of the LP_DEAD items from the page in vacrel
-	 */
-	if (lpdead_items > 0)
+	if (prunestate->num_offsets > 0)
 	{
-		VacDeadItems *dead_items = vacrel->dead_items;
-		ItemPointerData tmp;
-
 		vacrel->lpdead_item_pages++;
-		prunestate->has_lpdead_items = true;
-
-		ItemPointerSetBlockNumber(&tmp, blkno);
-
-		for (int i = 0; i < lpdead_items; i++)
-		{
-			ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
-			dead_items->items[dead_items->num_items++] = tmp;
-		}
-
-		Assert(dead_items->num_items <= dead_items->max_items);
-		pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
-									 dead_items->num_items);
 
 		/*
 		 * It was convenient to ignore LP_DEAD items in all_visible earlier on
@@ -1928,7 +1917,7 @@ retry:
 	/* Finally, add page-local counts to whole-VACUUM counts */
 	vacrel->tuples_deleted += tuples_deleted;
 	vacrel->tuples_frozen += tuples_frozen;
-	vacrel->lpdead_items += lpdead_items;
+	vacrel->lpdead_items += prunestate->num_offsets;
 	vacrel->live_tuples += live_tuples;
 	vacrel->recently_dead_tuples += recently_dead_tuples;
 }
@@ -1940,7 +1929,7 @@ retry:
  * lazy_scan_prune, which requires a full cleanup lock.  While pruning isn't
  * performed here, it's quite possible that an earlier opportunistic pruning
  * operation left LP_DEAD items behind.  We'll at least collect any such items
- * in the dead_items array for removal from indexes.
+ * in the dead_items for removal from indexes.
  *
  * For aggressive VACUUM callers, we may return false to indicate that a full
  * cleanup lock is required for processing by lazy_scan_prune.  This is only
@@ -2099,7 +2088,7 @@ lazy_scan_noprune(LVRelState *vacrel,
 	vacrel->NewRelfrozenXid = NoFreezePageRelfrozenXid;
 	vacrel->NewRelminMxid = NoFreezePageRelminMxid;
 
-	/* Save any LP_DEAD items found on the page in dead_items array */
+	/* Save any LP_DEAD items found on the page in dead_items */
 	if (vacrel->nindexes == 0)
 	{
 		/* Using one-pass strategy (since table has no indexes) */
@@ -2129,8 +2118,7 @@ lazy_scan_noprune(LVRelState *vacrel,
 	}
 	else
 	{
-		VacDeadItems *dead_items = vacrel->dead_items;
-		ItemPointerData tmp;
+		TidStore *dead_items = vacrel->dead_items;
 
 		/*
 		 * Page has LP_DEAD items, and so any references/TIDs that remain in
@@ -2139,17 +2127,10 @@ lazy_scan_noprune(LVRelState *vacrel,
 		 */
 		vacrel->lpdead_item_pages++;
 
-		ItemPointerSetBlockNumber(&tmp, blkno);
+		tidstore_add_tids(dead_items, blkno, deadoffsets, lpdead_items);
 
-		for (int i = 0; i < lpdead_items; i++)
-		{
-			ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
-			dead_items->items[dead_items->num_items++] = tmp;
-		}
-
-		Assert(dead_items->num_items <= dead_items->max_items);
-		pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
-									 dead_items->num_items);
+		pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
+									 tidstore_memory_usage(dead_items));
 
 		vacrel->lpdead_items += lpdead_items;
 
@@ -2198,7 +2179,7 @@ lazy_vacuum(LVRelState *vacrel)
 	if (!vacrel->do_index_vacuuming)
 	{
 		Assert(!vacrel->do_index_cleanup);
-		vacrel->dead_items->num_items = 0;
+		tidstore_reset(vacrel->dead_items);
 		return;
 	}
 
@@ -2227,7 +2208,7 @@ lazy_vacuum(LVRelState *vacrel)
 		BlockNumber threshold;
 
 		Assert(vacrel->num_index_scans == 0);
-		Assert(vacrel->lpdead_items == vacrel->dead_items->num_items);
+		Assert(vacrel->lpdead_items == tidstore_num_tids(vacrel->dead_items));
 		Assert(vacrel->do_index_vacuuming);
 		Assert(vacrel->do_index_cleanup);
 
@@ -2254,8 +2235,8 @@ lazy_vacuum(LVRelState *vacrel)
 		 * cases then this may need to be reconsidered.
 		 */
 		threshold = (double) vacrel->rel_pages * BYPASS_THRESHOLD_PAGES;
-		bypass = (vacrel->lpdead_item_pages < threshold &&
-				  vacrel->lpdead_items < MAXDEADITEMS(32L * 1024L * 1024L));
+		bypass = (vacrel->lpdead_item_pages < threshold) &&
+			tidstore_memory_usage(vacrel->dead_items) < (32L * 1024L * 1024L);
 	}
 
 	if (bypass)
@@ -2300,7 +2281,7 @@ lazy_vacuum(LVRelState *vacrel)
 	 * Forget the LP_DEAD items that we just vacuumed (or just decided to not
 	 * vacuum)
 	 */
-	vacrel->dead_items->num_items = 0;
+	tidstore_reset(vacrel->dead_items);
 }
 
 /*
@@ -2373,7 +2354,7 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
 	 * place).
 	 */
 	Assert(vacrel->num_index_scans > 0 ||
-		   vacrel->dead_items->num_items == vacrel->lpdead_items);
+		   tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items);
 	Assert(allindexes || vacrel->failsafe_active);
 
 	/*
@@ -2392,9 +2373,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
 /*
  *	lazy_vacuum_heap_rel() -- second pass over the heap for two pass strategy
  *
- * This routine marks LP_DEAD items in vacrel->dead_items array as LP_UNUSED.
- * Pages that never had lazy_scan_prune record LP_DEAD items are not visited
- * at all.
+ * This routine marks LP_DEAD items in vacrel->dead_items as LP_UNUSED. Pages
+ * that never had lazy_scan_prune record LP_DEAD items are not visited at all.
  *
  * We may also be able to truncate the line pointer array of the heap pages we
  * visit.  If there is a contiguous group of LP_UNUSED items at the end of the
@@ -2410,10 +2390,11 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
 static void
 lazy_vacuum_heap_rel(LVRelState *vacrel)
 {
-	int			index = 0;
 	BlockNumber vacuumed_pages = 0;
 	Buffer		vmbuffer = InvalidBuffer;
 	LVSavedErrInfo saved_err_info;
+	TidStoreIter *iter;
+	TidStoreIterResult *result;
 
 	Assert(vacrel->do_index_vacuuming);
 	Assert(vacrel->do_index_cleanup);
@@ -2428,7 +2409,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
 							 VACUUM_ERRCB_PHASE_VACUUM_HEAP,
 							 InvalidBlockNumber, InvalidOffsetNumber);
 
-	while (index < vacrel->dead_items->num_items)
+	iter = tidstore_begin_iterate(vacrel->dead_items);
+	while ((result = tidstore_iterate_next(iter)) != NULL)
 	{
 		BlockNumber blkno;
 		Buffer		buf;
@@ -2437,7 +2419,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
 
 		vacuum_delay_point();
 
-		blkno = ItemPointerGetBlockNumber(&vacrel->dead_items->items[index]);
+		blkno = result->blkno;
 		vacrel->blkno = blkno;
 
 		/*
@@ -2451,7 +2433,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
 		buf = ReadBufferExtended(vacrel->rel, MAIN_FORKNUM, blkno, RBM_NORMAL,
 								 vacrel->bstrategy);
 		LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
-		index = lazy_vacuum_heap_page(vacrel, blkno, buf, index, vmbuffer);
+		lazy_vacuum_heap_page(vacrel, blkno, result->offsets, result->num_offsets,
+							  buf, vmbuffer);
 
 		/* Now that we've vacuumed the page, record its available space */
 		page = BufferGetPage(buf);
@@ -2461,6 +2444,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
 		RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
 		vacuumed_pages++;
 	}
+	tidstore_end_iterate(iter);
 
 	vacrel->blkno = InvalidBlockNumber;
 	if (BufferIsValid(vmbuffer))
@@ -2470,36 +2454,31 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
 	 * We set all LP_DEAD items from the first heap pass to LP_UNUSED during
 	 * the second heap pass.  No more, no less.
 	 */
-	Assert(index > 0);
 	Assert(vacrel->num_index_scans > 1 ||
-		   (index == vacrel->lpdead_items &&
+		   (tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items &&
 			vacuumed_pages == vacrel->lpdead_item_pages));
 
 	ereport(DEBUG2,
-			(errmsg("table \"%s\": removed %lld dead item identifiers in %u pages",
-					vacrel->relname, (long long) index, vacuumed_pages)));
+			(errmsg("table \"%s\": removed " UINT64_FORMAT "dead item identifiers in %u pages",
+					vacrel->relname, tidstore_num_tids(vacrel->dead_items),
+					vacuumed_pages)));
 
 	/* Revert to the previous phase information for error traceback */
 	restore_vacuum_error_info(vacrel, &saved_err_info);
 }
 
 /*
- *	lazy_vacuum_heap_page() -- free page's LP_DEAD items listed in the
- *						  vacrel->dead_items array.
+ *	lazy_vacuum_heap_page() -- free page's LP_DEAD items.
  *
  * Caller must have an exclusive buffer lock on the buffer (though a full
  * cleanup lock is also acceptable).  vmbuffer must be valid and already have
  * a pin on blkno's visibility map page.
- *
- * index is an offset into the vacrel->dead_items array for the first listed
- * LP_DEAD item on the page.  The return value is the first index immediately
- * after all LP_DEAD items for the same page in the array.
  */
-static int
-lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
-					  int index, Buffer vmbuffer)
+static void
+lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
+					  OffsetNumber *deadoffsets, int num_offsets, Buffer buffer,
+					  Buffer vmbuffer)
 {
-	VacDeadItems *dead_items = vacrel->dead_items;
 	Page		page = BufferGetPage(buffer);
 	OffsetNumber unused[MaxHeapTuplesPerPage];
 	int			nunused = 0;
@@ -2518,16 +2497,11 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
 
 	START_CRIT_SECTION();
 
-	for (; index < dead_items->num_items; index++)
+	for (int i = 0; i < num_offsets; i++)
 	{
-		BlockNumber tblk;
-		OffsetNumber toff;
 		ItemId		itemid;
+		OffsetNumber	toff = deadoffsets[i];
 
-		tblk = ItemPointerGetBlockNumber(&dead_items->items[index]);
-		if (tblk != blkno)
-			break;				/* past end of tuples for this block */
-		toff = ItemPointerGetOffsetNumber(&dead_items->items[index]);
 		itemid = PageGetItemId(page, toff);
 
 		Assert(ItemIdIsDead(itemid) && !ItemIdHasStorage(itemid));
@@ -2597,7 +2571,6 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
 
 	/* Revert to the previous phase information for error traceback */
 	restore_vacuum_error_info(vacrel, &saved_err_info);
-	return index;
 }
 
 /*
@@ -2687,8 +2660,8 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
  *	lazy_vacuum_one_index() -- vacuum index relation.
  *
  *		Delete all the index tuples containing a TID collected in
- *		vacrel->dead_items array.  Also update running statistics.
- *		Exact details depend on index AM's ambulkdelete routine.
+ *		vacrel->dead_items.  Also update running statistics. Exact
+ *		details depend on index AM's ambulkdelete routine.
  *
  *		reltuples is the number of heap tuples to be passed to the
  *		bulkdelete callback.  It's always assumed to be estimated.
@@ -3094,48 +3067,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
 }
 
 /*
- * Returns the number of dead TIDs that VACUUM should allocate space to
- * store, given a heap rel of size vacrel->rel_pages, and given current
- * maintenance_work_mem setting (or current autovacuum_work_mem setting,
- * when applicable).
- *
- * See the comments at the head of this file for rationale.
- */
-static int
-dead_items_max_items(LVRelState *vacrel)
-{
-	int64		max_items;
-	int			vac_work_mem = IsAutoVacuumWorkerProcess() &&
-	autovacuum_work_mem != -1 ?
-	autovacuum_work_mem : maintenance_work_mem;
-
-	if (vacrel->nindexes > 0)
-	{
-		BlockNumber rel_pages = vacrel->rel_pages;
-
-		max_items = MAXDEADITEMS(vac_work_mem * 1024L);
-		max_items = Min(max_items, INT_MAX);
-		max_items = Min(max_items, MAXDEADITEMS(MaxAllocSize));
-
-		/* curious coding here to ensure the multiplication can't overflow */
-		if ((BlockNumber) (max_items / MaxHeapTuplesPerPage) > rel_pages)
-			max_items = rel_pages * MaxHeapTuplesPerPage;
-
-		/* stay sane if small maintenance_work_mem */
-		max_items = Max(max_items, MaxHeapTuplesPerPage);
-	}
-	else
-	{
-		/* One-pass case only stores a single heap page's TIDs at a time */
-		max_items = MaxHeapTuplesPerPage;
-	}
-
-	return (int) max_items;
-}
-
-/*
- * Allocate dead_items (either using palloc, or in dynamic shared memory).
- * Sets dead_items in vacrel for caller.
+ * Allocate a (local or shared) TidStore for storing dead TIDs. Sets dead_items
+ * in vacrel for caller.
  *
  * Also handles parallel initialization as part of allocating dead_items in
  * DSM when required.
@@ -3143,11 +3076,9 @@ dead_items_max_items(LVRelState *vacrel)
 static void
 dead_items_alloc(LVRelState *vacrel, int nworkers)
 {
-	VacDeadItems *dead_items;
-	int			max_items;
-
-	max_items = dead_items_max_items(vacrel);
-	Assert(max_items >= MaxHeapTuplesPerPage);
+	int			vac_work_mem = IsAutoVacuumWorkerProcess() &&
+		autovacuum_work_mem != -1 ?
+		autovacuum_work_mem * 1024L : maintenance_work_mem * 1024L;
 
 	/*
 	 * Initialize state for a parallel vacuum.  As of now, only one worker can
@@ -3174,7 +3105,7 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
 		else
 			vacrel->pvs = parallel_vacuum_init(vacrel->rel, vacrel->indrels,
 											   vacrel->nindexes, nworkers,
-											   max_items,
+											   vac_work_mem, MaxHeapTuplesPerPage,
 											   vacrel->verbose ? INFO : DEBUG2,
 											   vacrel->bstrategy);
 
@@ -3187,11 +3118,8 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
 	}
 
 	/* Serial VACUUM case */
-	dead_items = (VacDeadItems *) palloc(vac_max_items_to_alloc_size(max_items));
-	dead_items->max_items = max_items;
-	dead_items->num_items = 0;
-
-	vacrel->dead_items = dead_items;
+	vacrel->dead_items = tidstore_create(vac_work_mem, MaxHeapTuplesPerPage,
+										 NULL);
 }
 
 /*
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 8608e3fa5b..a526e607fe 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1165,7 +1165,7 @@ CREATE VIEW pg_stat_progress_vacuum AS
                       END AS phase,
         S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
         S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
-        S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+        S.param6 AS max_dead_tuple_bytes, S.param7 AS dead_tuple_bytes
     FROM pg_stat_get_progress_info('VACUUM') AS S
         LEFT JOIN pg_database D ON S.datid = D.oid;
 
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 7b1a4b127e..d8e680ca20 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -97,7 +97,6 @@ static bool vacuum_rel(Oid relid, RangeVar *relation, VacuumParams *params,
 static double compute_parallel_delay(void);
 static VacOptValue get_vacoptval_from_boolean(DefElem *def);
 static bool vac_tid_reaped(ItemPointer itemptr, void *state);
-static int	vac_cmp_itemptr(const void *left, const void *right);
 
 /*
  * Primary entry point for manual VACUUM and ANALYZE commands
@@ -2303,16 +2302,16 @@ get_vacoptval_from_boolean(DefElem *def)
  */
 IndexBulkDeleteResult *
 vac_bulkdel_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat,
-					  VacDeadItems *dead_items)
+					  TidStore *dead_items)
 {
 	/* Do bulk deletion */
 	istat = index_bulk_delete(ivinfo, istat, vac_tid_reaped,
 							  (void *) dead_items);
 
 	ereport(ivinfo->message_level,
-			(errmsg("scanned index \"%s\" to remove %d row versions",
+			(errmsg("scanned index \"%s\" to remove " UINT64_FORMAT " row versions",
 					RelationGetRelationName(ivinfo->index),
-					dead_items->num_items)));
+					tidstore_num_tids(dead_items))));
 
 	return istat;
 }
@@ -2343,82 +2342,15 @@ vac_cleanup_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat)
 	return istat;
 }
 
-/*
- * Returns the total required space for VACUUM's dead_items array given a
- * max_items value.
- */
-Size
-vac_max_items_to_alloc_size(int max_items)
-{
-	Assert(max_items <= MAXDEADITEMS(MaxAllocSize));
-
-	return offsetof(VacDeadItems, items) + sizeof(ItemPointerData) * max_items;
-}
-
 /*
  *	vac_tid_reaped() -- is a particular tid deletable?
  *
  *		This has the right signature to be an IndexBulkDeleteCallback.
- *
- *		Assumes dead_items array is sorted (in ascending TID order).
  */
 static bool
 vac_tid_reaped(ItemPointer itemptr, void *state)
 {
-	VacDeadItems *dead_items = (VacDeadItems *) state;
-	int64		litem,
-				ritem,
-				item;
-	ItemPointer res;
-
-	litem = itemptr_encode(&dead_items->items[0]);
-	ritem = itemptr_encode(&dead_items->items[dead_items->num_items - 1]);
-	item = itemptr_encode(itemptr);
-
-	/*
-	 * Doing a simple bound check before bsearch() is useful to avoid the
-	 * extra cost of bsearch(), especially if dead items on the heap are
-	 * concentrated in a certain range.  Since this function is called for
-	 * every index tuple, it pays to be really fast.
-	 */
-	if (item < litem || item > ritem)
-		return false;
-
-	res = (ItemPointer) bsearch((void *) itemptr,
-								(void *) dead_items->items,
-								dead_items->num_items,
-								sizeof(ItemPointerData),
-								vac_cmp_itemptr);
-
-	return (res != NULL);
-}
-
-/*
- * Comparator routines for use with qsort() and bsearch().
- */
-static int
-vac_cmp_itemptr(const void *left, const void *right)
-{
-	BlockNumber lblk,
-				rblk;
-	OffsetNumber loff,
-				roff;
-
-	lblk = ItemPointerGetBlockNumber((ItemPointer) left);
-	rblk = ItemPointerGetBlockNumber((ItemPointer) right);
-
-	if (lblk < rblk)
-		return -1;
-	if (lblk > rblk)
-		return 1;
-
-	loff = ItemPointerGetOffsetNumber((ItemPointer) left);
-	roff = ItemPointerGetOffsetNumber((ItemPointer) right);
-
-	if (loff < roff)
-		return -1;
-	if (loff > roff)
-		return 1;
+	TidStore *dead_items = (TidStore *) state;
 
-	return 0;
+	return tidstore_lookup_tid(dead_items, itemptr);
 }
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index bcd40c80a1..d653683693 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -9,12 +9,11 @@
  * In a parallel vacuum, we perform both index bulk deletion and index cleanup
  * with parallel worker processes.  Individual indexes are processed by one
  * vacuum process.  ParalleVacuumState contains shared information as well as
- * the memory space for storing dead items allocated in the DSM segment.  We
- * launch parallel worker processes at the start of parallel index
- * bulk-deletion and index cleanup and once all indexes are processed, the
- * parallel worker processes exit.  Each time we process indexes in parallel,
- * the parallel context is re-initialized so that the same DSM can be used for
- * multiple passes of index bulk-deletion and index cleanup.
+ * the shared TidStore. We launch parallel worker processes at the start of
+ * parallel index bulk-deletion and index cleanup and once all indexes are
+ * processed, the parallel worker processes exit.  Each time we process indexes
+ * in parallel, the parallel context is re-initialized so that the same DSM can
+ * be used for multiple passes of index bulk-deletion and index cleanup.
  *
  * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
  * Portions Copyright (c) 1994, Regents of the University of California
@@ -103,6 +102,9 @@ typedef struct PVShared
 
 	/* Counter for vacuuming and cleanup */
 	pg_atomic_uint32 idx;
+
+	/* Handle of the shared TidStore */
+	tidstore_handle	dead_items_handle;
 } PVShared;
 
 /* Status used during parallel index vacuum or cleanup */
@@ -166,7 +168,8 @@ struct ParallelVacuumState
 	PVIndStats *indstats;
 
 	/* Shared dead items space among parallel vacuum workers */
-	VacDeadItems *dead_items;
+	TidStore *dead_items;
+	dsa_area *dead_items_area;
 
 	/* Points to buffer usage area in DSM */
 	BufferUsage *buffer_usage;
@@ -222,20 +225,23 @@ static void parallel_vacuum_error_callback(void *arg);
  */
 ParallelVacuumState *
 parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
-					 int nrequested_workers, int max_items,
-					 int elevel, BufferAccessStrategy bstrategy)
+					 int nrequested_workers, int vac_work_mem,
+					 int max_offset, int elevel,
+					 BufferAccessStrategy bstrategy)
 {
 	ParallelVacuumState *pvs;
 	ParallelContext *pcxt;
 	PVShared   *shared;
-	VacDeadItems *dead_items;
+	TidStore	*dead_items;
 	PVIndStats *indstats;
 	BufferUsage *buffer_usage;
 	WalUsage   *wal_usage;
+	void		*area_space;
+	dsa_area	*dead_items_dsa;
 	bool	   *will_parallel_vacuum;
 	Size		est_indstats_len;
 	Size		est_shared_len;
-	Size		est_dead_items_len;
+	Size		dsa_minsize = dsa_minimum_size();
 	int			nindexes_mwm = 0;
 	int			parallel_workers = 0;
 	int			querylen;
@@ -283,9 +289,8 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
 	shm_toc_estimate_chunk(&pcxt->estimator, est_shared_len);
 	shm_toc_estimate_keys(&pcxt->estimator, 1);
 
-	/* Estimate size for dead_items -- PARALLEL_VACUUM_KEY_DEAD_ITEMS */
-	est_dead_items_len = vac_max_items_to_alloc_size(max_items);
-	shm_toc_estimate_chunk(&pcxt->estimator, est_dead_items_len);
+	/* Estimate size for dead tuple DSA -- PARALLEL_VACUUM_KEY_DSA */
+	shm_toc_estimate_chunk(&pcxt->estimator, dsa_minsize);
 	shm_toc_estimate_keys(&pcxt->estimator, 1);
 
 	/*
@@ -351,6 +356,16 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
 	shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_INDEX_STATS, indstats);
 	pvs->indstats = indstats;
 
+	/* Prepare DSA space for dead items */
+	area_space = shm_toc_allocate(pcxt->toc, dsa_minsize);
+	shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, area_space);
+	dead_items_dsa = dsa_create_in_place(area_space, dsa_minsize,
+										 LWTRANCHE_PARALLEL_VACUUM_DSA,
+										 pcxt->seg);
+	dead_items = tidstore_create(vac_work_mem, max_offset, dead_items_dsa);
+	pvs->dead_items = dead_items;
+	pvs->dead_items_area = dead_items_dsa;
+
 	/* Prepare shared information */
 	shared = (PVShared *) shm_toc_allocate(pcxt->toc, est_shared_len);
 	MemSet(shared, 0, est_shared_len);
@@ -360,6 +375,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
 		(nindexes_mwm > 0) ?
 		maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
 		maintenance_work_mem;
+	shared->dead_items_handle = tidstore_get_handle(dead_items);
 
 	pg_atomic_init_u32(&(shared->cost_balance), 0);
 	pg_atomic_init_u32(&(shared->active_nworkers), 0);
@@ -368,15 +384,6 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
 	shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_SHARED, shared);
 	pvs->shared = shared;
 
-	/* Prepare the dead_items space */
-	dead_items = (VacDeadItems *) shm_toc_allocate(pcxt->toc,
-												   est_dead_items_len);
-	dead_items->max_items = max_items;
-	dead_items->num_items = 0;
-	MemSet(dead_items->items, 0, sizeof(ItemPointerData) * max_items);
-	shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, dead_items);
-	pvs->dead_items = dead_items;
-
 	/*
 	 * Allocate space for each worker's BufferUsage and WalUsage; no need to
 	 * initialize
@@ -434,6 +441,9 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
 			istats[i] = NULL;
 	}
 
+	tidstore_destroy(pvs->dead_items);
+	dsa_detach(pvs->dead_items_area);
+
 	DestroyParallelContext(pvs->pcxt);
 	ExitParallelMode();
 
@@ -442,7 +452,7 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
 }
 
 /* Returns the dead items space */
-VacDeadItems *
+TidStore *
 parallel_vacuum_get_dead_items(ParallelVacuumState *pvs)
 {
 	return pvs->dead_items;
@@ -940,7 +950,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
 	Relation   *indrels;
 	PVIndStats *indstats;
 	PVShared   *shared;
-	VacDeadItems *dead_items;
+	TidStore	*dead_items;
+	void		*area_space;
+	dsa_area	*dead_items_area;
 	BufferUsage *buffer_usage;
 	WalUsage   *wal_usage;
 	int			nindexes;
@@ -984,10 +996,10 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
 											 PARALLEL_VACUUM_KEY_INDEX_STATS,
 											 false);
 
-	/* Set dead_items space */
-	dead_items = (VacDeadItems *) shm_toc_lookup(toc,
-												 PARALLEL_VACUUM_KEY_DEAD_ITEMS,
-												 false);
+	/* Set dead items */
+	area_space = shm_toc_lookup(toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, false);
+	dead_items_area = dsa_attach_in_place(area_space, seg);
+	dead_items = tidstore_attach(dead_items_area, shared->dead_items_handle);
 
 	/* Set cost-based vacuum delay */
 	VacuumCostActive = (VacuumCostDelay > 0);
@@ -1033,6 +1045,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
 	InstrEndParallelQuery(&buffer_usage[ParallelWorkerNumber],
 						  &wal_usage[ParallelWorkerNumber]);
 
+	tidstore_detach(pvs.dead_items);
+	dsa_detach(dead_items_area);
+
 	/* Pop the error context stack */
 	error_context_stack = errcallback.previous;
 
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index f5ea381c53..d88db3e1f8 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -3397,12 +3397,12 @@ check_autovacuum_work_mem(int *newval, void **extra, GucSource source)
 		return true;
 
 	/*
-	 * We clamp manually-set values to at least 1MB.  Since
+	 * We clamp manually-set values to at least 2MB.  Since
 	 * maintenance_work_mem is always set to at least this value, do the same
 	 * here.
 	 */
-	if (*newval < 1024)
-		*newval = 1024;
+	if (*newval < 2048)
+		*newval = 2048;
 
 	return true;
 }
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 55b3a04097..c223a7dc94 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -192,6 +192,8 @@ static const char *const BuiltinTrancheNames[] = {
 	"LogicalRepLauncherDSA",
 	/* LWTRANCHE_LAUNCHER_HASH: */
 	"LogicalRepLauncherHash",
+	/* LWTRANCHE_PARALLEL_VACUUM_DSA: */
+	"ParallelVacuumDSA",
 };
 
 StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index b46e3b8c55..27a88b9369 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2312,7 +2312,7 @@ struct config_int ConfigureNamesInt[] =
 			GUC_UNIT_KB
 		},
 		&maintenance_work_mem,
-		65536, 1024, MAX_KILOBYTES,
+		65536, 2048, MAX_KILOBYTES,
 		NULL, NULL, NULL
 	},
 
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index e5add41352..b209d3cf84 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -23,8 +23,8 @@
 #define PROGRESS_VACUUM_HEAP_BLKS_SCANNED		2
 #define PROGRESS_VACUUM_HEAP_BLKS_VACUUMED		3
 #define PROGRESS_VACUUM_NUM_INDEX_VACUUMS		4
-#define PROGRESS_VACUUM_MAX_DEAD_TUPLES			5
-#define PROGRESS_VACUUM_NUM_DEAD_TUPLES			6
+#define PROGRESS_VACUUM_MAX_DEAD_TUPLE_BYTES	5
+#define PROGRESS_VACUUM_DEAD_TUPLE_BYTES		6
 
 /* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
 #define PROGRESS_VACUUM_PHASE_SCAN_HEAP			1
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 689dbb7702..a3ebb169ef 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -17,6 +17,7 @@
 #include "access/htup.h"
 #include "access/genam.h"
 #include "access/parallel.h"
+#include "access/tidstore.h"
 #include "catalog/pg_class.h"
 #include "catalog/pg_statistic.h"
 #include "catalog/pg_type.h"
@@ -276,21 +277,6 @@ struct VacuumCutoffs
 	MultiXactId MultiXactCutoff;
 };
 
-/*
- * VacDeadItems stores TIDs whose index tuples are deleted by index vacuuming.
- */
-typedef struct VacDeadItems
-{
-	int			max_items;		/* # slots allocated in array */
-	int			num_items;		/* current # of entries */
-
-	/* Sorted array of TIDs to delete from indexes */
-	ItemPointerData items[FLEXIBLE_ARRAY_MEMBER];
-} VacDeadItems;
-
-#define MAXDEADITEMS(avail_mem) \
-	(((avail_mem) - offsetof(VacDeadItems, items)) / sizeof(ItemPointerData))
-
 /* GUC parameters */
 extern PGDLLIMPORT int default_statistics_target;	/* PGDLLIMPORT for PostGIS */
 extern PGDLLIMPORT int vacuum_freeze_min_age;
@@ -339,18 +325,17 @@ extern Relation vacuum_open_relation(Oid relid, RangeVar *relation,
 									 LOCKMODE lmode);
 extern IndexBulkDeleteResult *vac_bulkdel_one_index(IndexVacuumInfo *ivinfo,
 													IndexBulkDeleteResult *istat,
-													VacDeadItems *dead_items);
+													TidStore *dead_items);
 extern IndexBulkDeleteResult *vac_cleanup_one_index(IndexVacuumInfo *ivinfo,
 													IndexBulkDeleteResult *istat);
-extern Size vac_max_items_to_alloc_size(int max_items);
 
 /* in commands/vacuumparallel.c */
 extern ParallelVacuumState *parallel_vacuum_init(Relation rel, Relation *indrels,
 												 int nindexes, int nrequested_workers,
-												 int max_items, int elevel,
-												 BufferAccessStrategy bstrategy);
+												 int vac_work_mem, int max_offset,
+												 int elevel, BufferAccessStrategy bstrategy);
 extern void parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats);
-extern VacDeadItems *parallel_vacuum_get_dead_items(ParallelVacuumState *pvs);
+extern TidStore *parallel_vacuum_get_dead_items(ParallelVacuumState *pvs);
 extern void parallel_vacuum_bulkdel_all_indexes(ParallelVacuumState *pvs,
 												long num_table_tuples,
 												int num_index_scans);
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index 07002fdfbe..537b34b30c 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -207,6 +207,7 @@ typedef enum BuiltinTrancheIds
 	LWTRANCHE_PGSTATS_DATA,
 	LWTRANCHE_LAUNCHER_DSA,
 	LWTRANCHE_LAUNCHER_HASH,
+	LWTRANCHE_PARALLEL_VACUUM_DSA,
 	LWTRANCHE_FIRST_USER_DEFINED
 }			BuiltinTrancheIds;
 
diff --git a/src/test/regress/expected/cluster.out b/src/test/regress/expected/cluster.out
index 2eec483eaa..e04f50726f 100644
--- a/src/test/regress/expected/cluster.out
+++ b/src/test/regress/expected/cluster.out
@@ -526,7 +526,7 @@ create index cluster_sort on clstr_4 (hundred, thousand, tenthous);
 -- ensure we don't use the index in CLUSTER nor the checking SELECTs
 set enable_indexscan = off;
 -- Use external sort:
-set maintenance_work_mem = '1MB';
+set maintenance_work_mem = '2MB';
 cluster clstr_4 using cluster_sort;
 select * from
 (select hundred, lag(hundred) over () as lhundred,
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index 6cd57e3eaa..d1889b9d10 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -1214,7 +1214,7 @@ DROP TABLE unlogged_hash_table;
 -- CREATE INDEX hash_ovfl_index ON hash_ovfl_heap USING hash (x int4_ops);
 -- Test hash index build tuplesorting.  Force hash tuplesort using low
 -- maintenance_work_mem setting and fillfactor:
-SET maintenance_work_mem = '1MB';
+SET maintenance_work_mem = '2MB';
 CREATE INDEX hash_tuplesort_idx ON tenk1 USING hash (stringu1 name_ops) WITH (fillfactor = 10);
 EXPLAIN (COSTS OFF)
 SELECT count(*) FROM tenk1 WHERE stringu1 = 'TVAAAA';
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index e7a2f5856a..f6ae02eb14 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2020,8 +2020,8 @@ pg_stat_progress_vacuum| SELECT s.pid,
     s.param3 AS heap_blks_scanned,
     s.param4 AS heap_blks_vacuumed,
     s.param5 AS index_vacuum_count,
-    s.param6 AS max_dead_tuples,
-    s.param7 AS num_dead_tuples
+    s.param6 AS max_dead_tuple_bytes,
+    s.param7 AS dead_tuple_bytes
    FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
      LEFT JOIN pg_database d ON ((s.datid = d.oid)));
 pg_stat_recovery_prefetch| SELECT stats_reset,
diff --git a/src/test/regress/sql/cluster.sql b/src/test/regress/sql/cluster.sql
index a4cfaae807..a4cb5b98a5 100644
--- a/src/test/regress/sql/cluster.sql
+++ b/src/test/regress/sql/cluster.sql
@@ -258,7 +258,7 @@ create index cluster_sort on clstr_4 (hundred, thousand, tenthous);
 set enable_indexscan = off;
 
 -- Use external sort:
-set maintenance_work_mem = '1MB';
+set maintenance_work_mem = '2MB';
 cluster clstr_4 using cluster_sort;
 select * from
 (select hundred, lag(hundred) over () as lhundred,
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index a3738833b2..edb5e4b4f3 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -367,7 +367,7 @@ DROP TABLE unlogged_hash_table;
 
 -- Test hash index build tuplesorting.  Force hash tuplesort using low
 -- maintenance_work_mem setting and fillfactor:
-SET maintenance_work_mem = '1MB';
+SET maintenance_work_mem = '2MB';
 CREATE INDEX hash_tuplesort_idx ON tenk1 USING hash (stringu1 name_ops) WITH (fillfactor = 10);
 EXPLAIN (COSTS OFF)
 SELECT count(*) FROM tenk1 WHERE stringu1 = 'TVAAAA';
-- 
2.39.1

---

From: John Naylor <[email-masked]>
Date: Tue, 7 Feb 2023 17:22:59 +0700

On Tue, Feb 7, 2023 at 4:25 PM John Naylor <[email-masked]>
wrote:
> [v25]

This conflicted with a commit from earlier today, so rebased in v26 with no
further changes.

 
 					if (insertpos < count)
 					{
@@ -200,6 +206,9 @@
 												   count, insertpos);
 #endif
 					}
+#else
+					insertpos = count;
+#endif
 
 					n32->base.chunks[insertpos] = chunk;
 #ifdef RT_NODE_LEVEL_LEAF
From f6f476ba71864821cb5144f513165671c64db1b2 Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Tue, 6 Dec 2022 13:39:41 +0700
Subject: [PATCH v26 2/9] Move some bitmap logic out of bitmapset.c

 
-/*----------

 /*
 
-			w = RIGHTMOST_ONE(w);

 
 #else

 #endif

 
 /*
 
+/*----------

From cf3e16ed894fc0c6574c48eddad7c587e5dec688 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Mon, 24 Oct 2022 14:07:09 +0900
Subject: [PATCH v26 1/9] Introduce helper SIMD functions for small byte arrays

Masahiko Sawada

 

+/*
 
+/*

From 4a0d293937876ec348f30ce4ab94da14b925b020 Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Tue, 7 Feb 2023 13:06:00 +0700
Subject: [PATCH v26 5/9] Measure load time of bench_search_random_nodes

---

 

 		rt_set(rt, key, &key);
 	}

 
 	rt_free(rt);

From 0e328f6d85d30797af158f2a4070004fe40d93fe Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Fri, 16 Sep 2022 11:57:03 +0900
Subject: [PATCH v26 4/9] Tool for measuring radix tree performance

XXX: Not for commit
---

From 4c4cbb9b13da160b8883e6c7f861516f3eedac6a Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Wed, 14 Sep 2022 12:38:51 +0000
Subject: [PATCH v26 3/9] Add radixtree template

 
+size_t

 

From 0726bb6b4e0250a72ce399d945d250724b4a29ab Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Mon, 6 Feb 2023 21:04:14 +0700
Subject: [PATCH v26 6/9] Adjust some inlining declarations

---

From 6831fe27a2c9c5765113b7903403c426f09f55f6 Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Mon, 6 Feb 2023 22:04:50 +0700
Subject: [PATCH v26 7/9] Skip unnecessary searches in RT_NODE_INSERT_INNER

 

 

 
 	/* Update statistics */
 
 	/*

 #undef RT_NODE3_TYPE
 #undef RT_NODE32_TYPE
-- 
2.39.1

From f17e983832736a1daa64e67a10f9a64189b68210 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Fri, 23 Dec 2022 23:50:39 +0900
Subject: [PATCH v26 8/9] Add TIDStore, to store sets of TIDs (ItemPointerData)
 efficiently.

 

From ed8115b5f5c1b0745e35a0d6d72064ad9df4cf42 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Tue, 7 Feb 2023 17:19:29 +0700
Subject: [PATCH v26 9/9] Use TIDStore for storing dead tuple TID during lazy
 vacuum

 
      <row>
 
      <row>
 
 	/*
 
 	/*

 

 
 
 				/*
 
 		/*
 
 	/*

 
 
-	/*
 
 		/*

 		/*

 

 

 	if (bypass)
 
 /*
 
 	/*

 
 		vacuum_delay_point();

 		/*

 
 	ereport(DEBUG2,

 /*
 
 	START_CRIT_SECTION();

 

 /*
 
 /*
 
 	/*

 	/* Serial VACUUM case */
 
 /*
 
index aa79d9de4d..d8e680ca20 100644
 
 /*

 	return istat;
 }
 
-/*
-	res = (ItemPointer) bsearch(itemptr,
-								dead_items->items,
 
-	return 0;

 

 
 	/*

 

 

 
 
index ff6149a179..a371f6fbba 100644
 
 	/*
 
 	return true;
 }

 
 
-/*

 
 -- Use external sort:

---

From: Masahiko Sawada <[email-masked]>
Date: Thu, 9 Feb 2023 16:08:03 +0900

Hi,

On Tue, Feb 7, 2023 at 6:25 PM John Naylor <[email-masked]> wrote:
>
>
> On Tue, Jan 31, 2023 at 9:43 PM Masahiko Sawada <[email-masked]> wrote:
>
> Okay, that's a lot more simple, and closer to what I imagined. For v25, I squashed v24's additions and added a couple of my own. I've kept the CF status at "needs review" because no specific action is required at the moment.
>
> I did start to review the TID store some more, but that's on hold because something else came up: On a lark I decided to re-run some benchmarks to see if anything got lost in converting to a template, and that led me down a rabbit hole -- some good and bad news on that below.
>
> 0001:
>
> I removed the uint64 case, as discussed. There is now a brief commit message, but needs to be fleshed out a bit. I took another look at the Arm optimization that Nathan found some month ago, for forming the highbit mask, but that doesn't play nicely with how node32 uses it, so I decided against it. I added a comment to describe the reasoning in case someone else gets a similar idea.
>
> I briefly looked into "separate-commit TODO: move non-SIMD fallbacks to their own header to clean up the #ifdef maze.", but decided it wasn't such a clear win to justify starting the work now. It's still in the back of my mind, but I removed the reminder from the commit message.

The changes make sense to me.

>
> 0003:
>
> The template now requires the value to be passed as a pointer. That was a pretty trivial change, but affected multiple other patches, so not sent separately. Also adds a forgotten RT_ prefix to the bitmap macros and adds a top comment to the *_impl.h headers. There are some comment fixes. The changes were either trivial or discussed earlier, so also not sent separately.

Great.

>
> 0004/5: I wanted to measure the load time as well as search time in bench_search_random_nodes(). That's kept separate to make it easier to test other patch versions.
>
> The bad news is that the speed of loading TIDs in bench_seq/shuffle_search() has regressed noticeably. I can't reproduce this in any other bench function and was the reason for writing 0005 to begin with. More confusingly, my efforts to fix this improved *other* functions, but the former didn't budge at all. First the patches:
>
> 0006 adds and removes some "inline" declarations (where it made sense), and added some for "pg_noinline" based on Andres' advice some months ago.

Agreed.

>
> 0007 removes some dead code. RT_NODE_INSERT_INNER is only called during RT_SET_EXTEND, so it can't possibly find an existing key. This kind of change is much easier with the inner/node cases handled together in a template, as far as being sure of how those cases are different. I thought about trying the search in assert builds and verifying it doesn't exist, but thought yet another #ifdef would be too messy.

Agreed.

>
> v25-addendum-try-no-maintain-order.txt -- It makes optional keeping the key chunks in order for the linear-search nodes. I believe the TID store no longer cares about the ordering, but this is a text file for now because I don't want to clutter the CI with a behavior change. Also, the second ART paper (on concurrency) mentioned that some locking schemes don't allow these arrays to be shifted. So it might make sense to give up entirely on guaranteeing ordered iteration, or at least make it optional as in the patch.

I think it's still important for lazy vacuum that an iteration over a
TID store returns TIDs in ascending order, because otherwise a heap
vacuum does random writes. That being said, we can have
RT_ITERATE_NEXT() return key-value pairs in an order regardless of how
the key chunks are stored in a node.

> psql -c "select rt_load_ms, rt_search_ms from bench_seq_search(0, 1 * 1000 * 1000)"
> (min load time of three)
>
> v15:
> Note: The regression seems to have started in v17, which is the first with a full template.
>
> Nothing so far has helped here, and previous experience has shown that trying to profile 100ms will not be useful. Instead of putting more effort into diving deeper, it seems a better use of time to write a benchmark that calls the tid store itself. That's more realistic, since this function was intended to test load and search of tids, but the tid store doesn't quite operate so simply anymore. What do you think, Masahiko?

Yeah, that's more realistic. TidStore now encodes TIDs slightly
differently from the benchmark test.

I've attached the patch that adds a simple benchmark test using
TidStore. With this test, I got similar trends of results to yours
with gcc, but I've not analyzed them in depth yet.

query: select * from bench_tidstore_load(0, 10 * 1000 * 1000)

v15:
 load_ms
---------
     816

load_ms
---------
     839

 load_ms
---------
     820

BTW it would be better to remove the RT_DEBUG macro from bench_radix_tree.c.

>
> I'm inclined to keep 0006, because it might give a slight boost, and 0007 because it's never a bad idea to remove dead code.

Yeah, these two changes make sense to me too.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
From e056133360436e115a434a8a21685a99602a5b5d Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Wed, 8 Feb 2023 15:53:14 +0900
Subject: [PATCH] Add bench_tidstore_load()

---
 .../bench_radix_tree--1.0.sql                 | 10 ++++
 contrib/bench_radix_tree/bench_radix_tree.c   | 46 +++++++++++++++++++
 2 files changed, 56 insertions(+)

diff --git a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
index 95eedbbe10..fbf51c1086 100644
--- a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
+++ b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
@@ -75,3 +75,13 @@ OUT rt_sparseload_ms int8
 returns record
 as 'MODULE_PATHNAME'
 LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_tidstore_load(
+minblk int4,
+maxblk int4,
+OUT mem_allocated int8,
+OUT load_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
index 7d1e2eee57..3c2caa3b90 100644
--- a/contrib/bench_radix_tree/bench_radix_tree.c
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -9,6 +9,7 @@
  */
 #include "postgres.h"
 
+#include "access/tidstore.h"
 #include "common/pg_prng.h"
 #include "fmgr.h"
 #include "funcapi.h"
@@ -54,6 +55,7 @@ PG_FUNCTION_INFO_V1(bench_load_random_int);
 PG_FUNCTION_INFO_V1(bench_fixed_height_search);
 PG_FUNCTION_INFO_V1(bench_search_random_nodes);
 PG_FUNCTION_INFO_V1(bench_node128_load);
+PG_FUNCTION_INFO_V1(bench_tidstore_load);
 
 static uint64
 tid_to_key_off(ItemPointer tid, uint32 *off)
@@ -168,6 +170,50 @@ vac_cmp_itemptr(const void *left, const void *right)
 }
 #endif
 
+Datum
+bench_tidstore_load(PG_FUNCTION_ARGS)
+{
+	BlockNumber minblk = PG_GETARG_INT32(0);
+	BlockNumber maxblk = PG_GETARG_INT32(1);
+	TidStore	*ts;
+	OffsetNumber *offs;
+	TimestampTz start_time,
+				end_time;
+	long		secs;
+	int			usecs;
+	int64		load_ms;
+	TupleDesc	tupdesc;
+	Datum		values[2];
+	bool		nulls[2] = {false};
+
+	/* Build a tuple descriptor for our result type */
+	if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+		elog(ERROR, "return type must be a row type");
+
+	offs = palloc(sizeof(OffsetNumber) * TIDS_PER_BLOCK_FOR_LOAD);
+	for (int i = 0; i < TIDS_PER_BLOCK_FOR_LOAD; i++)
+		offs[i] = i + 1; /* FirstOffsetNumber is 1 */
+
+	ts = tidstore_create(1 * 1024L * 1024L * 1024L, MaxHeapTuplesPerPage, NULL);
+
+	elog(NOTICE, "sleeping for 2 seconds...");
+	pg_usleep(2 * 1000000L);
+
+	/* load tids */
+	start_time = GetCurrentTimestamp();
+	for (BlockNumber blkno = minblk; blkno < maxblk; blkno++)
+		tidstore_add_tids(ts, blkno, offs, TIDS_PER_BLOCK_FOR_LOAD);
+	end_time = GetCurrentTimestamp();
+	TimestampDifference(start_time, end_time, &secs, &usecs);
+	load_ms = secs * 1000 + usecs / 1000;
+
+	values[0] = Int64GetDatum(tidstore_memory_usage(ts));
+	values[1] = Int64GetDatum(load_ms);
+
+	tidstore_destroy(ts);
+	PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
 static Datum
 bench_search(FunctionCallInfo fcinfo, bool shuffle)
 {
-- 
2.31.1

---

From: John Naylor <[email-masked]>
Date: Thu, 9 Feb 2023 19:56:27 +0700

> the key chunks are stored in a node.

Okay, we can keep that possibility in mind if we need to go there.

It just occurred to me that these facts might be related. v17 was the first
use of the full template, and I decided then I liked one of your earlier
patches where replace_node() calls node_update_inner() better than calling
node_insert_inner() with a NULL parent, which was a bit hard to understand.
That now-dead code was actually used in the latter case for updating the
(original) parent. It's possible that trying to use separate paths
contributed to the regression. I'll try the other way and report back.

> with gcc, but I've not analyzed them in depth yet.

Thanks for that! I'll take a look.

> BTW it would be better to remove the RT_DEBUG macro from
bench_radix_tree.c.

Absolutely.

---

From: John Naylor <[email-masked]>
Date: Fri, 10 Feb 2023 13:51:24 +0700

> query: select * from bench_tidstore_load(0, 10 * 1000 * 1000)

How did you build the tid store and test on v15? I first tried to
apply v15-0009-PoC-lazy-vacuum-integration.patch, which conflicts with
vacuum now, so reset all that, but still getting build errors because the
tid store types and functions have changed.

---

From: Masahiko Sawada <[email-masked]>
Date: Fri, 10 Feb 2023 16:15:46 +0900

On Fri, Feb 10, 2023 at 3:51 PM John Naylor
<[email-masked]> wrote:
>
>
> On Thu, Feb 9, 2023 at 2:08 PM Masahiko Sawada <[email-masked]> wrote:
> >
> How did you build the tid store and test on v15? I first tried to apply v15-0009-PoC-lazy-vacuum-integration.patch, which conflicts with vacuum now, so reset all that, but still getting build errors because the tid store types and functions have changed.

I applied v26-0008-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch
on top of v15 radix tree and changed the TidStore so that it uses v15
(non-templated) radixtree. That way, we can test TidStore using v15
radix tree. I've attached the patch that I applied on top of
v26-0008-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch.

Regards,

---

From: John Naylor <[email-masked]>
Date: Sat, 11 Feb 2023 12:33:42 +0700

I didn't get any closer to radix-tree regression, but I did find some
inefficiencies in tidstore_add_tids() that are worth talking about first,
addressed in a rough fashion in the attached .txt addendums that I can
clean up and incorporate later.

To start, I can reproduce the regression with this test as well:

select * from bench_tidstore_load(0, 10 * 1000 * 1000);

v15 + v26 store + adjustments:
 mem_allocated | load_ms
---------------+---------
      98202152 |    1676

v26 0001-0008
 mem_allocated | load_ms
---------------+---------
      98202032 |    1826

...and reverting to the alternate way to update the parent didn't help:

v26 0001-6, 0008, insert_inner w/ null parent

 mem_allocated | load_ms
---------------+---------
      98202032 |    1825

...and I'm kind of glad that wasn't the problem, because going back to that
would be a pain for the shmem case.

Running perf doesn't show anything much different in the proportions (note
that rt_set must have been inlined when declared locally in v26):

v15 + v26 store + adjustments:
  65.88%  postgres  postgres             [.] tidstore_add_tids
  10.74%  postgres  postgres             [.] rt_set
   9.20%  postgres  postgres             [.] palloc0
   6.49%  postgres  postgres             [.] rt_node_insert_leaf

v26 0001-0008
  78.50%  postgres  postgres             [.] tidstore_add_tids
   8.88%  postgres  postgres             [.] palloc0
   6.24%  postgres  postgres             [.] local_rt_node_insert_leaf

v2699-0001: The first thing I noticed is that palloc0 is taking way more
time than it should, and it's because the compiler doesn't know the
values[] array is small. One reason we need to zero the array is to make
the algorithm agnostic about what order the offsets come in, as I requested
in a previous review. Thinking some more, I was way too paranoid about
that. As long as access methods scan the line pointer array in the usual
way, maybe we can just assert that the keys we create are in order, and
zero any unused array entries as we find them. (I admit I can't actually
think of a reason we would ever encounter offsets out of order.) Also, we
can keep track of the last key we need to consider for insertion into the
radix tree, and ignore the rest. That might shave a few cycles during the
exclusive lock when the max offset of an LP_DEAD item < 64 on a given page,
which I think would be common in the wild. I also got rid of the special
case for non-encoding, since shifting by zero should work the same way.
These together led to a nice speedup on the v26 branch:

 mem_allocated | load_ms
---------------+---------
      98202032 |    1386

v2699-0002: The next thing I noticed is forming a full ItemIdPointer to
pass to tid_to_key_off(). That's bad for tidstore_add_tids() because
ItemPointerSetBlockNumber() must do this in order to allow the struct to be
SHORTALIGN'd:

static inline void
BlockIdSet(BlockIdData *blockId, BlockNumber blockNumber)
{
blockId->bi_hi = blockNumber >> 16;
blockId->bi_lo = blockNumber & 0xffff;
}

Then, tid_to_key_off() calls ItemPointerGetBlockNumber(), which must
reverse the above process:

static inline BlockNumber
BlockIdGetBlockNumber(const BlockIdData *blockId)
{
return (((BlockNumber) blockId->bi_hi) << 16) | ((BlockNumber)
blockId->bi_lo);
}

There is no reason to do any of this if we're not reading/writing directly
to/from an on-disk tid etc. To avoid this, I created a new function
encode_key_off() [name could be better], which deals with the raw block
number that we already have. Then turn tid_to_key_off() into a wrapper
around that, since we still need the full conversion for
tidstore_lookup_tid().

v2699-0003: Get rid of all the remaining special cases for encoding/or not.
I am unaware of the need to optimize that case or treat it in any way
differently. I haven't tested this on an installation with non-default
blocksize and didn't measure this separately, but 0002+0003 gives:

 mem_allocated | load_ms
---------------+---------
      98202032 |    1259

If these are acceptable, I can incorporate them into a later patchset. In
any case, speeding up tidstore_add_tids() will make any regressions in the
backing radix tree more obvious. I will take a look at that next week.

--
John Naylor
EDB: http://www.enterprisedb.com
From 6bdd33fa4f55757b54d16ce00dc60a21b929606e Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Sat, 11 Feb 2023 10:45:21 +0700
Subject: [PATCH v2699 2/3] Do less work when encoding key/value

---
 src/backend/access/common/tidstore.c | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)

diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 5d24680737..3d384cf645 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -159,6 +159,7 @@ typedef struct TidStoreIter
 
 static void tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val);
 static inline BlockNumber key_get_blkno(TidStore *ts, uint64 key);
+static inline uint64 encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint32 *off);
 static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off);
 
 /*
@@ -367,7 +368,6 @@ void
 tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
 				  int num_offsets)
 {
-	ItemPointerData tid;
 	uint64	*values;
 	uint64	key;
 	uint64	prev_key;
@@ -381,16 +381,12 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
 	values = palloc(sizeof(uint64) * nkeys);
 	key = prev_key = key_base;
 
-	ItemPointerSetBlockNumber(&tid, blkno);
-
 	for (int i = 0; i < num_offsets; i++)
 	{
 		uint32	off;
 
-		ItemPointerSetOffsetNumber(&tid, offsets[i]);
-
 		/* encode the tid to key and val */
-		key = tid_to_key_off(ts, &tid, &off);
+		key = encode_key_off(ts, blkno, offsets[i], &off);
 
 		/* make sure we scanned the line pointer array in order */
 		Assert(key >= prev_key);
@@ -681,20 +677,29 @@ key_get_blkno(TidStore *ts, uint64 key)
 /* Encode a tid to key and offset */
 static inline uint64
 tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off)
+{
+	uint32 offset = ItemPointerGetOffsetNumber(tid);
+	BlockNumber block = ItemPointerGetBlockNumber(tid);
+
+	return encode_key_off(ts, block, offset, off);
+}
+
+/* encode a block and offset to a key and partial offset */
+static inline uint64
+encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint32 *off)
 {
 	uint64 key;
 	uint64 tid_i;
 
 	if (!ts->control->encode_tids)
 	{
-		*off = ItemPointerGetOffsetNumber(tid);
+		*off = offset;
 
 		/* Use the block number as the key */
-		return (int64) ItemPointerGetBlockNumber(tid);
+		return (int64) block;
 	}
 
-	tid_i = ItemPointerGetOffsetNumber(tid);
-	tid_i |= (uint64) ItemPointerGetBlockNumber(tid) << ts->control->offset_nbits;
+	tid_i = offset | ((uint64) block << ts->control->offset_nbits);
 
 	*off = tid_i & ((UINT64CONST(1) << TIDSTORE_VALUE_NBITS) - 1);
 	key = tid_i >> TIDSTORE_VALUE_NBITS;
-- 
2.39.1

From c0bc497f50318c8e31ccdf0c2a9186ffc736abeb Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Fri, 10 Feb 2023 19:56:01 +0700
Subject: [PATCH v2699 1/3] Miscellaneous optimizations for tidstore_add_tids()

- remove palloc0; it's expensive for lengths not known at compile-time
- optimize for case with only one key per heap block
- make some intializations const and branch-free
- when writing to the radix tree, stop at the last non-zero bitmap
---
 src/backend/access/common/tidstore.c | 56 ++++++++++++++++++----------
 1 file changed, 36 insertions(+), 20 deletions(-)

diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 4c72673ce9..5d24680737 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -368,51 +368,67 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
 				  int num_offsets)
 {
 	ItemPointerData tid;
-	uint64	key_base;
 	uint64	*values;
-	int	nkeys;
+	uint64	key;
+	uint64	prev_key;
+	uint64	off_bitmap = 0;
+	int idx;
+	const uint64 key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
+	const int nkeys = UINT64CONST(1) << ts->control->offset_key_nbits;
 
 	Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
 
-	if (ts->control->encode_tids)
-	{
-		key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
-		nkeys = UINT64CONST(1) << ts->control->offset_key_nbits;
-	}
-	else
-	{
-		key_base = (uint64) blkno;
-		nkeys = 1;
-	}
-	values = palloc0(sizeof(uint64) * nkeys);
+	values = palloc(sizeof(uint64) * nkeys);
+	key = prev_key = key_base;
 
 	ItemPointerSetBlockNumber(&tid, blkno);
+
 	for (int i = 0; i < num_offsets; i++)
 	{
-		uint64	key;
 		uint32	off;
-		int idx;
 
 		ItemPointerSetOffsetNumber(&tid, offsets[i]);
 
 		/* encode the tid to key and val */
 		key = tid_to_key_off(ts, &tid, &off);
 
-		idx = key - key_base;
-		Assert(idx >= 0 && idx < nkeys);
+		/* make sure we scanned the line pointer array in order */
+		Assert(key >= prev_key);
 
-		values[idx] |= UINT64CONST(1) << off;
+		if (key > prev_key)
+		{
+			idx = prev_key - key_base;
+			Assert(idx >= 0 && idx < nkeys);
+
+			/* write out offset bitmap for this key */
+			values[idx] = off_bitmap;
+
+			/* zero out any gaps up to the current key */
+			for (int empty_idx = idx + 1; empty_idx < key - key_base; empty_idx++)
+				values[empty_idx] = 0;
+
+			/* reset for current key -- the current offset will be handled below */
+			off_bitmap = 0;
+			prev_key = key;
+		}
+
+		off_bitmap |= UINT64CONST(1) << off;
 	}
 
+	/* save the final index for later */
+	idx = key - key_base;
+	/* write out last offset bitmap */
+	values[idx] = off_bitmap;
+
 	if (TidStoreIsShared(ts))
 		LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
 
 	/* insert the calculated key-values to the tree */
-	for (int i = 0; i < nkeys; i++)
+	for (int i = 0; i <= idx; i++)
 	{
 		if (values[i])
 		{
-			uint64 key = key_base + i;
+			key = key_base + i;
 
 			if (TidStoreIsShared(ts))
 				shared_rt_set(ts->tree.shared, key, &values[i]);
-- 
2.39.1

From 82c1f639aaa64cc943af3b53294a63d5d8f7a9b9 Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Sat, 11 Feb 2023 11:51:32 +0700
Subject: [PATCH v2699 3/3] Force all callers to encode, no matter how small
 the expected offset

---
 src/backend/access/common/tidstore.c | 36 +++++-----------------------
 1 file changed, 6 insertions(+), 30 deletions(-)

diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 3d384cf645..ff8e66936e 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -99,7 +99,6 @@ typedef struct TidStoreControl
 	size_t	max_bytes;		/* the maximum bytes a TidStore can use */
 	int		max_offset;		/* the maximum offset number */
 	int		offset_nbits;	/* the number of bits required for max_offset */
-	bool	encode_tids;	/* do we use tid encoding? */
 	int		offset_key_nbits;	/* the number of bits of a offset number
 								 * used for the key */
 
@@ -224,21 +223,15 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
 	ts->control->max_offset = max_offset;
 	ts->control->offset_nbits = pg_ceil_log2_32(max_offset);
 
+	if (ts->control->offset_nbits < TIDSTORE_VALUE_NBITS)
+		ts->control->offset_nbits = TIDSTORE_VALUE_NBITS;
+
 	/*
 	 * We use tid encoding if the number of bits for the offset number doesn't
 	 * fix in a value, uint64.
 	 */
-	if (ts->control->offset_nbits > TIDSTORE_VALUE_NBITS)
-	{
-		ts->control->encode_tids = true;
-		ts->control->offset_key_nbits =
-			ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
-	}
-	else
-	{
-		ts->control->encode_tids = false;
-		ts->control->offset_key_nbits = 0;
-	}
+	ts->control->offset_key_nbits =
+		ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
 
 	return ts;
 }
@@ -643,12 +636,6 @@ tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
 		uint64	tid_i;
 		OffsetNumber	off;
 
-		if (i > iter->ts->control->max_offset)
-		{
-			Assert(!iter->ts->control->encode_tids);
-			break;
-		}
-
 		if ((val & (UINT64CONST(1) << i)) == 0)
 			continue;
 
@@ -668,10 +655,7 @@ tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
 static inline BlockNumber
 key_get_blkno(TidStore *ts, uint64 key)
 {
-	if (ts->control->encode_tids)
-		return (BlockNumber) (key >> ts->control->offset_key_nbits);
-
-	return (BlockNumber) key;
+	return (BlockNumber) (key >> ts->control->offset_key_nbits);
 }
 
 /* Encode a tid to key and offset */
@@ -691,14 +675,6 @@ encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint32 *off)
 	uint64 key;
 	uint64 tid_i;
 
-	if (!ts->control->encode_tids)
-	{
-		*off = offset;
-
-		/* Use the block number as the key */
-		return (int64) block;
-	}
-
 	tid_i = offset | ((uint64) block << ts->control->offset_nbits);
 
 	*off = tid_i & ((UINT64CONST(1) << TIDSTORE_VALUE_NBITS) - 1);
-- 
2.39.1

---

From: Masahiko Sawada <[email-masked]>
Date: Mon, 13 Feb 2023 16:50:57 +0900

On Sat, Feb 11, 2023 at 2:33 PM John Naylor
<[email-masked]> wrote:
>
> I didn't get any closer to radix-tree regression,

Me neither. It seems that in v26, inserting chunks into node-32 is
slow but needs more analysis. I'll share if I found something
interesting.

> but I did find some inefficiencies in tidstore_add_tids() that are worth talking about first, addressed in a rough fashion in the attached .txt addendums that I can clean up and incorporate later.
>
> ...and I'm kind of glad that wasn't the problem, because going back to that would be a pain for the shmem case.
>
> Running perf doesn't show anything much different in the proportions (note that rt_set must have been inlined when declared locally in v26):
>
> v2699-0001: The first thing I noticed is that palloc0 is taking way more time than it should, and it's because the compiler doesn't know the values[] array is small. One reason we need to zero the array is to make the algorithm agnostic about what order the offsets come in, as I requested in a previous review. Thinking some more, I was way too paranoid about that. As long as access methods scan the line pointer array in the usual way, maybe we can just assert that the keys we create are in order, and zero any unused array entries as we find them. (I admit I can't actually think of a reason we would ever encounter offsets out of order.)

I can think that something like traversing a HOT chain could visit
offsets out of order. But fortunately we prune such collected TIDs
before heap vacuum in heap case.

> Also, we can keep track of the last key we need to consider for insertion into the radix tree, and ignore the rest. That might shave a few cycles during the exclusive lock when the max offset of an LP_DEAD item < 64 on a given page, which I think would be common in the wild. I also got rid of the special case for non-encoding, since shifting by zero should work the same way. These together led to a nice speedup on the v26 branch:
>
>  mem_allocated | load_ms
> v2699-0002: The next thing I noticed is forming a full ItemIdPointer to pass to tid_to_key_off(). That's bad for tidstore_add_tids() because ItemPointerSetBlockNumber() must do this in order to allow the struct to be SHORTALIGN'd:
>
> static inline void
> Then, tid_to_key_off() calls ItemPointerGetBlockNumber(), which must reverse the above process:
>
> return (((BlockNumber) blockId->bi_hi) << 16) | ((BlockNumber) blockId->bi_lo);
> }
>
> There is no reason to do any of this if we're not reading/writing directly to/from an on-disk tid etc. To avoid this, I created a new function encode_key_off() [name could be better], which deals with the raw block number that we already have. Then turn tid_to_key_off() into a wrapper around that, since we still need the full conversion for tidstore_lookup_tid().
>
> v2699-0003: Get rid of all the remaining special cases for encoding/or not. I am unaware of the need to optimize that case or treat it in any way differently. I haven't tested this on an installation with non-default blocksize and didn't measure this separately, but 0002+0003 gives:
>
>  mem_allocated | load_ms
> If these are acceptable, I can incorporate them into a later patchset.

These are nice improvements! I agree with all changes.

Regards,

---

From: John Naylor <[email-masked]>
Date: Tue, 14 Feb 2023 18:24:38 +0700

> slow but needs more analysis. I'll share if I found something

If that were the case, then the other benchmarks I ran would likely have
slowed down as well, but they are the same or faster. There is one
microbenchmark I didn't run before: "select * from
bench_fixed_height_search(15)" (15 to reduce noise from growing size class,
and despite the name it measures load time as well). Trying this now shows
no difference: a few runs range 19 to 21ms in each version. That also
reinforces that update_inner is fine and that the move to value pointer API
didn't regress.

Changing TIDS_PER_BLOCK_FOR_LOAD to 1 to stress the tree more gives (min of
5, perf run separate from measurements):

v15 + v26 store:

 mem_allocated | load_ms
      98202152 |     553

  19.71%  postgres  postgres             [.] tidstore_add_tids
+ 31.47%  postgres  postgres             [.] rt_set
= 51.18%

  20.62%  postgres  postgres             [.] rt_node_insert_leaf
   6.05%  postgres  postgres             [.] AllocSetAlloc
   4.74%  postgres  postgres             [.] AllocSetFree
   4.62%  postgres  postgres             [.] palloc
   2.23%  postgres  postgres             [.] SlabAlloc

v26:

 mem_allocated | load_ms
      98202032 |     617

  57.45%  postgres  postgres             [.] tidstore_add_tids

  20.67%  postgres  postgres             [.] local_rt_node_insert_leaf
   5.99%  postgres  postgres             [.] AllocSetAlloc
   3.55%  postgres  postgres             [.] palloc
   3.05%  postgres  postgres             [.] AllocSetFree
   2.05%  postgres  postgres             [.] SlabAlloc

So it seems the store itself got faster when we removed shared memory paths
from the v26 store to test it against v15.

I thought to favor the local memory case in the tidstore by controlling
inlining -- it's smaller and will be called much more often, so I tried the
following (done in 0007)

 #define RT_PREFIX shared_rt
 #define RT_SHMEM
-#define RT_SCOPE static
+#define RT_SCOPE static pg_noinline

That brings it down to

 mem_allocated | load_ms
      98202032 |     590

That's better, but not still not within noise level. Perhaps some slowdown
is unavoidable, but it would be nice to understand why.

> before heap vacuum in heap case.

Further, currently we *already* assume we populate the tid array in order
(for binary search), so we can just continue assuming that (with an assert
added since it's more public in this form). I'm not sure why such basic
common sense evaded me a few versions ago...

> These are nice improvements! I agree with all changes.

Great, I've squashed these into the tidstore patch (0004). Also added 0005,
which is just a simplification.

I squashed the earlier dead code removal into the radix tree patch.

v27-0008 measures tid store iteration performance and adds a stub function
to prevent spurious warnings, so the benchmarking module can always be
built.

Getting the list of offsets from the old array for a given block is always
trivial, but tidstore_iter_extract_tids() is doing a huge amount of
unnecessary work when TIDS_PER_BLOCK_FOR_LOAD is 1, enough to exceed the
load time:

 mem_allocated | load_ms | iter_ms
---------------+---------+---------
      98202032 |     589 |     915

Fortunately, it's an easy fix, done in 0009.

 mem_allocated | load_ms | iter_ms
---------------+---------+---------
      98202032 |     589 |     153

I'll soon resume more cosmetic review of the tid store, but this is enough
to post.

--
John Naylor
EDB: http://www.enterprisedb.com
From d577ef9d9755e7ca4d3722c1a044381a81d66244 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Mon, 24 Oct 2022 14:07:09 +0900
Subject: [PATCH v27 1/9] Introduce helper SIMD functions for small byte arrays

Masahiko Sawada

 

+/*
 
+/*

From 149a49f51f7a16b7c1eb762e704f1ec476ecb65a Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Tue, 6 Dec 2022 13:39:41 +0700
Subject: [PATCH v27 2/9] Move some bitmap logic out of bitmapset.c

 
-/*----------

 /*
 
-			w = RIGHTMOST_ONE(w);

 
 #else

 #endif

 
 /*
 
+/*----------
index 36d1dc0117..a0c60feade 100644
@@ -3669,7 +3669,6 @@ shmem_request_hook_type
 shmem_startup_hook_type
 sig_atomic_t
 sigjmp_buf
-signedbitmapword
 sigset_t
 size_t
 slist_head
-- 
2.39.1

From dba9497b5b587da873fbb2de89570ec8b36d604b Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Sun, 12 Feb 2023 15:17:40 +0700
Subject: [PATCH v27 5/9] Do bitmap conversion in one place rather than forcing
 callers to do it

---
 src/backend/access/common/tidstore.c | 31 +++++++++++++++-------------
 1 file changed, 17 insertions(+), 14 deletions(-)

diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index ff8e66936e..ad8c0866e2 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -70,6 +70,7 @@
  * and value, respectively.
  */
 #define TIDSTORE_VALUE_NBITS	6	/* log(64, 2) */
+#define TIDSTORE_OFFSET_MASK ((1 << TIDSTORE_VALUE_NBITS) - 1)
 
 /* A magic value used to identify our TidStores. */
 #define TIDSTORE_MAGIC 0x826f6a10
@@ -158,8 +159,8 @@ typedef struct TidStoreIter
 
 static void tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val);
 static inline BlockNumber key_get_blkno(TidStore *ts, uint64 key);
-static inline uint64 encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint32 *off);
-static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off);
+static inline uint64 encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit);
+static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit);
 
 /*
  * Create a TidStore. The returned object is allocated in backend-local memory.
@@ -376,10 +377,10 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
 
-		uint32	off;
+		uint64	off_bit;
 
 		/* encode the tid to key and val */
-		key = encode_key_off(ts, blkno, offsets[i], &off);
+		key = encode_key_off(ts, blkno, offsets[i], &off_bit);
 
 		/* make sure we scanned the line pointer array in order */
 		Assert(key >= prev_key);
@@ -401,7 +402,7 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
 			prev_key = key;
 		}
 
-		off_bitmap |= UINT64CONST(1) << off;
+		off_bitmap |= off_bit;
 	}
 
 	/* save the final index for later */
@@ -441,10 +442,10 @@ tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
 {
 	uint64 key;
 	uint64 val = 0;
-	uint32 off;
+	uint64 off_bit;
 	bool found;
 
-	key = tid_to_key_off(ts, tid, &off);
+	key = tid_to_key_off(ts, tid, &off_bit);
 
 	if (TidStoreIsShared(ts))
 		found = shared_rt_search(ts->tree.shared, key, &val);
@@ -454,7 +455,7 @@ tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
 	if (!found)
 		return false;
 
-	return (val & (UINT64CONST(1) << off)) != 0;
+	return (val & off_bit) != 0;
 }
 
 /*
@@ -660,26 +661,28 @@ key_get_blkno(TidStore *ts, uint64 key)
 
 /* Encode a tid to key and offset */
 static inline uint64
-tid_to_key_off(TidStore *ts, ItemPointer tid, uint32 *off)
+tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit)
 {
 	uint32 offset = ItemPointerGetOffsetNumber(tid);
 	BlockNumber block = ItemPointerGetBlockNumber(tid);
 
-	return encode_key_off(ts, block, offset, off);
+	return encode_key_off(ts, block, offset, off_bit);
 }
 
 /* encode a block and offset to a key and partial offset */
 static inline uint64
-encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint32 *off)
+encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit)
 {
 	uint64 key;
 	uint64 tid_i;
+	uint32 off_lower;
 
-	tid_i = offset | ((uint64) block << ts->control->offset_nbits);
+	off_lower = offset & TIDSTORE_OFFSET_MASK;
+	Assert(off_lower < (sizeof(uint64) * BITS_PER_BYTE));
 
-	*off = tid_i & ((UINT64CONST(1) << TIDSTORE_VALUE_NBITS) - 1);
+	*off_bit = UINT64CONST(1) << off_lower;
+	tid_i = offset | ((uint64) block << ts->control->offset_nbits);
 	key = tid_i >> TIDSTORE_VALUE_NBITS;
-	Assert(*off < (sizeof(uint64) * BITS_PER_BYTE));
 
 	return key;
 }
-- 
2.39.1

From bf9d659187537b250683af321b0167d69c7fb18a Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Wed, 14 Sep 2022 12:38:51 +0000
Subject: [PATCH v27 3/9] Add radixtree template

 src/include/lib/radixtree_insert_impl.h       |  328 +++
 20 files changed, 4082 insertions(+)

+size_t
index 0000000000..1cdb995e54
index 0000000000..d56e58dcac
@@ -0,0 +1,328 @@

 

From 1be520c83274bc3a2f068689e665c254c8e3c04e Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Fri, 23 Dec 2022 23:50:39 +0900
Subject: [PATCH v27 4/9] Add TIDStore, to store sets of TIDs (ItemPointerData)
 efficiently.

 src/backend/access/common/tidstore.c          | 685 ++++++++++++++++++
 16 files changed, 1030 insertions(+)

index b246ddc634..e44387d2c1 100644
 
index 0000000000..ff8e66936e
@@ -0,0 +1,685 @@

From b0515a40b3aa4709047c7b70b9c0cadded979d15 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Fri, 16 Sep 2022 11:57:03 +0900
Subject: [PATCH v27 6/9] Tool for measuring radix tree and tidstore
 performance

XXX: Not for commit
---
 .../bench_radix_tree--1.0.sql                 |  87 +++
 contrib/bench_radix_tree/bench_radix_tree.c   | 717 ++++++++++++++++++
 8 files changed, 894 insertions(+)

index 0000000000..fbf51c1086
@@ -0,0 +1,87 @@
index 0000000000..b5ad75364c
@@ -0,0 +1,717 @@
+//#define RT_DEBUG

From 72bb462b1dab005cbc2aff265baedbaaee62cb2b Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Tue, 14 Feb 2023 17:02:53 +0700
Subject: [PATCH v27 8/9] Measure iteration of tidstore

---
 .../bench_radix_tree--1.0.sql                 |  3 +-
 contrib/bench_radix_tree/bench_radix_tree.c   | 40 ++++++++++++++++---
 contrib/meson.build                           |  2 +-
 3 files changed, 38 insertions(+), 7 deletions(-)

diff --git a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
index fbf51c1086..ad66265e23 100644
--- a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
+++ b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
@@ -80,7 +80,8 @@ create function bench_tidstore_load(
 minblk int4,
 maxblk int4,
 OUT mem_allocated int8,
-OUT load_ms int8
+OUT load_ms int8,
+OUT iter_ms int8
 )
 returns record
 as 'MODULE_PATHNAME'
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
index b5ad75364c..6e5149e2c4 100644
--- a/contrib/bench_radix_tree/bench_radix_tree.c
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -176,15 +176,18 @@ bench_tidstore_load(PG_FUNCTION_ARGS)
 	BlockNumber minblk = PG_GETARG_INT32(0);
 	BlockNumber maxblk = PG_GETARG_INT32(1);
 	TidStore	*ts;
+	TidStoreIter *iter;
+	TidStoreIterResult *result;
 	OffsetNumber *offs;
 	TimestampTz start_time,
 				end_time;
 	long		secs;
 	int			usecs;
 	int64		load_ms;
+	int64		iter_ms;
 	TupleDesc	tupdesc;
-	Datum		values[2];
-	bool		nulls[2] = {false};
+	Datum		values[3];
+	bool		nulls[3] = {false};
 
 	/* Build a tuple descriptor for our result type */
 	if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
@@ -196,9 +199,6 @@ bench_tidstore_load(PG_FUNCTION_ARGS)
 
 	ts = tidstore_create(1 * 1024L * 1024L * 1024L, MaxHeapTuplesPerPage, NULL);
 
-	elog(NOTICE, "sleeping for 2 seconds...");
-	pg_usleep(2 * 1000000L);
-
 	/* load tids */
 	start_time = GetCurrentTimestamp();
 	for (BlockNumber blkno = minblk; blkno < maxblk; blkno++)
@@ -207,8 +207,22 @@ bench_tidstore_load(PG_FUNCTION_ARGS)
 	TimestampDifference(start_time, end_time, &secs, &usecs);
 	load_ms = secs * 1000 + usecs / 1000;
 
+	elog(NOTICE, "sleeping for 2 seconds...");
+	pg_usleep(2 * 1000000L);
+
+	/* iterate through tids */
+	iter = tidstore_begin_iterate(ts);
+	start_time = GetCurrentTimestamp();
+	while ((result = tidstore_iterate_next(iter)) != NULL)
+		;
+	tidstore_end_iterate(iter);
+	end_time = GetCurrentTimestamp();
+	TimestampDifference(start_time, end_time, &secs, &usecs);
+	iter_ms = secs * 1000 + usecs / 1000;
+
 	values[0] = Int64GetDatum(tidstore_memory_usage(ts));
 	values[1] = Int64GetDatum(load_ms);
+	values[2] = Int64GetDatum(iter_ms);
 
 	tidstore_destroy(ts);
 	PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
@@ -715,3 +729,19 @@ bench_node128_load(PG_FUNCTION_ARGS)
 	rt_free(rt);
 	PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
 }
+
+/* to silence warnings about unused iter functions */
+static void pg_attribute_unused()
+stub_iter()
+{
+	rt_radix_tree *rt;
+	rt_iter *iter;
+	uint64 key = 1;
+	uint64 value = 1;
+
+	rt = rt_create(CurrentMemoryContext);
+
+	iter = rt_begin_iterate(rt);
+	rt_iterate_next(iter, &key, &value);
+	rt_end_iterate(iter);
+}
\ No newline at end of file
diff --git a/contrib/meson.build b/contrib/meson.build
index 52253de793..421d469f8c 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,7 +12,7 @@ subdir('amcheck')
 subdir('auth_delay')
 subdir('auto_explain')
 subdir('basic_archive')
-#subdir('bench_radix_tree')
+subdir('bench_radix_tree')
 subdir('bloom')
 subdir('basebackup_to_shell')
 subdir('bool_plperl')
-- 
2.39.1

From 54ab02eb2188382185436059ff6e7ad95d970c5d Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Tue, 14 Feb 2023 17:00:31 +0700
Subject: [PATCH v27 7/9] Prevent inlining of interface functions for shmem

---
 src/backend/access/common/tidstore.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index ad8c0866e2..d1b4675ea4 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -84,7 +84,7 @@
 
 #define RT_PREFIX shared_rt
 #define RT_SHMEM
-#define RT_SCOPE static
+#define RT_SCOPE static pg_noinline
 #define RT_DECLARE
 #define RT_DEFINE
 #define RT_VALUE_TYPE uint64
-- 
2.39.1

From 8ccc66211973bcc44a6bad45c05302ca743c1489 Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Tue, 14 Feb 2023 17:53:37 +0700
Subject: [PATCH v27 9/9] Speed up tidstore_iter_extract_tids()

---
 src/backend/access/common/tidstore.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index d1b4675ea4..5a897c01f7 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -632,21 +632,21 @@ tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
 {
 	TidStoreIterResult *result = (&iter->result);
 
-	for (int i = 0; i < sizeof(uint64) * BITS_PER_BYTE; i++)
+	while (val)
 	{
 		uint64	tid_i;
 		OffsetNumber	off;
 
-		if ((val & (UINT64CONST(1) << i)) == 0)
-			continue;
-
 		tid_i = key << TIDSTORE_VALUE_NBITS;
-		tid_i |= i;
+		tid_i |= pg_rightmost_one_pos64(val);
 
 		off = tid_i & ((UINT64CONST(1) << iter->ts->control->offset_nbits) - 1);
 
 		Assert(result->num_offsets < iter->ts->control->max_offset);
 		result->offsets[result->num_offsets++] = off;
+
+		/* unset the rightmost bit */
+		val &= ~pg_rightmost_one64(val);
 	}
 
 	result->blkno = key_get_blkno(iter->ts, key);
-- 
2.39.1

---

From: John Naylor <[email-masked]>
Date: Tue, 14 Feb 2023 19:36:51 +0700

The benchmark module shouldn't have been un-commented-out, so attached a
revert of that.

Subject: [PATCH v28 02/10] Move some bitmap logic out of bitmapset.c

 
-/*----------

 /*
 
-			w = RIGHTMOST_ONE(w);

 
 #else

 #endif

 
 /*
 
+/*----------

From d577ef9d9755e7ca4d3722c1a044381a81d66244 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Mon, 24 Oct 2022 14:07:09 +0900
Subject: [PATCH v28 01/10] Introduce helper SIMD functions for small byte
 arrays

Masahiko Sawada

 

+/*
 
+/*

From dba9497b5b587da873fbb2de89570ec8b36d604b Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Sun, 12 Feb 2023 15:17:40 +0700
Subject: [PATCH v28 05/10] Do bitmap conversion in one place rather than
 forcing callers to do it

---

 
 
 /*

 

 

 
 /*

 

 
 	return key;
 }
-- 
2.39.1

From 1be520c83274bc3a2f068689e665c254c8e3c04e Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Fri, 23 Dec 2022 23:50:39 +0900
Subject: [PATCH v28 04/10] Add TIDStore, to store sets of TIDs
 (ItemPointerData) efficiently.

 

From bf9d659187537b250683af321b0167d69c7fb18a Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Wed, 14 Sep 2022 12:38:51 +0000
Subject: [PATCH v28 03/10] Add radixtree template

 
+size_t

 

From b0515a40b3aa4709047c7b70b9c0cadded979d15 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Fri, 16 Sep 2022 11:57:03 +0900
Subject: [PATCH v28 06/10] Tool for measuring radix tree and tidstore
 performance

XXX: Not for commit
---

From 54ab02eb2188382185436059ff6e7ad95d970c5d Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Tue, 14 Feb 2023 17:00:31 +0700
Subject: [PATCH v28 07/10] Prevent inlining of interface functions for shmem

---

From 8ccc66211973bcc44a6bad45c05302ca743c1489 Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Tue, 14 Feb 2023 17:53:37 +0700
Subject: [PATCH v28 09/10] Speed up tidstore_iter_extract_tids()

---

 

 

From 72bb462b1dab005cbc2aff265baedbaaee62cb2b Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Tue, 14 Feb 2023 17:02:53 +0700
Subject: [PATCH v28 08/10] Measure iteration of tidstore

---

 

 
 	tidstore_destroy(ts);

From 42ba46f8073ee33bc5df6766f74f4c57587b070a Mon Sep 17 00:00:00 2001
From: John Naylor <[email-masked]>
Date: Tue, 14 Feb 2023 19:31:34 +0700
Subject: [PATCH v28 10/10] Revert building benchmark module for CI

---

diff --git a/contrib/meson.build b/contrib/meson.build
index 421d469f8c..52253de793 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,7 +12,7 @@ subdir('amcheck')
 subdir('auth_delay')
 subdir('auto_explain')
 subdir('basic_archive')
-subdir('bench_radix_tree')
+#subdir('bench_radix_tree')
 subdir('bloom')
 subdir('basebackup_to_shell')
 subdir('bool_plperl')
-- 
2.39.1

---

From: Masahiko Sawada <[email-masked]>
Date: Thu, 16 Feb 2023 12:24:16 +0900

On Tue, Feb 14, 2023 at 8:24 PM John Naylor
<[email-masked]> wrote:
>
> On Mon, Feb 13, 2023 at 2:51 PM Masahiko Sawada <[email-masked]> wrote:
> >
> If that were the case, then the other benchmarks I ran would likely have slowed down as well, but they are the same or faster. There is one microbenchmark I didn't run before: "select * from bench_fixed_height_search(15)" (15 to reduce noise from growing size class, and despite the name it measures load time as well). Trying this now shows no difference: a few runs range 19 to 21ms in each version. That also reinforces that update_inner is fine and that the move to value pointer API didn't regress.
>
> Changing TIDS_PER_BLOCK_FOR_LOAD to 1 to stress the tree more gives (min of 5, perf run separate from measurements):
>
> v15 + v26 store:
>
>  mem_allocated | load_ms
> So it seems the store itself got faster when we removed shared memory paths from the v26 store to test it against v15.
>
> I thought to favor the local memory case in the tidstore by controlling inlining -- it's smaller and will be called much more often, so I tried the following (done in 0007)
>

The improvement makes sense to me. I've also done the same test (with
changing TIDS_PER_BLOCK_FOR_LOAD to 1):

w/o 0007 patch:
      98202032 |     334 |     445
(1 row)

w/ 0007 patch:
      98202032 |     316 |     434
(1 row)

On the other hand, with TIDS_PER_BLOCK_FOR_LOAD being 30, the load
performance didn't improve:

w/0 0007 patch:
      98202032 |     601 |     608
(1 row)

w/ 0007 patch:
      98202032 |     610 |     606
(1 row)

That being said, it might be within noise level, so I agree with 0007 patch.

> Perhaps some slowdown is unavoidable, but it would be nice to understand why.

True.

> Further, currently we *already* assume we populate the tid array in order (for binary search), so we can just continue assuming that (with an assert added since it's more public in this form). I'm not sure why such basic common sense evaded me a few versions ago...

Right. TidStore is implemented not only for heap, so loading
out-of-order TIDs might be important in the future.

> Great, I've squashed these into the tidstore patch (0004). Also added 0005, which is just a simplification.
>

I've attached some small patches to improve the radix tree and tidstrore:

We have the following WIP comment in test_radixtree:

// WIP: compiles with warnings because rt_attach is defined but not used
// #define RT_SHMEM

How about unsetting RT_SCOPE to suppress warnings for unused rt_attach
and friends?

FYI I've briefly tested the TidStore with blocksize = 32kb, and it
seems to work fine.

> I squashed the earlier dead code removal into the radix tree patch.

Thanks!

>
> v27-0008 measures tid store iteration performance and adds a stub function to prevent spurious warnings, so the benchmarking module can always be built.
>
> Getting the list of offsets from the old array for a given block is always trivial, but tidstore_iter_extract_tids() is doing a huge amount of unnecessary work when TIDS_PER_BLOCK_FOR_LOAD is 1, enough to exceed the load time:
>

Cool!

>
> I'll soon resume more cosmetic review of the tid store, but this is enough to post.

Thanks!

You removed the vacuum integration patch from v27, is there any reason for that?

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
From f06557689f33d9b11be1083362fcce19665b4014 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Thu, 16 Feb 2023 12:18:22 +0900
Subject: [PATCH v2899 2/2] Small improvements for radixtree and tests.

---
 src/include/lib/radixtree.h                      |  2 +-
 src/test/modules/test_radixtree/test_radixtree.c | 13 ++++++++++---
 2 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index 1cdb995e54..e546bd705c 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -1622,7 +1622,7 @@ RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
 	/* Descend the tree until we reach a leaf node */
 	while (shift >= 0)
 	{
-		RT_PTR_ALLOC new_child = RT_INVALID_PTR_ALLOC;;
+		RT_PTR_ALLOC new_child = RT_INVALID_PTR_ALLOC;
 
 		child = RT_PTR_GET_LOCAL(tree, stored_child);
 
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
index f944945db9..afe53382f3 100644
--- a/src/test/modules/test_radixtree/test_radixtree.c
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -107,13 +107,12 @@ static const test_spec test_specs[] = {
 
 /* define the radix tree implementation to test */
 #define RT_PREFIX rt
-#define RT_SCOPE static
+#define RT_SCOPE
 #define RT_DECLARE
 #define RT_DEFINE
 #define RT_USE_DELETE
 #define RT_VALUE_TYPE TestValueType
-// WIP: compiles with warnings because rt_attach is defined but not used
-// #define RT_SHMEM
+/* #define RT_SHMEM */
 #include "lib/radixtree.h"

@@ -142,6 +141,8 @@ test_empty(void)
 #ifdef RT_SHMEM
 	int			tranche_id = LWLockNewTrancheId();
 	dsa_area   *dsa;
+
+	LWLockRegisterTranche(tranche_id, "test_radix_tree");
 	dsa = dsa_create(tranche_id);
 
 	radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
@@ -188,6 +189,8 @@ test_basic(int children, bool test_inner)
 #ifdef RT_SHMEM
 	int			tranche_id = LWLockNewTrancheId();
 	dsa_area   *dsa;
+
+	LWLockRegisterTranche(tranche_id, "test_radix_tree");
 	dsa = dsa_create(tranche_id);
 #endif
 
@@ -358,6 +361,8 @@ test_node_types(uint8 shift)
 #ifdef RT_SHMEM
 	int			tranche_id = LWLockNewTrancheId();
 	dsa_area   *dsa;
+
+	LWLockRegisterTranche(tranche_id, "test_radix_tree");
 	dsa = dsa_create(tranche_id);
 #endif
 
@@ -406,6 +411,8 @@ test_pattern(const test_spec * spec)
 #ifdef RT_SHMEM
 	int			tranche_id = LWLockNewTrancheId();
 	dsa_area   *dsa;
+
+	LWLockRegisterTranche(tranche_id, "test_radix_tree");
 	dsa = dsa_create(tranche_id);
 #endif
 
-- 
2.31.1

From f6ed6e18b2281cee96af98a39bdfc453117e6a21 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Thu, 16 Feb 2023 12:17:59 +0900
Subject: [PATCH v2899 1/2] comment update and test the shared tidstore.

---
 src/backend/access/common/tidstore.c          | 19 +++-------
 .../modules/test_tidstore/test_tidstore.c     | 37 +++++++++++++++++--
 2 files changed, 40 insertions(+), 16 deletions(-)

diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 015e3dea81..8c05e60d92 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -64,13 +64,9 @@
  * |---------------------------------------------| key
  *
  * The maximum height of the radix tree is 5 in this case.
- *
- * If the number of bits for offset number fits in a 64-bit value, we don't
- * encode tids but directly use the block number and the offset number as key
- * and value, respectively.
  */
 #define TIDSTORE_VALUE_NBITS	6	/* log(64, 2) */
-#define TIDSTORE_OFFSET_MASK ((1 << TIDSTORE_VALUE_NBITS) - 1)
+#define TIDSTORE_OFFSET_MASK	((1 << TIDSTORE_VALUE_NBITS) - 1)
 
 /* A magic value used to identify our TidStores. */
 #define TIDSTORE_MAGIC 0x826f6a10
@@ -99,9 +95,10 @@ typedef struct TidStoreControl
 	/* These values are never changed after creation */
 	size_t	max_bytes;		/* the maximum bytes a TidStore can use */
 	int		max_offset;		/* the maximum offset number */
-	int		offset_nbits;	/* the number of bits required for max_offset */
-	int		offset_key_nbits;	/* the number of bits of a offset number
-								 * used for the key */
+	int		offset_nbits;	/* the number of bits required for an offset
+							 * number */
+	int		offset_key_nbits;	/* the number of bits of an offset number
+								 * used in a key */
 
 	/* The below fields are used only in shared case */
 
@@ -227,10 +224,6 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
 	if (ts->control->offset_nbits < TIDSTORE_VALUE_NBITS)
 		ts->control->offset_nbits = TIDSTORE_VALUE_NBITS;
 
-	/*
-	 * We use tid encoding if the number of bits for the offset number doesn't
-	 * fix in a value, uint64.
-	 */
 	ts->control->offset_key_nbits =
 		ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
 
@@ -379,7 +372,7 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
 	{
 		uint64	off_bit;
 
-		/* encode the tid to key and val */
+		/* encode the tid to a key and partial offset */
 		key = encode_key_off(ts, blkno, offsets[i], &off_bit);
 
 		/* make sure we scanned the line pointer array in order */
diff --git a/src/test/modules/test_tidstore/test_tidstore.c b/src/test/modules/test_tidstore/test_tidstore.c
index 9b849ae8e8..9a1217f833 100644
--- a/src/test/modules/test_tidstore/test_tidstore.c
+++ b/src/test/modules/test_tidstore/test_tidstore.c
@@ -18,10 +18,13 @@
 #include "miscadmin.h"
 #include "storage/block.h"
 #include "storage/itemptr.h"
+#include "storage/lwlock.h"
 #include "utils/memutils.h"
 
 PG_MODULE_MAGIC;
 
+/* #define TEST_SHARED_TIDSTORE 1 */
+
 #define TEST_TIDSTORE_MAX_BYTES (2 * 1024 * 1024L) /* 2MB */
 
 PG_FUNCTION_INFO_V1(test_tidstore);
@@ -59,6 +62,18 @@ test_basic(int max_offset)
 	OffsetNumber offs[TEST_TIDSTORE_NUM_OFFSETS];
 	int blk_idx;
 
+#ifdef TEST_SHARED_TIDSTORE
+	int tranche_id = LWLockNewTrancheId();
+	dsa_area *dsa;
+
+	LWLockRegisterTranche(tranche_id, "test_tidstore");
+	dsa = dsa_create(tranche_id);
+
+	ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, dsa);
+#else
+	ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
+#endif
+
 	/* prepare the offset array */
 	offs[0] = FirstOffsetNumber;
 	offs[1] = FirstOffsetNumber + 1;
@@ -66,8 +81,6 @@ test_basic(int max_offset)
 	offs[3] = max_offset - 1;
 	offs[4] = max_offset;
 
-	ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
-
 	/* add tids */
 	for (int i = 0; i < TEST_TIDSTORE_NUM_BLOCKS; i++)
 		tidstore_add_tids(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
@@ -144,6 +157,10 @@ test_basic(int max_offset)
 	}
 
 	tidstore_destroy(ts);
+
+#ifdef TEST_SHARED_TIDSTORE
+	dsa_detach(dsa);
+#endif
 }
 
 static void
@@ -153,9 +170,19 @@ test_empty(void)
 	TidStoreIter *iter;
 	ItemPointerData tid;
 
-	elog(NOTICE, "testing empty tidstore");
+#ifdef TEST_SHARED_TIDSTORE
+	int tranche_id = LWLockNewTrancheId();
+	dsa_area *dsa;
+
+	LWLockRegisterTranche(tranche_id, "test_tidstore");
+	dsa = dsa_create(tranche_id);
 
+	ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, dsa);
+#else
 	ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
+#endif
+
+	elog(NOTICE, "testing empty tidstore");
 
 	ItemPointerSet(&tid, 0, FirstOffsetNumber);
 	if (tidstore_lookup_tid(ts, &tid))
@@ -180,6 +207,10 @@ test_empty(void)
 	tidstore_end_iterate(iter);
 
 	tidstore_destroy(ts);
+
+#ifdef TEST_SHARED_TIDSTORE
+	dsa_detach(dsa);
+#endif
 }
 
 Datum
-- 
2.31.1

---

From: John Naylor <[email-masked]>
Date: Thu, 16 Feb 2023 16:22:56 +0700

> > Further, currently we *already* assume we populate the tid array in
order (for binary search), so we can just continue assuming that (with an
assert added since it's more public in this form). I'm not sure why such
basic common sense evaded me a few versions ago...
> out-of-order TIDs might be important in the future.

That's what I was probably thinking about some weeks ago, but I'm having a
hard time imagining how it would come up, even for something like the
conveyor-belt concept.

> How about unsetting RT_SCOPE to suppress warnings for unused rt_attach

Sounds good to me, and the other fixes make sense as well.

> FYI I've briefly tested the TidStore with blocksize = 32kb, and it

That was on my list, so great! How about the other end -- nominally we
allow 512b. (In practice it won't matter, but this would make sure I didn't
mess anything up when forcing all MaxTuplesPerPage to encode.)

> You removed the vacuum integration patch from v27, is there any reason
for that?

Just an oversight.

Now for some general comments on the tid store...

+ * explicitly call tidstore_detach to free up backend-local memory
associated

Do we need to do anything for this todo?

It might help readability to have a concept of "off_upper/off_lower", just
so we can describe things more clearly. The key is block + off_upper, and
the value is a bitmap of all the off_lower bits. I hinted at that in my
addition of encode_key_off(). Along those lines, maybe
s/TIDSTORE_OFFSET_MASK/TIDSTORE_OFFSET_LOWER_MASK/. Actually, I'm not even
sure the TIDSTORE_ prefix is valuable for these local macros.

The word "value" as a variable name is pretty generic in this context, and
it might be better to call it the off_lower_bitmap, at least in some
places. The "key" doesn't have a good short term for naming, but in
comments we should make sure we're clear it's "block# + off_upper".

I'm not a fan of the name "tid_i", even as a temp variable -- maybe
"compressed_tid"?

maybe s/tid_to_key_off/encode_tid/ and s/encode_key_off/encode_block_offset/

It might be worth using typedefs for key and value type. Actually, since
key type is fixed for the foreseeable future, maybe the radix tree template
should define a key typedef?

The term "result" is probably fine within the tidstore, but as a public
name used by vacuum, it's not very descriptive. I don't have a good idea,
though.

Some files in backend/access use CamelCase for public functions, although
it's not consistent. I think doing that for tidstore would help
readability, since they would stand out from rt_* functions and vacuum
functions. It's a matter of taste, though.

I don't understand the control flow in tidstore_iterate_next(), or when
BlockNumberIsValid() is true. If this is the best way to code this, it
needs more commentary.

Some comments on vacuum:

I think we'd better get some real-world testing of this, fairly soon.

I had an idea: If it's not too much effort, it might be worth splitting it
into two parts: one that just adds the store (not caring about its memory
limits or progress reporting etc). During index scan, check both the new
store and the array and log a warning (we don't want to exit or crash,
better to try to investigate while live if possible) if the result doesn't
match. Then perhaps set up an instance and let something like TPC-C run for
a few days. The second patch would just restore the rest of the current
patch. That would help reassure us it's working as designed. Soon I plan to
do some measurements with vacuuming large tables to get some concrete
numbers that the community can get excited about.

We also want to verify that progress reporting works as designed and has no
weird corner cases.

  * autovacuum_work_mem) memory space to keep track of dead TIDs.  We
initially
...
+ * create a TidStore with the maximum bytes that can be used by the
TidStore.

This kind of implies that we allocate the maximum bytes upfront. I think
this sentence can be removed. We already mentioned in the previous
paragraph that we set an upper bound.

- (errmsg("table \"%s\": removed %lld dead item identifiers in %u pages",
- vacrel->relname, (long long) index, vacuumed_pages)));
+ (errmsg("table \"%s\": removed " UINT64_FORMAT "dead item identifiers in
%u pages",
+ vacrel->relname, tidstore_num_tids(vacrel->dead_items),
+ vacuumed_pages)));

I don't think the format string has to change, since num_tids was changed
back to int64 in an earlier patch version?

- * the memory space for storing dead items allocated in the DSM segment.
We
[a lot of whitespace adjustment]
+ * the shared TidStore. We launch parallel worker processes at the start of

The old comment still seems mostly ok? Maybe just s/DSM segment/DSA area/
or something else minor.

If we're starting from the minimum, "estimate" doesn't really describe it
anymore? Maybe "Initial size"?
What does dsa_minimum_size() work out to in practice? 1MB?
Also, I think PARALLEL_VACUUM_KEY_DSA is left over from an earlier patch.

Lastly, on the radix tree:

I find extend, set, and set_extend hard to keep straight when studying the
code. Maybe EXTEND -> EXTEND_UP , SET_EXTEND -> EXTEND_DOWN ?

RT_ITER_UPDATE_KEY is unused, but I somehow didn't notice when turning it
into a template.

+ /*

+/*

These comments don't really help readers unfamiliar with the code. The
iteration coding in general needs clearer description.

In the test:

The small size was changed to 3 -- if this test needs to know the max size
for each kind (class?), I wonder why it didn't fail. Should it? Maybe we
need symbols for the various fanouts.

I also want to mention now that we better decide soon if we want to support
shrinking of nodes for v16, even if the tidstore never shrinks. We'll need
to do it at some point, but I'm not sure if doing it now would make more
work for future changes targeting highly concurrent workloads. If so, doing
it now would just be wasted work. On the other hand, someone might have a
use that needs deletion before someone else needs concurrency. Just in
case, I have a start of node-shrinking logic, but needs some work because
we need the (local pointer) parent to update to the new smaller node, just
like the growing case.

---

From: Andres Freund <[email-masked]>
Date: Thu, 16 Feb 2023 08:44:08 -0800

Hi,

> hard time imagining how it would come up, even for something like the

We really ought to replace the tid bitmap used for bitmap heap scans. The
hashtable we use is a pretty awful data structure for it. And that's not
filled in-order, for example.

Greetings,

Andres Freund

---

From: John Naylor <[email-masked]>
Date: Fri, 17 Feb 2023 15:00:24 +0700

> filled in-order, for example.

I took a brief look at that and agree we should sometime make it work there
as well.

v26 tidstore_add_tids() appears to assume that it's only called once per
blocknumber. While the order of offsets doesn't matter there for a single
block, calling it again with the same block would wipe out the earlier
offsets, IIUC. To do an actual "add tid" where the order doesn't matter, it
seems we would need to (acquire lock if needed), read the current bitmap
and OR in the new bit if it exists, then write it back out.

That sounds slow, so it might still be good for vacuum to call a function
that passes a block and an array of offsets that are assumed ordered (as in
v28), but with a more accurate name, like tidstore_set_block_offsets().

---

From: Masahiko Sawada <[email-masked]>
Date: Mon, 20 Feb 2023 14:56:15 +0900

On Thu, Feb 16, 2023 at 6:23 PM John Naylor
<[email-masked]> wrote:
>
> On Thu, Feb 16, 2023 at 10:24 AM Masahiko Sawada <[email-masked]> wrote:
> >
> That's what I was probably thinking about some weeks ago, but I'm having a hard time imagining how it would come up, even for something like the conveyor-belt concept.
>

Thanks, I merged them.

> That was on my list, so great! How about the other end -- nominally we allow 512b. (In practice it won't matter, but this would make sure I didn't mess anything up when forcing all MaxTuplesPerPage to encode.)

According to the doc, the minimum block size is 1kB. It seems to work
fine with 1kB blocks.

> Do we need to do anything for this todo?

Since it's practically no problem, I think we can live with it for
now. dshash also has the same todo.

>
> It might help readability to have a concept of "off_upper/off_lower", just so we can describe things more clearly. The key is block + off_upper, and the value is a bitmap of all the off_lower bits. I hinted at that in my addition of encode_key_off(). Along those lines, maybe s/TIDSTORE_OFFSET_MASK/TIDSTORE_OFFSET_LOWER_MASK/. Actually, I'm not even sure the TIDSTORE_ prefix is valuable for these local macros.
>
> The word "value" as a variable name is pretty generic in this context, and it might be better to call it the off_lower_bitmap, at least in some places. The "key" doesn't have a good short term for naming, but in comments we should make sure we're clear it's "block# + off_upper".
>
> I'm not a fan of the name "tid_i", even as a temp variable -- maybe "compressed_tid"?
>
> It might be worth using typedefs for key and value type. Actually, since key type is fixed for the foreseeable future, maybe the radix tree template should define a key typedef?
>
> The term "result" is probably fine within the tidstore, but as a public name used by vacuum, it's not very descriptive. I don't have a good idea, though.
>
> Some files in backend/access use CamelCase for public functions, although it's not consistent. I think doing that for tidstore would help readability, since they would stand out from rt_* functions and vacuum functions. It's a matter of taste, though.
>
> I don't understand the control flow in tidstore_iterate_next(), or when BlockNumberIsValid() is true. If this is the best way to code this, it needs more commentary.

The attached 0008 patch addressed all above comments on tidstore.

> I had an idea: If it's not too much effort, it might be worth splitting it into two parts: one that just adds the store (not caring about its memory limits or progress reporting etc). During index scan, check both the new store and the array and log a warning (we don't want to exit or crash, better to try to investigate while live if possible) if the result doesn't match. Then perhaps set up an instance and let something like TPC-C run for a few days. The second patch would just restore the rest of the current patch. That would help reassure us it's working as designed.

Yeah, I did a similar thing in an earlier version of tidstore patch.
Since we're trying to introduce two new components: radix tree and
tidstore, I sometimes find it hard to investigate failures happening
during lazy (parallel) vacuum due to a bug either in tidstore or radix
tree. If there is a bug in lazy vacuum, we cannot even do initdb. So
it might be a good idea to do such checks in USE_ASSERT_CHECKING (or
with another macro say DEBUG_TIDSTORE) builds. For example, TidStore
stores tids to both the radix tree and array, and checks if the
results match when lookup or iteration. It will use more memory but it
would not be a big problem in USE_ASSERT_CHECKING builds. It would
also be great if we can enable such checks on some bf animals.

> Soon I plan to do some measurements with vacuuming large tables to get some concrete numbers that the community can get excited about.

Thanks!

>
> We also want to verify that progress reporting works as designed and has no weird corner cases.
>
> This kind of implies that we allocate the maximum bytes upfront. I think this sentence can be removed. We already mentioned in the previous paragraph that we set an upper bound.

Agreed.

> I don't think the format string has to change, since num_tids was changed back to int64 in an earlier patch version?

I think we need to change the format to INT64_FORMAT.

> The old comment still seems mostly ok? Maybe just s/DSM segment/DSA area/ or something else minor.
>
> If we're starting from the minimum, "estimate" doesn't really describe it anymore? Maybe "Initial size"?

Right. The attached 0009 patch addressed comments on vacuum
integration except for the correctness checking.

> I find extend, set, and set_extend hard to keep straight when studying the code. Maybe EXTEND -> EXTEND_UP , SET_EXTEND -> EXTEND_DOWN ?
>
> RT_ITER_UPDATE_KEY is unused, but I somehow didn't notice when turning it into a template.

It was used in radixtree_iter_impl.h. But I removed it as it was not necessary.

> These comments don't really help readers unfamiliar with the code. The iteration coding in general needs clearer description.
>

I agree with all of the above comments. The attached 0007 patch
addressed comments on the radix tree.

> The small size was changed to 3 -- if this test needs to know the max size for each kind (class?), I wonder why it didn't fail. Should it? Maybe we need symbols for the various fanouts.
>

Since this information is used to the number of keys inserted, it
doesn't check the node kind. So we just didn't test node-3. It might
be better to expose and use both RT_SIZE_CLASS and RT_SIZE_CLASS_INFO.

> I also want to mention now that we better decide soon if we want to support shrinking of nodes for v16, even if the tidstore never shrinks. We'll need to do it at some point, but I'm not sure if doing it now would make more work for future changes targeting highly concurrent workloads. If so, doing it now would just be wasted work. On the other hand, someone might have a use that needs deletion before someone else needs concurrency. Just in case, I have a start of node-shrinking logic, but needs some work because we need the (local pointer) parent to update to the new smaller node, just like the growing case.

Thanks, that's also on my todo list. TBH I'm not sure we should
improve the deletion at this stage as there is no use case of deletion
in the core. I'd prefer to focus on improving the quality of the
current radix tree and tidstore now, and I think we can support
node-shrinking once we are confident with the current implementation.

On Fri, Feb 17, 2023 at 5:00 PM John Naylor
<[email-masked]> wrote:
>
>That sounds slow, so it might still be good for vacuum to call a function that passes a block and an array of offsets that are assumed ordered (as in v28), but with a more accurate name, like tidstore_set_block_offsets().

tidstore_set_block_offsets() sounds better. I used
TidStoreSetBlockOffsets() in the latest patch set.

Regards,

---

From: Masahiko Sawada <[email-masked]>
Date: Wed, 22 Feb 2023 15:15:30 +0900

> also be great if we can enable such checks on some bf animals.

I've tried this idea. Enabling this check on all debug builds (i.e.,
with USE_ASSERT_CHECKING macro) seems not a good idea so I use a
special macro for that, TIDSTORE_DEBUG. I think we can define this
macro on some bf animals (or possibly a new bf animal).

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
From 107aa2af2966c10ce750e6b410ae570462423aab Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email-masked]>
Date: Wed, 22 Feb 2023 14:43:15 +0900
Subject: [PATCH v29 11/11] Debug TIDStore.

---
 src/backend/access/common/tidstore.c | 242 ++++++++++++++++++++++++++-
 1 file changed, 238 insertions(+), 4 deletions(-)

diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 9360520482..438bf0c800 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -28,12 +28,20 @@
 #include "postgres.h"
 
 #include "access/tidstore.h"
+#include "catalog/index.h"
 #include "miscadmin.h"
 #include "port/pg_bitutils.h"
 #include "storage/lwlock.h"
 #include "utils/dsa.h"
 #include "utils/memutils.h"
 
+#define TIDSTORE_DEBUG
+
+/* Enable TidStore debugging only when USE_ASSERT_CHECKING */
+#if defined(TIDSTORE_DEBUG) && !defined(USE_ASSERT_CHECKING)
+#undef TIDSTORE_DEBUG
+#endif
+
 /*
  * For encoding purposes, a tid is represented as a pair of 64-bit key and
  * 64-bit value.
@@ -115,6 +123,12 @@ typedef struct TidStoreControl
 	/* handles for TidStore and radix tree */
 	TidStoreHandle		handle;
 	shared_rt_handle	tree_handle;
+
+#ifdef TIDSTORE_DEBUG
+	dsm_handle		tids_handle;
+	int64			max_tids;
+	bool			tids_unordered;
+#endif
 } TidStoreControl;
 
 /* Per-backend state for a TidStore */
@@ -135,6 +149,11 @@ struct TidStore
 
 	/* DSA area for TidStore if used */
 	dsa_area	*area;
+
+#ifdef TIDSTORE_DEBUG
+	dsm_segment		*tids_seg;
+	ItemPointerData	*tids;
+#endif
 };
 #define TidStoreIsShared(ts) ((ts)->area != NULL)
 
@@ -157,6 +176,11 @@ typedef struct TidStoreIter
 	tidkey		next_tidkey;
 	offsetbm	next_off_bitmap;
 
+#ifdef TIDSTORE_DEBUG
+	/* iterator index for the ts->tids array */
+	int64		tids_idx;
+#endif
+
 	/*
 	 * output for the caller. Must be last because variable-size.
 	 */
@@ -169,6 +193,15 @@ static inline tidkey encode_blk_off(TidStore *ts, BlockNumber block,
 									OffsetNumber offset, offsetbm *off_bit);
 static inline tidkey encode_tid(TidStore *ts, ItemPointer tid, offsetbm *off_bit);
 
+/* debug functions available only when TIDSTORE_DEBUG */
+#ifdef TIDSTORE_DEBUG
+static void ts_debug_set_block_offsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+									   int num_offsets);
+static void ts_debug_iter_check_tids(TidStoreIter *iter);
+static bool ts_debug_is_member(TidStore *ts, ItemPointer tid);
+static int itemptr_cmp(const void *left, const void *right);
+#endif
+
 /*
  * Create a TidStore. The returned object is allocated in backend-local memory.
  * The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
@@ -237,6 +270,26 @@ TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
 	ts->control->upper_off_nbits =
 		ts->control->max_off_nbits - LOWER_OFFSET_NBITS;
 
+#ifdef TIDSTORE_DEBUG
+	{
+		int64		max_tids = max_bytes / sizeof(ItemPointerData);
+
+		/* Allocate the array of tids too */
+		if (TidStoreIsShared(ts))
+		{
+			ts->tids_seg = dsm_create(sizeof(ItemPointerData) * max_tids, 0);
+			ts->tids = dsm_segment_address(ts->tids_seg);
+			ts->control->tids_handle = dsm_segment_handle(ts->tids_seg);
+			ts->control->max_tids = max_tids;
+		}
+		else
+		{
+			ts->tids = palloc(sizeof(ItemPointerData) * max_tids);
+			ts->control->max_tids = max_tids;
+		}
+	}
+#endif
+
 	return ts;
 }
 
@@ -266,6 +319,11 @@ TidStoreAttach(dsa_area *area, TidStoreHandle handle)
 	ts->tree.shared = shared_rt_attach(area, ts->control->tree_handle);
 	ts->area = area;
 
+#ifdef TIDSTORE_DEBUG
+	ts->tids_seg = dsm_attach(ts->control->tids_handle);
+	ts->tids = (ItemPointer) dsm_segment_address(ts->tids_seg);
+#endif
+
 	return ts;
 }
 
@@ -280,6 +338,11 @@ TidStoreDetach(TidStore *ts)
 {
 	Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
 
+#ifdef TIDSTORE_DEBUG
+	if (TidStoreIsShared(ts))
+		dsm_detach(ts->tids_seg);
+#endif
+
 	shared_rt_detach(ts->tree.shared);
 	pfree(ts);
 }
@@ -315,6 +378,13 @@ TidStoreDestroy(TidStore *ts)
 		local_rt_free(ts->tree.local);
 	}
 
+#ifdef TIDSTORE_DEBUG
+	if (TidStoreIsShared(ts))
+		dsm_detach(ts->tids_seg);
+	else
+		 pfree(ts->tids);
+#endif
+
 	pfree(ts);
 }
 
@@ -434,6 +504,11 @@ TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
 		}
 	}
 
+#ifdef TIDSTORE_DEBUG
+	/* Insert tids into the tid array too */
+	ts_debug_set_block_offsets(ts, blkno, offsets, num_offsets);
+#endif
+
 	/* update statistics */
 	ts->control->num_tids += num_offsets;
 
@@ -451,6 +526,11 @@ TidStoreIsMember(TidStore *ts, ItemPointer tid)
 	offsetbm off_bitmap = 0;
 	offsetbm off_bit;
 	bool found;
+	bool ret;
+
+#ifdef TIDSTORE_DEBUG
+	bool ret_debug = ts_debug_is_member(ts, tid);
+#endif
 
 	key = encode_tid(ts, tid, &off_bit);
 
@@ -460,9 +540,20 @@ TidStoreIsMember(TidStore *ts, ItemPointer tid)
 		found = local_rt_search(ts->tree.local, key, &off_bitmap);
 
 	if (!found)
+	{
+#ifdef TIDSTORE_DEBUG
+		Assert(!ret_debug);
+#endif
 		return false;
+	}
+
+	ret = (off_bitmap & off_bit) != 0;
 
-	return (off_bitmap & off_bit) != 0;
+#ifdef TIDSTORE_DEBUG
+	Assert(ret == ret_debug);
+#endif
+
+	return ret;
 }
 
 /*
@@ -494,6 +585,10 @@ TidStoreBeginIterate(TidStore *ts)
 	if (TidStoreNumTids(ts) == 0)
 		iter->finished = true;
 
+#ifdef TIDSTORE_DEBUG
+	iter->tids_idx = 0;
+#endif
+
 	return iter;
 }
 
@@ -515,6 +610,7 @@ TidStoreIterResult *
 TidStoreIterateNext(TidStoreIter *iter)
 {
 	tidkey key;
+	bool	iter_found;
 	offsetbm off_bitmap = 0;
 	TidStoreIterResult *output = &(iter->output);
 
@@ -532,7 +628,7 @@ TidStoreIterateNext(TidStoreIter *iter)
 	if (iter->next_off_bitmap > 0)
 		iter_decode_key_off(iter, iter->next_tidkey, iter->next_off_bitmap);
 
-	while (tidstore_iter(iter, &key, &off_bitmap))
+	while ((iter_found = tidstore_iter(iter, &key, &off_bitmap)))
 	{
 		BlockNumber blkno = key_get_blkno(iter->ts, key);
 
@@ -545,14 +641,20 @@ TidStoreIterateNext(TidStoreIter *iter)
 			 */
 			iter->next_tidkey = key;
 			iter->next_off_bitmap = off_bitmap;
-			return output;
+			break;
 		}
 
 		/* Collect tids decoded from the key and offset bitmap */
 		iter_decode_key_off(iter, key, off_bitmap);
 	}
 
-	iter->finished = true;
+	if (!iter_found)
+		iter->finished = true;
+
+#ifdef TIDSTORE_DEBUG
+	ts_debug_iter_check_tids(iter);
+#endif
+
 	return output;
 }
 
@@ -699,3 +801,135 @@ encode_blk_off(TidStore *ts, BlockNumber block, OffsetNumber offset,
 
 	return key;
 }
+
+#ifdef TIDSTORE_DEBUG
+/* Comparator routines for ItemPointer */
+static int
+itemptr_cmp(const void *left, const void *right)
+{
+	BlockNumber lblk,
+		rblk;
+	OffsetNumber loff,
+		roff;
+
+	lblk = ItemPointerGetBlockNumber((ItemPointer) left);
+	rblk = ItemPointerGetBlockNumber((ItemPointer) right);
+
+	if (lblk < rblk)
+		return -1;
+	if (lblk > rblk)
+		return 1;
+
+	loff = ItemPointerGetOffsetNumber((ItemPointer) left);
+	roff = ItemPointerGetOffsetNumber((ItemPointer) right);
+
+	if (loff < roff)
+		return -1;
+	if (loff > roff)
+		return 1;
+
+	return 0;
+}
+
+/* Insert tids to the tid array for debugging */
+static void
+ts_debug_set_block_offsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+						   int num_offsets)
+{
+	if (ts->control->num_tids > 0 &&
+		blkno < ItemPointerGetBlockNumber(&(ts->tids[ts->control->num_tids - 1])))
+	{
+		/* The array will be sorted at ts_debug_is_member() */
+		ts->control->tids_unordered = true;
+	}
+
+	for (int i = 0; i < num_offsets; i++)
+	{
+		ItemPointer tid;
+		int idx = ts->control->num_tids + i;
+
+		/* Enlarge the tid array if necessary */
+		if (idx >= ts->control->max_tids)
+		{
+			ts->control->max_tids *= 2;
+
+			if (TidStoreIsShared(ts))
+			{
+				dsm_segment *new_seg =
+					dsm_create(sizeof(ItemPointerData) * ts->control->max_tids, 0);
+				ItemPointer new_tids = dsm_segment_address(new_seg);
+
+				/* copy tids from old to new array */
+				memcpy(new_tids, ts->tids,
+					   sizeof(ItemPointerData) * (ts->control->max_tids / 2));
+
+				dsm_detach(ts->tids_seg);
+				ts->tids = new_tids;
+			}
+			else
+				ts->tids = repalloc(ts->tids,
+									sizeof(ItemPointerData) * ts->control->max_tids);
+		}
+
+		tid = &(ts->tids[idx]);
+		ItemPointerSetBlockNumber(tid, blkno);
+		ItemPointerSetOffsetNumber(tid, offsets[i]);
+	}
+}
+
+/* Return true if the given tid is present in the tid array */
+static bool
+ts_debug_is_member(TidStore *ts, ItemPointer tid)
+{
+	int64	litem,
+		ritem,
+		item;
+	ItemPointer res;
+
+	if (ts->control->num_tids == 0)
+		return false;
+
+	/* Make sure the tid array is sorted */
+	if (ts->control->tids_unordered)
+	{
+		qsort(ts->tids, ts->control->num_tids, sizeof(ItemPointerData), itemptr_cmp);
+		ts->control->tids_unordered = false;
+	}
+
+	litem = itemptr_encode(&ts->tids[0]);
+	ritem = itemptr_encode(&ts->tids[ts->control->num_tids - 1]);
+	item = itemptr_encode(tid);
+
+	/*
+	 * Doing a simple bound check before bsearch() is useful to avoid the
+	 * extra cost of bsearch(), especially if dead items on the heap are
+	 * concentrated in a certain range.	Since this function is called for
+	 * every index tuple, it pays to be really fast.
+	 */
+	if (item < litem || item > ritem)
+		return false;
+
+	res = bsearch(tid, ts->tids, ts->control->num_tids, sizeof(ItemPointerData),
+				  itemptr_cmp);
+
+	return (res != NULL);
+}
+
+/* Verify if the iterator output matches the tids in the array for debugging */
+static void
+ts_debug_iter_check_tids(TidStoreIter *iter)
+{
+	BlockNumber blkno = iter->output.blkno;
+
+	for (int i = 0; i < iter->output.num_offsets; i++)
+	{
+		ItemPointer tid = &(iter->ts->tids[iter->tids_idx + i]);
+
+		Assert((iter->tids_idx + i) < iter->ts->control->max_tids);
+		Assert(ItemPointerGetBlockNumber(tid) == blkno);
+		Assert(ItemPointerGetOffsetNumber(tid) == iter->output.offsets[i]);
+	}
+
+	iter->tids_idx += iter->output.num_offsets;
+}
+#endif
-- 
2.31.1

---

From: John Naylor <[email-masked]>
Date: Wed, 22 Feb 2023 14:35:00 +0700

> > Yeah, I did a similar thing in an earlier version of tidstore patch.

Okay, if you had checks against the old array lookup in development, that
gives us better confidence.

> macro on some bf animals (or possibly a new bf animal).

 I don't think any vacuum calls in regression tests would stress any of
this code very much, so it's not worth carrying the old way forward. I was
thinking of only doing this as a short-time sanity check for testing a
real-world workload.

---

From: Masahiko Sawada <[email-masked]>
Date: Wed, 22 Feb 2023 17:29:23 +0900

On Wed, Feb 22, 2023 at 4:35 PM John Naylor
<[email-masked]> wrote:
>
>
> On Wed, Feb 22, 2023 at 1:16 PM Masahiko Sawada <[email-masked]> wrote:
> >
> Okay, if you had checks against the old array lookup in development, that gives us better confidence.
>
>  I don't think any vacuum calls in regression tests would stress any of this code very much, so it's not worth carrying the old way forward. I was thinking of only doing this as a short-time sanity check for testing a real-world workload.

I guess that It would also be helpful at least until the GA release.
People will be able to test them easily on their workloads or their
custom test scenarios.

Regards,

---

From: John Naylor <[email-masked]>
Date: Wed, 22 Feb 2023 16:55:30 +0700

> People will be able to test them easily on their workloads or their

That doesn't seem useful to me. If we've done enough testing to reassure us
the new way always gives the same answer, the old way is not needed at
commit time. If there is any doubt it will always give the same answer,
then the whole patchset won't be committed.

TPC-C was just an example. It should have testing comparing the old and new
methods. If you have already done that to some degree, that might be
enough. After performance tests, I'll also try some vacuums that use the
comparison patch.

---

From: John Naylor <[email-masked]>
Date: Thu, 23 Feb 2023 16:40:58 +0700

I ran a couple "in situ" tests on server hardware using UUID columns, since
they are common in the real world and have bad correlation to heap
order, so are a challenge for index vacuum.

=== test 1, delete everything from a small table, with very small
maintenance_work_mem:

alter system set shared_buffers ='4GB';
alter system set max_wal_size ='10GB';
alter system set checkpoint_timeout ='30 min';
alter system set autovacuum =off;

-- unrealistically low
alter system set maintenance_work_mem = '32MB';

create table if not exists test (x uuid);
truncate table test;
insert into test (x) select gen_random_uuid() from
generate_series(1,50*1000*1000);
create index on test (x);

delete from test;
vacuum (verbose, truncate off) test;
--

master:
INFO:  finished vacuuming "john.naylor.public.test": index scans: 9
system usage: CPU: user: 70.04 s, system: 19.85 s, elapsed: 802.06 s

v29 patch:
INFO:  finished vacuuming "john.naylor.public.test": index scans: 1
system usage: CPU: user: 9.80 s, system: 2.62 s, elapsed: 36.68 s

This is a bit artificial, but it's easy to construct cases where the array
leads to multiple index scans but the new tid store can fit everythin
without breaking a sweat. I didn't save the progress reporting, but v29 was
using about 11MB for tid storage.

=== test 2: try to stress tid lookup with production maintenance_work_mem:
1. use unlogged table to reduce noise
2. vacuum freeze first to reduce heap scan time
3. delete some records at the beginning and end of heap to defeat binary
search's pre-check

alter system set shared_buffers ='4GB';
alter system set max_wal_size ='10GB';
alter system set checkpoint_timeout ='30 min';
alter system set autovacuum =off;

alter system set maintenance_work_mem = '1GB';

create unlogged table if not exists test (x uuid);
truncate table test;
insert into test (x) select gen_random_uuid() from
generate_series(1,1000*1000*1000);
vacuum_freeze test;

select pg_size_pretty(pg_table_size('test'));
 pg_size_pretty
----------------
 41 GB

create index on test (x);

select pg_size_pretty(pg_total_relation_size('test'));
 pg_size_pretty
----------------
 71 GB

select max(ctid) from test;
     max
--------------
 (5405405,75)

delete from test where ctid <  '(100000,0)'::tid;
delete from test where ctid > '(5300000,0)'::tid;

vacuum (verbose, truncate off) test;

both:
INFO:  vacuuming "john.naylor.public.test"
INFO:  finished vacuuming "john.naylor.public.test": index scans: 1
index scan needed: 205406 pages from table (3.80% of total) had 38000000
dead item identifiers removed

--
master:
system usage: CPU: user: 134.32 s, system: 19.24 s, elapsed: 286.14 s

v29 patch:
system usage: CPU: user:  97.71 s, system: 45.78 s, elapsed: 573.94 s

The entire vacuum took 25% less wall clock time. Reminder that this is
without wal logging, and also unscientific because only one run.

--
I took 10 seconds of perf data while index vacuuming was going on (showing
calls > 2%):

master:
  40.59%  postgres  postgres            [.] vac_cmp_itemptr
  24.97%  postgres  libc-2.17.so        [.] bsearch
   6.67%  postgres  postgres            [.] btvacuumpage
   4.61%  postgres  [kernel.kallsyms]   [k] copy_user_enhanced_fast_string
   3.48%  postgres  postgres            [.] PageIndexMultiDelete
   2.67%  postgres  postgres            [.] vac_tid_reaped
   2.03%  postgres  postgres            [.] compactify_tuples
   2.01%  postgres  libc-2.17.so        [.] __memcpy_ssse3_back

v29 patch:

  29.22%  postgres  postgres            [.] TidStoreIsMember
   9.30%  postgres  postgres            [.] btvacuumpage
   7.76%  postgres  postgres            [.] PageIndexMultiDelete
   6.31%  postgres  [kernel.kallsyms]   [k] copy_user_enhanced_fast_string
   5.60%  postgres  postgres            [.] compactify_tuples
   4.26%  postgres  libc-2.17.so        [.] __memcpy_ssse3_back
   4.12%  postgres  postgres            [.] hash_search_with_hash_value

--
master:
psql -c "select phase, heap_blks_total, heap_blks_scanned, max_dead_tuples,
num_dead_tuples from pg_stat_progress_vacuum"
       phase       | heap_blks_total | heap_blks_scanned | max_dead_tuples
| num_dead_tuples
-------------------+-----------------+-------------------+-----------------+-----------------
 vacuuming indexes |         5405406 |           5405406 |       178956969
|        38000000

v29 patch:
psql  -c "select phase, heap_blks_total, heap_blks_scanned,
max_dead_tuple_bytes, dead_tuple_bytes from pg_stat_progress_vacuum"
       phase       | heap_blks_total | heap_blks_scanned |
max_dead_tuple_bytes | dead_tuple_bytes
-------------------+-----------------+-------------------+----------------------+------------------
 vacuuming indexes |         5405406 |           5405406 |
1073670144 |          8678064

Here, the old array pessimistically needs 1GB allocated (as for any table >
~5GB), but only fills 228MB for tid lookup. The patch reports 8.7MB. Tables
that only fit, say, 30-50 tuples per page will have less extreme
differences in memory use. Same for the case where only a couple dead items
occur per page, with many uninteresting pages in between. Even so, the
allocation will be much more accurately sized in the patch, especially in
non-parallel vacuum.

There are other cases that could be tested (I mentioned some above), but
this is enough to show the improvements possible.

I still need to do some cosmetic follow-up to v29 as well as a status
report, and I will try to get back to that soon.

---

From: Masahiko Sawada <[email-masked]>
Date: Fri, 24 Feb 2023 14:50:19 +0900

On Wed, Feb 22, 2023 at 6:55 PM John Naylor
<[email-masked]> wrote:
>
>
> On Wed, Feb 22, 2023 at 3:29 PM Masahiko Sawada <[email-masked]> wrote:
> >
> That doesn't seem useful to me. If we've done enough testing to reassure us the new way always gives the same answer, the old way is not needed at commit time. If there is any doubt it will always give the same answer, then the whole patchset won't be committed.

True. Even if we're done enough testing we cannot claim there is no
bug. My idea is to make the bug investigation easier but on
reflection, it seems not the best idea given this purpose. Instead, it
seems to be better to add more necessary assertions. What do you think
about the attached patch? Please note that it also includes the
changes for minimum memory requirement.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 9360520482..fc20e58a95 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -75,6 +75,14 @@ typedef uint64 offsetbm;
 #define LOWER_OFFSET_NBITS	6	/* log(sizeof(offsetbm), 2) */
 #define LOWER_OFFSET_MASK	((1 << LOWER_OFFSET_NBITS) - 1)
 
+/*
+ * The minimum amount of memory required by TidStore is 2MB, the current minimum
+ * valid value for the maintenance_work_mem GUC. This is required to allocate the
+ * DSA initial segment, 1MB, and some meta data. This number is applied also to
+ * the local TidStore cases for simplicity.
+ */
+#define TIDSTORE_MIN_MEMORY	(2 * 1024 * 1024L)		/* 2MB */
+
 /* A magic value used to identify our TidStore. */
 #define TIDSTORE_MAGIC 0x826f6a10
 
@@ -101,7 +109,7 @@ typedef struct TidStoreControl
 
 	/* These values are never changed after creation */
 	size_t	max_bytes;		/* the maximum bytes a TidStore can use */
-	int		max_off;		/* the maximum offset number */
+	OffsetNumber	max_off;		/* the maximum offset number */
 	int		max_off_nbits;	/* the number of bits required for offset
 							 * numbers */
 	int		upper_off_nbits;	/* the number of bits of offset numbers
@@ -174,10 +182,17 @@ static inline tidkey encode_tid(TidStore *ts, ItemPointer tid, offsetbm *off_bit
  * The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
  */
 TidStore *
-TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
+TidStoreCreate(size_t max_bytes, OffsetNumber max_off, dsa_area *area)
 {
 	TidStore	*ts;
 
+	Assert(max_off <= MaxOffsetNumber);
+
+	/* Sanity check for the max_bytes */
+	if (max_bytes < TIDSTORE_MIN_MEMORY)
+		elog(ERROR, "memory for TidStore must be at least %ld, but %zu provided",
+			 TIDSTORE_MIN_MEMORY, max_bytes);
+
 	ts = palloc0(sizeof(TidStore));
 
 	/*
@@ -192,8 +207,8 @@ TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
 	 * In local TidStore cases, the radix tree uses slab allocators for each kind
 	 * of node class. The most memory consuming case while adding Tids associated
 	 * with one page (i.e. during TidStoreSetBlockOffsets()) is that we allocate a new
-	 * slab block for a new radix tree node, which is approximately 70kB. Therefore,
-	 * we deduct 70kB from the max_bytes.
+	 * slab block for a new radix tree node, which is approximately 70kB at most.
+	 * Therefore, we deduct 70kB from the max_bytes.
 	 *
 	 * In shared cases, DSA allocates the memory segments big enough to follow
 	 * a geometric series that approximately doubles the total DSA size (see
@@ -378,6 +393,7 @@ TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
 	const int nkeys = UINT64CONST(1) << ts->control->upper_off_nbits;
 
 	Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+	Assert(BlockNumberIsValid(blkno));
 
 	bitmaps = palloc(sizeof(offsetbm) * nkeys);
 	key = prev_key = key_base;
@@ -386,6 +402,8 @@ TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
 	{
 		offsetbm	off_bit;
 
+		Assert(offsets[i] <= ts->control->max_off);
+
 		/* encode the tid to a key and partial offset */
 		key = encode_blk_off(ts, blkno, offsets[i], &off_bit);
 
@@ -452,6 +470,8 @@ TidStoreIsMember(TidStore *ts, ItemPointer tid)
 	offsetbm off_bit;
 	bool found;
 
+	Assert(ItemPointerIsValid(tid));
+
 	key = encode_tid(ts, tid, &off_bit);
 
 	if (TidStoreIsShared(ts))
@@ -535,6 +555,7 @@ TidStoreIterateNext(TidStoreIter *iter)
 	while (tidstore_iter(iter, &key, &off_bitmap))
 	{
 		BlockNumber blkno = key_get_blkno(iter->ts, key);
+		Assert(BlockNumberIsValid(blkno));
 
 		if (BlockNumberIsValid(output->blkno) && output->blkno != blkno)
 		{
@@ -586,6 +607,7 @@ TidStoreNumTids(TidStore *ts)
 	num_tids = ts->control->num_tids;
 	LWLockRelease(&ts->control->lock);
 
+	Assert(num_tids >= 0);
 	return num_tids;
 }
 
diff --git a/src/include/access/tidstore.h b/src/include/access/tidstore.h
index 66f0fdd482..d1cc93cbb6 100644
--- a/src/include/access/tidstore.h
+++ b/src/include/access/tidstore.h
@@ -30,7 +30,7 @@ typedef struct TidStoreIterResult
 	OffsetNumber	offsets[FLEXIBLE_ARRAY_MEMBER];
 } TidStoreIterResult;
 
-extern TidStore *TidStoreCreate(size_t max_bytes, int max_off, dsa_area *dsa);
+extern TidStore *TidStoreCreate(size_t max_bytes, OffsetNumber max_off, dsa_area *dsa);
 extern TidStore *TidStoreAttach(dsa_area *dsa, dsa_pointer handle);
 extern void TidStoreDetach(TidStore *ts);
 extern void TidStoreDestroy(TidStore *ts);

---

From: Masahiko Sawada <[email-masked]>
Date: Fri, 24 Feb 2023 17:40:27 +0900

On Thu, Feb 23, 2023 at 6:41 PM John Naylor
<[email-masked]> wrote:
>
> I ran a couple "in situ" tests on server hardware using UUID columns, since they are common in the real world and have bad correlation to heap order, so are a challenge for index vacuum.

Thank you for the test!

>
> === test 1, delete everything from a small table, with very small maintenance_work_mem:
>
> insert into test (x) select gen_random_uuid() from generate_series(1,50*1000*1000);
> This is a bit artificial, but it's easy to construct cases where the array leads to multiple index scans but the new tid store can fit everythin without breaking a sweat. I didn't save the progress reporting, but v29 was using about 11MB for tid storage.

Cool.

> 3. delete some records at the beginning and end of heap to defeat binary search's pre-check
>
> insert into test (x) select gen_random_uuid() from generate_series(1,1000*1000*1000);
> vacuum_freeze test;
>
> index scan needed: 205406 pages from table (3.80% of total) had 38000000 dead item identifiers removed
>
> --
> master:

In v29 vacuum took twice as long (286 s vs. 573 s)?

>
> The entire vacuum took 25% less wall clock time. Reminder that this is without wal logging, and also unscientific because only one run.
>
> --
> I took 10 seconds of perf data while index vacuuming was going on (showing calls > 2%):
>
> master:
> psql -c "select phase, heap_blks_total, heap_blks_scanned, max_dead_tuples, num_dead_tuples from pg_stat_progress_vacuum"
>        phase       | heap_blks_total | heap_blks_scanned | max_dead_tuples | num_dead_tuples
>  vacuuming indexes |         5405406 |           5405406 |       178956969 |        38000000
>
> v29 patch:
> psql  -c "select phase, heap_blks_total, heap_blks_scanned, max_dead_tuple_bytes, dead_tuple_bytes from pg_stat_progress_vacuum"
>        phase       | heap_blks_total | heap_blks_scanned | max_dead_tuple_bytes | dead_tuple_bytes
>  vacuuming indexes |         5405406 |           5405406 |           1073670144 |          8678064
>
> Here, the old array pessimistically needs 1GB allocated (as for any table > ~5GB), but only fills 228MB for tid lookup. The patch reports 8.7MB. Tables that only fit, say, 30-50 tuples per page will have less extreme differences in memory use. Same for the case where only a couple dead items occur per page, with many uninteresting pages in between. Even so, the allocation will be much more accurately sized in the patch, especially in non-parallel vacuum.

Agreed.

Regards,

---

From: John Naylor <[email-masked]>
Date: Tue, 28 Feb 2023 00:07:23 +0700

> In v29 vacuum took twice as long (286 s vs. 573 s)?

Not sure what happened there, and clearly I was looking at the wrong number
:/
I scripted the test for reproducibility and ran it three times. Also
included some variations (attached):

UUID times look comparable here, so no speedup or regression:

master:
system usage: CPU: user: 216.05 s, system: 35.81 s, elapsed: 634.22 s
system usage: CPU: user: 173.71 s, system: 31.24 s, elapsed: 599.04 s
system usage: CPU: user: 171.16 s, system: 30.21 s, elapsed: 583.21 s

v29:
system usage: CPU: user:  93.47 s, system: 40.92 s, elapsed: 594.10 s
system usage: CPU: user:  99.58 s, system: 44.73 s, elapsed: 606.80 s
system usage: CPU: user:  96.29 s, system: 42.74 s, elapsed: 600.10 s

Then, I tried sequential integers, which is a much more favorable access
pattern in general, and the new tid storage shows substantial improvement:

master:
system usage: CPU: user: 100.39 s, system: 7.79 s, elapsed: 121.57 s
system usage: CPU: user: 104.90 s, system: 8.81 s, elapsed: 124.24 s
system usage: CPU: user:  95.04 s, system: 7.55 s, elapsed: 116.44 s

v29:
system usage: CPU: user:  24.57 s, system: 8.53 s, elapsed: 61.07 s
system usage: CPU: user:  23.18 s, system: 8.25 s, elapsed: 58.99 s
system usage: CPU: user:  23.20 s, system: 8.98 s, elapsed: 66.86 s

That's fast enough that I thought an improvement would show up even with
standard WAL logging (no separate attachment, since it's a trivial change).
Seems a bit faster:

master:
system usage: CPU: user: 152.27 s, system: 11.76 s, elapsed: 216.86 s
system usage: CPU: user: 137.25 s, system: 11.07 s, elapsed: 213.62 s
system usage: CPU: user: 149.48 s, system: 12.15 s, elapsed: 220.96 s

v29:
system usage: CPU: user: 40.88 s, system: 15.99 s, elapsed: 170.98 s
system usage: CPU: user: 41.33 s, system: 15.45 s, elapsed: 166.75 s
system usage: CPU: user: 41.51 s, system: 18.20 s, elapsed: 203.94 s

There is more we could test here, but I feel better about these numbers.

In the next few days, I'll resume style review and list the remaining
issues we need to address.

---

From: John Naylor <[email-masked]>
Date: Tue, 28 Feb 2023 13:42:27 +0700

On Fri, Feb 24, 2023 at 12:50 PM Masahiko Sawada <[email-masked]>
wrote:
> > That doesn't seem useful to me. If we've done enough testing to
reassure us the new way always gives the same answer, the old way is not
needed at commit time. If there is any doubt it will always give the same
answer, then the whole patchset won't be committed.

> My idea is to make the bug investigation easier but on
> reflection, it seems not the best idea given this purpose.

My concern with TIDSTORE_DEBUG is that it adds new code that mimics the old
tid array. As I've said, that doesn't seem like a good thing to carry
forward forevermore, in any form. Plus, comparing new code with new code is
not the same thing as comparing existing code with new code. That was my
idea upthread.

Maybe the effort my idea requires is too much vs. the likelihood of finding
a problem. In any case, it's clear that if I want that level of paranoia,
I'm going to have to do it myself.

> What do you think

Most of the asserts look logical, or at least harmless.

I agree with using the specific type for offsets here, but I'm not sure why
this change belongs in this patch. If we decided against the new asserts,
this would be easy to lose.

This change, however, defies common sense:

+/*
+ * The minimum amount of memory required by TidStore is 2MB, the current
minimum
+ * valid value for the maintenance_work_mem GUC. This is required to
allocate the
+ * DSA initial segment, 1MB, and some meta data. This number is applied
also to
+ * the local TidStore cases for simplicity.
+ */
+#define TIDSTORE_MIN_MEMORY (2 * 1024 * 1024L) /* 2MB */

Aside from the fact that this elog's something that would never get past
development, the #define just adds a hard-coded copy of something that is
already hard-coded somewhere else, whose size depends on an implementation
detail in a third place.

This also assumes that all users of tid store are limited by
maintenance_work_mem. Andres thought of an example of some day unifying
with tidbitmap.c, and maybe other applications will be limited by work_mem.

But now that I'm looking at the guc tables, I am reminded that work_mem's
minimum is 64kB, so this highlights a design problem: There is obviously no
requirement that the minimum work_mem has to be >= a single DSA segment,
even though operations like parallel hash and parallel bitmap heap scan are
limited by work_mem. It would be nice to find out what happens with these
parallel features when work_mem is tiny (maybe parallelism is not even
considered?).

---

From: Masahiko Sawada <[email-masked]>
Date: Tue, 28 Feb 2023 22:20:17 +0900

On Tue, Feb 28, 2023 at 3:42 PM John Naylor
<[email-masked]> wrote:
>
>
> On Fri, Feb 24, 2023 at 12:50 PM Masahiko Sawada <[email-masked]> wrote:
> >
> My concern with TIDSTORE_DEBUG is that it adds new code that mimics the old tid array. As I've said, that doesn't seem like a good thing to carry forward forevermore, in any form. Plus, comparing new code with new code is not the same thing as comparing existing code with new code. That was my idea upthread.
>
> Maybe the effort my idea requires is too much vs. the likelihood of finding a problem. In any case, it's clear that if I want that level of paranoia, I'm going to have to do it myself.
>
> > What do you think
> I agree with using the specific type for offsets here, but I'm not sure why this change belongs in this patch. If we decided against the new asserts, this would be easy to lose.

Right. I'll separate this change as a separate patch.

> Aside from the fact that this elog's something that would never get past development, the #define just adds a hard-coded copy of something that is already hard-coded somewhere else, whose size depends on an implementation detail in a third place.
>
> This also assumes that all users of tid store are limited by maintenance_work_mem. Andres thought of an example of some day unifying with tidbitmap.c, and maybe other applications will be limited by work_mem.
>
> But now that I'm looking at the guc tables, I am reminded that work_mem's minimum is 64kB, so this highlights a design problem: There is obviously no requirement that the minimum work_mem has to be >= a single DSA segment, even though operations like parallel hash and parallel bitmap heap scan are limited by work_mem.

Right.

>  It would be nice to find out what happens with these parallel features when work_mem is tiny (maybe parallelism is not even considered?).

IIUC both don't care about the allocated DSA segment size. Parallel
hash accounts actual tuple (+ header) size as used memory but doesn't
consider how much DSA segment is allocated behind. Both parallel hash
and parallel bitmap scan can work even with work_mem = 64kB, but when
checking the total DSA segment size allocated during these operations,
it was 1MB.

I realized that there is a similar memory limit design issue also on
the non-shared tidstore cases. We deduct 70kB from max_bytes but it
won't work fine with work_mem = 64kB.  Probably we need to reconsider
it. FYI 70kB comes from the maximum slab block size for node256.

Regards,

---

From: Masahiko Sawada <[email-masked]>
Date: Wed, 1 Mar 2023 00:09:16 +0900

> it. FYI 70kB comes from the maximum slab block size for node256.

Currently, we calculate the slab block size enough to allocate 32
chunks from there. For node256, the leaf node is 2,088 bytes and the
slab block size is 66,816 bytes. One idea to fix this issue to
decrease it. For example, with 16 chunks the slab block size is 33,408
bytes and with 8 chunks it's 16,704 bytes. I ran a brief benchmark
test with 70kB block size and 16kB block size:

* 70kB slab blocks:
select * from bench_search_random_nodes(20 * 1000 * 1000, '0xFFFFFF');
height = 2, n3 = 0, n15 = 0, n32 = 0, n125 = 0, n256 = 65793
 mem_allocated | load_ms | search_ms
---------------+---------+-----------
     143085184 |    1216 |       750
(1 row)

* 16kB slab blocks:
select * from bench_search_random_nodes(20 * 1000 * 1000, '0xFFFFFF');
height = 2, n3 = 0, n15 = 0, n32 = 0, n125 = 0, n256 = 65793
 mem_allocated | load_ms | search_ms
---------------+---------+-----------
     157601248 |    1220 |       786
(1 row)

There is a performance difference a bit but a smaller slab block size
seems to be acceptable if there is no other better way.

Regards,

---

From: John Naylor <[email-masked]>
Date: Wed, 1 Mar 2023 13:37:01 +0700

> > > My concern with TIDSTORE_DEBUG is that it adds new code that mimics
the old tid array. As I've said, that doesn't seem like a good thing to
carry forward forevermore, in any form. Plus, comparing new code with new
code is not the same thing as comparing existing code with new code. That
was my idea upthread.
> > >
> > > Maybe the effort my idea requires is too much vs. the likelihood of
finding a problem. In any case, it's clear that if I want that level of
paranoia, I'm going to have to do it myself.
> > > I agree with using the specific type for offsets here, but I'm not
sure why this change belongs in this patch. If we decided against the new
asserts, this would be easy to lose.
allocate the
> > > Aside from the fact that this elog's something that would never get
past development, the #define just adds a hard-coded copy of something that
is already hard-coded somewhere else, whose size depends on an
implementation detail in a third place.
> > >
> > > But now that I'm looking at the guc tables, I am reminded that
work_mem's minimum is 64kB, so this highlights a design problem: There is
obviously no requirement that the minimum work_mem has to be >= a single
DSA segment, even though operations like parallel hash and parallel bitmap
heap scan are limited by work_mem.
> >
> > Right.
> >
> > >  It would be nice to find out what happens with these parallel
features when work_mem is tiny (maybe parallelism is not even considered?).
> decrease it.

I think we're trying to solve the wrong problem here. I need to study this
more, but it seems that code that needs to stay within a memory limit only
needs to track what's been allocated in chunks within a block, since
writing there is what invokes a page fault. If we're not keeping track of
each and every chunk space, for speed, it doesn't follow that we need to
keep every block allocation within the configured limit. I'm guessing we
can just ask the context if the block space has gone *over* the limit, and
we can assume that the last allocation we perform will only fault one
additional page. We need to have a clear answer on this before doing
anything else.

If that's correct, and I'm not positive yet, we can get rid of all the
fragile assumptions about things the tid store has no business knowing
about, as well as the guc change. I'm not sure how this affects progress
reporting, because it would be nice if it didn't report dead_tuple_bytes
bigger than max_dead_tuple_bytes.

---

From: Masahiko Sawada <[email-masked]>
Date: Wed, 1 Mar 2023 20:58:48 +0900

On Wed, Mar 1, 2023 at 3:37 PM John Naylor <[email-masked]> wrote:
>
> On Tue, Feb 28, 2023 at 10:09 PM Masahiko Sawada <[email-masked]> wrote:
> >
> I think we're trying to solve the wrong problem here. I need to study this more, but it seems that code that needs to stay within a memory limit only needs to track what's been allocated in chunks within a block, since writing there is what invokes a page fault.

Right. I guess we've discussed what we use for calculating the *used*
memory amount but I don't remember.

I think I was confused by the fact that we use some different
approaches to calculate the amount of used memory. Parallel hash and
tidbitmap use the allocated chunk size whereas hash_agg_check_limits()
in nodeAgg.c uses MemoryContextMemAllocated(), which uses the
allocated block size.

> If we're not keeping track of each and every chunk space, for speed, it doesn't follow that we need to keep every block allocation within the configured limit. I'm guessing we can just ask the context if the block space has gone *over* the limit, and we can assume that the last allocation we perform will only fault one additional page. We need to have a clear answer on this before doing anything else.
>
> If that's correct, and I'm not positive yet, we can get rid of all the fragile assumptions about things the tid store has no business knowing about, as well as the guc change.

True.

> I'm not sure how this affects progress reporting, because it would be nice if it didn't report dead_tuple_bytes bigger than max_dead_tuple_bytes.

Yes, the progress reporting could be confusable. Particularly, in
shared tidstore cases, the dead_tuple_bytes could be much bigger than
max_dead_tuple_bytes. Probably what we need might be functions for
MemoryContext and dsa_area to get the amount of memory that has been
allocated, by not tracking every chunk space. For example, the
functions would be like what SlabStats() does; iterate over every
block and calculates the total/free memory usage.

Regards,

---

From: John Naylor <[email-masked]>
Date: Fri, 3 Mar 2023 18:03:53 +0700

On Wed, Mar 1, 2023 at 6:59 PM Masahiko Sawada <[email-masked]>
wrote:
> > I think we're trying to solve the wrong problem here. I need to study
this more, but it seems that code that needs to stay within a memory limit
only needs to track what's been allocated in chunks within a block, since
writing there is what invokes a page fault.
> in nodeAgg.c uses MemoryContextMemAllocated(), which uses the

That's good to know. The latter says:

 * After adding a new group to the hash table, check whether we need to
enter
 * spill mode. Allocations may happen without adding new groups (for
instance,
 * if the transition state size grows), so this check is imperfect.

I'm willing to claim that vacuum can be imperfect also, given the tid
store's properties: 1) on average much more efficient in used space, and 2)
no longer bound by the 1GB limit.

> > I'm not sure how this affects progress reporting, because it would be
nice if it didn't report dead_tuple_bytes bigger than max_dead_tuple_bytes.
> block and calculates the total/free memory usage.

I'm not sure we need to invent new infrastructure for this. Looking at v29
in vacuumlazy.c, the order of operations for memory accounting is:

First, get the block-level space -- stop and vacuum indexes if we exceed
the limit:

/*
 * Consider if we definitely have enough space to process TIDs on page
 * already.  If we are close to overrunning the available space for
 * dead_items TIDs, pause and do a cycle of vacuuming before we tackle
 * this page.
 */
if (TidStoreIsFull(vacrel->dead_items)) --> which is basically "if
(TidStoreMemoryUsage(ts) > ts->control->max_bytes)"

Then, after pruning the current page, store the tids and then get the
block-level space again:

else if (prunestate.num_offsets > 0)
{
  /* Save details of the LP_DEAD items from the page in dead_items */
  TidStoreSetBlockOffsets(...);

  pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
                               TidStoreMemoryUsage(dead_items));
}

Since the block-level measurement is likely overestimating quite a bit, I
propose to simply reverse the order of the actions here, effectively
reporting progress for the *last page* and not the current one: First
update progress with the current memory usage, then add tids for this page.
If this allocated a new block, only a small bit of that will be written to.
If this block pushes it over the limit, we will detect that up at the top
of the loop. It's kind of like our earlier attempts at a "fudge factor",
but simpler and less brittle. And, as far as OS pages we have actually
written to, I think it'll effectively respect the memory limit, at least in
the local mem case. And the numbers will make sense.

Thoughts?

But now that I'm looking more closely at the details of memory accounting,
I don't like that TidStoreMemoryUsage() is called twice per page pruned
(see above). Maybe it wouldn't noticeably slow things down, but it's a bit
sloppy. It seems like we should call it once per loop and save the result
somewhere. If that's the right way to go, that possibly indicates that
TidStoreIsFull() is not a useful interface, at least in this form.

---

From: Masahiko Sawada <[email-masked]>
Date: Mon, 6 Mar 2023 15:27:44 +0900

On Fri, Mar 3, 2023 at 8:04 PM John Naylor <[email-masked]> wrote:
>
> On Wed, Mar 1, 2023 at 6:59 PM Masahiko Sawada <[email-masked]> wrote:
> >
>  * After adding a new group to the hash table, check whether we need to enter
>  * spill mode. Allocations may happen without adding new groups (for instance,
> I'm willing to claim that vacuum can be imperfect also, given the tid store's properties: 1) on average much more efficient in used space, and 2) no longer bound by the 1GB limit.
>
> I'm not sure we need to invent new infrastructure for this. Looking at v29 in vacuumlazy.c, the order of operations for memory accounting is:
>
> First, get the block-level space -- stop and vacuum indexes if we exceed the limit:
>
> /*
> if (TidStoreIsFull(vacrel->dead_items)) --> which is basically "if (TidStoreMemoryUsage(ts) > ts->control->max_bytes)"
>
> Then, after pruning the current page, store the tids and then get the block-level space again:
>
> Since the block-level measurement is likely overestimating quite a bit, I propose to simply reverse the order of the actions here, effectively reporting progress for the *last page* and not the current one: First update progress with the current memory usage, then add tids for this page. If this allocated a new block, only a small bit of that will be written to. If this block pushes it over the limit, we will detect that up at the top of the loop. It's kind of like our earlier attempts at a "fudge factor", but simpler and less brittle. And, as far as OS pages we have actually written to, I think it'll effectively respect the memory limit, at least in the local mem case. And the numbers will make sense.
>
> Thoughts?

It looks to work but it still doesn't work in a case where a shared
tidstore is created with a 64kB memory limit, right?
TidStoreMemoryUsage() returns 1MB and TidStoreIsFull() returns true
from the beginning.

BTW I realized that since the caller can pass dsa_area to tidstore
(and radix tree), if other data are allocated in the same DSA are,
TidStoreMemoryUsage() (and RT_MEMORY_USAGE()) returns the memory usage
that includes not only itself but also other data. Probably it's
better to comment that the passed dsa_area should be dedicated to a
tidstore (or a radix tree).

>
> But now that I'm looking more closely at the details of memory accounting, I don't like that TidStoreMemoryUsage() is called twice per page pruned (see above). Maybe it wouldn't noticeably slow things down, but it's a bit sloppy. It seems like we should call it once per loop and save the result somewhere. If that's the right way to go, that possibly indicates that TidStoreIsFull() is not a useful interface, at least in this form.

Agreed.

Regards,

---

From: John Naylor <[email-masked]>
Date: Mon, 6 Mar 2023 23:00:52 +0700

On Mon, Mar 6, 2023 at 1:28 PM Masahiko Sawada <[email-masked]>
wrote:

> > Since the block-level measurement is likely overestimating quite a bit,
I propose to simply reverse the order of the actions here, effectively
> TidStoreMemoryUsage() returns 1MB and TidStoreIsFull() returns true

I have two ideas:

1. Make it optional to track chunk memory space by a template parameter. It
might be tiny compared to everything else that vacuum does. That would
allow other users to avoid that overhead.
2. When context block usage exceeds the limit (rare), make the additional
effort to get the precise usage -- I'm not sure such a top-down facility
exists, and I'm not feeling well enough today to study this further.

---

From: Masahiko Sawada <[email-masked]>
Date: Tue, 7 Mar 2023 10:24:32 +0900

On Tue, Mar 7, 2023 at 1:01 AM John Naylor <[email-masked]> wrote:
>
> On Mon, Mar 6, 2023 at 1:28 PM Masahiko Sawada <[email-masked]> wrote:
>
> 1. Make it optional to track chunk memory space by a template parameter. It might be tiny compared to everything else that vacuum does. That would allow other users to avoid that overhead.
> 2. When context block usage exceeds the limit (rare), make the additional effort to get the precise usage -- I'm not sure such a top-down facility exists, and I'm not feeling well enough today to study this further.

I prefer option (1) as it's straight forward. I mentioned a similar
idea before[1]. RT_MEMORY_USAGE() is defined only when the macro is
defined. It might be worth checking if there is visible overhead of
tracking chunk memory space. IIRC we've not evaluated it yet.

[1] https://www.postgresql.org/message-id/CAD21AoDK3gbX-jVxT6Pfso1Na0Krzr8Q15498Aj6tmXgzMFksA%40mail.gmail.com

Regards,

---

From: John Naylor <[email-masked]>
Date: Wed, 8 Mar 2023 11:40:09 +0700

On Tue, Mar 7, 2023 at 8:25 AM Masahiko Sawada <[email-masked]>
wrote:

> > 1. Make it optional to track chunk memory space by a template
parameter. It might be tiny compared to everything else that vacuum does.
That would allow other users to avoid that overhead.
> > 2. When context block usage exceeds the limit (rare), make the
additional effort to get the precise usage -- I'm not sure such a top-down
facility exists, and I'm not feeling well enough today to study this
further.
> tracking chunk memory space. IIRC we've not evaluated it yet.

Ok, let's try this -- I can test and profile later this week.

---

From: Masahiko Sawada <[email-masked]>
Date: Thu, 9 Mar 2023 15:51:05 +0900

On Wed, Mar 8, 2023 at 1:40 PM John Naylor <[email-masked]> wrote:

> On Tue, Mar 7, 2023 at 8:25 AM Masahiko Sawada <[email-masked]> wrote:
>

Thanks!

I've attached the new version patches. I merged improvements and fixes
I did in the v29 patch. 0007 through 0010 are updates from v29. The
main change made in v30 is to make the memory measurement and
RT_MEMORY_USAGE() optional, which is done in 0007 patch. The 0008 and
0009 patches are the updates for tidstore and the vacuum integration
patches. Here are results of quick tests (an average of 3 executions):

query: select * from bench_load_random_int(10 * 1000 * 1000)

* w/ RT_MEASURE_MEMORY_USAGE:
 mem_allocated | load_ms
---------------+---------
    1996512000 |    3305
(1 row)

* w/o RT_MEASURE_MEMORY_USAGE:
 mem_allocated | load_ms
---------------+---------
             0 |    3258
(1 row)

It seems to be within a noise level but I agree to make it optional.

Apart from the memory measurement stuff, I've done another todo item
on my list; adding min max classes for node3 and node125. I've done
that in 0010 patch, and here is a quick test result:

query: select * from bench_load_random_int(10 * 1000 * 1000)

* w/ 0000 patch
 mem_allocated | load_ms
    1268630080 |    3275
(1 row)

* w/o 0000 patch
 mem_allocated | load_ms
    1996512000 |    3214
(1 row)

That's a good improvement on the memory usage, without a noticeable
performance overhead. FYI CLASS_3_MIN has 1 fanout and is 24 bytes in
size, and CLASS_125_MIN has 61 fanouts and is 768 bytes in size.

Regards,

---

From: John Naylor <[email-masked]>
Date: Fri, 10 Mar 2023 13:42:33 +0700

On Thu, Mar 9, 2023 at 1:51 PM Masahiko Sawada <[email-masked]>
wrote:

> I did in the v29 patch.

I haven't yet had a chance to look at those closely, since I've had to
devote time to other commitments. I remember I wasn't particularly
impressed that v29-0008 mixed my requested name-casing changes with a bunch
of other random things. Separating those out would be an obvious way to
make it easier for me to look at, whenever I can get back to this. I need
to look at the iteration changes as well, in addition to testing memory
measurement (thanks for the new results, they look encouraging).

> on my list; adding min max classes for node3 and node125. I've done

This didn't help us move us closer to something committable the first time
you coded this without making sure it was a good idea. It's still not
helping and arguably makes it worse. To be fair, I did speak positively
about _considering_ additional size classes some months ago, but that has a
very obvious maintenance cost, something we can least afford right now.

I'm frankly baffled you thought this was important enough to work on again,
yet thought it was a waste of time to try to prove to ourselves that
autovacuum in a realistic, non-deterministic workload gave the same answer
as the current tid lookup. Even if we had gone that far, it doesn't seem
like a good idea to add non-essential code to critical paths right now.

We're rapidly running out of time, and we're at the point in the cycle
where it's impossible to get meaningful review from anyone not already
intimately familiar with the patch series. I only want to see progress on
addressing possible (especially architectural) objections from the
community, because if they don't notice them now, they surely will after
commit. I have my own list of possible objections as well as bikeshedding
points, which I'll clean up and share next week. I plan to invite Andres to
look at that list and give his impressions, because it's a lot quicker than
reading the patches. Based on that, I'll hopefully be able to decide
whether we have enough time to address any feedback and do remaining
polishing in time for feature freeze.

I'd suggest sharing your todo list in the meanwhile, it'd be good to
discuss what's worth doing and what is not.

---

From: Masahiko Sawada <[email-masked]>
Date: Fri, 10 Mar 2023 23:30:04 +0900

On Fri, Mar 10, 2023 at 3:42 PM John Naylor
<[email-masked]> wrote:
>
> On Thu, Mar 9, 2023 at 1:51 PM Masahiko Sawada <[email-masked]> wrote:
>
> I haven't yet had a chance to look at those closely, since I've had to devote time to other commitments. I remember I wasn't particularly impressed that v29-0008 mixed my requested name-casing changes with a bunch of other random things. Separating those out would be an obvious way to make it easier for me to look at, whenever I can get back to this. I need to look at the iteration changes as well, in addition to testing memory measurement (thanks for the new results, they look encouraging).

Okay, I'll separate them again.

> This didn't help us move us closer to something committable the first time you coded this without making sure it was a good idea. It's still not helping and arguably makes it worse. To be fair, I did speak positively about _considering_ additional size classes some months ago, but that has a very obvious maintenance cost, something we can least afford right now.
>
> I'm frankly baffled you thought this was important enough to work on again, yet thought it was a waste of time to try to prove to ourselves that autovacuum in a realistic, non-deterministic workload gave the same answer as the current tid lookup. Even if we had gone that far, it doesn't seem like a good idea to add non-essential code to critical paths right now.

I didn't think that proving tidstore and the current tid lookup return
the same result was a waste of time. I've shared a patch to do that in
tidstore before. I agreed not to add it to the tree but we can test
that using this patch. Actually I've done a test that ran pgbench
workload for a few days.

IIUC it's still important to consider whether to have node1 since it
could be a good alternative for the path compression. The prototype
also implemented it. Of course we can leave it for future improvement.
But considering this item with the performance tests helps us to prove
our decoupling approach is promising.

> We're rapidly running out of time, and we're at the point in the cycle where it's impossible to get meaningful review from anyone not already intimately familiar with the patch series. I only want to see progress on addressing possible (especially architectural) objections from the community, because if they don't notice them now, they surely will after commit.

Right, we've been making many design decisions. Some of them are
agreed just between you and me and some are agreed with other hackers.
There are some irrevertible design decisions due to the remaining
time.

>  I have my own list of possible objections as well as bikeshedding points, which I'll clean up and share next week.

Thanks.

>  I plan to invite Andres to look at that list and give his impressions, because it's a lot quicker than reading the patches. Based on that, I'll hopefully be able to decide whether we have enough time to address any feedback and do remaining polishing in time for feature freeze.
>
> I'd suggest sharing your todo list in the meanwhile, it'd be good to discuss what's worth doing and what is not.

Apart from more rounds of reviews and tests, my todo items that need
discussion and possibly implementation are:

* The memory measurement in radix trees and the memory limit in
tidstores. I've implemented it in v30-0007 through 0009 but we need to
review it. This is the highest priority for me.

* Additional size classes. It's important for an alternative of path
compression as well as supporting our decoupling approach. Middle
priority.

* Node shrinking support. Low priority.

Regards,

---

From: Masahiko Sawada <[email-masked]>
Date: Sat, 11 Mar 2023 00:26:18 +0900

> Okay, I'll separate them again.

Attached new patch series. In addition to separate them again, I've
fixed a conflict with HEAD.

Regards,

---

From: John Naylor <[email-masked]>
Date: Sat, 11 Mar 2023 22:54:40 +0700

Agreed.

> compression as well as supporting our decoupling approach. Middle

I'm going to push back a bit and claim this doesn't bring much gain, while
it does have a complexity cost. The node1 from Andres's prototype is 32
bytes in size, same as our node3, so it's roughly equivalent as a way to
ameliorate the lack of path compression. I say "roughly" because the loop
in node3 is probably noticeably slower. A new size class will by definition
still use that loop.

About a smaller node125-type class: I'm actually not even sure we need to
have any sub-max node bigger about 64 (node size 768 bytes). I'd just let
65+ go to the max node -- there won't be many of them, at least in
synthetic workloads we've seen so far.

> * Node shrinking support. Low priority.

This is an architectural wart that's been neglected since the tid store
doesn't perform deletion. We'll need it sometime. If we're not going to
make this work, why ship a deletion API at all?

I took a look at this a couple weeks ago, and fixing it wouldn't be that
hard. I even had an idea of how to detect when to shrink size class within
a node kind, while keeping the header at 5 bytes. I'd be willing to put
effort into that, but to have a chance of succeeding, I'm unwilling to make
it more difficult by adding more size classes at this point.

---

From: Masahiko Sawada <[email-masked]>
Date: Mon, 13 Mar 2023 10:41:16 +0900

On Sun, Mar 12, 2023 at 12:54 AM John Naylor
<[email-masked]> wrote:
>
> On Fri, Mar 10, 2023 at 9:30 PM Masahiko Sawada <[email-masked]> wrote:
> >
> I'm going to push back a bit and claim this doesn't bring much gain, while it does have a complexity cost. The node1 from Andres's prototype is 32 bytes in size, same as our node3, so it's roughly equivalent as a way to ameliorate the lack of path compression.

But does it mean that our node1 would help reduce the memory further
since since our base node type (i.e. RT_NODE) is smaller than the base
node type of Andres's prototype? The result I shared before showed
1.2GB vs. 1.9GB.

> I say "roughly" because the loop in node3 is probably noticeably slower. A new size class will by definition still use that loop.

I've evaluated the performance of node1 but the result seems to show
the opposite. I used the test query:

select * from bench_search_random_nodes(100 * 1000 * 1000,
'0xFF000000000000FF');

Which make the radix tree that has node1 like:

max_val = 18446744073709551615
num_keys = 65536
height = 7, n1 = 1536, n3 = 0, n15 = 0, n32 = 0, n61 = 0, n256 = 257

All internal nodes except for the root node are node1. The radix tree
that doesn't have node1 is:

max_val = 18446744073709551615
num_keys = 65536
height = 7, n3 = 1536, n15 = 0, n32 = 0, n125 = 0, n256 = 257

Here is the result:

* w/ node1
        573448 |    1848 |      1707
(1 row)

* w/o node1
        598024 |    2014 |      1825
(1 row)

Am I missing something?

>
> About a smaller node125-type class: I'm actually not even sure we need to have any sub-max node bigger about 64 (node size 768 bytes). I'd just let 65+ go to the max node -- there won't be many of them, at least in synthetic workloads we've seen so far.

Makes sense to me.

> This is an architectural wart that's been neglected since the tid store doesn't perform deletion. We'll need it sometime. If we're not going to make this work, why ship a deletion API at all?
>
> I took a look at this a couple weeks ago, and fixing it wouldn't be that hard. I even had an idea of how to detect when to shrink size class within a node kind, while keeping the header at 5 bytes. I'd be willing to put effort into that, but to have a chance of succeeding, I'm unwilling to make it more difficult by adding more size classes at this point.

I think that the deletion (and locking support) doesn't have use cases
in the core (i.e. tidstore) but is implemented so that external
extensions can use it. There might not be such extensions. Given the
lack of use cases in the core (and the rest of time), I think it's
okay even if the implementation of such API is minimal and not
optimized enough.  For instance, the implementation of dshash.c is
minimalist, and doesn't have resizing. We can improve them in the
future if extensions or other core features want.

Personally I think we should focus on addressing feedback that we
would get and improving the existing use cases for the rest of time.
That's why considering min-max size class has a higher priority than
the node shrinking support in my todo list.

FYI, I've run TPC-C workload over the weekend, and didn't get any
failures of the assertion proving tidstore and the current tid lookup
return the same result.

Regards,

---

From: John Naylor <[email-masked]>
Date: Mon, 13 Mar 2023 20:28:09 +0700

wrote:

> > I'm going to push back a bit and claim this doesn't bring much gain,
while it does have a complexity cost. The node1 from Andres's prototype is
32 bytes in size, same as our node3, so it's roughly equivalent as a way to
ameliorate the lack of path compression.
> node type of Andres's prototype? The result I shared before showed

The benefit is found in a synthetic benchmark with random integers. I
highly doubt that anyone would be willing to force us to keep
binary-searching the 1GB array for one more cycle on account of not adding
a size class here. I'll repeat myself and say that there are also
maintenance costs.

In contrast, I'm fairly certain that our attempts thus far at memory
accounting/limiting are not quite up to par, and lacking enough to
jeopardize the feature. We're already discussing that, so I'll say no more.

> > I say "roughly" because the loop in node3 is probably noticeably
slower. A new size class will by definition still use that loop.
> the opposite.

As an aside, I meant the loop in our node3 might make your node1 slower
than the prototype's node1, which was coded for 1 member only.

> >
> > I took a look at this a couple weeks ago, and fixing it wouldn't be
that hard. I even had an idea of how to detect when to shrink size class
within a node kind, while keeping the header at 5 bytes. I'd be willing to
put effort into that, but to have a chance of succeeding, I'm unwilling to
make it more difficult by adding more size classes at this point.
> extensions can use it.

I think these cases are a bit different: Doing anything with a data
structure stored in shared memory without a synchronization scheme is
completely unthinkable and insane. I'm not yet sure if
deleting-without-shrinking is a showstopper, or if it's preferable in v16
than no deletion at all.

Anything we don't implement now is a limit on future use cases, and thus a
cause for objection. On the other hand, anything we implement also
represents more stuff that will have to be rewritten for high-concurrency.

> failures of the assertion proving tidstore and the current tid lookup

Great!

---

From: Masahiko Sawada <[email-masked]>
Date: Mon, 13 Mar 2023 23:55:29 +0900

On Mon, Mar 13, 2023 at 10:28 PM John Naylor
<[email-masked]> wrote:
>
> On Mon, Mar 13, 2023 at 8:41 AM Masahiko Sawada <[email-masked]> wrote:
> >
> The benefit is found in a synthetic benchmark with random integers. I highly doubt that anyone would be willing to force us to keep binary-searching the 1GB array for one more cycle on account of not adding a size class here. I'll repeat myself and say that there are also maintenance costs.
>
> In contrast, I'm fairly certain that our attempts thus far at memory accounting/limiting are not quite up to par, and lacking enough to jeopardize the feature. We're already discussing that, so I'll say no more.

I agree that memory accounting/limiting stuff is the highest priority.
So what kinds of size classes do you think we need? node3, 15, 32, 61
and 256?

> As an aside, I meant the loop in our node3 might make your node1 slower than the prototype's node1, which was coded for 1 member only.

Agreed.

> I think these cases are a bit different: Doing anything with a data structure stored in shared memory without a synchronization scheme is completely unthinkable and insane.

Right.

> I'm not yet sure if deleting-without-shrinking is a showstopper, or if it's preferable in v16 than no deletion at all.
>
> Anything we don't implement now is a limit on future use cases, and thus a cause for objection. On the other hand, anything we implement also represents more stuff that will have to be rewritten for high-concurrency.

Okay. Given that adding shrinking support also requires maintenance
costs (and probably new test cases?) and there are no use cases in the
core, I'm not sure it's worth supporting it at this stage. So I prefer
either shipping the deletion API as it is and removing the deletion
API. I think that it's a discussion point that we'd like to hear
feedback from other hackers.

Regards,

---

From: John Naylor <[email-masked]>
Date: Tue, 14 Mar 2023 18:27:37 +0700

I wrote:

> > > Since the block-level measurement is likely overestimating quite a
bit, I propose to simply reverse the order of the actions here, effectively
> 1. Make it optional to track chunk memory space by a template parameter.
It might be tiny compared to everything else that vacuum does. That would
allow other users to avoid that overhead.

Since then, Masahiko incorporated #1 into v31, and that's what I'm looking
at now. Unfortunately, If I had spent five minutes reminding myself what
the original objections were to this approach, I could have saved us some
effort. Back in July (!), Andres raised two points: GetMemoryChunkSpace()
is slow [1], and fragmentation [2] (leading to underestimation).

In v31, in the local case at least, the underestimation is actually worse
than tracking chunk space, since it ignores chunk header and alignment.
I'm not sure about the DSA case. This doesn't seem great.

It shouldn't be a surprise why a simple increment of raw allocation size is
comparable in speed -- GetMemoryChunkSpace() calls the right function
through a pointer, which is slower. If we were willing to underestimate for
the sake of speed, that takes away the reason for making memory tracking
optional.

Further, if the option is not specified, in v31 there is no way to get the
memory use at all, which seems odd. Surely the caller should be able to ask
the context/area, if it wants to.

I still like my idea at the top of the page -- at least for vacuum and
m_w_m. It's still not completely clear if it's right but I've got nothing
better. It also ignores the work_mem issue, but I've given up anticipating
all future cases at the moment.

I'll put this item and a couple other things together in a separate email
tomorrow.

[1]
https://www.postgresql.org/message-id/20220704211822.kfxtzpcdmslzm2dy%40awork3.anarazel.de
[2]
https://www.postgresql.org/message-id/20220704220038.at2ane5xkymzzssb%40awork3.anarazel.de

---

From: Masahiko Sawada <[email-masked]>
Date: Wed, 15 Mar 2023 11:32:06 +0900

On Tue, Mar 14, 2023 at 8:27 PM John Naylor
<[email-masked]> wrote:
> Since then, Masahiko incorporated #1 into v31, and that's what I'm looking at now. Unfortunately, If I had spent five minutes reminding myself what the original objections were to this approach, I could have saved us some effort. Back in July (!), Andres raised two points: GetMemoryChunkSpace() is slow [1], and fragmentation [2] (leading to underestimation).
>
> In v31, in the local case at least, the underestimation is actually worse than tracking chunk space, since it ignores chunk header and alignment.  I'm not sure about the DSA case. This doesn't seem great.

Right.

>
> It shouldn't be a surprise why a simple increment of raw allocation size is comparable in speed -- GetMemoryChunkSpace() calls the right function through a pointer, which is slower. If we were willing to underestimate for the sake of speed, that takes away the reason for making memory tracking optional.
>
> Further, if the option is not specified, in v31 there is no way to get the memory use at all, which seems odd. Surely the caller should be able to ask the context/area, if it wants to.

There are precedents that don't provide a way to return memory usage,
such as simplehash.h and dshash.c.

>
> I still like my idea at the top of the page -- at least for vacuum and m_w_m. It's still not completely clear if it's right but I've got nothing better. It also ignores the work_mem issue, but I've given up anticipating all future cases at the moment.
>

What does it mean by "the precise usage" in your idea? Quoting from
the email you referred to, Andres said:

---
One thing I was wondering about is trying to choose node types in
roughly-power-of-two struct sizes. It's pretty easy to end up with significant
fragmentation in the slabs right now when inserting as you go, because some of
the smaller node types will be freed but not enough to actually free blocks of
memory. If we instead have ~power-of-two sizes we could just use a single slab
of the max size, and carve out the smaller node types out of that largest
allocation.

Btw, that fragmentation is another reason why I think it's better to track
memory usage via memory contexts, rather than doing so based on
GetMemoryChunkSpace().
---

IIUC he suggested measuring memory usage in block-level in order to
count blocks that are not actually freed but some of its chunks are
freed. That's why we used MemoryContextMemAllocated(). On the other
hand, recently you pointed out[1]:

---
I think we're trying to solve the wrong problem here. I need to study
this more, but it seems that code that needs to stay within a memory
limit only needs to track what's been allocated in chunks within a
block, since writing there is what invokes a page fault.
---

IIUC you suggested measuring memory usage by tracking how much memory
chunks are allocated within a block. If your idea at the top of the
page follows this method, it still doesn't deal with the point Andres
mentioned.

> I'll put this item and a couple other things together in a separate email tomorrow.

Thanks!

Regards,

[1] https://www.postgresql.org/message-id/CAFBsxsEnzivaJ13iCGdDoUMsXJVGOaahuBe_y%3Dq6ow%3DLTzyDvA%40mail.gmail.com

---

From: John Naylor <[email-masked]>
Date: Fri, 17 Mar 2023 14:02:53 +0700

On Wed, Mar 15, 2023 at 9:32 AM Masahiko Sawada <[email-masked]>
wrote:
> > > > > Since the block-level measurement is likely overestimating quite
a bit, I propose to simply reverse the order of the actions here,
effectively reporting progress for the *last page* and not the current one:
First update progress with the current memory usage, then add tids for this
page. If this allocated a new block, only a small bit of that will be
written to. If this block pushes it over the limit, we will detect that up
at the top of the loop. It's kind of like our earlier attempts at a "fudge
factor", but simpler and less brittle. And, as far as OS pages we have
actually written to, I think it'll effectively respect the memory limit, at
least in the local mem case. And the numbers will make sense.

> page follows this method, it still doesn't deal with the point Andres

Right, but that idea was orthogonal to how we measure memory use, and in
fact mentions blocks specifically. The re-ordering was just to make sure
that progress reporting didn't show current-use > max-use.

However, the big question remains DSA, since a new segment can be as large
as the entire previous set of allocations. It seems it just wasn't designed
for things where memory growth is unpredictable.

I'm starting to wonder if we need to give DSA a bit more info at the start.
Imagine a "soft" limit given to the DSA area when it is initialized. If the
total segment usage exceeds this, it stops doubling and instead new
segments get smaller. Modifying an example we used for the fudge-factor
idea some time ago:

m_w_m = 1GB, so calculate the soft limit to be 512MB and pass it to the DSA
area.

2*(1+2+4+8+16+32+64+128) + 256 = 766MB (74.8% of 1GB) -> hit soft limit, so
"stairstep down" the new segment sizes:

766 + 2*(128) + 64 = 1086MB -> stop

That's just an undeveloped idea, however, so likely v17 development, even
assuming it's not a bad idea (could be).

And sadly, unless we find some other, simpler answer soon for tracking and
limiting shared memory, the tid store is looking like v17 material.

---

From: Masahiko Sawada <[email-masked]>
Date: Fri, 17 Mar 2023 16:49:33 +0900

On Fri, Mar 17, 2023 at 4:03 PM John Naylor
<[email-masked]> wrote:
>
> On Wed, Mar 15, 2023 at 9:32 AM Masahiko Sawada <[email-masked]> wrote:
> >
> Right, but that idea was orthogonal to how we measure memory use, and in fact mentions blocks specifically. The re-ordering was just to make sure that progress reporting didn't show current-use > max-use.

Right. I still like your re-ordering idea. It's true that the most
area of the last allocated block before heap scanning stops is not
actually used yet. I'm guessing we can just check if the context
memory has gone over the limit. But I'm concerned it might not work
well in systems where overcommit memory is disabled.

>
> However, the big question remains DSA, since a new segment can be as large as the entire previous set of allocations. It seems it just wasn't designed for things where memory growth is unpredictable.
>
> I'm starting to wonder if we need to give DSA a bit more info at the start. Imagine a "soft" limit given to the DSA area when it is initialized. If the total segment usage exceeds this, it stops doubling and instead new segments get smaller. Modifying an example we used for the fudge-factor idea some time ago:
>
> m_w_m = 1GB, so calculate the soft limit to be 512MB and pass it to the DSA area.
>
> 2*(1+2+4+8+16+32+64+128) + 256 = 766MB (74.8% of 1GB) -> hit soft limit, so "stairstep down" the new segment sizes:
>
> That's just an undeveloped idea, however, so likely v17 development, even assuming it's not a bad idea (could be).

This is an interesting idea. But I'm concerned we don't have enough
time to get confident with adding this new concept to DSA.

>
> And sadly, unless we find some other, simpler answer soon for tracking and limiting shared memory, the tid store is looking like v17 material.

Another problem we need to deal with is the supported minimum memory
in shared tidstore cases. Since the initial DSA segment size is 1MB,
memory usage of a shared tidstore will start from 1MB+. This is higher
than the minimum values of both work_mem and maintenance_work_mem,
64kB and 1MB respectively. Increasing the minimum m_w_m to 2MB seems
to be acceptable in the community but not for work_mem. One idea is to
deny the memory limit less than 2MB so it won't work with small m_w_m
settings. While it might be an acceptable restriction at this stage
(where there is no use case of using tidstore with work_mem in the
core) but it will be a blocker for the future adoptions such as
unifying with tidbitmap.c. Another idea is that the process can
specify the initial segment size at dsa_create() so that DSA can start
with a smaller segment, say 32kB. That way, a tidstore with a 32kB
limit gets full once it allocates the next DSA segment, 32kB. . But a
downside of this idea is to increase the number of segments behind
DSA. Assuming it's a relatively rare case where we use such a low
work_mem, it might be acceptable. FYI, the total number of DSM
segments available on the system is calculated by:

#define PG_DYNSHMEM_FIXED_SLOTS         64
#define PG_DYNSHMEM_SLOTS_PER_BACKEND   5

maxitems = PG_DYNSHMEM_FIXED_SLOTS
    + PG_DYNSHMEM_SLOTS_PER_BACKEND * MaxBackends;

Regards,

---

From: Masahiko Sawada <[email-masked]>
Date: Mon, 20 Mar 2023 14:24:38 +0900

> > However, the big question remains DSA, since a new segment can be as large as the entire previous set of allocations. It seems it just wasn't designed for things where memory growth is unpredicta…

aset.c also has a similar characteristic; allocates an 8K block upon
the first allocation in a context, and doubles that size for each
successive block request. But we can specify the initial block size
and max blocksize. This made me think of another idea to specify both
to DSA and both values are calculated based on m_w_m. For example, we
can create a DSA in parallel_vacuum_init() as follows:

initial block size = min(m_w_m / 4, 1MB)
max block size = max(m_w_m / 8, 8MB)

In most cases, we can start with a 1MB initial segment, the same as
before. For small memory cases, say 1MB, we start with a 256KB initial
segment and heap scanning stops after DSA allocated 1.5MB (= 256kB +
256kB + 512kB + 512kB). For larger memory, we can have heap scan stop
after DSA allocates 1.25 times more memory than m_w_m. For example, if
m_w_m = 1GB, the both initial and maximum segment sizes are 1MB and
128MB respectively, and then DSA allocates the segments as follows
until heap scanning stops:

2 * (1 + 2 + 4 + 8 + 16 + 32 + 64 + 128) + (128 * 5) = 1150MB

dsa_allocate() will be extended to have the initial and maximum block
sizes like AllocSetContextCreate().

Regards,

---

From: John Naylor <[email-masked]>
Date: Fri, 7 Apr 2023 16:55:41 +0700

> filled in-order, for example.

I spent some time studying tidbitmap.c, and not only does it make sense to
use a radix tree there, but since it has more complex behavior and stricter
runtime requirements, it should really be the thing driving the design and
tradeoffs, not vacuum:

- With lazy expansion and single-value leaves, the root of a radix tree can
point to a single leaf. That might get rid of the need to track TBMStatus,
since setting a single-leaf tree should be cheap.

- Fixed-size PagetableEntry's are pretty large, but the tid compression
scheme used in this thread (in addition to being complex) is not a great
fit for tidbitmap because it makes it more difficult to track per-block
metadata (see also next point). With the "combined pointer-value slots"
technique, if a page's max tid offset is 63 or less, the offsets can be
stored directly in the pointer for the exact case. The lowest bit can tag
to indicate a pointer to a single-value leaf. That would complicate
operations like union/intersection and tracking "needs recheck", but it
would reduce memory use and node-traversal in common cases.

- Managing lossy storage. With pure blocknumber keys, replacing exact
storage for a range of 256 pages amounts to replacing a last-level node
with a single leaf containing one lossy PagetableEntry. The leader could
iterate over the nodes, and rank the last-level nodes by how much storage
they (possibly with leaf children) are using, and come up with an optimal
lossy-conversion plan.

The above would address the points (not including better iteration and
parallel bitmap index scans) raised in

https://www.postgresql.org/message-id/[email-masked]

Ironically, by targeting a more difficult use case, it's easier since there
is less freedom. There are many ways to beat a binary search, but fewer
good ways to improve bitmap heap scan. I'd like to put aside vacuum for
some time and try killing two birds with one stone, building upon our work
thus far.

Note: I've moved the CF entry to the next CF, and set to waiting on
author for now. Since no action is currently required from Masahiko, I've
added myself as author as well. If tackling bitmap heap scan shows promise,
we could RWF and resurrect at a later time.

---

From: Masahiko Sawada <[email-masked]>
Date: Mon, 17 Apr 2023 22:48:27 +0900

On Fri, Apr 7, 2023 at 6:55 PM John Naylor <[email-masked]> wrote:
> I spent some time studying tidbitmap.c, and not only does it make sense to use a radix tree there, but since it has more complex behavior and stricter runtime requirements, it should really be the thing driving the design and tradeoffs, not vacuum:
>
> - With lazy expansion and single-value leaves, the root of a radix tree can point to a single leaf. That might get rid of the need to track TBMStatus, since setting a single-leaf tree should be cheap.
>

Instead of introducing single-value leaves to the radix tree as
another structure, can we store pointers to PagetableEntry as values?

> - Fixed-size PagetableEntry's are pretty large, but the tid compression scheme used in this thread (in addition to being complex) is not a great fit for tidbitmap because it makes it more difficult to track per-block metadata (see also next point). With the "combined pointer-value slots" technique, if a page's max tid offset is 63 or less, the offsets can be stored directly in the pointer for the exact case. The lowest bit can tag to indicate a pointer to a single-value leaf. That would complicate operations like union/intersection and tracking "needs recheck", but it would reduce memory use and node-traversal in common cases.
>
> - Managing lossy storage. With pure blocknumber keys, replacing exact storage for a range of 256 pages amounts to replacing a last-level node with a single leaf containing one lossy PagetableEntry. The leader could iterate over the nodes, and rank the last-level nodes by how much storage they (possibly with leaf children) are using, and come up with an optimal lossy-conversion plan.
>
> The above would address the points (not including better iteration and parallel bitmap index scans) raised in
>
> Ironically, by targeting a more difficult use case, it's easier since there is less freedom. There are many ways to beat a binary search, but fewer good ways to improve bitmap heap scan. I'd like to put aside vacuum for some time and try killing two birds with one stone, building upon our work thus far.
>
> Note: I've moved the CF entry to the next CF, and set to waiting on author for now. Since no action is currently required from Masahiko, I've added myself as author as well. If tackling bitmap heap scan shows promise, we could RWF and resurrect at a later time.

Thanks. I'm going to continue researching the memory limitation and
try lazy path expansion until PG17 development begins.

Regards,

---

From: Masahiko Sawada <[email-masked]>
Date: Tue, 18 Apr 2023 00:20:16 +0900

> fixed a conflict with HEAD.

I've attached updated version patches to make cfbot happy. Also, I've
splitted fixup patches further(from 0007 except for 0016 and 0018) to
make reviews easy. These patches have the prefix radix tree, tidstore,
and vacuum, indicating the part it changes. 0016 patch is to change
DSA so that we can specify both the initial and max segment size and
0017 makes use of it in vacuumparallel.c I'm still researching a
better solution for memory limitation but it's the best solution for
me for now.

Regards,

---

From: John Naylor <[email-masked]>
Date: Wed, 19 Apr 2023 14:02:14 +0700

On Mon, Apr 17, 2023 at 8:49 PM Masahiko Sawada <[email-masked]>
wrote:

> > - With lazy expansion and single-value leaves, the root of a radix tree
can point to a single leaf. That might get rid of the need to track
TBMStatus, since setting a single-leaf tree should be cheap.
> another structure, can we store pointers to PagetableEntry as values?

Well, that's pretty much what a single-value leaf is. Now that I've had
time to pause and regroup, I've looked into some aspects we previously put
off for future work, and this is one of them.

The concept is really quite trivial, and it's the simplest and most
flexible way to implement ART. Our, or at least my, documented reason not
to go that route was due to "an extra pointer traversal", but that's
partially mitigated by "lazy expansion", which is actually fairly easy to
do with single-value leaves. The two techniques complement each other in a
natural way. (Path compression, on the other hand, is much more complex.)

> Thanks. I'm going to continue researching the memory limitation and

Sounds like the best thing to nail down at this point.

> try lazy path expansion until PG17 development begins.

This doesn't seem like a useful thing to try and attach into the current
patch (if that's what you mean), as the current insert/delete paths are
quite complex. Using bitmap heap scan as a motivating use case, I hope to
refocus complexity to where it's most needed, and aggressively simplify
where possible.

---

From: Masahiko Sawada <[email-masked]>
Date: Mon, 24 Apr 2023 14:45:04 +0900

On Wed, Apr 19, 2023 at 4:02 PM John Naylor
<[email-masked]> wrote:
>
> On Mon, Apr 17, 2023 at 8:49 PM Masahiko Sawada <[email-masked]> wrote:
>
> Well, that's pretty much what a single-value leaf is. Now that I've had time to pause and regroup, I've looked into some aspects we previously put off for future work, and this is one of them.
>
> The concept is really quite trivial, and it's the simplest and most flexible way to implement ART. Our, or at least my, documented reason not to go that route was due to "an extra pointer traversal", but that's partially mitigated by "lazy expansion", which is actually fairly easy to do with single-value leaves. The two techniques complement each other in a natural way. (Path compression, on the other hand, is much more complex.)
>
> This doesn't seem like a useful thing to try and attach into the current patch (if that's what you mean), as the current insert/delete paths are quite complex. Using bitmap heap scan as a motivating use case, I hope to refocus complexity to where it's most needed, and aggressively simplify where possible.
>

I agree that we don't want to make the current patch complex further.

Thinking about the memory limitation more, I think that combination of
the idea of specifying the initial and max DSA segment size and
dsa_set_size_limit() works well. There are two points in terms of
memory limitation; when the memory usage reaches the limit we want (1)
to minimize the last allocated memory block that is allocated but not
used yet and (2) to minimize the amount of memory that exceeds the
memory limit. Since we can specify the maximum DSA segment size, the
last allocated block before reaching the memory limit is small. Also,
thanks to dsa_set_size_limit(), the total DSA size will stop at the
limit, so (memory_usage >= memory_limit) returns true without any
exceeding memory.

Given that we need to configure the initial and maximum DSA segment
size and set the DSA limit for TidStore memory accounting and
limiting, it would be better to create the DSA for TidStore by
TidStoreCreate() API, rather than creating DSA in the caller and pass
it to TidStoreCreate().

Regards,

---

From: John Naylor <[email-masked]>
Date: Mon, 8 May 2023 17:23:23 +0700

[just getting some thoughts out there before I have something concrete]

Thinking some more, this needn't be complicated at all. We'd just need to
reserve some bits of a bitmapword for the tag, as well as flags for
"ischunk" and "recheck". The other bits can be used for offsets.
Getting/storing the offsets basically amounts to adjusting the shift by a
constant. That way, this "embeddable PTE" could serve as both "PTE embedded
in a node pointer" and also the first member of a full PTE. A full PTE is
now just an array of embedded PTEs, except only the first one has the flags
we need. That reduces the number of places that have to be different.
Storing any set of offsets all less than ~60 would save
allocation/traversal in a large number of real cases. Furthermore, that
would reduce a full PTE to 40 bytes because there would be no padding.

This all assumes the key (block number) is no longer stored in the PTE,

...is not a good trade off because it requires each leaf to have the key,
and would thus reduce the utility of embedded leaves. We just need to make
sure storing a single value is not costly, and I suspect it's not.
(Currently the overhead avoided is allocating and zeroing a few kilobytes
for a hash table). If it is not, then we don't need a special case in
tidbitmap, which would be a great simplification. If it is, there are other
ways to mitigate.

---

From: John Naylor <[email-masked]>
Date: Wed, 24 May 2023 06:16:54 +0700

I wrote:
> the current insert/delete paths are quite complex. Using bitmap heap scan
as a motivating use case, I hope to refocus complexity to where it's most
needed, and aggressively simplify where possible.

Sometime in the not-too-distant future, I will start a new thread focusing
on bitmap heap scan, but for now, I just want to share some progress on
making the radix tree usable not only for that, but hopefully a wider range
of applications, while making the code simpler and the binary smaller. The
attached patches are incomplete (e.g. no iteration) and quite a bit messy,
so tar'd and gzip'd for the curious (should apply on top of v32 0001-03 +
0007-09 ).

0001

This combines a few concepts that I didn't bother separating out after the
fact:
- Split insert_impl.h into multiple functions for improved readability and
maintainability.
- Use single-value leaves as the basis for storing values, with the goal to
get to "combined pointer-value slots" for efficiency and flexibility.
- With the latter in mind, searching the child within a node now returns
the address of the slot. This allows the same interface whether the slot
contains a child pointer or a value.
- Starting with RT_SET, start turning some iterative algorithms into
recursive ones. This is a more natural way to traverse a tree structure,
and we already see an advantage: Previously when growing a node, we
searched within the parent to update its reference to the new node, because
we didn't know the slot we descended from. Now we can simply update a
single variable.
- Since we recursively pass the "shift" down the stack, it doesn't have to
be stored in any node -- only the "top-level" start shift is stored in the
tree control struct. This was easy to code since the node's shift value was
hardly ever accessed anyway! The node header shrinks from 5 bytes to 4.

0002

Back in v15, we tried keeping DSA/local pointers as members of a struct. I
did not like the result, but still thought it was a good idea. RT_DELETE is
a complex function and I didn't want to try rewriting it without a pointer
abstraction, so I've resurrected this idea, but in a simpler, less
intrusive way. A key difference from v15 is using a union type for the
non-shmem case.

0004

Rewrite RT_DELETE using recursion. I find this simpler than the previous
open-coded stack.

0005-06

Deletion has an inefficiency: One function searches for the child to see if
it's there, then another function searches for it again to delete it. Since
0001, a successful child search returns the address of the slot, so we can
save it. For the two smaller "linear search" node kinds we can then use a
single subtraction to compute the chunk/slot index for deletion. Also,
split RT_NODE_DELETE_INNER into separate functions, for a similar reason as
the insert case in 0001.

0007

Anticipate node shrinking: If only one node-kind needs to be freed, we can
move a branch to that one code path, rather than every place where RT_FREE
is inlined.

0009

Teach node256 how to shrink *. Since we know the number of children in a
node256 can't possibly be zero, we can use uint8 to store the count and
interpret an overflow to zero as 256 for this node. The node header shrinks
from 4 bytes to 3.

* Other nodes will follow in due time, but only after I figure out how to
do it nicely (ideas welcome!) -- currently node32's two size classes work
fine for growing, but the code should be simplified before extending to
other cases.)

0010

Limited support for "combined pointer-value slots". At compile-time, choose
either that or "single-value leaves" based on the size of the value type
template parameter. Values that are pointer-sized or less can fit in the
last-level child slots of nominal "inner nodes" without duplicated
leaf-node code. Node256 now must act like the previous 'node256 leaf',
since zero is a valid value. Aside from that, this was a small change.

What I've shared here could work (in principal, since it uses uint64
values) for tidstore, possibly faster (untested) because of better code
density, but as mentioned I want to shoot for higher. For tidbitmap.c, I
want to extend this idea and branch at run-time on a per-value basis, so
that a page-table entry that fits in a pointer can go there, and if not,
it'll be a full leaf. (This technique enables more flexibility in
lossifying pages as well.) Run-time info will require e.g. an additional
bit per slot. Since the node header is now 3 bytes, we can spare one more
byte in the node3 case. In addition, we can and should also bump it back up
to node4, still keeping the metadata within 8 bytes (no struct padding).

I've started in this patchset to refer to the node kinds as "4/16/48/256",
regardless of their actual fanout. This is for readability (by matching the
language in the paper) and maintainability (should *not* ever change
again). The size classes (including multiple classes per kind) could be
determined by macros and #ifdef's. For example, in non-SIMD architectures,
it's likely slow to search an array of 32 key chunks, so in that case the
compiler should choose size classes similar to these four nominal kinds.

---

From: Masahiko Sawada <[email-masked]>
Date: Mon, 5 Jun 2023 06:31:41 -0400

Hi,

On Tue, May 23, 2023 at 7:17 PM John Naylor
<[email-masked]> wrote:
>
> I wrote:
> > the current insert/delete paths are quite complex. Using bitmap heap scan as a motivating use case, I hope to refocus complexity to where it's most needed, and aggressively simplify where possible.
>
> Sometime in the not-too-distant future, I will start a new thread focusing on bitmap heap scan, but for now, I just want to share some progress on making the radix tree usable not only for that, but hopefully a wider range of applications, while making the code simpler and the binary smaller. The attached patches are incomplete (e.g. no iteration) and quite a bit messy, so tar'd and gzip'd for the curious (should apply on top of v32 0001-03 + 0007-09 ).
>

Thank you for making progress on this. I agree with these directions
overall. I have some comments and questions:

> - With the latter in mind, searching the child within a node now returns the address of the slot. This allows the same interface whether the slot contains a child pointer or a value.

Probably we can apply similar changes to the iteration as well.

> * Other nodes will follow in due time, but only after I figure out how to do it nicely (ideas welcome!) -- currently node32's two size classes work fine for growing, but the code should be simplified before extending to other cases.)

Within the size class, we just alloc a new node of lower size class
and do memcpy(). I guess it will be almost same as what we do for
growing. It might be a good idea to support node shrinking within the
size class for node32 (and node125 if we support). I don't think
shrinking class-3 to class-1 makes sense.

>
> Limited support for "combined pointer-value slots". At compile-time, choose either that or "single-value leaves" based on the size of the value type template parameter. Values that are pointer-sized or less can fit in the last-level child slots of nominal "inner nodes" without duplicated leaf-node code. Node256 now must act like the previous 'node256 leaf', since zero is a valid value. Aside from that, this was a small change.

Yes, but it also means that we use pointer-sized value anyway even if
the value size is less than that, which wastes the memory, no?

>
> What I've shared here could work (in principal, since it uses uint64 values) for tidstore, possibly faster (untested) because of better code density, but as mentioned I want to shoot for higher. For tidbitmap.c, I want to extend this idea and branch at run-time on a per-value basis, so that a page-table entry that fits in a pointer can go there, and if not, it'll be a full leaf. (This technique enables more flexibility in lossifying pages as well.) Run-time info will require e.g. an additional bit per slot. Since the node header is now 3 bytes, we can spare one more byte in the node3 case. In addition, we can and should also bump it back up to node4, still keeping the metadata within 8 bytes (no struct padding).

Sounds good.

> I've started in this patchset to refer to the node kinds as "4/16/48/256", regardless of their actual fanout. This is for readability (by matching the language in the paper) and maintainability (should *not* ever change again). The size classes (including multiple classes per kind) could be determined by macros and #ifdef's. For example, in non-SIMD architectures, it's likely slow to search an array of 32 key chunks, so in that case the compiler should choose size classes similar to these four nominal kinds.

If we want to use the node kinds used in the paper, I think we should
change the number in RT_NODE_KIND_X too. Otherwise, it would be
confusing when reading the code without referring to the paper.
Particularly, this part is very confusing:

        case RT_NODE_KIND_3:
            RT_ADD_CHILD_4(tree, ref, node, chunk, child);
            break;
        case RT_NODE_KIND_32:
            RT_ADD_CHILD_16(tree, ref, node, chunk, child);
            break;
        case RT_NODE_KIND_125:
            RT_ADD_CHILD_48(tree, ref, node, chunk, child);
            break;
        case RT_NODE_KIND_256:
            RT_ADD_CHILD_256(tree, ref, node, chunk, child);
            break;

Regards,

---

From: John Naylor <[email-masked]>
Date: Tue, 6 Jun 2023 12:13:36 +0700

On Mon, Jun 5, 2023 at 5:32 PM Masahiko Sawada <[email-masked]>
wrote:
>
> > Sometime in the not-too-distant future, I will start a new thread
focusing on bitmap heap scan, but for now, I just want to share some
progress on making the radix tree usable not only for that, but hopefully a
wider range of applications, while making the code simpler and the binary
smaller. The attached patches are incomplete (e.g. no iteration) and quite
a bit messy, so tar'd and gzip'd for the curious (should apply on top of
v32 0001-03 + 0007-09 ).
> overall. I have some comments and questions:

Glad to hear it and thanks for looking!

> > * Other nodes will follow in due time, but only after I figure out how
to do it nicely (ideas welcome!) -- currently node32's two size classes
work fine for growing, but the code should be simplified before extending
to other cases.)
> growing.

Oh, the memcpy part is great, very simple. I mean the (compile-time) "class
info" table lookups are a bit awkward. I'm thinking the hard-coded numbers
like this:

.fanout = 3,
.inner_size = sizeof(RT_NODE_INNER_3) + 3 * sizeof(RT_PTR_ALLOC),

...may be better with a #defined symbol that can also be used elsewhere.

> I don't think

Agreed. The smallest kind should just be freed when empty.

> > Limited support for "combined pointer-value slots". At compile-time,
choose either that or "single-value leaves" based on the size of the value
type template parameter. Values that are pointer-sized or less can fit in
the last-level child slots of nominal "inner nodes" without duplicated
leaf-node code. Node256 now must act like the previous 'node256 leaf',
since zero is a valid value. Aside from that, this was a small change.
> the value size is less than that, which wastes the memory, no?

At a low-level, that makes sense, but I've found an interesting global
effect showing the opposite: _less_ memory, which may compensate:

psql -c "select * from bench_search_random_nodes(1*1000*1000)"
num_keys = 992660

(using a low enough number that the experimental change n125->n63 doesn't
affect anything)
height = 4, n3 = 375258, n15 = 137490, n32 = 0, n63 = 0, n256 = 1025

v31:
      47800768 |     253 |       134

(unreleased code "similar" to v33, but among other things restores the
separate "extend down" function)
 mem_allocated | load_ms | search_ms
---------------+---------+-----------
      42926048 |     221 |       127

I'd need to make sure, but apparently just going from 6 non-empty memory
contexts to 3 (remember all values are embedded here) reduces memory
fragmentation significantly in this test. (That should also serve as a
demonstration that additional size classes have both runtime costs as well
as benefits. We need to have a balance.)

So, I'm inclined to think the only reason to prefer "multi-value leaves" is
if 1) the value type is _bigger_ than a pointer 2) there is no convenient
abbreviation (like tid bitmaps have) and 3) the use case really needs to
avoid another memory access. Under those circumstances, though, the new
code plus lazy expansion etc might suit and be easier to maintain. That
said, I've mostly left alone the "leaf" types and functions, as well as
added some detritus like "const bool = false;". It would look a *lot* nicer
if we gave up on multi-value leaves entirely, but there's no rush and I
don't want to close that door entirely just yet.

> Sounds good.

The additional bit per slot would require per-node logic and additional
branches, which is not great. I'm now thinking a much easier way to get
there is to give up (at least for now) on promising that "run-time
embeddable values" can use the full pointer-size (unlike value types found
embeddable at compile-time). Reserving the lowest pointer bit for a tag
"value or pointer-to-leaf" would have a much smaller code footprint. That
also has a curious side-effect for TID offsets: They are one-based so
reserving the zero bit would actually simplify things: getting rid of the
+1/-1 logic when converting bits to/from offsets.

In addition, without a new bitmap, the smallest node can actually be up to
a node5 with no struct padding, with a node2 as a subclass. (Those numbers
coincidentally were also one scenario in the paper, when calculating
worst-case memory usage). That's worth considering.

> > I've started in this patchset to refer to the node kinds as
"4/16/48/256", regardless of their actual fanout.

> change the number in RT_NODE_KIND_X too.

Oh absolutely, this is nowhere near ready for cosmetic review :-)

---

From: Masahiko Sawada <[email-masked]>
Date: Tue, 13 Jun 2023 14:46:57 +0900

On Tue, Jun 6, 2023 at 2:13 PM John Naylor <[email-masked]> wrote:
>
> On Mon, Jun 5, 2023 at 5:32 PM Masahiko Sawada <[email-masked]> wrote:
> >
> Oh, the memcpy part is great, very simple. I mean the (compile-time) "class info" table lookups are a bit awkward. I'm thinking the hard-coded numbers like this:
>
> .fanout = 3,

FWIW, exposing these definitions would be good in terms of testing too
since we can use them in regression tests.

> At a low-level, that makes sense, but I've found an interesting global effect showing the opposite: _less_ memory, which may compensate:
>
> (using a low enough number that the experimental change n125->n63 doesn't affect anything)
> (unreleased code "similar" to v33, but among other things restores the separate "extend down" function)
> I'd need to make sure, but apparently just going from 6 non-empty memory contexts to 3 (remember all values are embedded here) reduces memory fragmentation significantly in this test. (That should also serve as a demonstration that additional size classes have both runtime costs as well as benefits. We need to have a balance.)

Interesting. The result would probably vary if we change the slab
block sizes. I'd like to experiment if the code is available.

>
> So, I'm inclined to think the only reason to prefer "multi-value leaves" is if 1) the value type is _bigger_ than a pointer 2) there is no convenient abbreviation (like tid bitmaps have) and 3) the use case really needs to avoid another memory access. Under those circumstances, though, the new code plus lazy expansion etc might suit and be easier to maintain.

Indeed.

> The additional bit per slot would require per-node logic and additional branches, which is not great. I'm now thinking a much easier way to get there is to give up (at least for now) on promising that "run-time embeddable values" can use the full pointer-size (unlike value types found embeddable at compile-time). Reserving the lowest pointer bit for a tag "value or pointer-to-leaf" would have a much smaller code footprint.

Do you mean we can make sure that the value doesn't set the lowest
bit? Or is it an optimization for TIDStore?

> In addition, without a new bitmap, the smallest node can actually be up to a node5 with no struct padding, with a node2 as a subclass. (Those numbers coincidentally were also one scenario in the paper, when calculating worst-case memory usage). That's worth considering.

Agreed.

FWIW please let me know if there are some experiments I can help with.

Regards,

---

From: John Naylor <[email-masked]>
Date: Wed, 14 Jun 2023 13:23:07 +0700

> > I'd need to make sure, but apparently just going from 6 non-empty
memory contexts to 3 (remember all values are embedded here) reduces memory
fragmentation significantly in this test. (That should also serve as a
demonstration that additional size classes have both runtime costs as well
as benefits. We need to have a balance.)
> block sizes. I'd like to experiment if the code is available.

I cleaned up a few things and attached v34 so you can do that if you like.
(Note: what I said about node63/n125 not making a difference in that one
test is not quite true since slab keeps a few empty blocks around. I did
some rough mental math and I think it doesn't change the conclusion any.)

0001-0007 is basically v33, but can apply on master.

0008 just adds back RT_EXTEND_DOWN. I left it out to simplify moving to
recursion.

> > Oh, the memcpy part is great, very simple. I mean the (compile-time)
"class info" table lookups are a bit awkward. I'm thinking the hard-coded
> since we can use them in regression tests.

I added some definitions in 0012. It kind of doesn't matter now what sizes
are the test unless it also can test that it stays within the expected
size, if that makes sense. It is helpful during debugging to force growth
to stop at a certain size.

> > > and do memcpy().

Not anymore. ;-) To be technical, it didn't "just" memcpy(), since it then
fell through to find the insert position and memmove(). In some parts of
Andres' prototype, no memmove() is necessary, because it memcpy()'s around
the insert position, and puts the new child in the right place. I've done
this in 0009.

The memcpy you mention was done for 1) simplicity 2) to avoid memset'ing.
Well, it was never necessary to memset the whole node in the first place.
Only the header, slot index array, and isset arrays need to be zeroed, so
in 0011 we always do only that. That combines alloc and init functionality,
and it's simple everywhere.

In 0010 I restored iteration functionality -- it can no longer get the
shift from the node, because it's not there as of v33. I was not
particularly impressed that there were no basic iteration tests, and in
fact the test_pattern test relied on functioning iteration. I added some
basic tests. I'm not entirely pleased with testing overall, but I think
it's at least sufficient for the job. I had the idea to replace "shift"
everywhere and use "level" as a fundamental concept. This is clearer. I do
want to make sure the compiler can compute the shift efficiently where
necessary. I think that can wait until much later.

0013 standardizes (mostly) on 4/16/48/256 for naming convention, regardless
of actual size, as I started to do earlier.

0014 is part cleanup of shrinking, and part making grow-node-48 more
consistent with the rest.

"value or pointer-to-leaf" would have a much smaller code footprint.
> bit? Or is it an optimization for TIDStore?

It will be up to the caller (the user of the template) -- if an
abbreviation is possible that fits in the upper 63 bits (with something to
guard for 32-bit platforms), the developer will be able to specify a
conversion function so that the caller only sees the full value when
searching and setting. Without such a function, the template will fall back
to the size of the value type to determine how the value is stored.