public inbox for [email protected]
help / color / mirror / Atom feedFrom: Masahiko Sawada <[email protected]>
To: John Naylor <[email protected]>
Cc: Nathan Bossart <[email protected]>
Cc: Andres Freund <[email protected]>
Cc: Matthias van de Meent <[email protected]>
Cc: Yura Sokolov <[email protected]>
Cc: PostgreSQL-development <[email protected]>
Subject: Re: [PoC] Improve dead tuple storage for lazy vacuum
Date: Tue, 18 Apr 2023 00:20:16 +0900
Message-ID: <CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com> (raw)
In-Reply-To: <CAD21AoDBmD5q=eO+K=gyuVt53XvwpJ2dgxPwrtZ-eVOjVmtJjg@mail.gmail.com>
References: <CAFBsxsGW_aMSw0Odpd=T1UGRyfX2CdjdVqCNBMik-wfnGC0vAA@mail.gmail.com>
<CAD21AoAyxDaNrvWwSVfPerQkP034FG0PjYb=7j51QVTiPAtqgQ@mail.gmail.com>
<CAFBsxsFm1eNPv-_Z2-Gy0aFMMiqgvNym3ojwK0scbEaBbW4FHw@mail.gmail.com>
<CAD21AoB9M2Faw7K9CLON41iT6eXoC_YwTcKJ1V4Z33J0o0MySg@mail.gmail.com>
<CAD21AoA4QZ6uTksTzneO7WGXQ-0jnfv=4mGLfgF7V9tdNgV11Q@mail.gmail.com>
<CAFBsxsFV=-ER1GYzXkBnS68imeoAdAS69vEvMsrVpBhYFNopHg@mail.gmail.com>
<CAD21AoD3w76wERs_Lq7_uA6+gTaoOERPji+Yz8Ac6aui4JwvTg@mail.gmail.com>
<CAFBsxsHbvfdMe7qranpiEwaWPwTY97EgM9mXnCEj9gTo6BY=UA@mail.gmail.com>
<CAD21AoCCQ7T1XNCKtnNe+qiqsdgMsmZsugDpbYQdBVCAi5=zgg@mail.gmail.com>
<CAFBsxsGCW=CfgzatJOQ9iuz5UbrMaqPNpP7UZ-ON+2Pgh=zCwg@mail.gmail.com>
<20220921180126.GA453160@nathanxps13>
<CAFBsxsFk7btGg8G_Yjy76be8AJ9PKZH677zK5NKx4uL2EPRtcg@mail.gmail.com>
<CAFBsxsF6OT+4Hkjm3J1+qtE_OuegWO3D-UQ5hO0krTTrRZ=8hw@mail.gmail.com>
<CAD21AoBBF5Cr6Y+scEzoAcPQt6UQqwMaGnCODKvj+=CMUsY-Tw@mail.gmail.com>
<CAFBsxsG2SDMpmON_ovVY6S+oCT-Mn=9giMm4_YquHdkZfn7XBQ@mail.gmail.com>
<CAD21AoDg8S1QP-YR4132oPazPSvhdn-uArDF1AGEQiAYKhHfqA@mail.gmail.com>
<CAFBsxsFEVckVzsBsfgGzGR4Yz=Jp=UxOtjYvTjOz6fOoLXtOig@mail.gmail.com>
<CAFBsxsHDNrNSK4PDfKDArzVDEE2zozgMQZ0NbwMJmOfAC-xhbA@mail.gmail.com>
<CAFBsxsFPs258sP1E9oMT2EN3GJtMB7O8ZKrSDnsA0qdf1nFbcg@mail.gmail.com>
<CAD21AoB+z3+T2EnOBVXok2aeRywrDZDFkwwEaWKZSrFz9s1+mw@mail.gmail.com>
<CAFBsxsEunCjqXo6Q1rjw6fqmduh5DncTDaDw0NhLr2fYBQTj3w@mail.gmail.com>
<CAD21AoAT0p0crCjiY+UwXbkQG2bRNFbe0UF0Ba5EfC_w458JBQ@mail.gmail.com>
<CAFBsxsGV9ssgP-itGa_TVux0v=_B16-28ff+q_G2WLCgBstSLg@mail.gmail.com>
<CAD21AoBp2ErXCc01t0+jwBrNjRA=FkKPz1-yNNFcNC-PRsG5TA@mail.gmail.com>
<CAFBsxsG9NmS=XKtofS5sxddsA64RVLkYnQ4+8pbXkwADGDGjoQ@mail.gmail.com>
<CAD21AoB8oE4WtgXhs+YFwOH3ODAOvEN48afFLYL=vqMk0d25TA@mail.gmail.com>
<CAD21AoCLvVkvf=0Jyu+wV9=GZTcEva+R0q-FzMU9viT9asoevA@mail.gmail.com>
<CAFBsxsF67qpj16kRitgRLJFw=omWsWNwCkaJmNJM9PWvZ4zBLA@mail.gmail.com>
<CAFBsxsHgP5LP9q=Rq_3Q2S6Oyut67z+Vpemux+KseSPXhJF7sg@mail.gmail.com>
<CAFBsxsHdKnkSzyKwEgzL16-FQCEHuQ1jT5T6L=28DWT_sMfgPg@mail.gmail.com>
<CAD21AoDyJWxmPJ8cZj3Y8P1o6EEjjkbmgho4Jb2ABjfhc19Kkw@mail.gmail.com>
<CAFBsxsFSSXCyU1RtGi1EpONi+QRpurrgnwHR8mNK8Wgi=Stgvg@mail.gmail.com>
<CAFBsxsFMuyUd48SXy1VMFvpVBHh6B6300_UfTSiPckKJtcwWMA@mail.gmail.com>
<CAD21AoCe1Ch+y1_nDzTXF5w6VT=nnP33QXM5V9z-PE3-x0f_xw@mail.gmail.com>
<CAFBsxsF8_ce1Vc9HTqjdh4KknPDekjMJTW6JPq99U+GhdzxuqA@mail.gmail.com>
<CAD21AoA_VjvG6LDYvJyuezNPtH12bx0My5Ws88o1Gu4h9qX5GQ@mail.gmail.com>
<CAFBsxsG1Diefx7iUOW6Sgv+CGZSVc88tfQyhmsf93gU_C7N14g@mail.gmail.com>
<CAD21AoD3-bUAUPhZC9qnVY__fhNJSjHkxNxmzKmJwTuFvdDfrQ@mail.gmail.com>
<CAFBsxsEPQkCr50co+EA-+bvQNaVv-Cr99JxT-CMGBBqs0bGftQ@mail.gmail.com>
<CAD21AoALOcuJVgwn+5EFSQccMh_hQAfBtiC5hLDMgpRwOxY4rA@mail.gmail.com>
<CAFBsxsEdQPnvgyJRpVS+cWxQpe-Uq23ou0G0ukfK9-_pCxszZA@mail.gmail.com>
<CAD21AoA-=5rpABAGCu2tJ=YzfTtNgJ5WxkZR+4SS0W54ccN5Ww@mail.gmail.com>
<CAFBsxsF2-xJkUMoQCO0hda_SHKXsz2WLWbp8ep3J_bKFWiozkQ@mail.gmail.com>
<CAD21AoDqvkT7LjnjAeJ1urfRsmf0oeJkj5_=qKomFw1sYQ45mQ@mail.gmail.com>
<CAFBsxsGYGuEkKgF6h0BegtQoAkJ+B2enOMPB4u5T7RdVFAvQEQ@mail.gmail.com>
<CAD21AoBjmj0fDyTJCYvs3W2GQb9bUT5pwQLOU_=BUTOWhj9ToQ@mail.gmail.com>
<CAFBsxsG0wn9h7JJcMR03fdTmvwUaU0c9ps3_Bro98ndN7ysO=g@mail.gmail.com>
<CAD21AoAtR3NN2E1u5sv5RdZapDR07AmLQS2GseFMyvYeWDoBRg@mail.gmail.com>
<CAD21AoCSkEDYAqW3+JtSQQY+R2tDCrXA=PgNAUXUdUQ5gBOp0Q@mail.gmail.com>
<CAFBsxsHbBm_M22gLBO+AZT4mfMq3L_oX3wdKZxjeNnT7fHsYMQ@mail.gmail.com>
<CAD21AoDePt8x14Vrg+xLFd2akRAKHS_8j2uVmyoRVMYOze-VSw@mail.gmail.com>
<CAFBsxsFps1MReb3HQYEGc=M=1NXRjSu3TSaEVEXkVrCR3g-hxg@mail.gmail.com>
<CAD21AoDqO7v5cvFUyWD7_pwceJ1XfhA81FVLTspKn0sFxtqNPA@mail.gmail.com>
<CAFBsxsESfmZpWwk3XxWkpdg_DjawH2vizKW91fyawQ+wVFkOiQ@mail.gmail.com>
<CAD21AoD-sOZY5KFTQw1PQDvRExFGhda4SrW8C1tFr7jgJ_6ByQ@mail.gmail.com>
<CAFBsxsGCv8+v6d6Fgi+K0Re+CXsW_m9uTimZL5Kt7asUJKgz4A@mail.gmail.com>
<CAD21AoAFiw2DS07nPhiS14eVXpnF-bBmjvab4y+Ct8igU2z_8A@mail.gmail.com>
<CAD21AoDCuTO6_wKF9mPm2hb9Fo-dc9kwbec6iXpwptsYaoFdyA@mail.gmail.com>
<CAFBsxsGkhZ8YFdh0eg6ss3daGtekwjm2ZggS1=fewfkdStHeAA@mail.gmail.com>
<CAD21AoDYZjDm+gixXptOjWqZ7N7hkrh4r7YUPuqPJDoGR8r_5g@mail.gmail.com>
<CAFBsxsHLqMNnMFK7uirb-oAtJOtRhDUtz+ewiHsBLwdre6bjXQ@mail.gmail.com>
<CAD21AoC=sazrXJUwH61CMz_+GjM__5BhQzMjN4-=gtOXWvkbAg@mail.gmail.com>
<CAFBsxsGs8fnpD2Lb+M7UYKA9smzyXfsD5gE0G_dMoeN2KeSdgQ@mail.gmail.com>
<CAD21AoDe3izpQLGAFWcmxiFM6pqocWYawCB6vRO_wK8sh=csdA@mail.gmail.com>
<CAD21AoBJtOzhC6pCKrZg0vNiUPbyM-uZ=Xz5BNS_UFgGatj1cw@mail.gmail.com>
<CAFBsxsEnzivaJ13iCGdDoUMsXJVGOaahuBe_y=q6ow=LTzyDvA@mail.gmail.com>
<CAD21AoCVm9iWMGriq6UhoOa7ZrpvkjQvzYyP1+CRBZyac3KwRg@mail.gmail.com>
<CAFBsxsFH3f6r3hSmXSX1bvsDe7EFGExa9Sf3Ysd5DKJ=C1bxgQ@mail.gmail.com>
<CAD21AoAYDcPFrYwHphnPHVXqJODKtxJRM_u8fgztRP06d31f1A@mail.gmail.com>
<CAFBsxsEhHdWO9X=w_D4u8KyZXu_L8119ozTWpoWYMUbpVCZOqw@mail.gmail.com>
<CAD21AoCqeCeE2cxZ4cS=qGSvUKUgPf_6OPTBMh_wo8HekRM_uw@mail.gmail.com>
<CAFBsxsFF4q7vxOgcJ==tCtuZ2o34sGcuo0=V=TSdkatXgpN7ug@mail.gmail.com>
<CAD21AoC+kjqkSnLLcbqtu=Td5HSx2H+iAwNHCeZcWqjy1w7k5g@mail.gmail.com>
<CAFBsxsEQQgt=OvK3CHWbsCggOwMmQ6CHjGT_0ziA2CZAdkFExw@mail.gmail.com>
<CAD21AoCbnN6JiXm4sWqAh5kH656oiHhi=HjFwtzKTr=g53YmnA@mail.gmail.com>
<CAD21AoDBmD5q=eO+K=gyuVt53XvwpJ2dgxPwrtZ-eVOjVmtJjg@mail.gmail.com>
On Sat, Mar 11, 2023 at 12:26 AM Masahiko Sawada <[email protected]> wrote:
>
> On Fri, Mar 10, 2023 at 11:30 PM Masahiko Sawada <[email protected]> wrote:
> >
> > On Fri, Mar 10, 2023 at 3:42 PM John Naylor
> > <[email protected]> wrote:
> > >
> > > On Thu, Mar 9, 2023 at 1:51 PM Masahiko Sawada <[email protected]> wrote:
> > >
> > > > I've attached the new version patches. I merged improvements and fixes
> > > > I did in the v29 patch.
> > >
> > > I haven't yet had a chance to look at those closely, since I've had to devote time to other commitments. I remember I wasn't particularly impressed that v29-0008 mixed my requested name-casing changes with a bunch of other random things. Separating those out would be an obvious way to make it easier for me to look at, whenever I can get back to this. I need to look at the iteration changes as well, in addition to testing memory measurement (thanks for the new results, they look encouraging).
> >
> > Okay, I'll separate them again.
>
> Attached new patch series. In addition to separate them again, I've
> fixed a conflict with HEAD.
>
I've attached updated version patches to make cfbot happy. Also, I've
splitted fixup patches further(from 0007 except for 0016 and 0018) to
make reviews easy. These patches have the prefix radix tree, tidstore,
and vacuum, indicating the part it changes. 0016 patch is to change
DSA so that we can specify both the initial and max segment size and
0017 makes use of it in vacuumparallel.c I'm still researching a
better solution for memory limitation but it's the best solution for
me for now.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] v32-0015-vacuum-Miscellaneous-updates.patch (4.9K, ../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/2-v32-0015-vacuum-Miscellaneous-updates.patch)
download | inline diff:
From 16e55ffde1cb152dc94cf38a9f6c8442b78be284 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 18:07:04 +0900
Subject: [PATCH v32 15/18] vacuum: Miscellaneous updates
fix typos, comment updates, etc.
---
doc/src/sgml/monitoring.sgml | 2 +-
src/backend/access/heap/vacuumlazy.c | 17 ++++++++---------
src/backend/commands/vacuumparallel.c | 13 +++++++------
3 files changed, 16 insertions(+), 16 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 9b64614beb..67ab9fa2bc 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -7331,7 +7331,7 @@ FROM pg_stat_get_backend_idset() AS backendid;
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>num_dead_tuple_bytes</structfield> <type>bigint</type>
+ <structfield>dead_tuple_bytes</structfield> <type>bigint</type>
</para>
<para>
Amount of dead tuple data collected since the last index vacuum cycle.
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index be487aced6..228daad750 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -10,11 +10,10 @@
* of dead TIDs at once.
*
* We are willing to use at most maintenance_work_mem (or perhaps
- * autovacuum_work_mem) memory space to keep track of dead TIDs. We initially
- * create a TidStore with the maximum bytes that can be used by the TidStore.
- * If the TidStore is full, we must call lazy_vacuum to vacuum indexes (and to
- * vacuum the pages that we've pruned). This frees up the memory space dedicated
- * to storing dead TIDs.
+ * autovacuum_work_mem) memory space to keep track of dead TIDs. If the
+ * TidStore is full, we must call lazy_vacuum to vacuum indexes (and to vacuum
+ * the pages that we've pruned). This frees up the memory space dedicated to
+ * to store dead TIDs.
*
* In practice VACUUM will often complete its initial pass over the target
* heap relation without ever running out of space to store TIDs. This means
@@ -2392,7 +2391,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
Buffer vmbuffer = InvalidBuffer;
LVSavedErrInfo saved_err_info;
TidStoreIter *iter;
- TidStoreIterResult *result;
+ TidStoreIterResult *iter_result;
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2417,7 +2416,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
vacuum_delay_point();
- blkno = result->blkno;
+ blkno = iter_result->blkno;
vacrel->blkno = blkno;
/*
@@ -2431,8 +2430,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
buf = ReadBufferExtended(vacrel->rel, MAIN_FORKNUM, blkno, RBM_NORMAL,
vacrel->bstrategy);
LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
- lazy_vacuum_heap_page(vacrel, blkno, result->offsets, result->num_offsets,
- buf, vmbuffer);
+ lazy_vacuum_heap_page(vacrel, blkno, iter_result->offsets,
+ iter_result->num_offsets, buf, vmbuffer);
/* Now that we've vacuumed the page, record its available space */
page = BufferGetPage(buf);
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index be83ceb871..8385d375db 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -9,11 +9,12 @@
* In a parallel vacuum, we perform both index bulk deletion and index cleanup
* with parallel worker processes. Individual indexes are processed by one
* vacuum process. ParalleVacuumState contains shared information as well as
- * the shared TidStore. We launch parallel worker processes at the start of
- * parallel index bulk-deletion and index cleanup and once all indexes are
- * processed, the parallel worker processes exit. Each time we process indexes
- * in parallel, the parallel context is re-initialized so that the same DSM can
- * be used for multiple passes of index bulk-deletion and index cleanup.
+ * the memory space for storing dead items allocated in the DSA area. We
+ * launch parallel worker processes at the start of parallel index
+ * bulk-deletion and index cleanup and once all indexes are processed, the
+ * parallel worker processes exit. Each time we process indexes in parallel,
+ * the parallel context is re-initialized so that the same DSM can be used for
+ * multiple passes of index bulk-deletion and index cleanup.
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -299,7 +300,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_estimate_chunk(&pcxt->estimator, est_shared_len);
shm_toc_estimate_keys(&pcxt->estimator, 1);
- /* Estimate size for dead tuple DSA -- PARALLEL_VACUUM_KEY_DSA */
+ /* Initial size for dead tuple DSA -- PARALLEL_VACUUM_KEY_DEAD_ITEMS */
shm_toc_estimate_chunk(&pcxt->estimator, dsa_minsize);
shm_toc_estimate_keys(&pcxt->estimator, 1);
--
2.31.1
[application/octet-stream] v32-0016-Make-initial-and-maximum-DSA-segment-size-config.patch (9.6K, ../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/3-v32-0016-Make-initial-and-maximum-DSA-segment-size-config.patch)
download | inline diff:
From bc7b41a404cc1c8050c400919836951f78456aef Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 21:59:12 +0900
Subject: [PATCH v32 16/18] Make initial and maximum DSA segment size
configurable
---
src/backend/utils/mmgr/dsa.c | 64 +++++++++++++++++-------------------
src/include/utils/dsa.h | 45 ++++++++++++++++++++++---
2 files changed, 71 insertions(+), 38 deletions(-)
diff --git a/src/backend/utils/mmgr/dsa.c b/src/backend/utils/mmgr/dsa.c
index 80555aefff..b6238bf4a3 100644
--- a/src/backend/utils/mmgr/dsa.c
+++ b/src/backend/utils/mmgr/dsa.c
@@ -60,14 +60,6 @@
#include "utils/freepage.h"
#include "utils/memutils.h"
-/*
- * The size of the initial DSM segment that backs a dsa_area created by
- * dsa_create. After creating some number of segments of this size we'll
- * double this size, and so on. Larger segments may be created if necessary
- * to satisfy large requests.
- */
-#define DSA_INITIAL_SEGMENT_SIZE ((size_t) (1 * 1024 * 1024))
-
/*
* How many segments to create before we double the segment size. If this is
* low, then there is likely to be a lot of wasted space in the largest
@@ -77,17 +69,6 @@
*/
#define DSA_NUM_SEGMENTS_AT_EACH_SIZE 2
-/*
- * The number of bits used to represent the offset part of a dsa_pointer.
- * This controls the maximum size of a segment, the maximum possible
- * allocation size and also the maximum number of segments per area.
- */
-#if SIZEOF_DSA_POINTER == 4
-#define DSA_OFFSET_WIDTH 27 /* 32 segments of size up to 128MB */
-#else
-#define DSA_OFFSET_WIDTH 40 /* 1024 segments of size up to 1TB */
-#endif
-
/*
* The maximum number of DSM segments that an area can own, determined by
* the number of bits remaining (but capped at 1024).
@@ -98,9 +79,6 @@
/* The bitmask for extracting the offset from a dsa_pointer. */
#define DSA_OFFSET_BITMASK (((dsa_pointer) 1 << DSA_OFFSET_WIDTH) - 1)
-/* The maximum size of a DSM segment. */
-#define DSA_MAX_SEGMENT_SIZE ((size_t) 1 << DSA_OFFSET_WIDTH)
-
/* Number of pages (see FPM_PAGE_SIZE) per regular superblock. */
#define DSA_PAGES_PER_SUPERBLOCK 16
@@ -319,6 +297,10 @@ typedef struct
dsa_segment_index segment_bins[DSA_NUM_SEGMENT_BINS];
/* The object pools for each size class. */
dsa_area_pool pools[DSA_NUM_SIZE_CLASSES];
+ /* initial allocation segment size */
+ size_t init_segment_size;
+ /* maximum allocation segment size */
+ size_t max_segment_size;
/* The total size of all active segments. */
size_t total_segment_size;
/* The maximum total size of backing storage we are allowed. */
@@ -413,7 +395,9 @@ static dsa_segment_map *make_new_segment(dsa_area *area, size_t requested_pages)
static dsa_area *create_internal(void *place, size_t size,
int tranche_id,
dsm_handle control_handle,
- dsm_segment *control_segment);
+ dsm_segment *control_segment,
+ size_t init_segment_size,
+ size_t max_segment_size);
static dsa_area *attach_internal(void *place, dsm_segment *segment,
dsa_handle handle);
static void check_for_freed_segments(dsa_area *area);
@@ -429,7 +413,8 @@ static void check_for_freed_segments_locked(dsa_area *area);
* we require the caller to provide one.
*/
dsa_area *
-dsa_create(int tranche_id)
+dsa_create_extended(int tranche_id, size_t init_segment_size,
+ size_t max_segment_size)
{
dsm_segment *segment;
dsa_area *area;
@@ -438,7 +423,7 @@ dsa_create(int tranche_id)
* Create the DSM segment that will hold the shared control object and the
* first segment of usable space.
*/
- segment = dsm_create(DSA_INITIAL_SEGMENT_SIZE, 0);
+ segment = dsm_create(init_segment_size, 0);
/*
* All segments backing this area are pinned, so that DSA can explicitly
@@ -450,9 +435,10 @@ dsa_create(int tranche_id)
/* Create a new DSA area with the control object in this segment. */
area = create_internal(dsm_segment_address(segment),
- DSA_INITIAL_SEGMENT_SIZE,
+ init_segment_size,
tranche_id,
- dsm_segment_handle(segment), segment);
+ dsm_segment_handle(segment), segment,
+ init_segment_size, max_segment_size);
/* Clean up when the control segment detaches. */
on_dsm_detach(segment, &dsa_on_dsm_detach_release_in_place,
@@ -478,13 +464,15 @@ dsa_create(int tranche_id)
* See dsa_create() for a note about the tranche arguments.
*/
dsa_area *
-dsa_create_in_place(void *place, size_t size,
- int tranche_id, dsm_segment *segment)
+dsa_create_in_place_extended(void *place, size_t size,
+ int tranche_id, dsm_segment *segment,
+ size_t init_segment_size, size_t max_segment_size)
{
dsa_area *area;
area = create_internal(place, size, tranche_id,
- DSM_HANDLE_INVALID, NULL);
+ DSM_HANDLE_INVALID, NULL,
+ init_segment_size, max_segment_size);
/*
* Clean up when the control segment detaches, if a containing DSM segment
@@ -1215,7 +1203,8 @@ static dsa_area *
create_internal(void *place, size_t size,
int tranche_id,
dsm_handle control_handle,
- dsm_segment *control_segment)
+ dsm_segment *control_segment,
+ size_t init_segment_size, size_t max_segment_size)
{
dsa_area_control *control;
dsa_area *area;
@@ -1225,6 +1214,11 @@ create_internal(void *place, size_t size,
size_t metadata_bytes;
int i;
+ /* Validate the initial and maximum block sizes */
+ Assert(init_segment_size >= 1024);
+ Assert(max_segment_size >= init_segment_size);
+ Assert(max_segment_size <= DSA_MAX_SEGMENT_SIZE);
+
/* Sanity check on the space we have to work in. */
if (size < dsa_minimum_size())
elog(ERROR, "dsa_area space must be at least %zu, but %zu provided",
@@ -1254,8 +1248,10 @@ create_internal(void *place, size_t size,
control->segment_header.prev = DSA_SEGMENT_INDEX_NONE;
control->segment_header.usable_pages = usable_pages;
control->segment_header.freed = false;
- control->segment_header.size = DSA_INITIAL_SEGMENT_SIZE;
+ control->segment_header.size = size;
control->handle = control_handle;
+ control->init_segment_size = init_segment_size;
+ control->max_segment_size = max_segment_size;
control->max_total_segment_size = (size_t) -1;
control->total_segment_size = size;
control->segment_handles[0] = control_handle;
@@ -2124,9 +2120,9 @@ make_new_segment(dsa_area *area, size_t requested_pages)
* move to huge pages in the future. Then we work back to the number of
* pages we can fit.
*/
- total_size = DSA_INITIAL_SEGMENT_SIZE *
+ total_size = area->control->init_segment_size *
((size_t) 1 << (new_index / DSA_NUM_SEGMENTS_AT_EACH_SIZE));
- total_size = Min(total_size, DSA_MAX_SEGMENT_SIZE);
+ total_size = Min(total_size, area->control->max_segment_size);
total_size = Min(total_size,
area->control->max_total_segment_size -
area->control->total_segment_size);
diff --git a/src/include/utils/dsa.h b/src/include/utils/dsa.h
index 2af215484f..90b7b0d93f 100644
--- a/src/include/utils/dsa.h
+++ b/src/include/utils/dsa.h
@@ -77,6 +77,28 @@ typedef pg_atomic_uint64 dsa_pointer_atomic;
/* A sentinel value for dsa_pointer used to indicate failure to allocate. */
#define InvalidDsaPointer ((dsa_pointer) 0)
+/*
+ * The default size of the initial DSM segment that backs a dsa_area created
+ * by dsa_create. After creating some number of segments of this size we'll
+ * double this size, and so on. Larger segments may be created if necessary
+ * to satisfy large requests.
+ */
+#define DSA_INITIAL_SEGMENT_SIZE ((size_t) (1 * 1024 * 1024))
+
+/*
+ * The number of bits used to represent the offset part of a dsa_pointer.
+ * This controls the maximum size of a segment, the maximum possible
+ * allocation size and also the maximum number of segments per area.
+ */
+#if SIZEOF_DSA_POINTER == 4
+#define DSA_OFFSET_WIDTH 27 /* 32 segments of size up to 128MB */
+#else
+#define DSA_OFFSET_WIDTH 40 /* 1024 segments of size up to 1TB */
+#endif
+
+/* The maximum size of a DSM segment. */
+#define DSA_MAX_SEGMENT_SIZE ((size_t) 1 << DSA_OFFSET_WIDTH)
+
/* Check if a dsa_pointer value is valid. */
#define DsaPointerIsValid(x) ((x) != InvalidDsaPointer)
@@ -88,6 +110,19 @@ typedef pg_atomic_uint64 dsa_pointer_atomic;
#define dsa_allocate0(area, size) \
dsa_allocate_extended(area, size, DSA_ALLOC_ZERO)
+/* Create dsa_area with default segment sizes */
+#define dsa_create(tranch_id) \
+ dsa_create_extended(tranch_id, DSA_INITIAL_SEGMENT_SIZE, \
+ DSA_MAX_SEGMENT_SIZE)
+
+/*
+ * Create dsa_area with default segment sizes in an existing share memory
+ * space.
+ */
+#define dsa_create_in_place(place, size, tranch_id, segment) \
+ dsa_create_in_place_extended(place, size, tranch_id, segment, \
+ DSA_INITIAL_SEGMENT_SIZE, DSA_MAX_SEGMENT_SIZE)
+
/*
* The type used for dsa_area handles. dsa_handle values can be shared with
* other processes, so that they can attach to them. This provides a way to
@@ -102,10 +137,12 @@ typedef dsm_handle dsa_handle;
/* Sentinel value to use for invalid dsa_handles. */
#define DSA_HANDLE_INVALID ((dsa_handle) DSM_HANDLE_INVALID)
-
-extern dsa_area *dsa_create(int tranche_id);
-extern dsa_area *dsa_create_in_place(void *place, size_t size,
- int tranche_id, dsm_segment *segment);
+extern dsa_area *dsa_create_extended(int tranche_id, size_t init_segment_size,
+ size_t max_segment_size);
+extern dsa_area *dsa_create_in_place_extended(void *place, size_t size,
+ int tranche_id, dsm_segment *segment,
+ size_t init_segment_size,
+ size_t max_segment_size);
extern dsa_area *dsa_attach(dsa_handle handle);
extern dsa_area *dsa_attach_in_place(void *place, dsm_segment *segment);
extern void dsa_release_in_place(void *place);
--
2.31.1
[application/octet-stream] v32-0014-tidstore-Miscellaneous-updates.patch (7.3K, ../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/4-v32-0014-tidstore-Miscellaneous-updates.patch)
download | inline diff:
From f9be0044ee6e35dd44bceca59d733ba8cdf5373e Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 18:01:46 +0900
Subject: [PATCH v32 14/18] tidstore: Miscellaneous updates.
comment updates, fix typos, etc.
---
src/backend/access/common/tidstore.c | 78 +++++++++++--------
.../modules/test_tidstore/test_tidstore.c | 1 +
2 files changed, 47 insertions(+), 32 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 15b77b5bcb..9360520482 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -3,18 +3,19 @@
* tidstore.c
* Tid (ItemPointerData) storage implementation.
*
- * This module provides a in-memory data structure to store Tids (ItemPointer).
- * Internally, a tid is encoded as a pair of 64-bit key and 64-bit value, and
- * stored in the radix tree.
+ * TidStore is a in-memory data structure to store tids (ItemPointerData).
+ * Internally, a tid is encoded as a pair of 64-bit key and 64-bit value,
+ * and stored in the radix tree.
*
* TidStore can be shared among parallel worker processes by passing DSA area
* to TidStoreCreate(). Other backends can attach to the shared TidStore by
* TidStoreAttach().
*
- * Regarding the concurrency, it basically relies on the concurrency support in
- * the radix tree, but we acquires the lock on a TidStore in some cases, for
- * example, when to reset the store and when to access the number tids in the
- * store (num_tids).
+ * Regarding the concurrency support, we use a single LWLock for the TidStore.
+ * The TidStore is exclusively locked when inserting encoded tids to the
+ * radix tree or when resetting itself. When searching on the TidStore or
+ * doing the iteration, it is not locked but the underlying radix tree is
+ * locked in shared mode.
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -34,16 +35,18 @@
#include "utils/memutils.h"
/*
- * For encoding purposes, tids are represented as a pair of 64-bit key and
- * 64-bit value. First, we construct 64-bit unsigned integer by combining
- * the block number and the offset number. The number of bits used for the
- * offset number is specified by max_offsets in tidstore_create(). We are
- * frugal with the bits, because smaller keys could help keeping the radix
- * tree shallow.
+ * For encoding purposes, a tid is represented as a pair of 64-bit key and
+ * 64-bit value.
*
- * For example, a tid of heap with 8kB blocks uses the lowest 9 bits for
- * the offset number and uses the next 32 bits for the block number. That
- * is, only 41 bits are used:
+ * First, we construct a 64-bit unsigned integer by combining the block
+ * number and the offset number. The number of bits used for the offset number
+ * is specified by max_off in TidStoreCreate(). We are frugal with the bits,
+ * because smaller keys could help keeping the radix tree shallow.
+ *
+ * For example, a tid of heap on a 8kB block uses the lowest 9 bits for
+ * the offset number and uses the next 32 bits for the block number. 9 bits
+ * are enough for the offset number, because MaxHeapTuplesPerPage < 2^9
+ * on 8kB blocks. That is, only 41 bits are used:
*
* uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
*
@@ -52,25 +55,27 @@
* u = unused bit
* (high on the left, low on the right)
*
- * 9 bits are enough for the offset number, because MaxHeapTuplesPerPage < 2^9
- * on 8kB blocks.
- *
- * The 64-bit value is the bitmap representation of the lowest 6 bits
- * (TIDSTORE_VALUE_NBITS) of the integer, and the rest 35 bits are used
- * as the key:
+ * Then, 64-bit value is the bitmap representation of the lowest 6 bits
+ * (LOWER_OFFSET_NBITS) of the integer, and 64-bit key consists of the
+ * upper 3 bits of the offset number and the block number, 35 bits in
+ * total:
*
* uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
* |----| value
- * |---------------------------------------------| key
+ * |--------------------------------------| key
*
* The maximum height of the radix tree is 5 in this case.
+ *
+ * If the number of bits required for offset numbers fits in LOWER_OFFSET_NBITS,
+ * 64-bit value is the bitmap representation of the offset number, and the
+ * 64-bit key is the block number.
*/
typedef uint64 tidkey;
typedef uint64 offsetbm;
#define LOWER_OFFSET_NBITS 6 /* log(sizeof(offsetbm), 2) */
#define LOWER_OFFSET_MASK ((1 << LOWER_OFFSET_NBITS) - 1)
-/* A magic value used to identify our TidStores. */
+/* A magic value used to identify our TidStore. */
#define TIDSTORE_MAGIC 0x826f6a10
#define RT_PREFIX local_rt
@@ -152,8 +157,10 @@ typedef struct TidStoreIter
tidkey next_tidkey;
offsetbm next_off_bitmap;
- /* output for the caller */
- TidStoreIterResult result;
+ /*
+ * output for the caller. Must be last because variable-size.
+ */
+ TidStoreIterResult output;
} TidStoreIter;
static void iter_decode_key_off(TidStoreIter *iter, tidkey key, offsetbm off_bitmap);
@@ -205,7 +212,7 @@ TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
dp = dsa_allocate0(area, sizeof(TidStoreControl));
ts->control = (TidStoreControl *) dsa_get_address(area, dp);
- ts->control->max_bytes = (uint64) (max_bytes * ratio);
+ ts->control->max_bytes = (size_t) (max_bytes * ratio);
ts->area = area;
ts->control->magic = TIDSTORE_MAGIC;
@@ -353,7 +360,11 @@ TidStoreReset(TidStore *ts)
}
}
-/* Add Tids on a block to TidStore */
+/*
+ * Set the given tids on the blkno to TidStore.
+ *
+ * NB: the offset numbers in offsets must be sorted in ascending order.
+ */
void
TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
int num_offsets)
@@ -564,7 +575,7 @@ TidStoreEndIterate(TidStoreIter *iter)
int64
TidStoreNumTids(TidStore *ts)
{
- uint64 num_tids;
+ int64 num_tids;
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
@@ -624,11 +635,14 @@ TidStoreGetHandle(TidStore *ts)
return ts->control->handle;
}
-/* Extract tids from the given key-value pair */
+/*
+ * Decode the key and offset bitmap to tids and store them to the iteration
+ * result.
+ */
static void
iter_decode_key_off(TidStoreIter *iter, tidkey key, offsetbm off_bitmap)
{
- TidStoreIterResult *result = (&iter->result);
+ TidStoreIterResult *output = (&iter->output);
while (off_bitmap)
{
@@ -661,7 +675,7 @@ key_get_blkno(TidStore *ts, tidkey key)
static inline tidkey
encode_tid(TidStore *ts, ItemPointer tid, offsetbm *off_bit)
{
- uint32 offset = ItemPointerGetOffsetNumber(tid);
+ OffsetNumber offset = ItemPointerGetOffsetNumber(tid);
BlockNumber block = ItemPointerGetBlockNumber(tid);
return encode_blk_off(ts, block, offset, off_bit);
diff --git a/src/test/modules/test_tidstore/test_tidstore.c b/src/test/modules/test_tidstore/test_tidstore.c
index 12d3027624..8659e6780e 100644
--- a/src/test/modules/test_tidstore/test_tidstore.c
+++ b/src/test/modules/test_tidstore/test_tidstore.c
@@ -222,6 +222,7 @@ test_tidstore(PG_FUNCTION_ARGS)
elog(NOTICE, "testing basic operations");
test_basic(MaxHeapTuplesPerPage);
test_basic(10);
+ test_basic(MaxHeapTuplesPerPage * 2);
PG_RETURN_VOID();
}
--
2.31.1
[application/octet-stream] v32-0018-Revert-building-benchmark-module-for-CI.patch (694B, ../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/5-v32-0018-Revert-building-benchmark-module-for-CI.patch)
download | inline diff:
From 9e42c43a7d081c06c02f0029e610c29d911732e3 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 14 Feb 2023 19:31:34 +0700
Subject: [PATCH v32 18/18] Revert building benchmark module for CI
---
contrib/meson.build | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/contrib/meson.build b/contrib/meson.build
index 421d469f8c..52253de793 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,7 +12,7 @@ subdir('amcheck')
subdir('auth_delay')
subdir('auto_explain')
subdir('basic_archive')
-subdir('bench_radix_tree')
+#subdir('bench_radix_tree')
subdir('bloom')
subdir('basebackup_to_shell')
subdir('bool_plperl')
--
2.31.1
[application/octet-stream] v32-0017-tidstore-vacuum-Specify-the-init-and-max-DSA-seg.patch (4.8K, ../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/6-v32-0017-tidstore-vacuum-Specify-the-init-and-max-DSA-seg.patch)
download | inline diff:
From 11fda58f829c03d2a7c6476affc61862a078f741 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 22:27:04 +0900
Subject: [PATCH v32 17/18] tidstore, vacuum: Specify the init and max DSA
segment size based on m_w_m
---
src/backend/access/common/tidstore.c | 32 +++++----------------------
src/backend/commands/vacuumparallel.c | 21 ++++++++++++++----
2 files changed, 23 insertions(+), 30 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 9360520482..571d15c5c3 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -180,39 +180,15 @@ TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
ts = palloc0(sizeof(TidStore));
- /*
- * Create the radix tree for the main storage.
- *
- * Memory consumption depends on the number of stored tids, but also on the
- * distribution of them, how the radix tree stores, and the memory management
- * that backed the radix tree. The maximum bytes that a TidStore can
- * use is specified by the max_bytes in TidStoreCreate(). We want the total
- * amount of memory consumption by a TidStore not to exceed the max_bytes.
- *
- * In local TidStore cases, the radix tree uses slab allocators for each kind
- * of node class. The most memory consuming case while adding Tids associated
- * with one page (i.e. during TidStoreSetBlockOffsets()) is that we allocate a new
- * slab block for a new radix tree node, which is approximately 70kB. Therefore,
- * we deduct 70kB from the max_bytes.
- *
- * In shared cases, DSA allocates the memory segments big enough to follow
- * a geometric series that approximately doubles the total DSA size (see
- * make_new_segment() in dsa.c). We simulated the how DSA increases segment
- * size and the simulation revealed, the 75% threshold for the maximum bytes
- * perfectly works in case where the max_bytes is a power-of-2, and the 60%
- * threshold works for other cases.
- */
if (area != NULL)
{
dsa_pointer dp;
- float ratio = ((max_bytes & (max_bytes - 1)) == 0) ? 0.75 : 0.6;
ts->tree.shared = shared_rt_create(CurrentMemoryContext, area,
LWTRANCHE_SHARED_TIDSTORE);
dp = dsa_allocate0(area, sizeof(TidStoreControl));
ts->control = (TidStoreControl *) dsa_get_address(area, dp);
- ts->control->max_bytes = (size_t) (max_bytes * ratio);
ts->area = area;
ts->control->magic = TIDSTORE_MAGIC;
@@ -223,11 +199,15 @@ TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
else
{
ts->tree.local = local_rt_create(CurrentMemoryContext);
-
ts->control = (TidStoreControl *) palloc0(sizeof(TidStoreControl));
- ts->control->max_bytes = max_bytes - (70 * 1024);
}
+ /*
+ * max_bytes is forced to be at least 64kB, the current minimum valid value
+ * for the work_mem GUC.
+ */
+ ts->control->max_bytes = Max(64 * 1024L, max_bytes);
+
ts->control->max_off = max_off;
ts->control->max_off_nbits = pg_ceil_log2_32(max_off);
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 8385d375db..17699aa007 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -252,6 +252,8 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
Size est_indstats_len;
Size est_shared_len;
Size dsa_minsize = dsa_minimum_size();
+ Size init_segsize;
+ Size max_segsize;
int nindexes_mwm = 0;
int parallel_workers = 0;
int querylen;
@@ -367,12 +369,23 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_INDEX_STATS, indstats);
pvs->indstats = indstats;
- /* Prepare DSA space for dead items */
+ /*
+ * Prepare DSA space for dead items.
+ *
+ * Since total DSA size grows while following a geometric series by default,
+ * we specify both the initial DSA segment and maximum DSA segment sizes
+ * based on the memory available for parallel vacuum. Typically, the initial
+ * segment size is 1MB and the maximum segment size is vac_work_mem / 8, and
+ * heap scan stops after allocating 1.125 times more memory than vac_work_mem.
+ */
+ init_segsize = Min(vac_work_mem / 4, (1024 * 1024));
+ max_segsize = Max(vac_work_mem / 8, (8 * 1024 * 1024));
area_space = shm_toc_allocate(pcxt->toc, dsa_minsize);
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, area_space);
- dead_items_dsa = dsa_create_in_place(area_space, dsa_minsize,
- LWTRANCHE_PARALLEL_VACUUM_DSA,
- pcxt->seg);
+ dead_items_dsa = dsa_create_in_place_extended(area_space, dsa_minsize,
+ LWTRANCHE_PARALLEL_VACUUM_DSA,
+ pcxt->seg,
+ init_segsize, max_segsize);
dead_items = TidStoreCreate(vac_work_mem, max_offset, dead_items_dsa);
pvs->dead_items = dead_items;
pvs->dead_items_area = dead_items_dsa;
--
2.31.1
[application/octet-stream] v32-0010-radix-tree-fix-radix-tree-test-code.patch (6.8K, ../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/7-v32-0010-radix-tree-fix-radix-tree-test-code.patch)
download | inline diff:
From 591bede6738ca9e5c7264db7ff1d3dd9ba29247f Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 17:35:14 +0900
Subject: [PATCH v32 10/18] radix tree: fix radix tree test code
fix tests for key insertion in ascending or descending order.
Also, we missed tests for MIN and MAX size classes.
---
.../expected/test_radixtree.out | 6 +-
.../modules/test_radixtree/test_radixtree.c | 103 ++++++++++++------
2 files changed, 71 insertions(+), 38 deletions(-)
diff --git a/src/test/modules/test_radixtree/expected/test_radixtree.out b/src/test/modules/test_radixtree/expected/test_radixtree.out
index ce645cb8b5..7ad1ce3605 100644
--- a/src/test/modules/test_radixtree/expected/test_radixtree.out
+++ b/src/test/modules/test_radixtree/expected/test_radixtree.out
@@ -4,8 +4,10 @@ CREATE EXTENSION test_radixtree;
-- an error if something fails.
--
SELECT test_radixtree();
-NOTICE: testing basic operations with leaf node 4
-NOTICE: testing basic operations with inner node 4
+NOTICE: testing basic operations with leaf node 3
+NOTICE: testing basic operations with inner node 3
+NOTICE: testing basic operations with leaf node 15
+NOTICE: testing basic operations with inner node 15
NOTICE: testing basic operations with leaf node 32
NOTICE: testing basic operations with inner node 32
NOTICE: testing basic operations with leaf node 125
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
index afe53382f3..5a169854d9 100644
--- a/src/test/modules/test_radixtree/test_radixtree.c
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -43,12 +43,15 @@ typedef uint64 TestValueType;
*/
static const bool rt_test_stats = false;
-static int rt_node_kind_fanouts[] = {
- 0,
- 4, /* RT_NODE_KIND_4 */
- 32, /* RT_NODE_KIND_32 */
- 125, /* RT_NODE_KIND_125 */
- 256 /* RT_NODE_KIND_256 */
+/*
+ * XXX: should we expose and use RT_SIZE_CLASS and RT_SIZE_CLASS_INFO?
+ */
+static int rt_node_class_fanouts[] = {
+ 3, /* RT_CLASS_3 */
+ 15, /* RT_CLASS_32_MIN */
+ 32, /* RT_CLASS_32_MAX */
+ 125, /* RT_CLASS_125 */
+ 256 /* RT_CLASS_256 */
};
/*
* A struct to define a pattern of integers, for use with the test_pattern()
@@ -260,10 +263,9 @@ test_basic(int children, bool test_inner)
* Check if keys from start to end with the shift exist in the tree.
*/
static void
-check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end,
- int incr)
+check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end)
{
- for (int i = start; i < end; i++)
+ for (int i = start; i <= end; i++)
{
uint64 key = ((uint64) i << shift);
TestValueType val;
@@ -277,22 +279,26 @@ check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end,
}
}
+/*
+ * Insert 256 key-value pairs, and check if keys are properly inserted on each
+ * node class.
+ */
+/* Test keys [0, 256) */
+#define NODE_TYPE_TEST_KEY_MIN 0
+#define NODE_TYPE_TEST_KEY_MAX 256
static void
-test_node_types_insert(rt_radix_tree *radixtree, uint8 shift, bool insert_asc)
+test_node_types_insert_asc(rt_radix_tree *radixtree, uint8 shift)
{
- uint64 num_entries;
- int ninserted = 0;
- int start = insert_asc ? 0 : 256;
- int incr = insert_asc ? 1 : -1;
- int end = insert_asc ? 256 : 0;
- int node_kind_idx = 1;
+ uint64 num_entries;
+ int node_class_idx = 0;
+ uint64 key_checked = 0;
- for (int i = start; i != end; i += incr)
+ for (int i = NODE_TYPE_TEST_KEY_MIN; i < NODE_TYPE_TEST_KEY_MAX; i++)
{
uint64 key = ((uint64) i << shift);
bool found;
- found = rt_set(radixtree, key, (TestValueType*) &key);
+ found = rt_set(radixtree, key, (TestValueType *) &key);
if (found)
elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " is found", key);
@@ -300,24 +306,49 @@ test_node_types_insert(rt_radix_tree *radixtree, uint8 shift, bool insert_asc)
* After filling all slots in each node type, check if the values
* are stored properly.
*/
- if (ninserted == rt_node_kind_fanouts[node_kind_idx] - 1)
+ if ((i + 1) == rt_node_class_fanouts[node_class_idx])
{
- int check_start = insert_asc
- ? rt_node_kind_fanouts[node_kind_idx - 1]
- : rt_node_kind_fanouts[node_kind_idx];
- int check_end = insert_asc
- ? rt_node_kind_fanouts[node_kind_idx]
- : rt_node_kind_fanouts[node_kind_idx - 1];
-
- check_search_on_node(radixtree, shift, check_start, check_end, incr);
- node_kind_idx++;
+ check_search_on_node(radixtree, shift, key_checked, i);
+ key_checked = i;
+ node_class_idx++;
}
-
- ninserted++;
}
num_entries = rt_num_entries(radixtree);
+ if (num_entries != 256)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+/*
+ * Similar to test_node_types_insert_asc(), but inserts keys in descending order.
+ */
+static void
+test_node_types_insert_desc(rt_radix_tree *radixtree, uint8 shift)
+{
+ uint64 num_entries;
+ int node_class_idx = 0;
+ uint64 key_checked = NODE_TYPE_TEST_KEY_MAX - 1;
+
+ for (int i = NODE_TYPE_TEST_KEY_MAX - 1; i >= NODE_TYPE_TEST_KEY_MIN; i--)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_set(radixtree, key, (TestValueType *) &key);
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " is found", key);
+ if ((i + 1) == rt_node_class_fanouts[node_class_idx])
+ {
+ check_search_on_node(radixtree, shift, i, key_checked);
+ key_checked = i;
+ node_class_idx++;
+ }
+ }
+
+ num_entries = rt_num_entries(radixtree);
if (num_entries != 256)
elog(ERROR,
"rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
@@ -329,7 +360,7 @@ test_node_types_delete(rt_radix_tree *radixtree, uint8 shift)
{
uint64 num_entries;
- for (int i = 0; i < 256; i++)
+ for (int i = NODE_TYPE_TEST_KEY_MIN; i < NODE_TYPE_TEST_KEY_MAX; i++)
{
uint64 key = ((uint64) i << shift);
bool found;
@@ -379,9 +410,9 @@ test_node_types(uint8 shift)
* then delete all entries to make it empty, and insert and search entries
* again.
*/
- test_node_types_insert(radixtree, shift, true);
+ test_node_types_insert_asc(radixtree, shift);
test_node_types_delete(radixtree, shift);
- test_node_types_insert(radixtree, shift, false);
+ test_node_types_insert_desc(radixtree, shift);
rt_free(radixtree);
#ifdef RT_SHMEM
@@ -664,10 +695,10 @@ test_radixtree(PG_FUNCTION_ARGS)
{
test_empty();
- for (int i = 1; i < lengthof(rt_node_kind_fanouts); i++)
+ for (int i = 0; i < lengthof(rt_node_class_fanouts); i++)
{
- test_basic(rt_node_kind_fanouts[i], false);
- test_basic(rt_node_kind_fanouts[i], true);
+ test_basic(rt_node_class_fanouts[i], false);
+ test_basic(rt_node_class_fanouts[i], true);
}
for (int shift = 0; shift <= (64 - 8); shift += 8)
--
2.31.1
[application/octet-stream] v32-0011-tidstore-vacuum-Use-camel-case-for-TidStore-APIs.patch (25.0K, ../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/8-v32-0011-tidstore-vacuum-Use-camel-case-for-TidStore-APIs.patch)
download | inline diff:
From 6f4ff3584cbbf4db3ed7268ebc360df0ad328696 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 17:47:10 +0900
Subject: [PATCH v32 11/18] tidstore, vacuum: Use camel case for TidStore APIs
---
src/backend/access/common/tidstore.c | 64 +++++++++---------
src/backend/access/heap/vacuumlazy.c | 44 ++++++------
src/backend/commands/vacuum.c | 4 +-
src/backend/commands/vacuumparallel.c | 12 ++--
src/include/access/tidstore.h | 34 +++++-----
.../modules/test_tidstore/test_tidstore.c | 67 ++++++++++---------
6 files changed, 114 insertions(+), 111 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 8c05e60d92..283a326d13 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -7,9 +7,9 @@
* Internally, a tid is encoded as a pair of 64-bit key and 64-bit value, and
* stored in the radix tree.
*
- * A TidStore can be shared among parallel worker processes by passing DSA area
- * to tidstore_create(). Other backends can attach to the shared TidStore by
- * tidstore_attach().
+ * TidStore can be shared among parallel worker processes by passing DSA area
+ * to TidStoreCreate(). Other backends can attach to the shared TidStore by
+ * TidStoreAttach().
*
* Regarding the concurrency, it basically relies on the concurrency support in
* the radix tree, but we acquires the lock on a TidStore in some cases, for
@@ -106,7 +106,7 @@ typedef struct TidStoreControl
LWLock lock;
/* handles for TidStore and radix tree */
- tidstore_handle handle;
+ TidStoreHandle handle;
shared_rt_handle tree_handle;
} TidStoreControl;
@@ -164,7 +164,7 @@ static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_b
* The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
*/
TidStore *
-tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
+TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
{
TidStore *ts;
@@ -176,12 +176,12 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
* Memory consumption depends on the number of stored tids, but also on the
* distribution of them, how the radix tree stores, and the memory management
* that backed the radix tree. The maximum bytes that a TidStore can
- * use is specified by the max_bytes in tidstore_create(). We want the total
+ * use is specified by the max_bytes in TidStoreCreate(). We want the total
* amount of memory consumption by a TidStore not to exceed the max_bytes.
*
* In local TidStore cases, the radix tree uses slab allocators for each kind
* of node class. The most memory consuming case while adding Tids associated
- * with one page (i.e. during tidstore_add_tids()) is that we allocate a new
+ * with one page (i.e. during TidStoreSetBlockOffsets()) is that we allocate a new
* slab block for a new radix tree node, which is approximately 70kB. Therefore,
* we deduct 70kB from the max_bytes.
*
@@ -235,7 +235,7 @@ tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
* allocated in backend-local memory using the CurrentMemoryContext.
*/
TidStore *
-tidstore_attach(dsa_area *area, tidstore_handle handle)
+TidStoreAttach(dsa_area *area, TidStoreHandle handle)
{
TidStore *ts;
dsa_pointer control;
@@ -266,7 +266,7 @@ tidstore_attach(dsa_area *area, tidstore_handle handle)
* to the operating system.
*/
void
-tidstore_detach(TidStore *ts)
+TidStoreDetach(TidStore *ts)
{
Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
@@ -279,12 +279,12 @@ tidstore_detach(TidStore *ts)
*
* TODO: The caller must be certain that no other backend will attempt to
* access the TidStore before calling this function. Other backend must
- * explicitly call tidstore_detach to free up backend-local memory associated
- * with the TidStore. The backend that calls tidstore_destroy must not call
- * tidstore_detach.
+ * explicitly call TidStoreDetach() to free up backend-local memory associated
+ * with the TidStore. The backend that calls TidStoreDestroy() must not call
+ * TidStoreDetach().
*/
void
-tidstore_destroy(TidStore *ts)
+TidStoreDestroy(TidStore *ts)
{
if (TidStoreIsShared(ts))
{
@@ -309,11 +309,11 @@ tidstore_destroy(TidStore *ts)
}
/*
- * Forget all collected Tids. It's similar to tidstore_destroy but we don't free
+ * Forget all collected Tids. It's similar to TidStoreDestroy() but we don't free
* entire TidStore but recreate only the radix tree storage.
*/
void
-tidstore_reset(TidStore *ts)
+TidStoreReset(TidStore *ts)
{
if (TidStoreIsShared(ts))
{
@@ -352,8 +352,8 @@ tidstore_reset(TidStore *ts)
/* Add Tids on a block to TidStore */
void
-tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
- int num_offsets)
+TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets)
{
uint64 *values;
uint64 key;
@@ -431,7 +431,7 @@ tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
/* Return true if the given tid is present in the TidStore */
bool
-tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
+TidStoreIsMember(TidStore *ts, ItemPointer tid)
{
uint64 key;
uint64 val = 0;
@@ -452,14 +452,16 @@ tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
}
/*
- * Prepare to iterate through a TidStore. Since the radix tree is locked during the
- * iteration, so tidstore_end_iterate() needs to called when finished.
+ * Prepare to iterate through a TidStore. Since the radix tree is locked during
+ * the iteration, so TidStoreEndIterate() needs to be called when finished.
+ *
+ * The TidStoreIter struct is created in the caller's memory context.
*
* Concurrent updates during the iteration will be blocked when inserting a
* key-value to the radix tree.
*/
TidStoreIter *
-tidstore_begin_iterate(TidStore *ts)
+TidStoreBeginIterate(TidStore *ts)
{
TidStoreIter *iter;
@@ -477,7 +479,7 @@ tidstore_begin_iterate(TidStore *ts)
iter->tree_iter.local = local_rt_begin_iterate(ts->tree.local);
/* If the TidStore is empty, there is no business */
- if (tidstore_num_tids(ts) == 0)
+ if (TidStoreNumTids(ts) == 0)
iter->finished = true;
return iter;
@@ -498,7 +500,7 @@ tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
* numbers in each result is also sorted in ascending order.
*/
TidStoreIterResult *
-tidstore_iterate_next(TidStoreIter *iter)
+TidStoreIterateNext(TidStoreIter *iter)
{
uint64 key;
uint64 val;
@@ -544,7 +546,7 @@ tidstore_iterate_next(TidStoreIter *iter)
* or when existing an iteration.
*/
void
-tidstore_end_iterate(TidStoreIter *iter)
+TidStoreEndIterate(TidStoreIter *iter)
{
if (TidStoreIsShared(iter->ts))
shared_rt_end_iterate(iter->tree_iter.shared);
@@ -557,7 +559,7 @@ tidstore_end_iterate(TidStoreIter *iter)
/* Return the number of tids we collected so far */
int64
-tidstore_num_tids(TidStore *ts)
+TidStoreNumTids(TidStore *ts)
{
uint64 num_tids;
@@ -575,16 +577,16 @@ tidstore_num_tids(TidStore *ts)
/* Return true if the current memory usage of TidStore exceeds the limit */
bool
-tidstore_is_full(TidStore *ts)
+TidStoreIsFull(TidStore *ts)
{
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
- return (tidstore_memory_usage(ts) > ts->control->max_bytes);
+ return (TidStoreMemoryUsage(ts) > ts->control->max_bytes);
}
/* Return the maximum memory TidStore can use */
size_t
-tidstore_max_memory(TidStore *ts)
+TidStoreMaxMemory(TidStore *ts)
{
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
@@ -593,7 +595,7 @@ tidstore_max_memory(TidStore *ts)
/* Return the memory usage of TidStore */
size_t
-tidstore_memory_usage(TidStore *ts)
+TidStoreMemoryUsage(TidStore *ts)
{
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
@@ -611,8 +613,8 @@ tidstore_memory_usage(TidStore *ts)
/*
* Get a handle that can be used by other processes to attach to this TidStore
*/
-tidstore_handle
-tidstore_get_handle(TidStore *ts)
+TidStoreHandle
+TidStoreGetHandle(TidStore *ts)
{
Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 2c72088e69..be487aced6 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -842,7 +842,7 @@ lazy_scan_heap(LVRelState *vacrel)
/* Report that we're scanning the heap, advertising total # of blocks */
initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
initprog_val[1] = rel_pages;
- initprog_val[2] = tidstore_max_memory(vacrel->dead_items);
+ initprog_val[2] = TidStoreMaxMemory(vacrel->dead_items);
pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
/* Set up an initial range of skippable blocks using the visibility map */
@@ -909,7 +909,7 @@ lazy_scan_heap(LVRelState *vacrel)
* dead_items TIDs, pause and do a cycle of vacuuming before we tackle
* this page.
*/
- if (tidstore_is_full(vacrel->dead_items))
+ if (TidStoreIsFull(vacrel->dead_items))
{
/*
* Before beginning index vacuuming, we release any pin we may
@@ -1078,16 +1078,16 @@ lazy_scan_heap(LVRelState *vacrel)
* with prunestate-driven visibility map and FSM steps (just like
* the two-pass strategy).
*/
- Assert(tidstore_num_tids(dead_items) == 0);
+ Assert(TidStoreNumTids(dead_items) == 0);
}
else if (prunestate.num_offsets > 0)
{
/* Save details of the LP_DEAD items from the page in dead_items */
- tidstore_add_tids(dead_items, blkno, prunestate.deadoffsets,
- prunestate.num_offsets);
+ TidStoreSetBlockOffsets(dead_items, blkno, prunestate.deadoffsets,
+ prunestate.num_offsets);
pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
- tidstore_memory_usage(dead_items));
+ TidStoreMemoryUsage(dead_items));
}
/*
@@ -1258,7 +1258,7 @@ lazy_scan_heap(LVRelState *vacrel)
* Do index vacuuming (call each index's ambulkdelete routine), then do
* related heap vacuuming
*/
- if (tidstore_num_tids(dead_items) > 0)
+ if (TidStoreNumTids(dead_items) > 0)
lazy_vacuum(vacrel);
/*
@@ -2125,10 +2125,10 @@ lazy_scan_noprune(LVRelState *vacrel,
*/
vacrel->lpdead_item_pages++;
- tidstore_add_tids(dead_items, blkno, deadoffsets, lpdead_items);
+ TidStoreSetBlockOffsets(dead_items, blkno, deadoffsets, lpdead_items);
pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
- tidstore_memory_usage(dead_items));
+ TidStoreMemoryUsage(dead_items));
vacrel->lpdead_items += lpdead_items;
@@ -2177,7 +2177,7 @@ lazy_vacuum(LVRelState *vacrel)
if (!vacrel->do_index_vacuuming)
{
Assert(!vacrel->do_index_cleanup);
- tidstore_reset(vacrel->dead_items);
+ TidStoreReset(vacrel->dead_items);
return;
}
@@ -2206,7 +2206,7 @@ lazy_vacuum(LVRelState *vacrel)
BlockNumber threshold;
Assert(vacrel->num_index_scans == 0);
- Assert(vacrel->lpdead_items == tidstore_num_tids(vacrel->dead_items));
+ Assert(vacrel->lpdead_items == TidStoreNumTids(vacrel->dead_items));
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2234,7 +2234,7 @@ lazy_vacuum(LVRelState *vacrel)
*/
threshold = (double) vacrel->rel_pages * BYPASS_THRESHOLD_PAGES;
bypass = (vacrel->lpdead_item_pages < threshold) &&
- tidstore_memory_usage(vacrel->dead_items) < (32L * 1024L * 1024L);
+ TidStoreMemoryUsage(vacrel->dead_items) < (32L * 1024L * 1024L);
}
if (bypass)
@@ -2279,7 +2279,7 @@ lazy_vacuum(LVRelState *vacrel)
* Forget the LP_DEAD items that we just vacuumed (or just decided to not
* vacuum)
*/
- tidstore_reset(vacrel->dead_items);
+ TidStoreReset(vacrel->dead_items);
}
/*
@@ -2352,7 +2352,7 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
* place).
*/
Assert(vacrel->num_index_scans > 0 ||
- tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items);
+ TidStoreNumTids(vacrel->dead_items) == vacrel->lpdead_items);
Assert(allindexes || VacuumFailsafeActive);
/*
@@ -2407,8 +2407,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
VACUUM_ERRCB_PHASE_VACUUM_HEAP,
InvalidBlockNumber, InvalidOffsetNumber);
- iter = tidstore_begin_iterate(vacrel->dead_items);
- while ((result = tidstore_iterate_next(iter)) != NULL)
+ iter = TidStoreBeginIterate(vacrel->dead_items);
+ while ((iter_result = TidStoreIterateNext(iter)) != NULL)
{
BlockNumber blkno;
Buffer buf;
@@ -2442,7 +2442,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
vacuumed_pages++;
}
- tidstore_end_iterate(iter);
+ TidStoreEndIterate(iter);
vacrel->blkno = InvalidBlockNumber;
if (BufferIsValid(vmbuffer))
@@ -2453,12 +2453,12 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
* the second heap pass. No more, no less.
*/
Assert(vacrel->num_index_scans > 1 ||
- (tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items &&
+ (TidStoreNumTids(vacrel->dead_items) == vacrel->lpdead_items &&
vacuumed_pages == vacrel->lpdead_item_pages));
ereport(DEBUG2,
- (errmsg("table \"%s\": removed " UINT64_FORMAT "dead item identifiers in %u pages",
- vacrel->relname, tidstore_num_tids(vacrel->dead_items),
+ (errmsg("table \"%s\": removed " INT64_FORMAT "dead item identifiers in %u pages",
+ vacrel->relname, TidStoreNumTids(vacrel->dead_items),
vacuumed_pages)));
/* Revert to the previous phase information for error traceback */
@@ -3125,8 +3125,8 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
}
/* Serial VACUUM case */
- vacrel->dead_items = tidstore_create(vac_work_mem, MaxHeapTuplesPerPage,
- NULL);
+ vacrel->dead_items = TidStoreCreate(vac_work_mem, MaxHeapTuplesPerPage,
+ NULL);
}
/*
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index f3922b72dc..84f71fb14a 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -2486,7 +2486,7 @@ vac_bulkdel_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat,
ereport(ivinfo->message_level,
(errmsg("scanned index \"%s\" to remove " UINT64_FORMAT " row versions",
RelationGetRelationName(ivinfo->index),
- tidstore_num_tids(dead_items))));
+ TidStoreNumTids(dead_items))));
return istat;
}
@@ -2527,5 +2527,5 @@ vac_tid_reaped(ItemPointer itemptr, void *state)
{
TidStore *dead_items = (TidStore *) state;
- return tidstore_lookup_tid(dead_items, itemptr);
+ return TidStoreIsMember(dead_items, itemptr);
}
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index c363f45e32..be83ceb871 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -110,7 +110,7 @@ typedef struct PVShared
pg_atomic_uint32 idx;
/* Handle of the shared TidStore */
- tidstore_handle dead_items_handle;
+ TidStoreHandle dead_items_handle;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -372,7 +372,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
dead_items_dsa = dsa_create_in_place(area_space, dsa_minsize,
LWTRANCHE_PARALLEL_VACUUM_DSA,
pcxt->seg);
- dead_items = tidstore_create(vac_work_mem, max_offset, dead_items_dsa);
+ dead_items = TidStoreCreate(vac_work_mem, max_offset, dead_items_dsa);
pvs->dead_items = dead_items;
pvs->dead_items_area = dead_items_dsa;
@@ -385,7 +385,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
- shared->dead_items_handle = tidstore_get_handle(dead_items);
+ shared->dead_items_handle = TidStoreGetHandle(dead_items);
/* Use the same buffer size for all workers */
shared->ring_nbuffers = GetAccessStrategyBufferCount(bstrategy);
@@ -454,7 +454,7 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
istats[i] = NULL;
}
- tidstore_destroy(pvs->dead_items);
+ TidStoreDestroy(pvs->dead_items);
dsa_detach(pvs->dead_items_area);
DestroyParallelContext(pvs->pcxt);
@@ -1013,7 +1013,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
/* Set dead items */
area_space = shm_toc_lookup(toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, false);
dead_items_area = dsa_attach_in_place(area_space, seg);
- dead_items = tidstore_attach(dead_items_area, shared->dead_items_handle);
+ dead_items = TidStoreAttach(dead_items_area, shared->dead_items_handle);
/* Set cost-based vacuum delay */
VacuumUpdateCosts();
@@ -1061,7 +1061,7 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
InstrEndParallelQuery(&buffer_usage[ParallelWorkerNumber],
&wal_usage[ParallelWorkerNumber]);
- tidstore_detach(pvs.dead_items);
+ TidStoreDetach(dead_items);
dsa_detach(dead_items_area);
/* Pop the error context stack */
diff --git a/src/include/access/tidstore.h b/src/include/access/tidstore.h
index a35a52124a..f0a432d0da 100644
--- a/src/include/access/tidstore.h
+++ b/src/include/access/tidstore.h
@@ -17,7 +17,7 @@
#include "storage/itemptr.h"
#include "utils/dsa.h"
-typedef dsa_pointer tidstore_handle;
+typedef dsa_pointer TidStoreHandle;
typedef struct TidStore TidStore;
typedef struct TidStoreIter TidStoreIter;
@@ -29,21 +29,21 @@ typedef struct TidStoreIterResult
int num_offsets;
} TidStoreIterResult;
-extern TidStore *tidstore_create(size_t max_bytes, int max_offset, dsa_area *dsa);
-extern TidStore *tidstore_attach(dsa_area *dsa, dsa_pointer handle);
-extern void tidstore_detach(TidStore *ts);
-extern void tidstore_destroy(TidStore *ts);
-extern void tidstore_reset(TidStore *ts);
-extern void tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
- int num_offsets);
-extern bool tidstore_lookup_tid(TidStore *ts, ItemPointer tid);
-extern TidStoreIter * tidstore_begin_iterate(TidStore *ts);
-extern TidStoreIterResult *tidstore_iterate_next(TidStoreIter *iter);
-extern void tidstore_end_iterate(TidStoreIter *iter);
-extern int64 tidstore_num_tids(TidStore *ts);
-extern bool tidstore_is_full(TidStore *ts);
-extern size_t tidstore_max_memory(TidStore *ts);
-extern size_t tidstore_memory_usage(TidStore *ts);
-extern tidstore_handle tidstore_get_handle(TidStore *ts);
+extern TidStore *TidStoreCreate(size_t max_bytes, int max_off, dsa_area *dsa);
+extern TidStore *TidStoreAttach(dsa_area *dsa, dsa_pointer handle);
+extern void TidStoreDetach(TidStore *ts);
+extern void TidStoreDestroy(TidStore *ts);
+extern void TidStoreReset(TidStore *ts);
+extern void TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets);
+extern bool TidStoreIsMember(TidStore *ts, ItemPointer tid);
+extern TidStoreIter * TidStoreBeginIterate(TidStore *ts);
+extern TidStoreIterResult *TidStoreIterateNext(TidStoreIter *iter);
+extern void TidStoreEndIterate(TidStoreIter *iter);
+extern int64 TidStoreNumTids(TidStore *ts);
+extern bool TidStoreIsFull(TidStore *ts);
+extern size_t TidStoreMaxMemory(TidStore *ts);
+extern size_t TidStoreMemoryUsage(TidStore *ts);
+extern TidStoreHandle TidStoreGetHandle(TidStore *ts);
#endif /* TIDSTORE_H */
diff --git a/src/test/modules/test_tidstore/test_tidstore.c b/src/test/modules/test_tidstore/test_tidstore.c
index 9a1217f833..12d3027624 100644
--- a/src/test/modules/test_tidstore/test_tidstore.c
+++ b/src/test/modules/test_tidstore/test_tidstore.c
@@ -37,10 +37,10 @@ check_tid(TidStore *ts, BlockNumber blkno, OffsetNumber off, bool expect)
ItemPointerSet(&tid, blkno, off);
- found = tidstore_lookup_tid(ts, &tid);
+ found = TidStoreIsMember(ts, &tid);
if (found != expect)
- elog(ERROR, "lookup TID (%u, %u) returned %d, expected %d",
+ elog(ERROR, "TidStoreIsMember for TID (%u, %u) returned %d, expected %d",
blkno, off, found, expect);
}
@@ -69,9 +69,9 @@ test_basic(int max_offset)
LWLockRegisterTranche(tranche_id, "test_tidstore");
dsa = dsa_create(tranche_id);
- ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, dsa);
+ ts = TidStoreCreate(TEST_TIDSTORE_MAX_BYTES, max_offset, dsa);
#else
- ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
+ ts = TidStoreCreate(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
#endif
/* prepare the offset array */
@@ -83,7 +83,7 @@ test_basic(int max_offset)
/* add tids */
for (int i = 0; i < TEST_TIDSTORE_NUM_BLOCKS; i++)
- tidstore_add_tids(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
+ TidStoreSetBlockOffsets(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
/* lookup test */
for (OffsetNumber off = FirstOffsetNumber ; off < max_offset; off++)
@@ -105,30 +105,30 @@ test_basic(int max_offset)
}
/* test the number of tids */
- if (tidstore_num_tids(ts) != (TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS))
- elog(ERROR, "tidstore_num_tids returned " UINT64_FORMAT ", expected %d",
- tidstore_num_tids(ts),
+ if (TidStoreNumTids(ts) != (TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS))
+ elog(ERROR, "TidStoreNumTids returned " UINT64_FORMAT ", expected %d",
+ TidStoreNumTids(ts),
TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS);
/* iteration test */
- iter = tidstore_begin_iterate(ts);
+ iter = TidStoreBeginIterate(ts);
blk_idx = 0;
- while ((iter_result = tidstore_iterate_next(iter)) != NULL)
+ while ((iter_result = TidStoreIterateNext(iter)) != NULL)
{
/* check the returned block number */
if (blks_sorted[blk_idx] != iter_result->blkno)
- elog(ERROR, "tidstore_iterate_next returned block number %u, expected %u",
+ elog(ERROR, "TidStoreIterateNext returned block number %u, expected %u",
iter_result->blkno, blks_sorted[blk_idx]);
/* check the returned offset numbers */
if (TEST_TIDSTORE_NUM_OFFSETS != iter_result->num_offsets)
- elog(ERROR, "tidstore_iterate_next returned %u offsets, expected %u",
+ elog(ERROR, "TidStoreIterateNext %u offsets, expected %u",
iter_result->num_offsets, TEST_TIDSTORE_NUM_OFFSETS);
for (int i = 0; i < iter_result->num_offsets; i++)
{
if (offs[i] != iter_result->offsets[i])
- elog(ERROR, "tidstore_iterate_next returned offset number %u on block %u, expected %u",
+ elog(ERROR, "TidStoreIterateNext offset number %u on block %u, expected %u",
iter_result->offsets[i], iter_result->blkno, offs[i]);
}
@@ -136,15 +136,15 @@ test_basic(int max_offset)
}
if (blk_idx != TEST_TIDSTORE_NUM_BLOCKS)
- elog(ERROR, "tidstore_iterate_next returned %d blocks, expected %d",
+ elog(ERROR, "TidStoreIterateNext returned %d blocks, expected %d",
blk_idx, TEST_TIDSTORE_NUM_BLOCKS);
/* remove all tids */
- tidstore_reset(ts);
+ TidStoreReset(ts);
/* test the number of tids */
- if (tidstore_num_tids(ts) != 0)
- elog(ERROR, "tidstore_num_tids on empty store returned non-zero");
+ if (TidStoreNumTids(ts) != 0)
+ elog(ERROR, "TidStoreNumTids on empty store returned non-zero");
/* lookup test for empty store */
for (OffsetNumber off = FirstOffsetNumber ; off < MaxHeapTuplesPerPage;
@@ -156,7 +156,7 @@ test_basic(int max_offset)
check_tid(ts, MaxBlockNumber, off, false);
}
- tidstore_destroy(ts);
+ TidStoreDestroy(ts);
#ifdef TEST_SHARED_TIDSTORE
dsa_detach(dsa);
@@ -177,36 +177,37 @@ test_empty(void)
LWLockRegisterTranche(tranche_id, "test_tidstore");
dsa = dsa_create(tranche_id);
- ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, dsa);
+ ts = TidStoreCreate(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, dsa);
#else
- ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
+ ts = TidStoreCreate(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
#endif
elog(NOTICE, "testing empty tidstore");
ItemPointerSet(&tid, 0, FirstOffsetNumber);
- if (tidstore_lookup_tid(ts, &tid))
- elog(ERROR, "tidstore_lookup_tid for (0,1) on empty store returned true");
+ if (TidStoreIsMember(ts, &tid))
+ elog(ERROR, "TidStoreIsMember for TID (%u,%u) on empty store returned true",
+ 0, FirstOffsetNumber);
ItemPointerSet(&tid, MaxBlockNumber, MaxOffsetNumber);
- if (tidstore_lookup_tid(ts, &tid))
- elog(ERROR, "tidstore_lookup_tid for (%u,%u) on empty store returned true",
+ if (TidStoreIsMember(ts, &tid))
+ elog(ERROR, "TidStoreIsMember for TID (%u,%u) on empty store returned true",
MaxBlockNumber, MaxOffsetNumber);
- if (tidstore_num_tids(ts) != 0)
- elog(ERROR, "tidstore_num_entries on empty store returned non-zero");
+ if (TidStoreNumTids(ts) != 0)
+ elog(ERROR, "TidStoreNumTids on empty store returned non-zero");
- if (tidstore_is_full(ts))
- elog(ERROR, "tidstore_is_full on empty store returned true");
+ if (TidStoreIsFull(ts))
+ elog(ERROR, "TidStoreIsFull on empty store returned true");
- iter = tidstore_begin_iterate(ts);
+ iter = TidStoreBeginIterate(ts);
- if (tidstore_iterate_next(iter) != NULL)
- elog(ERROR, "tidstore_iterate_next on empty store returned TIDs");
+ if (TidStoreIterateNext(iter) != NULL)
+ elog(ERROR, "TidStoreIterateNext on empty store returned TIDs");
- tidstore_end_iterate(iter);
+ TidStoreEndIterate(iter);
- tidstore_destroy(ts);
+ TidStoreDestroy(ts);
#ifdef TEST_SHARED_TIDSTORE
dsa_detach(dsa);
--
2.31.1
[application/octet-stream] v32-0012-tidstore-Use-concept-of-off_upper-and-off_lower.patch (12.6K, ../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/9-v32-0012-tidstore-Use-concept-of-off_upper-and-off_lower.patch)
download | inline diff:
From 3f38c7722deb260e5cc4ac003ab37cfe959b1954 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 17:54:49 +0900
Subject: [PATCH v32 12/18] tidstore: Use concept of off_upper and off_lower.
The key is block number + the upper of offset number, whereas the
value is the bitmap representation of the lower offset number.
Updated function and variable names accordingly.
---
src/backend/access/common/tidstore.c | 191 ++++++++++++++-------------
1 file changed, 99 insertions(+), 92 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index 283a326d13..d9fe3d5f15 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -65,8 +65,10 @@
*
* The maximum height of the radix tree is 5 in this case.
*/
-#define TIDSTORE_VALUE_NBITS 6 /* log(64, 2) */
-#define TIDSTORE_OFFSET_MASK ((1 << TIDSTORE_VALUE_NBITS) - 1)
+typedef uint64 tidkey;
+typedef uint64 offsetbm;
+#define LOWER_OFFSET_NBITS 6 /* log(sizeof(offsetbm), 2) */
+#define LOWER_OFFSET_MASK ((1 << LOWER_OFFSET_NBITS) - 1)
/* A magic value used to identify our TidStores. */
#define TIDSTORE_MAGIC 0x826f6a10
@@ -75,7 +77,7 @@
#define RT_SCOPE static
#define RT_DECLARE
#define RT_DEFINE
-#define RT_VALUE_TYPE uint64
+#define RT_VALUE_TYPE tidkey
#include "lib/radixtree.h"
#define RT_PREFIX shared_rt
@@ -83,7 +85,7 @@
#define RT_SCOPE static
#define RT_DECLARE
#define RT_DEFINE
-#define RT_VALUE_TYPE uint64
+#define RT_VALUE_TYPE tidkey
#include "lib/radixtree.h"
/* The control object for a TidStore */
@@ -94,10 +96,10 @@ typedef struct TidStoreControl
/* These values are never changed after creation */
size_t max_bytes; /* the maximum bytes a TidStore can use */
- int max_offset; /* the maximum offset number */
- int offset_nbits; /* the number of bits required for an offset
- * number */
- int offset_key_nbits; /* the number of bits of an offset number
+ int max_off; /* the maximum offset number */
+ int max_off_nbits; /* the number of bits required for offset
+ * numbers */
+ int upper_off_nbits; /* the number of bits of offset numbers
* used in a key */
/* The below fields are used only in shared case */
@@ -147,17 +149,18 @@ typedef struct TidStoreIter
bool finished;
/* save for the next iteration */
- uint64 next_key;
- uint64 next_val;
+ tidkey next_tidkey;
+ offsetbm next_off_bitmap;
/* output for the caller */
TidStoreIterResult result;
} TidStoreIter;
-static void tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val);
-static inline BlockNumber key_get_blkno(TidStore *ts, uint64 key);
-static inline uint64 encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit);
-static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit);
+static void iter_decode_key_off(TidStoreIter *iter, tidkey key, offsetbm off_bitmap);
+static inline BlockNumber key_get_blkno(TidStore *ts, tidkey key);
+static inline tidkey encode_blk_off(TidStore *ts, BlockNumber block,
+ OffsetNumber offset, offsetbm *off_bit);
+static inline tidkey encode_tid(TidStore *ts, ItemPointer tid, offsetbm *off_bit);
/*
* Create a TidStore. The returned object is allocated in backend-local memory.
@@ -218,14 +221,14 @@ TidStoreCreate(size_t max_bytes, int max_off, dsa_area *area)
ts->control->max_bytes = max_bytes - (70 * 1024);
}
- ts->control->max_offset = max_offset;
- ts->control->offset_nbits = pg_ceil_log2_32(max_offset);
+ ts->control->max_off = max_off;
+ ts->control->max_off_nbits = pg_ceil_log2_32(max_off);
- if (ts->control->offset_nbits < TIDSTORE_VALUE_NBITS)
- ts->control->offset_nbits = TIDSTORE_VALUE_NBITS;
+ if (ts->control->max_off_nbits < LOWER_OFFSET_NBITS)
+ ts->control->max_off_nbits = LOWER_OFFSET_NBITS;
- ts->control->offset_key_nbits =
- ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
+ ts->control->upper_off_nbits =
+ ts->control->max_off_nbits - LOWER_OFFSET_NBITS;
return ts;
}
@@ -355,25 +358,25 @@ void
TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
int num_offsets)
{
- uint64 *values;
- uint64 key;
- uint64 prev_key;
- uint64 off_bitmap = 0;
+ offsetbm *bitmaps;
+ tidkey key;
+ tidkey prev_key;
+ offsetbm off_bitmap = 0;
int idx;
- const uint64 key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
- const int nkeys = UINT64CONST(1) << ts->control->offset_key_nbits;
+ const tidkey key_base = ((uint64) blkno) << ts->control->upper_off_nbits;
+ const int nkeys = UINT64CONST(1) << ts->control->upper_off_nbits;
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
- values = palloc(sizeof(uint64) * nkeys);
+ bitmaps = palloc(sizeof(offsetbm) * nkeys);
key = prev_key = key_base;
for (int i = 0; i < num_offsets; i++)
{
- uint64 off_bit;
+ offsetbm off_bit;
/* encode the tid to a key and partial offset */
- key = encode_key_off(ts, blkno, offsets[i], &off_bit);
+ key = encode_blk_off(ts, blkno, offsets[i], &off_bit);
/* make sure we scanned the line pointer array in order */
Assert(key >= prev_key);
@@ -384,11 +387,11 @@ TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
Assert(idx >= 0 && idx < nkeys);
/* write out offset bitmap for this key */
- values[idx] = off_bitmap;
+ bitmaps[idx] = off_bitmap;
/* zero out any gaps up to the current key */
for (int empty_idx = idx + 1; empty_idx < key - key_base; empty_idx++)
- values[empty_idx] = 0;
+ bitmaps[empty_idx] = 0;
/* reset for current key -- the current offset will be handled below */
off_bitmap = 0;
@@ -401,7 +404,7 @@ TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
/* save the final index for later */
idx = key - key_base;
/* write out last offset bitmap */
- values[idx] = off_bitmap;
+ bitmaps[idx] = off_bitmap;
if (TidStoreIsShared(ts))
LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
@@ -409,14 +412,14 @@ TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
/* insert the calculated key-values to the tree */
for (int i = 0; i <= idx; i++)
{
- if (values[i])
+ if (bitmaps[i])
{
key = key_base + i;
if (TidStoreIsShared(ts))
- shared_rt_set(ts->tree.shared, key, &values[i]);
+ shared_rt_set(ts->tree.shared, key, &bitmaps[i]);
else
- local_rt_set(ts->tree.local, key, &values[i]);
+ local_rt_set(ts->tree.local, key, &bitmaps[i]);
}
}
@@ -426,29 +429,29 @@ TidStoreSetBlockOffsets(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
if (TidStoreIsShared(ts))
LWLockRelease(&ts->control->lock);
- pfree(values);
+ pfree(bitmaps);
}
/* Return true if the given tid is present in the TidStore */
bool
TidStoreIsMember(TidStore *ts, ItemPointer tid)
{
- uint64 key;
- uint64 val = 0;
- uint64 off_bit;
+ tidkey key;
+ offsetbm off_bitmap = 0;
+ offsetbm off_bit;
bool found;
- key = tid_to_key_off(ts, tid, &off_bit);
+ key = encode_tid(ts, tid, &off_bit);
if (TidStoreIsShared(ts))
- found = shared_rt_search(ts->tree.shared, key, &val);
+ found = shared_rt_search(ts->tree.shared, key, &off_bitmap);
else
- found = local_rt_search(ts->tree.local, key, &val);
+ found = local_rt_search(ts->tree.local, key, &off_bitmap);
if (!found)
return false;
- return (val & off_bit) != 0;
+ return (off_bitmap & off_bit) != 0;
}
/*
@@ -486,12 +489,12 @@ TidStoreBeginIterate(TidStore *ts)
}
static inline bool
-tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
+tidstore_iter(TidStoreIter *iter, tidkey *key, offsetbm *off_bitmap)
{
if (TidStoreIsShared(iter->ts))
- return shared_rt_iterate_next(iter->tree_iter.shared, key, val);
+ return shared_rt_iterate_next(iter->tree_iter.shared, key, off_bitmap);
- return local_rt_iterate_next(iter->tree_iter.local, key, val);
+ return local_rt_iterate_next(iter->tree_iter.local, key, off_bitmap);
}
/*
@@ -502,43 +505,46 @@ tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
TidStoreIterResult *
TidStoreIterateNext(TidStoreIter *iter)
{
- uint64 key;
- uint64 val;
- TidStoreIterResult *result = &(iter->result);
+ tidkey key;
+ offsetbm off_bitmap = 0;
+ TidStoreIterResult *output = &(iter->output);
if (iter->finished)
return NULL;
- if (BlockNumberIsValid(result->blkno))
- {
- /* Process the previously collected key-value */
- result->num_offsets = 0;
- tidstore_iter_extract_tids(iter, iter->next_key, iter->next_val);
- }
+ /* Initialize the outputs */
+ output->blkno = InvalidBlockNumber;
+ output->num_offsets = 0;
- while (tidstore_iter_kv(iter, &key, &val))
- {
- BlockNumber blkno;
+ /*
+ * Decode the key and offset bitmap that are collected in the previous
+ * time, if exists.
+ */
+ if (iter->next_off_bitmap > 0)
+ iter_decode_key_off(iter, iter->next_tidkey, iter->next_off_bitmap);
- blkno = key_get_blkno(iter->ts, key);
+ while (tidstore_iter(iter, &key, &off_bitmap))
+ {
+ BlockNumber blkno = key_get_blkno(iter->ts, key);
- if (BlockNumberIsValid(result->blkno) && result->blkno != blkno)
+ if (BlockNumberIsValid(output->blkno) && output->blkno != blkno)
{
/*
- * We got a key-value pair for a different block. So return the
- * collected tids, and remember the key-value for the next iteration.
+ * We got tids for a different block. We return the collected
+ * tids so far, and remember the key-value for the next
+ * iteration.
*/
- iter->next_key = key;
- iter->next_val = val;
- return result;
+ iter->next_tidkey = key;
+ iter->next_off_bitmap = off_bitmap;
+ return output;
}
- /* Collect tids extracted from the key-value pair */
- tidstore_iter_extract_tids(iter, key, val);
+ /* Collect tids decoded from the key and offset bitmap */
+ iter_decode_key_off(iter, key, off_bitmap);
}
iter->finished = true;
- return result;
+ return output;
}
/*
@@ -623,61 +629,62 @@ TidStoreGetHandle(TidStore *ts)
/* Extract tids from the given key-value pair */
static void
-tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
+iter_decode_key_off(TidStoreIter *iter, tidkey key, offsetbm off_bitmap)
{
TidStoreIterResult *result = (&iter->result);
- while (val)
+ while (off_bitmap)
{
- uint64 tid_i;
+ uint64 compressed_tid;
OffsetNumber off;
- tid_i = key << TIDSTORE_VALUE_NBITS;
- tid_i |= pg_rightmost_one_pos64(val);
+ compressed_tid = key << LOWER_OFFSET_NBITS;
+ compressed_tid |= pg_rightmost_one_pos64(off_bitmap);
- off = tid_i & ((UINT64CONST(1) << iter->ts->control->offset_nbits) - 1);
+ off = compressed_tid & ((UINT64CONST(1) << iter->ts->control->max_off_nbits) - 1);
- Assert(result->num_offsets < iter->ts->control->max_offset);
- result->offsets[result->num_offsets++] = off;
+ Assert(output->num_offsets < iter->ts->control->max_off);
+ output->offsets[output->num_offsets++] = off;
/* unset the rightmost bit */
- val &= ~pg_rightmost_one64(val);
+ off_bitmap &= ~pg_rightmost_one64(off_bitmap);
}
- result->blkno = key_get_blkno(iter->ts, key);
+ output->blkno = key_get_blkno(iter->ts, key);
}
/* Get block number from the given key */
static inline BlockNumber
-key_get_blkno(TidStore *ts, uint64 key)
+key_get_blkno(TidStore *ts, tidkey key)
{
- return (BlockNumber) (key >> ts->control->offset_key_nbits);
+ return (BlockNumber) (key >> ts->control->upper_off_nbits);
}
-/* Encode a tid to key and offset */
-static inline uint64
-tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit)
+/* Encode a tid to key and partial offset */
+static inline tidkey
+encode_tid(TidStore *ts, ItemPointer tid, offsetbm *off_bit)
{
uint32 offset = ItemPointerGetOffsetNumber(tid);
BlockNumber block = ItemPointerGetBlockNumber(tid);
- return encode_key_off(ts, block, offset, off_bit);
+ return encode_blk_off(ts, block, offset, off_bit);
}
/* encode a block and offset to a key and partial offset */
-static inline uint64
-encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit)
+static inline tidkey
+encode_blk_off(TidStore *ts, BlockNumber block, OffsetNumber offset,
+ offsetbm *off_bit)
{
- uint64 key;
- uint64 tid_i;
+ tidkey key;
+ uint64 compressed_tid;
uint32 off_lower;
- off_lower = offset & TIDSTORE_OFFSET_MASK;
- Assert(off_lower < (sizeof(uint64) * BITS_PER_BYTE));
+ off_lower = offset & LOWER_OFFSET_MASK;
+ Assert(off_lower < (sizeof(offsetbm) * BITS_PER_BYTE));
*off_bit = UINT64CONST(1) << off_lower;
- tid_i = offset | ((uint64) block << ts->control->offset_nbits);
- key = tid_i >> TIDSTORE_VALUE_NBITS;
+ compressed_tid = offset | ((uint64) block << ts->control->max_off_nbits);
+ key = compressed_tid >> LOWER_OFFSET_NBITS;
return key;
}
--
2.31.1
[application/octet-stream] v32-0009-radix-tree-Review-tree-iteration-code.patch (13.9K, ../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/10-v32-0009-radix-tree-Review-tree-iteration-code.patch)
download | inline diff:
From 989dd2cb442c1c2a6182bb5f7785c52f4d5cdb5e Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 17:33:21 +0900
Subject: [PATCH v32 09/18] radix tree: Review tree iteration code
Cleanup the routines and improve comments and variable names.
---
src/include/lib/radixtree.h | 152 ++++++++++++++------------
src/include/lib/radixtree_iter_impl.h | 85 +++++++-------
2 files changed, 118 insertions(+), 119 deletions(-)
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index 088d1dfd9d..8bea606c62 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -83,7 +83,7 @@
* RT_SET - Set a key-value pair
* RT_BEGIN_ITERATE - Begin iterating through all key-value pairs
* RT_ITERATE_NEXT - Return next key-value pair, if any
- * RT_END_ITER - End iteration
+ * RT_END_ITERATE - End iteration
* RT_MEMORY_USAGE - Get the memory usage
*
* Interface for Shared Memory
@@ -191,7 +191,7 @@
#define RT_NODE_INSERT_LEAF RT_MAKE_NAME(node_insert_leaf)
#define RT_NODE_INNER_ITERATE_NEXT RT_MAKE_NAME(node_inner_iterate_next)
#define RT_NODE_LEAF_ITERATE_NEXT RT_MAKE_NAME(node_leaf_iterate_next)
-#define RT_UPDATE_ITER_STACK RT_MAKE_NAME(update_iter_stack)
+#define RT_ITER_SET_NODE_FROM RT_MAKE_NAME(iter_set_node_from)
#define RT_ITER_UPDATE_KEY RT_MAKE_NAME(iter_update_key)
#define RT_VERIFY_NODE RT_MAKE_NAME(verify_node)
@@ -650,36 +650,40 @@ typedef struct RT_RADIX_TREE
* Iteration support.
*
* Iterating the radix tree returns each pair of key and value in the ascending
- * order of the key. To support this, the we iterate nodes of each level.
+ * order of the key.
*
- * RT_NODE_ITER struct is used to track the iteration within a node.
+ * RT_NODE_ITER is the struct for iteration of one radix tree node.
*
* RT_ITER is the struct for iteration of the radix tree, and uses RT_NODE_ITER
- * in order to track the iteration of each level. During iteration, we also
- * construct the key whenever updating the node iteration information, e.g., when
- * advancing the current index within the node or when moving to the next node
- * at the same level.
- *
- * XXX: Currently we allow only one process to do iteration. Therefore, rt_node_iter
- * has the local pointers to nodes, rather than RT_PTR_ALLOC.
- * We need either a safeguard to disallow other processes to begin the iteration
- * while one process is doing or to allow multiple processes to do the iteration.
+ * for each level to track the iteration within the node.
*/
typedef struct RT_NODE_ITER
{
- RT_PTR_LOCAL node; /* current node being iterated */
- int current_idx; /* current position. -1 for initial value */
+ /*
+ * Local pointer to the node we are iterating over.
+ *
+ * Since the radix tree doesn't support the shared iteration among multiple
+ * processes, we use RT_PTR_LOCAL rather than RT_PTR_ALLOC.
+ */
+ RT_PTR_LOCAL node;
+
+ /*
+ * The next index of the chunk array in RT_NODE_KIND_3 and
+ * RT_NODE_KIND_32 nodes, or the next chunk in RT_NODE_KIND_125 and
+ * RT_NODE_KIND_256 nodes. 0 for the initial value.
+ */
+ int idx;
} RT_NODE_ITER;
typedef struct RT_ITER
{
RT_RADIX_TREE *tree;
- /* Track the iteration on nodes of each level */
- RT_NODE_ITER stack[RT_MAX_LEVEL];
- int stack_len;
+ /* Track the nodes for each level. level = 0 is for a leaf node */
+ RT_NODE_ITER node_iters[RT_MAX_LEVEL];
+ int top_level;
- /* The key is constructed during iteration */
+ /* The key constructed during the iteration */
uint64 key;
} RT_ITER;
@@ -1804,16 +1808,9 @@ RT_DELETE(RT_RADIX_TREE *tree, uint64 key)
}
#endif
-static inline void
-RT_ITER_UPDATE_KEY(RT_ITER *iter, uint8 chunk, uint8 shift)
-{
- iter->key &= ~(((uint64) RT_CHUNK_MASK) << shift);
- iter->key |= (((uint64) chunk) << shift);
-}
-
/*
- * Advance the slot in the inner node. Return the child if exists, otherwise
- * null.
+ * Scan the inner node and return the next child node if exist, otherwise
+ * return NULL.
*/
static inline RT_PTR_LOCAL
RT_NODE_INNER_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter)
@@ -1824,8 +1821,8 @@ RT_NODE_INNER_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter)
}
/*
- * Advance the slot in the leaf node. On success, return true and the value
- * is set to value_p, otherwise return false.
+ * Scan the leaf node, and return true and the next value is set to value_p
+ * if exists. Otherwise return false.
*/
static inline bool
RT_NODE_LEAF_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter,
@@ -1837,29 +1834,50 @@ RT_NODE_LEAF_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter,
}
/*
- * Update each node_iter for inner nodes in the iterator node stack.
+ * While descending the radix tree from the 'from' node to the bottom, we
+ * set the next node to iterate for each level.
*/
static void
-RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
+RT_ITER_SET_NODE_FROM(RT_ITER *iter, RT_PTR_LOCAL from)
{
- int level = from;
- RT_PTR_LOCAL node = from_node;
+ int level = from->shift / RT_NODE_SPAN;
+ RT_PTR_LOCAL node = from;
for (;;)
{
- RT_NODE_ITER *node_iter = &(iter->stack[level--]);
+ RT_NODE_ITER *node_iter = &(iter->node_iters[level--]);
+
+#ifdef USE_ASSERT_CHECKING
+ if (node_iter->node)
+ {
+ /* We must have finished the iteration on the previous node */
+ if (RT_NODE_IS_LEAF(node_iter->node))
+ {
+ uint64 dummy;
+ Assert(!RT_NODE_LEAF_ITERATE_NEXT(iter, node_iter, &dummy));
+ }
+ else
+ Assert(!RT_NODE_INNER_ITERATE_NEXT(iter, node_iter));
+ }
+#endif
+ /* Set the node to the node iterator of this level */
node_iter->node = node;
- node_iter->current_idx = -1;
+ node_iter->idx = 0;
- /* We don't advance the leaf node iterator here */
if (RT_NODE_IS_LEAF(node))
- return;
+ {
+ /* We will visit the leaf node when RT_ITERATE_NEXT() */
+ break;
+ }
- /* Advance to the next slot in the inner node */
+ /*
+ * Get the first child node from the node, which corresponds to the
+ * lowest chunk within the node.
+ */
node = RT_NODE_INNER_ITERATE_NEXT(iter, node_iter);
- /* We must find the first children in the node */
+ /* The first child must be found */
Assert(node);
}
}
@@ -1873,14 +1891,11 @@ RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
RT_SCOPE RT_ITER *
RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
{
- MemoryContext old_ctx;
RT_ITER *iter;
RT_PTR_LOCAL root;
- int top_level;
- old_ctx = MemoryContextSwitchTo(tree->context);
-
- iter = (RT_ITER *) palloc0(sizeof(RT_ITER));
+ iter = (RT_ITER *) MemoryContextAllocZero(tree->context,
+ sizeof(RT_ITER));
iter->tree = tree;
RT_LOCK_SHARED(tree);
@@ -1890,16 +1905,13 @@ RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
return iter;
root = RT_PTR_GET_LOCAL(tree, iter->tree->ctl->root);
- top_level = root->shift / RT_NODE_SPAN;
- iter->stack_len = top_level;
+ iter->top_level = root->shift / RT_NODE_SPAN;
/*
- * Descend to the left most leaf node from the root. The key is being
- * constructed while descending to the leaf.
+ * Set the next node to iterate for each level from the level of the
+ * root node.
*/
- RT_UPDATE_ITER_STACK(iter, root, top_level);
-
- MemoryContextSwitchTo(old_ctx);
+ RT_ITER_SET_NODE_FROM(iter, root);
return iter;
}
@@ -1911,6 +1923,8 @@ RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
RT_SCOPE bool
RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p)
{
+ Assert(value_p != NULL);
+
/* Empty tree */
if (!iter->tree->ctl->root)
return false;
@@ -1918,43 +1932,38 @@ RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p)
for (;;)
{
RT_PTR_LOCAL child = NULL;
- RT_VALUE_TYPE value;
- int level;
- bool found;
-
- /* Advance the leaf node iterator to get next key-value pair */
- found = RT_NODE_LEAF_ITERATE_NEXT(iter, &(iter->stack[0]), &value);
- if (found)
+ /* Get the next chunk of the leaf node */
+ if (RT_NODE_LEAF_ITERATE_NEXT(iter, &(iter->node_iters[0]), value_p))
{
*key_p = iter->key;
- *value_p = value;
return true;
}
/*
- * We've visited all values in the leaf node, so advance inner node
- * iterators from the level=1 until we find the next child node.
+ * We've visited all values in the leaf node, so advance all inner node
+ * iterators by visiting inner nodes from the level = 1 until we find the
+ * next inner node that has a child node.
*/
- for (level = 1; level <= iter->stack_len; level++)
+ for (int level = 1; level <= iter->top_level; level++)
{
- child = RT_NODE_INNER_ITERATE_NEXT(iter, &(iter->stack[level]));
+ child = RT_NODE_INNER_ITERATE_NEXT(iter, &(iter->node_iters[level]));
if (child)
break;
}
- /* the iteration finished */
+ /* We've visited all nodes, so the iteration finished */
if (!child)
- return false;
+ break;
/*
- * Set the node to the node iterator and update the iterator stack
- * from this node.
+ * Found the new child node. We update the next node to iterate for each
+ * level from the level of this child node.
*/
- RT_UPDATE_ITER_STACK(iter, child, level - 1);
+ RT_ITER_SET_NODE_FROM(iter, child);
- /* Node iterators are updated, so try again from the leaf */
+ /* Find key-value from the leaf node again */
}
return false;
@@ -2508,8 +2517,7 @@ RT_DUMP(RT_RADIX_TREE *tree)
#undef RT_NODE_INSERT_LEAF
#undef RT_NODE_INNER_ITERATE_NEXT
#undef RT_NODE_LEAF_ITERATE_NEXT
-#undef RT_UPDATE_ITER_STACK
-#undef RT_ITER_UPDATE_KEY
+#undef RT_RT_ITER_SET_NODE_FROM
#undef RT_VERIFY_NODE
#undef RT_DEBUG
diff --git a/src/include/lib/radixtree_iter_impl.h b/src/include/lib/radixtree_iter_impl.h
index 98c78eb237..5c1034768e 100644
--- a/src/include/lib/radixtree_iter_impl.h
+++ b/src/include/lib/radixtree_iter_impl.h
@@ -27,12 +27,10 @@
#error node level must be either inner or leaf
#endif
- bool found = false;
- uint8 key_chunk;
+ uint8 key_chunk = 0;
#ifdef RT_NODE_LEVEL_LEAF
- RT_VALUE_TYPE value;
-
+ Assert(value_p != NULL);
Assert(RT_NODE_IS_LEAF(node_iter->node));
#else
RT_PTR_LOCAL child = NULL;
@@ -50,99 +48,92 @@
{
RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node_iter->node;
- node_iter->current_idx++;
- if (node_iter->current_idx >= n3->base.n.count)
- break;
+ if (node_iter->idx >= n3->base.n.count)
+ return false;
+
#ifdef RT_NODE_LEVEL_LEAF
- value = n3->values[node_iter->current_idx];
+ *value_p = n3->values[node_iter->idx];
#else
- child = RT_PTR_GET_LOCAL(iter->tree, n3->children[node_iter->current_idx]);
+ child = RT_PTR_GET_LOCAL(iter->tree, n3->children[node_iter->idx]);
#endif
- key_chunk = n3->base.chunks[node_iter->current_idx];
- found = true;
+ key_chunk = n3->base.chunks[node_iter->idx];
+ node_iter->idx++;
break;
}
case RT_NODE_KIND_32:
{
RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node_iter->node;
- node_iter->current_idx++;
- if (node_iter->current_idx >= n32->base.n.count)
- break;
+ if (node_iter->idx >= n32->base.n.count)
+ return false;
#ifdef RT_NODE_LEVEL_LEAF
- value = n32->values[node_iter->current_idx];
+ *value_p = n32->values[node_iter->idx];
#else
- child = RT_PTR_GET_LOCAL(iter->tree, n32->children[node_iter->current_idx]);
+ child = RT_PTR_GET_LOCAL(iter->tree, n32->children[node_iter->idx]);
#endif
- key_chunk = n32->base.chunks[node_iter->current_idx];
- found = true;
+ key_chunk = n32->base.chunks[node_iter->idx];
+ node_iter->idx++;
break;
}
case RT_NODE_KIND_125:
{
RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node_iter->node;
- int i;
+ int chunk;
- for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ for (chunk = node_iter->idx; chunk < RT_NODE_MAX_SLOTS; chunk++)
{
- if (RT_NODE_125_IS_CHUNK_USED((RT_NODE_BASE_125 *) n125, i))
+ if (RT_NODE_125_IS_CHUNK_USED((RT_NODE_BASE_125 *) n125, chunk))
break;
}
- if (i >= RT_NODE_MAX_SLOTS)
- break;
+ if (chunk >= RT_NODE_MAX_SLOTS)
+ return false;
- node_iter->current_idx = i;
#ifdef RT_NODE_LEVEL_LEAF
- value = RT_NODE_LEAF_125_GET_VALUE(n125, i);
+ *value_p = RT_NODE_LEAF_125_GET_VALUE(n125, chunk);
#else
- child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_125_GET_CHILD(n125, chunk));
#endif
- key_chunk = i;
- found = true;
+ key_chunk = chunk;
+ node_iter->idx = chunk + 1;
break;
}
case RT_NODE_KIND_256:
{
RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node_iter->node;
- int i;
+ int chunk;
- for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ for (chunk = node_iter->idx; chunk < RT_NODE_MAX_SLOTS; chunk++)
{
#ifdef RT_NODE_LEVEL_LEAF
- if (RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+ if (RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
#else
- if (RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ if (RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
#endif
break;
}
- if (i >= RT_NODE_MAX_SLOTS)
- break;
+ if (chunk >= RT_NODE_MAX_SLOTS)
+ return false;
- node_iter->current_idx = i;
#ifdef RT_NODE_LEVEL_LEAF
- value = RT_NODE_LEAF_256_GET_VALUE(n256, i);
+ *value_p = RT_NODE_LEAF_256_GET_VALUE(n256, chunk);
#else
- child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_256_GET_CHILD(n256, chunk));
#endif
- key_chunk = i;
- found = true;
+ key_chunk = chunk;
+ node_iter->idx = chunk + 1;
break;
}
}
- if (found)
- {
- RT_ITER_UPDATE_KEY(iter, key_chunk, node_iter->node->shift);
-#ifdef RT_NODE_LEVEL_LEAF
- *value_p = value;
-#endif
- }
+ /* Update the part of the key */
+ iter->key &= ~(((uint64) RT_CHUNK_MASK) << node_iter->node->shift);
+ iter->key |= (((uint64) key_chunk) << node_iter->node->shift);
#ifdef RT_NODE_LEVEL_LEAF
- return found;
+ return true;
#else
return child;
#endif
--
2.31.1
[application/octet-stream] v32-0013-tidstore-Embed-output-offsets-in-TidStoreIterRes.patch (1.9K, ../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/11-v32-0013-tidstore-Embed-output-offsets-in-TidStoreIterRes.patch)
download | inline diff:
From 453dc7fd8078ba202569417d4ed65ce6e7f4a850 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 18:00:44 +0900
Subject: [PATCH v32 13/18] tidstore: Embed output offsets in
TidStoreIterResult.
---
src/backend/access/common/tidstore.c | 7 ++-----
src/include/access/tidstore.h | 3 ++-
2 files changed, 4 insertions(+), 6 deletions(-)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
index d9fe3d5f15..15b77b5bcb 100644
--- a/src/backend/access/common/tidstore.c
+++ b/src/backend/access/common/tidstore.c
@@ -470,12 +470,10 @@ TidStoreBeginIterate(TidStore *ts)
Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
- iter = palloc0(sizeof(TidStoreIter));
+ iter = palloc0(sizeof(TidStoreIter) +
+ sizeof(OffsetNumber) * ts->control->max_off);
iter->ts = ts;
- iter->result.blkno = InvalidBlockNumber;
- iter->result.offsets = palloc(sizeof(OffsetNumber) * ts->control->max_offset);
-
if (TidStoreIsShared(ts))
iter->tree_iter.shared = shared_rt_begin_iterate(ts->tree.shared);
else
@@ -559,7 +557,6 @@ TidStoreEndIterate(TidStoreIter *iter)
else
local_rt_end_iterate(iter->tree_iter.local);
- pfree(iter->result.offsets);
pfree(iter);
}
diff --git a/src/include/access/tidstore.h b/src/include/access/tidstore.h
index f0a432d0da..66f0fdd482 100644
--- a/src/include/access/tidstore.h
+++ b/src/include/access/tidstore.h
@@ -22,11 +22,12 @@ typedef dsa_pointer TidStoreHandle;
typedef struct TidStore TidStore;
typedef struct TidStoreIter TidStoreIter;
+/* Result struct for TidStoreIterateNext */
typedef struct TidStoreIterResult
{
BlockNumber blkno;
- OffsetNumber *offsets;
int num_offsets;
+ OffsetNumber offsets[FLEXIBLE_ARRAY_MEMBER];
} TidStoreIterResult;
extern TidStore *TidStoreCreate(size_t max_bytes, int max_off, dsa_area *dsa);
--
2.31.1
[application/octet-stream] v32-0008-radix-tree-remove-resolved-TODO.patch (706B, ../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/12-v32-0008-radix-tree-remove-resolved-TODO.patch)
download | inline diff:
From 84bad553eecc97bbc3d7ccacc90723ae22b7888f Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 17:29:32 +0900
Subject: [PATCH v32 08/18] radix tree: remove resolved TODO
---
src/include/lib/radixtree.h | 1 -
1 file changed, 1 deletion(-)
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index c277d5a484..088d1dfd9d 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -612,7 +612,6 @@ static const RT_SIZE_CLASS_ELEM RT_SIZE_CLASS_INFO[] = {
#endif
/* Contains the actual tree and ancillary info */
-// WIP: this name is a bit strange
typedef struct RT_RADIX_TREE_CONTROL
{
#ifdef RT_SHMEM
--
2.31.1
[application/octet-stream] v32-0007-radix-tree-rename-RT_EXTEND-and-RT_SET_EXTEND-to.patch (2.6K, ../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/13-v32-0007-radix-tree-rename-RT_EXTEND-and-RT_SET_EXTEND-to.patch)
download | inline diff:
From e25dc39fd502ae5c6c1c44a798a24dc5c6a1c7b0 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 17:26:52 +0900
Subject: [PATCH v32 07/18] radix tree: rename RT_EXTEND and RT_SET_EXTEND to
RT_EXTEND_UP/DOWN
---
src/include/lib/radixtree.h | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
index e546bd705c..c277d5a484 100644
--- a/src/include/lib/radixtree.h
+++ b/src/include/lib/radixtree.h
@@ -152,8 +152,8 @@
#define RT_INIT_NODE RT_MAKE_NAME(init_node)
#define RT_FREE_NODE RT_MAKE_NAME(free_node)
#define RT_FREE_RECURSE RT_MAKE_NAME(free_recurse)
-#define RT_EXTEND RT_MAKE_NAME(extend)
-#define RT_SET_EXTEND RT_MAKE_NAME(set_extend)
+#define RT_EXTEND_UP RT_MAKE_NAME(extend_up)
+#define RT_EXTEND_DOWN RT_MAKE_NAME(extend_down)
#define RT_SWITCH_NODE_KIND RT_MAKE_NAME(grow_node_kind)
#define RT_COPY_NODE RT_MAKE_NAME(copy_node)
#define RT_REPLACE_NODE RT_MAKE_NAME(replace_node)
@@ -1243,7 +1243,7 @@ RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
* it can store the key.
*/
static pg_noinline void
-RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
+RT_EXTEND_UP(RT_RADIX_TREE *tree, uint64 key)
{
int target_shift;
RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
@@ -1282,7 +1282,7 @@ RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
* Insert inner and leaf nodes from 'node' to bottom.
*/
static pg_noinline void
-RT_SET_EXTEND(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
+RT_EXTEND_DOWN(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node)
{
int shift = node->shift;
@@ -1613,7 +1613,7 @@ RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
/* Extend the tree if necessary */
if (key > tree->ctl->max_val)
- RT_EXTEND(tree, key);
+ RT_EXTEND_UP(tree, key);
stored_child = tree->ctl->root;
parent = RT_PTR_GET_LOCAL(tree, stored_child);
@@ -1631,7 +1631,7 @@ RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
if (!RT_NODE_SEARCH_INNER(child, key, &new_child))
{
- RT_SET_EXTEND(tree, key, value_p, parent, stored_child, child);
+ RT_EXTEND_DOWN(tree, key, value_p, parent, stored_child, child);
RT_UNLOCK(tree);
return false;
}
@@ -2470,8 +2470,8 @@ RT_DUMP(RT_RADIX_TREE *tree)
#undef RT_INIT_NODE
#undef RT_FREE_NODE
#undef RT_FREE_RECURSE
-#undef RT_EXTEND
-#undef RT_SET_EXTEND
+#undef RT_EXTEND_UP
+#undef RT_EXTEND_DOWN
#undef RT_SWITCH_NODE_KIND
#undef RT_COPY_NODE
#undef RT_REPLACE_NODE
--
2.31.1
[application/octet-stream] v32-0006-Use-TIDStore-for-storing-dead-tuple-TID-during-l.patch (48.3K, ../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/14-v32-0006-Use-TIDStore-for-storing-dead-tuple-TID-during-l.patch)
download | inline diff:
From 1f6c4aa27d734b8c81369541481b0d3abd0d5dec Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 17 Apr 2023 17:22:03 +0900
Subject: [PATCH v32 06/18] Use TIDStore for storing dead tuple TID during lazy
vacuum
Previously, we used an array of ItemPointerData to store dead tuple
TIDs, which was not space efficient and slow to lookup. Also, we had
the 1GB limit on its size.
Now we use TIDStore to store dead tuple TIDs. Since the TIDStore,
backed by the radix tree, incrementaly allocates the memory, we get
rid of the 1GB limit.
Since we are no longer able to exactly estimate the maximum number of
TIDs can be stored the pg_stat_progress_vacuum shows the progress
information based on the amount of memory in bytes. The column names
are also changed to max_dead_tuple_bytes and num_dead_tuple_bytes.
In addition, since the TIDStore use the radix tree internally, the
minimum amount of memory required by TIDStore is 1MB, the inital DSA
segment size. Due to that, we increase the minimum value of
maintenance_work_mem (also autovacuum_work_mem) from 1MB to 2MB.
XXX: needs to bump catalog version
---
doc/src/sgml/monitoring.sgml | 8 +-
src/backend/access/heap/vacuumlazy.c | 278 ++++++++-------------
src/backend/catalog/system_views.sql | 2 +-
src/backend/commands/vacuum.c | 78 +-----
src/backend/commands/vacuumparallel.c | 73 +++---
src/backend/postmaster/autovacuum.c | 6 +-
src/backend/storage/lmgr/lwlock.c | 2 +
src/backend/utils/misc/guc_tables.c | 2 +-
src/include/commands/progress.h | 4 +-
src/include/commands/vacuum.h | 25 +-
src/include/storage/lwlock.h | 1 +
src/test/regress/expected/cluster.out | 2 +-
src/test/regress/expected/create_index.out | 2 +-
src/test/regress/expected/rules.out | 4 +-
src/test/regress/sql/cluster.sql | 2 +-
src/test/regress/sql/create_index.sql | 2 +-
16 files changed, 177 insertions(+), 314 deletions(-)
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index be4448fe6e..9b64614beb 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -7320,10 +7320,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>max_dead_tuples</structfield> <type>bigint</type>
+ <structfield>max_dead_tuple_bytes</structfield> <type>bigint</type>
</para>
<para>
- Number of dead tuples that we can store before needing to perform
+ Amount of dead tuple data that we can store before needing to perform
an index vacuum cycle, based on
<xref linkend="guc-maintenance-work-mem"/>.
</para></entry>
@@ -7331,10 +7331,10 @@ FROM pg_stat_get_backend_idset() AS backendid;
<row>
<entry role="catalog_table_entry"><para role="column_definition">
- <structfield>num_dead_tuples</structfield> <type>bigint</type>
+ <structfield>num_dead_tuple_bytes</structfield> <type>bigint</type>
</para>
<para>
- Number of dead tuples collected since the last index vacuum cycle.
+ Amount of dead tuple data collected since the last index vacuum cycle.
</para></entry>
</row>
</tbody>
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 0a9ebd22bd..2c72088e69 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -3,18 +3,18 @@
* vacuumlazy.c
* Concurrent ("lazy") vacuuming.
*
- * The major space usage for vacuuming is storage for the array of dead TIDs
+ * The major space usage for vacuuming is TidStore, a storage for dead TIDs
* that are to be removed from indexes. We want to ensure we can vacuum even
* the very largest relations with finite memory space usage. To do that, we
- * set upper bounds on the number of TIDs we can keep track of at once.
+ * set upper bounds on the maximum memory that can be used for keeping track
+ * of dead TIDs at once.
*
* We are willing to use at most maintenance_work_mem (or perhaps
* autovacuum_work_mem) memory space to keep track of dead TIDs. We initially
- * allocate an array of TIDs of that size, with an upper limit that depends on
- * table size (this limit ensures we don't allocate a huge area uselessly for
- * vacuuming small tables). If the array threatens to overflow, we must call
- * lazy_vacuum to vacuum indexes (and to vacuum the pages that we've pruned).
- * This frees up the memory space dedicated to storing dead TIDs.
+ * create a TidStore with the maximum bytes that can be used by the TidStore.
+ * If the TidStore is full, we must call lazy_vacuum to vacuum indexes (and to
+ * vacuum the pages that we've pruned). This frees up the memory space dedicated
+ * to storing dead TIDs.
*
* In practice VACUUM will often complete its initial pass over the target
* heap relation without ever running out of space to store TIDs. This means
@@ -40,6 +40,7 @@
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
#include "access/multixact.h"
+#include "access/tidstore.h"
#include "access/transam.h"
#include "access/visibilitymap.h"
#include "access/xact.h"
@@ -186,7 +187,7 @@ typedef struct LVRelState
* lazy_vacuum_heap_rel, which marks the same LP_DEAD line pointers as
* LP_UNUSED during second heap pass.
*/
- VacDeadItems *dead_items; /* TIDs whose index tuples we'll delete */
+ TidStore *dead_items; /* TIDs whose index tuples we'll delete */
BlockNumber rel_pages; /* total number of pages */
BlockNumber scanned_pages; /* # pages examined (not skipped via VM) */
BlockNumber removed_pages; /* # pages removed by relation truncation */
@@ -218,11 +219,14 @@ typedef struct LVRelState
typedef struct LVPagePruneState
{
bool hastup; /* Page prevents rel truncation? */
- bool has_lpdead_items; /* includes existing LP_DEAD items */
+
+ /* collected offsets of LP_DEAD items including existing ones */
+ OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
+ int num_offsets;
/*
* State describes the proper VM bit states to set for the page following
- * pruning and freezing. all_visible implies !has_lpdead_items, but don't
+ * pruning and freezing. all_visible implies num_offsets == 0, but don't
* trust all_frozen result unless all_visible is also set to true.
*/
bool all_visible; /* Every item visible to all? */
@@ -257,8 +261,9 @@ static bool lazy_scan_noprune(LVRelState *vacrel, Buffer buf,
static void lazy_vacuum(LVRelState *vacrel);
static bool lazy_vacuum_all_indexes(LVRelState *vacrel);
static void lazy_vacuum_heap_rel(LVRelState *vacrel);
-static int lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
- Buffer buffer, int index, Buffer vmbuffer);
+static void lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
+ OffsetNumber *offsets, int num_offsets,
+ Buffer buffer, Buffer vmbuffer);
static bool lazy_check_wraparound_failsafe(LVRelState *vacrel);
static void lazy_cleanup_all_indexes(LVRelState *vacrel);
static IndexBulkDeleteResult *lazy_vacuum_one_index(Relation indrel,
@@ -485,11 +490,11 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
}
/*
- * Allocate dead_items array memory using dead_items_alloc. This handles
- * parallel VACUUM initialization as part of allocating shared memory
- * space used for dead_items. (But do a failsafe precheck first, to
- * ensure that parallel VACUUM won't be attempted at all when relfrozenxid
- * is already dangerously old.)
+ * Allocate dead_items memory using dead_items_alloc. This handles parallel
+ * VACUUM initialization as part of allocating shared memory space used for
+ * dead_items. (But do a failsafe precheck first, to ensure that parallel
+ * VACUUM won't be attempted at all when relfrozenxid is already dangerously
+ * old.)
*/
lazy_check_wraparound_failsafe(vacrel);
dead_items_alloc(vacrel, params->nworkers);
@@ -795,7 +800,7 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* have collected the TIDs whose index tuples need to be removed.
*
* Finally, invokes lazy_vacuum_heap_rel to vacuum heap pages, which
- * largely consists of marking LP_DEAD items (from collected TID array)
+ * largely consists of marking LP_DEAD items (from vacrel->dead_items)
* as LP_UNUSED. This has to happen in a second, final pass over the
* heap, to preserve a basic invariant that all index AMs rely on: no
* extant index tuple can ever be allowed to contain a TID that points to
@@ -823,21 +828,21 @@ lazy_scan_heap(LVRelState *vacrel)
blkno,
next_unskippable_block,
next_fsm_block_to_vacuum = 0;
- VacDeadItems *dead_items = vacrel->dead_items;
+ TidStore *dead_items = vacrel->dead_items;
Buffer vmbuffer = InvalidBuffer;
bool next_unskippable_allvis,
skipping_current_range;
const int initprog_index[] = {
PROGRESS_VACUUM_PHASE,
PROGRESS_VACUUM_TOTAL_HEAP_BLKS,
- PROGRESS_VACUUM_MAX_DEAD_TUPLES
+ PROGRESS_VACUUM_MAX_DEAD_TUPLE_BYTES
};
int64 initprog_val[3];
/* Report that we're scanning the heap, advertising total # of blocks */
initprog_val[0] = PROGRESS_VACUUM_PHASE_SCAN_HEAP;
initprog_val[1] = rel_pages;
- initprog_val[2] = dead_items->max_items;
+ initprog_val[2] = tidstore_max_memory(vacrel->dead_items);
pgstat_progress_update_multi_param(3, initprog_index, initprog_val);
/* Set up an initial range of skippable blocks using the visibility map */
@@ -904,8 +909,7 @@ lazy_scan_heap(LVRelState *vacrel)
* dead_items TIDs, pause and do a cycle of vacuuming before we tackle
* this page.
*/
- Assert(dead_items->max_items >= MaxHeapTuplesPerPage);
- if (dead_items->max_items - dead_items->num_items < MaxHeapTuplesPerPage)
+ if (tidstore_is_full(vacrel->dead_items))
{
/*
* Before beginning index vacuuming, we release any pin we may
@@ -967,7 +971,7 @@ lazy_scan_heap(LVRelState *vacrel)
continue;
}
- /* Collect LP_DEAD items in dead_items array, count tuples */
+ /* Collect LP_DEAD items in dead_items, count tuples */
if (lazy_scan_noprune(vacrel, buf, blkno, page, &hastup,
&recordfreespace))
{
@@ -1009,14 +1013,14 @@ lazy_scan_heap(LVRelState *vacrel)
* Prune, freeze, and count tuples.
*
* Accumulates details of remaining LP_DEAD line pointers on page in
- * dead_items array. This includes LP_DEAD line pointers that we
- * pruned ourselves, as well as existing LP_DEAD line pointers that
- * were pruned some time earlier. Also considers freezing XIDs in the
- * tuple headers of remaining items with storage.
+ * dead_items. This includes LP_DEAD line pointers that we pruned
+ * ourselves, as well as existing LP_DEAD line pointers that were pruned
+ * some time earlier. Also considers freezing XIDs in the tuple headers
+ * of remaining items with storage.
*/
lazy_scan_prune(vacrel, buf, blkno, page, &prunestate);
- Assert(!prunestate.all_visible || !prunestate.has_lpdead_items);
+ Assert(!prunestate.all_visible || (prunestate.num_offsets == 0));
/* Remember the location of the last page with nonremovable tuples */
if (prunestate.hastup)
@@ -1032,14 +1036,12 @@ lazy_scan_heap(LVRelState *vacrel)
* performed here can be thought of as the one-pass equivalent of
* a call to lazy_vacuum().
*/
- if (prunestate.has_lpdead_items)
+ if (prunestate.num_offsets > 0)
{
Size freespace;
- lazy_vacuum_heap_page(vacrel, blkno, buf, 0, vmbuffer);
-
- /* Forget the LP_DEAD items that we just vacuumed */
- dead_items->num_items = 0;
+ lazy_vacuum_heap_page(vacrel, blkno, prunestate.deadoffsets,
+ prunestate.num_offsets, buf, vmbuffer);
/*
* Periodically perform FSM vacuuming to make newly-freed
@@ -1076,7 +1078,16 @@ lazy_scan_heap(LVRelState *vacrel)
* with prunestate-driven visibility map and FSM steps (just like
* the two-pass strategy).
*/
- Assert(dead_items->num_items == 0);
+ Assert(tidstore_num_tids(dead_items) == 0);
+ }
+ else if (prunestate.num_offsets > 0)
+ {
+ /* Save details of the LP_DEAD items from the page in dead_items */
+ tidstore_add_tids(dead_items, blkno, prunestate.deadoffsets,
+ prunestate.num_offsets);
+
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
+ tidstore_memory_usage(dead_items));
}
/*
@@ -1143,7 +1154,7 @@ lazy_scan_heap(LVRelState *vacrel)
* There should never be LP_DEAD items on a page with PD_ALL_VISIBLE
* set, however.
*/
- else if (prunestate.has_lpdead_items && PageIsAllVisible(page))
+ else if ((prunestate.num_offsets > 0) && PageIsAllVisible(page))
{
elog(WARNING, "page containing LP_DEAD items is marked as all-visible in relation \"%s\" page %u",
vacrel->relname, blkno);
@@ -1191,7 +1202,7 @@ lazy_scan_heap(LVRelState *vacrel)
* Final steps for block: drop cleanup lock, record free space in the
* FSM
*/
- if (prunestate.has_lpdead_items && vacrel->do_index_vacuuming)
+ if ((prunestate.num_offsets > 0) && vacrel->do_index_vacuuming)
{
/*
* Wait until lazy_vacuum_heap_rel() to save free space. This
@@ -1247,7 +1258,7 @@ lazy_scan_heap(LVRelState *vacrel)
* Do index vacuuming (call each index's ambulkdelete routine), then do
* related heap vacuuming
*/
- if (dead_items->num_items > 0)
+ if (tidstore_num_tids(dead_items) > 0)
lazy_vacuum(vacrel);
/*
@@ -1522,9 +1533,9 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* The approach we take now is to restart pruning when the race condition is
* detected. This allows heap_page_prune() to prune the tuples inserted by
* the now-aborted transaction. This is a little crude, but it guarantees
- * that any items that make it into the dead_items array are simple LP_DEAD
- * line pointers, and that every remaining item with tuple storage is
- * considered as a candidate for freezing.
+ * that any items that make it into the dead_items are simple LP_DEAD line
+ * pointers, and that every remaining item with tuple storage is considered
+ * as a candidate for freezing.
*/
static void
lazy_scan_prune(LVRelState *vacrel,
@@ -1541,13 +1552,11 @@ lazy_scan_prune(LVRelState *vacrel,
HTSV_Result res;
int tuples_deleted,
tuples_frozen,
- lpdead_items,
live_tuples,
recently_dead_tuples;
int nnewlpdead;
HeapPageFreeze pagefrz;
int64 fpi_before = pgWalUsage.wal_fpi;
- OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
@@ -1569,7 +1578,6 @@ retry:
pagefrz.NoFreezePageRelminMxid = vacrel->NewRelminMxid;
tuples_deleted = 0;
tuples_frozen = 0;
- lpdead_items = 0;
live_tuples = 0;
recently_dead_tuples = 0;
@@ -1578,9 +1586,9 @@ retry:
*
* We count tuples removed by the pruning step as tuples_deleted. Its
* final value can be thought of as the number of tuples that have been
- * deleted from the table. It should not be confused with lpdead_items;
- * lpdead_items's final value can be thought of as the number of tuples
- * that were deleted from indexes.
+ * deleted from the table. It should not be confused with
+ * prunestate->deadoffsets; prunestate->deadoffsets's final value can
+ * be thought of as the number of tuples that were deleted from indexes.
*/
tuples_deleted = heap_page_prune(rel, buf, vacrel->vistest,
InvalidTransactionId, 0, &nnewlpdead,
@@ -1591,7 +1599,7 @@ retry:
* requiring freezing among remaining tuples with storage
*/
prunestate->hastup = false;
- prunestate->has_lpdead_items = false;
+ prunestate->num_offsets = 0;
prunestate->all_visible = true;
prunestate->all_frozen = true;
prunestate->visibility_cutoff_xid = InvalidTransactionId;
@@ -1636,7 +1644,7 @@ retry:
* (This is another case where it's useful to anticipate that any
* LP_DEAD items will become LP_UNUSED during the ongoing VACUUM.)
*/
- deadoffsets[lpdead_items++] = offnum;
+ prunestate->deadoffsets[prunestate->num_offsets++] = offnum;
continue;
}
@@ -1873,7 +1881,7 @@ retry:
*/
#ifdef USE_ASSERT_CHECKING
/* Note that all_frozen value does not matter when !all_visible */
- if (prunestate->all_visible && lpdead_items == 0)
+ if (prunestate->all_visible && prunestate->num_offsets == 0)
{
TransactionId cutoff;
bool all_frozen;
@@ -1886,28 +1894,9 @@ retry:
}
#endif
- /*
- * Now save details of the LP_DEAD items from the page in vacrel
- */
- if (lpdead_items > 0)
+ if (prunestate->num_offsets > 0)
{
- VacDeadItems *dead_items = vacrel->dead_items;
- ItemPointerData tmp;
-
vacrel->lpdead_item_pages++;
- prunestate->has_lpdead_items = true;
-
- ItemPointerSetBlockNumber(&tmp, blkno);
-
- for (int i = 0; i < lpdead_items; i++)
- {
- ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
- dead_items->items[dead_items->num_items++] = tmp;
- }
-
- Assert(dead_items->num_items <= dead_items->max_items);
- pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
- dead_items->num_items);
/*
* It was convenient to ignore LP_DEAD items in all_visible earlier on
@@ -1926,7 +1915,7 @@ retry:
/* Finally, add page-local counts to whole-VACUUM counts */
vacrel->tuples_deleted += tuples_deleted;
vacrel->tuples_frozen += tuples_frozen;
- vacrel->lpdead_items += lpdead_items;
+ vacrel->lpdead_items += prunestate->num_offsets;
vacrel->live_tuples += live_tuples;
vacrel->recently_dead_tuples += recently_dead_tuples;
}
@@ -1938,7 +1927,7 @@ retry:
* lazy_scan_prune, which requires a full cleanup lock. While pruning isn't
* performed here, it's quite possible that an earlier opportunistic pruning
* operation left LP_DEAD items behind. We'll at least collect any such items
- * in the dead_items array for removal from indexes.
+ * in the dead_items for removal from indexes.
*
* For aggressive VACUUM callers, we may return false to indicate that a full
* cleanup lock is required for processing by lazy_scan_prune. This is only
@@ -2097,7 +2086,7 @@ lazy_scan_noprune(LVRelState *vacrel,
vacrel->NewRelfrozenXid = NoFreezePageRelfrozenXid;
vacrel->NewRelminMxid = NoFreezePageRelminMxid;
- /* Save any LP_DEAD items found on the page in dead_items array */
+ /* Save any LP_DEAD items found on the page in dead_items */
if (vacrel->nindexes == 0)
{
/* Using one-pass strategy (since table has no indexes) */
@@ -2127,8 +2116,7 @@ lazy_scan_noprune(LVRelState *vacrel,
}
else
{
- VacDeadItems *dead_items = vacrel->dead_items;
- ItemPointerData tmp;
+ TidStore *dead_items = vacrel->dead_items;
/*
* Page has LP_DEAD items, and so any references/TIDs that remain in
@@ -2137,17 +2125,10 @@ lazy_scan_noprune(LVRelState *vacrel,
*/
vacrel->lpdead_item_pages++;
- ItemPointerSetBlockNumber(&tmp, blkno);
+ tidstore_add_tids(dead_items, blkno, deadoffsets, lpdead_items);
- for (int i = 0; i < lpdead_items; i++)
- {
- ItemPointerSetOffsetNumber(&tmp, deadoffsets[i]);
- dead_items->items[dead_items->num_items++] = tmp;
- }
-
- Assert(dead_items->num_items <= dead_items->max_items);
- pgstat_progress_update_param(PROGRESS_VACUUM_NUM_DEAD_TUPLES,
- dead_items->num_items);
+ pgstat_progress_update_param(PROGRESS_VACUUM_DEAD_TUPLE_BYTES,
+ tidstore_memory_usage(dead_items));
vacrel->lpdead_items += lpdead_items;
@@ -2196,7 +2177,7 @@ lazy_vacuum(LVRelState *vacrel)
if (!vacrel->do_index_vacuuming)
{
Assert(!vacrel->do_index_cleanup);
- vacrel->dead_items->num_items = 0;
+ tidstore_reset(vacrel->dead_items);
return;
}
@@ -2225,7 +2206,7 @@ lazy_vacuum(LVRelState *vacrel)
BlockNumber threshold;
Assert(vacrel->num_index_scans == 0);
- Assert(vacrel->lpdead_items == vacrel->dead_items->num_items);
+ Assert(vacrel->lpdead_items == tidstore_num_tids(vacrel->dead_items));
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2252,8 +2233,8 @@ lazy_vacuum(LVRelState *vacrel)
* cases then this may need to be reconsidered.
*/
threshold = (double) vacrel->rel_pages * BYPASS_THRESHOLD_PAGES;
- bypass = (vacrel->lpdead_item_pages < threshold &&
- vacrel->lpdead_items < MAXDEADITEMS(32L * 1024L * 1024L));
+ bypass = (vacrel->lpdead_item_pages < threshold) &&
+ tidstore_memory_usage(vacrel->dead_items) < (32L * 1024L * 1024L);
}
if (bypass)
@@ -2298,7 +2279,7 @@ lazy_vacuum(LVRelState *vacrel)
* Forget the LP_DEAD items that we just vacuumed (or just decided to not
* vacuum)
*/
- vacrel->dead_items->num_items = 0;
+ tidstore_reset(vacrel->dead_items);
}
/*
@@ -2371,7 +2352,7 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
* place).
*/
Assert(vacrel->num_index_scans > 0 ||
- vacrel->dead_items->num_items == vacrel->lpdead_items);
+ tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items);
Assert(allindexes || VacuumFailsafeActive);
/*
@@ -2390,9 +2371,8 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
/*
* lazy_vacuum_heap_rel() -- second pass over the heap for two pass strategy
*
- * This routine marks LP_DEAD items in vacrel->dead_items array as LP_UNUSED.
- * Pages that never had lazy_scan_prune record LP_DEAD items are not visited
- * at all.
+ * This routine marks LP_DEAD items in vacrel->dead_items as LP_UNUSED. Pages
+ * that never had lazy_scan_prune record LP_DEAD items are not visited at all.
*
* We may also be able to truncate the line pointer array of the heap pages we
* visit. If there is a contiguous group of LP_UNUSED items at the end of the
@@ -2408,10 +2388,11 @@ lazy_vacuum_all_indexes(LVRelState *vacrel)
static void
lazy_vacuum_heap_rel(LVRelState *vacrel)
{
- int index = 0;
BlockNumber vacuumed_pages = 0;
Buffer vmbuffer = InvalidBuffer;
LVSavedErrInfo saved_err_info;
+ TidStoreIter *iter;
+ TidStoreIterResult *result;
Assert(vacrel->do_index_vacuuming);
Assert(vacrel->do_index_cleanup);
@@ -2426,7 +2407,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
VACUUM_ERRCB_PHASE_VACUUM_HEAP,
InvalidBlockNumber, InvalidOffsetNumber);
- while (index < vacrel->dead_items->num_items)
+ iter = tidstore_begin_iterate(vacrel->dead_items);
+ while ((result = tidstore_iterate_next(iter)) != NULL)
{
BlockNumber blkno;
Buffer buf;
@@ -2435,7 +2417,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
vacuum_delay_point();
- blkno = ItemPointerGetBlockNumber(&vacrel->dead_items->items[index]);
+ blkno = result->blkno;
vacrel->blkno = blkno;
/*
@@ -2449,7 +2431,8 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
buf = ReadBufferExtended(vacrel->rel, MAIN_FORKNUM, blkno, RBM_NORMAL,
vacrel->bstrategy);
LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE);
- index = lazy_vacuum_heap_page(vacrel, blkno, buf, index, vmbuffer);
+ lazy_vacuum_heap_page(vacrel, blkno, result->offsets, result->num_offsets,
+ buf, vmbuffer);
/* Now that we've vacuumed the page, record its available space */
page = BufferGetPage(buf);
@@ -2459,6 +2442,7 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
RecordPageWithFreeSpace(vacrel->rel, blkno, freespace);
vacuumed_pages++;
}
+ tidstore_end_iterate(iter);
vacrel->blkno = InvalidBlockNumber;
if (BufferIsValid(vmbuffer))
@@ -2468,36 +2452,31 @@ lazy_vacuum_heap_rel(LVRelState *vacrel)
* We set all LP_DEAD items from the first heap pass to LP_UNUSED during
* the second heap pass. No more, no less.
*/
- Assert(index > 0);
Assert(vacrel->num_index_scans > 1 ||
- (index == vacrel->lpdead_items &&
+ (tidstore_num_tids(vacrel->dead_items) == vacrel->lpdead_items &&
vacuumed_pages == vacrel->lpdead_item_pages));
ereport(DEBUG2,
- (errmsg("table \"%s\": removed %lld dead item identifiers in %u pages",
- vacrel->relname, (long long) index, vacuumed_pages)));
+ (errmsg("table \"%s\": removed " UINT64_FORMAT "dead item identifiers in %u pages",
+ vacrel->relname, tidstore_num_tids(vacrel->dead_items),
+ vacuumed_pages)));
/* Revert to the previous phase information for error traceback */
restore_vacuum_error_info(vacrel, &saved_err_info);
}
/*
- * lazy_vacuum_heap_page() -- free page's LP_DEAD items listed in the
- * vacrel->dead_items array.
+ * lazy_vacuum_heap_page() -- free page's LP_DEAD items.
*
* Caller must have an exclusive buffer lock on the buffer (though a full
* cleanup lock is also acceptable). vmbuffer must be valid and already have
* a pin on blkno's visibility map page.
- *
- * index is an offset into the vacrel->dead_items array for the first listed
- * LP_DEAD item on the page. The return value is the first index immediately
- * after all LP_DEAD items for the same page in the array.
*/
-static int
-lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
- int index, Buffer vmbuffer)
+static void
+lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno,
+ OffsetNumber *deadoffsets, int num_offsets, Buffer buffer,
+ Buffer vmbuffer)
{
- VacDeadItems *dead_items = vacrel->dead_items;
Page page = BufferGetPage(buffer);
OffsetNumber unused[MaxHeapTuplesPerPage];
int nunused = 0;
@@ -2516,16 +2495,11 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
START_CRIT_SECTION();
- for (; index < dead_items->num_items; index++)
+ for (int i = 0; i < num_offsets; i++)
{
- BlockNumber tblk;
- OffsetNumber toff;
ItemId itemid;
+ OffsetNumber toff = deadoffsets[i];
- tblk = ItemPointerGetBlockNumber(&dead_items->items[index]);
- if (tblk != blkno)
- break; /* past end of tuples for this block */
- toff = ItemPointerGetOffsetNumber(&dead_items->items[index]);
itemid = PageGetItemId(page, toff);
Assert(ItemIdIsDead(itemid) && !ItemIdHasStorage(itemid));
@@ -2595,7 +2569,6 @@ lazy_vacuum_heap_page(LVRelState *vacrel, BlockNumber blkno, Buffer buffer,
/* Revert to the previous phase information for error traceback */
restore_vacuum_error_info(vacrel, &saved_err_info);
- return index;
}
/*
@@ -2692,8 +2665,8 @@ lazy_cleanup_all_indexes(LVRelState *vacrel)
* lazy_vacuum_one_index() -- vacuum index relation.
*
* Delete all the index tuples containing a TID collected in
- * vacrel->dead_items array. Also update running statistics.
- * Exact details depend on index AM's ambulkdelete routine.
+ * vacrel->dead_items. Also update running statistics. Exact
+ * details depend on index AM's ambulkdelete routine.
*
* reltuples is the number of heap tuples to be passed to the
* bulkdelete callback. It's always assumed to be estimated.
@@ -3101,48 +3074,8 @@ count_nondeletable_pages(LVRelState *vacrel, bool *lock_waiter_detected)
}
/*
- * Returns the number of dead TIDs that VACUUM should allocate space to
- * store, given a heap rel of size vacrel->rel_pages, and given current
- * maintenance_work_mem setting (or current autovacuum_work_mem setting,
- * when applicable).
- *
- * See the comments at the head of this file for rationale.
- */
-static int
-dead_items_max_items(LVRelState *vacrel)
-{
- int64 max_items;
- int vac_work_mem = IsAutoVacuumWorkerProcess() &&
- autovacuum_work_mem != -1 ?
- autovacuum_work_mem : maintenance_work_mem;
-
- if (vacrel->nindexes > 0)
- {
- BlockNumber rel_pages = vacrel->rel_pages;
-
- max_items = MAXDEADITEMS(vac_work_mem * 1024L);
- max_items = Min(max_items, INT_MAX);
- max_items = Min(max_items, MAXDEADITEMS(MaxAllocSize));
-
- /* curious coding here to ensure the multiplication can't overflow */
- if ((BlockNumber) (max_items / MaxHeapTuplesPerPage) > rel_pages)
- max_items = rel_pages * MaxHeapTuplesPerPage;
-
- /* stay sane if small maintenance_work_mem */
- max_items = Max(max_items, MaxHeapTuplesPerPage);
- }
- else
- {
- /* One-pass case only stores a single heap page's TIDs at a time */
- max_items = MaxHeapTuplesPerPage;
- }
-
- return (int) max_items;
-}
-
-/*
- * Allocate dead_items (either using palloc, or in dynamic shared memory).
- * Sets dead_items in vacrel for caller.
+ * Allocate a (local or shared) TidStore for storing dead TIDs. Sets dead_items
+ * in vacrel for caller.
*
* Also handles parallel initialization as part of allocating dead_items in
* DSM when required.
@@ -3150,11 +3083,9 @@ dead_items_max_items(LVRelState *vacrel)
static void
dead_items_alloc(LVRelState *vacrel, int nworkers)
{
- VacDeadItems *dead_items;
- int max_items;
-
- max_items = dead_items_max_items(vacrel);
- Assert(max_items >= MaxHeapTuplesPerPage);
+ int vac_work_mem = IsAutoVacuumWorkerProcess() &&
+ autovacuum_work_mem != -1 ?
+ autovacuum_work_mem * 1024L : maintenance_work_mem * 1024L;
/*
* Initialize state for a parallel vacuum. As of now, only one worker can
@@ -3181,7 +3112,7 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
else
vacrel->pvs = parallel_vacuum_init(vacrel->rel, vacrel->indrels,
vacrel->nindexes, nworkers,
- max_items,
+ vac_work_mem, MaxHeapTuplesPerPage,
vacrel->verbose ? INFO : DEBUG2,
vacrel->bstrategy);
@@ -3194,11 +3125,8 @@ dead_items_alloc(LVRelState *vacrel, int nworkers)
}
/* Serial VACUUM case */
- dead_items = (VacDeadItems *) palloc(vac_max_items_to_alloc_size(max_items));
- dead_items->max_items = max_items;
- dead_items->num_items = 0;
-
- vacrel->dead_items = dead_items;
+ vacrel->dead_items = tidstore_create(vac_work_mem, MaxHeapTuplesPerPage,
+ NULL);
}
/*
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 2129c916aa..134df925ce 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -1190,7 +1190,7 @@ CREATE VIEW pg_stat_progress_vacuum AS
END AS phase,
S.param2 AS heap_blks_total, S.param3 AS heap_blks_scanned,
S.param4 AS heap_blks_vacuumed, S.param5 AS index_vacuum_count,
- S.param6 AS max_dead_tuples, S.param7 AS num_dead_tuples
+ S.param6 AS max_dead_tuple_bytes, S.param7 AS dead_tuple_bytes
FROM pg_stat_get_progress_info('VACUUM') AS S
LEFT JOIN pg_database D ON S.datid = D.oid;
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index a843f9ad92..f3922b72dc 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -119,7 +119,6 @@ static bool vacuum_rel(Oid relid, RangeVar *relation, VacuumParams *params,
static double compute_parallel_delay(void);
static VacOptValue get_vacoptval_from_boolean(DefElem *def);
static bool vac_tid_reaped(ItemPointer itemptr, void *state);
-static int vac_cmp_itemptr(const void *left, const void *right);
/*
* GUC check function to ensure GUC value specified is within the allowable
@@ -2478,16 +2477,16 @@ get_vacoptval_from_boolean(DefElem *def)
*/
IndexBulkDeleteResult *
vac_bulkdel_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat,
- VacDeadItems *dead_items)
+ TidStore *dead_items)
{
/* Do bulk deletion */
istat = index_bulk_delete(ivinfo, istat, vac_tid_reaped,
(void *) dead_items);
ereport(ivinfo->message_level,
- (errmsg("scanned index \"%s\" to remove %d row versions",
+ (errmsg("scanned index \"%s\" to remove " UINT64_FORMAT " row versions",
RelationGetRelationName(ivinfo->index),
- dead_items->num_items)));
+ tidstore_num_tids(dead_items))));
return istat;
}
@@ -2518,82 +2517,15 @@ vac_cleanup_one_index(IndexVacuumInfo *ivinfo, IndexBulkDeleteResult *istat)
return istat;
}
-/*
- * Returns the total required space for VACUUM's dead_items array given a
- * max_items value.
- */
-Size
-vac_max_items_to_alloc_size(int max_items)
-{
- Assert(max_items <= MAXDEADITEMS(MaxAllocSize));
-
- return offsetof(VacDeadItems, items) + sizeof(ItemPointerData) * max_items;
-}
-
/*
* vac_tid_reaped() -- is a particular tid deletable?
*
* This has the right signature to be an IndexBulkDeleteCallback.
- *
- * Assumes dead_items array is sorted (in ascending TID order).
*/
static bool
vac_tid_reaped(ItemPointer itemptr, void *state)
{
- VacDeadItems *dead_items = (VacDeadItems *) state;
- int64 litem,
- ritem,
- item;
- ItemPointer res;
-
- litem = itemptr_encode(&dead_items->items[0]);
- ritem = itemptr_encode(&dead_items->items[dead_items->num_items - 1]);
- item = itemptr_encode(itemptr);
-
- /*
- * Doing a simple bound check before bsearch() is useful to avoid the
- * extra cost of bsearch(), especially if dead items on the heap are
- * concentrated in a certain range. Since this function is called for
- * every index tuple, it pays to be really fast.
- */
- if (item < litem || item > ritem)
- return false;
-
- res = (ItemPointer) bsearch(itemptr,
- dead_items->items,
- dead_items->num_items,
- sizeof(ItemPointerData),
- vac_cmp_itemptr);
-
- return (res != NULL);
-}
-
-/*
- * Comparator routines for use with qsort() and bsearch().
- */
-static int
-vac_cmp_itemptr(const void *left, const void *right)
-{
- BlockNumber lblk,
- rblk;
- OffsetNumber loff,
- roff;
-
- lblk = ItemPointerGetBlockNumber((ItemPointer) left);
- rblk = ItemPointerGetBlockNumber((ItemPointer) right);
-
- if (lblk < rblk)
- return -1;
- if (lblk > rblk)
- return 1;
-
- loff = ItemPointerGetOffsetNumber((ItemPointer) left);
- roff = ItemPointerGetOffsetNumber((ItemPointer) right);
-
- if (loff < roff)
- return -1;
- if (loff > roff)
- return 1;
+ TidStore *dead_items = (TidStore *) state;
- return 0;
+ return tidstore_lookup_tid(dead_items, itemptr);
}
diff --git a/src/backend/commands/vacuumparallel.c b/src/backend/commands/vacuumparallel.c
index 87ea5c5242..c363f45e32 100644
--- a/src/backend/commands/vacuumparallel.c
+++ b/src/backend/commands/vacuumparallel.c
@@ -9,12 +9,11 @@
* In a parallel vacuum, we perform both index bulk deletion and index cleanup
* with parallel worker processes. Individual indexes are processed by one
* vacuum process. ParalleVacuumState contains shared information as well as
- * the memory space for storing dead items allocated in the DSM segment. We
- * launch parallel worker processes at the start of parallel index
- * bulk-deletion and index cleanup and once all indexes are processed, the
- * parallel worker processes exit. Each time we process indexes in parallel,
- * the parallel context is re-initialized so that the same DSM can be used for
- * multiple passes of index bulk-deletion and index cleanup.
+ * the shared TidStore. We launch parallel worker processes at the start of
+ * parallel index bulk-deletion and index cleanup and once all indexes are
+ * processed, the parallel worker processes exit. Each time we process indexes
+ * in parallel, the parallel context is re-initialized so that the same DSM can
+ * be used for multiple passes of index bulk-deletion and index cleanup.
*
* Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
@@ -109,6 +108,9 @@ typedef struct PVShared
/* Counter for vacuuming and cleanup */
pg_atomic_uint32 idx;
+
+ /* Handle of the shared TidStore */
+ tidstore_handle dead_items_handle;
} PVShared;
/* Status used during parallel index vacuum or cleanup */
@@ -175,7 +177,8 @@ struct ParallelVacuumState
PVIndStats *indstats;
/* Shared dead items space among parallel vacuum workers */
- VacDeadItems *dead_items;
+ TidStore *dead_items;
+ dsa_area *dead_items_area;
/* Points to buffer usage area in DSM */
BufferUsage *buffer_usage;
@@ -231,20 +234,23 @@ static void parallel_vacuum_error_callback(void *arg);
*/
ParallelVacuumState *
parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
- int nrequested_workers, int max_items,
- int elevel, BufferAccessStrategy bstrategy)
+ int nrequested_workers, int vac_work_mem,
+ int max_offset, int elevel,
+ BufferAccessStrategy bstrategy)
{
ParallelVacuumState *pvs;
ParallelContext *pcxt;
PVShared *shared;
- VacDeadItems *dead_items;
+ TidStore *dead_items;
PVIndStats *indstats;
BufferUsage *buffer_usage;
WalUsage *wal_usage;
+ void *area_space;
+ dsa_area *dead_items_dsa;
bool *will_parallel_vacuum;
Size est_indstats_len;
Size est_shared_len;
- Size est_dead_items_len;
+ Size dsa_minsize = dsa_minimum_size();
int nindexes_mwm = 0;
int parallel_workers = 0;
int querylen;
@@ -293,9 +299,8 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_estimate_chunk(&pcxt->estimator, est_shared_len);
shm_toc_estimate_keys(&pcxt->estimator, 1);
- /* Estimate size for dead_items -- PARALLEL_VACUUM_KEY_DEAD_ITEMS */
- est_dead_items_len = vac_max_items_to_alloc_size(max_items);
- shm_toc_estimate_chunk(&pcxt->estimator, est_dead_items_len);
+ /* Estimate size for dead tuple DSA -- PARALLEL_VACUUM_KEY_DSA */
+ shm_toc_estimate_chunk(&pcxt->estimator, dsa_minsize);
shm_toc_estimate_keys(&pcxt->estimator, 1);
/*
@@ -361,6 +366,16 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_INDEX_STATS, indstats);
pvs->indstats = indstats;
+ /* Prepare DSA space for dead items */
+ area_space = shm_toc_allocate(pcxt->toc, dsa_minsize);
+ shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, area_space);
+ dead_items_dsa = dsa_create_in_place(area_space, dsa_minsize,
+ LWTRANCHE_PARALLEL_VACUUM_DSA,
+ pcxt->seg);
+ dead_items = tidstore_create(vac_work_mem, max_offset, dead_items_dsa);
+ pvs->dead_items = dead_items;
+ pvs->dead_items_area = dead_items_dsa;
+
/* Prepare shared information */
shared = (PVShared *) shm_toc_allocate(pcxt->toc, est_shared_len);
MemSet(shared, 0, est_shared_len);
@@ -370,6 +385,7 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
(nindexes_mwm > 0) ?
maintenance_work_mem / Min(parallel_workers, nindexes_mwm) :
maintenance_work_mem;
+ shared->dead_items_handle = tidstore_get_handle(dead_items);
/* Use the same buffer size for all workers */
shared->ring_nbuffers = GetAccessStrategyBufferCount(bstrategy);
@@ -381,15 +397,6 @@ parallel_vacuum_init(Relation rel, Relation *indrels, int nindexes,
shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_SHARED, shared);
pvs->shared = shared;
- /* Prepare the dead_items space */
- dead_items = (VacDeadItems *) shm_toc_allocate(pcxt->toc,
- est_dead_items_len);
- dead_items->max_items = max_items;
- dead_items->num_items = 0;
- MemSet(dead_items->items, 0, sizeof(ItemPointerData) * max_items);
- shm_toc_insert(pcxt->toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, dead_items);
- pvs->dead_items = dead_items;
-
/*
* Allocate space for each worker's BufferUsage and WalUsage; no need to
* initialize
@@ -447,6 +454,9 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
istats[i] = NULL;
}
+ tidstore_destroy(pvs->dead_items);
+ dsa_detach(pvs->dead_items_area);
+
DestroyParallelContext(pvs->pcxt);
ExitParallelMode();
@@ -455,7 +465,7 @@ parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats)
}
/* Returns the dead items space */
-VacDeadItems *
+TidStore *
parallel_vacuum_get_dead_items(ParallelVacuumState *pvs)
{
return pvs->dead_items;
@@ -954,7 +964,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
Relation *indrels;
PVIndStats *indstats;
PVShared *shared;
- VacDeadItems *dead_items;
+ TidStore *dead_items;
+ void *area_space;
+ dsa_area *dead_items_area;
BufferUsage *buffer_usage;
WalUsage *wal_usage;
int nindexes;
@@ -998,10 +1010,10 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
PARALLEL_VACUUM_KEY_INDEX_STATS,
false);
- /* Set dead_items space */
- dead_items = (VacDeadItems *) shm_toc_lookup(toc,
- PARALLEL_VACUUM_KEY_DEAD_ITEMS,
- false);
+ /* Set dead items */
+ area_space = shm_toc_lookup(toc, PARALLEL_VACUUM_KEY_DEAD_ITEMS, false);
+ dead_items_area = dsa_attach_in_place(area_space, seg);
+ dead_items = tidstore_attach(dead_items_area, shared->dead_items_handle);
/* Set cost-based vacuum delay */
VacuumUpdateCosts();
@@ -1049,6 +1061,9 @@ parallel_vacuum_main(dsm_segment *seg, shm_toc *toc)
InstrEndParallelQuery(&buffer_usage[ParallelWorkerNumber],
&wal_usage[ParallelWorkerNumber]);
+ tidstore_detach(pvs.dead_items);
+ dsa_detach(dead_items_area);
+
/* Pop the error context stack */
error_context_stack = errcallback.previous;
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 53c8f8d79c..74915bee9b 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -3474,12 +3474,12 @@ check_autovacuum_work_mem(int *newval, void **extra, GucSource source)
return true;
/*
- * We clamp manually-set values to at least 1MB. Since
+ * We clamp manually-set values to at least 2MB. Since
* maintenance_work_mem is always set to at least this value, do the same
* here.
*/
- if (*newval < 1024)
- *newval = 1024;
+ if (*newval < 2048)
+ *newval = 2048;
return true;
}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 55b3a04097..c223a7dc94 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -192,6 +192,8 @@ static const char *const BuiltinTrancheNames[] = {
"LogicalRepLauncherDSA",
/* LWTRANCHE_LAUNCHER_HASH: */
"LogicalRepLauncherHash",
+ /* LWTRANCHE_PARALLEL_VACUUM_DSA: */
+ "ParallelVacuumDSA",
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index cab3ddbe11..0bbdf04980 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2353,7 +2353,7 @@ struct config_int ConfigureNamesInt[] =
GUC_UNIT_KB
},
&maintenance_work_mem,
- 65536, 1024, MAX_KILOBYTES,
+ 65536, 2048, MAX_KILOBYTES,
NULL, NULL, NULL
},
diff --git a/src/include/commands/progress.h b/src/include/commands/progress.h
index e5add41352..b209d3cf84 100644
--- a/src/include/commands/progress.h
+++ b/src/include/commands/progress.h
@@ -23,8 +23,8 @@
#define PROGRESS_VACUUM_HEAP_BLKS_SCANNED 2
#define PROGRESS_VACUUM_HEAP_BLKS_VACUUMED 3
#define PROGRESS_VACUUM_NUM_INDEX_VACUUMS 4
-#define PROGRESS_VACUUM_MAX_DEAD_TUPLES 5
-#define PROGRESS_VACUUM_NUM_DEAD_TUPLES 6
+#define PROGRESS_VACUUM_MAX_DEAD_TUPLE_BYTES 5
+#define PROGRESS_VACUUM_DEAD_TUPLE_BYTES 6
/* Phases of vacuum (as advertised via PROGRESS_VACUUM_PHASE) */
#define PROGRESS_VACUUM_PHASE_SCAN_HEAP 1
diff --git a/src/include/commands/vacuum.h b/src/include/commands/vacuum.h
index 17e9b4f68e..b48c6ebf2d 100644
--- a/src/include/commands/vacuum.h
+++ b/src/include/commands/vacuum.h
@@ -17,6 +17,7 @@
#include "access/htup.h"
#include "access/genam.h"
#include "access/parallel.h"
+#include "access/tidstore.h"
#include "catalog/pg_class.h"
#include "catalog/pg_statistic.h"
#include "catalog/pg_type.h"
@@ -280,21 +281,6 @@ struct VacuumCutoffs
MultiXactId MultiXactCutoff;
};
-/*
- * VacDeadItems stores TIDs whose index tuples are deleted by index vacuuming.
- */
-typedef struct VacDeadItems
-{
- int max_items; /* # slots allocated in array */
- int num_items; /* current # of entries */
-
- /* Sorted array of TIDs to delete from indexes */
- ItemPointerData items[FLEXIBLE_ARRAY_MEMBER];
-} VacDeadItems;
-
-#define MAXDEADITEMS(avail_mem) \
- (((avail_mem) - offsetof(VacDeadItems, items)) / sizeof(ItemPointerData))
-
/* GUC parameters */
extern PGDLLIMPORT int default_statistics_target; /* PGDLLIMPORT for PostGIS */
extern PGDLLIMPORT int vacuum_freeze_min_age;
@@ -347,10 +333,9 @@ extern Relation vacuum_open_relation(Oid relid, RangeVar *relation,
LOCKMODE lmode);
extern IndexBulkDeleteResult *vac_bulkdel_one_index(IndexVacuumInfo *ivinfo,
IndexBulkDeleteResult *istat,
- VacDeadItems *dead_items);
+ TidStore *dead_items);
extern IndexBulkDeleteResult *vac_cleanup_one_index(IndexVacuumInfo *ivinfo,
IndexBulkDeleteResult *istat);
-extern Size vac_max_items_to_alloc_size(int max_items);
/* In postmaster/autovacuum.c */
extern void AutoVacuumUpdateCostLimit(void);
@@ -359,10 +344,10 @@ extern void VacuumUpdateCosts(void);
/* in commands/vacuumparallel.c */
extern ParallelVacuumState *parallel_vacuum_init(Relation rel, Relation *indrels,
int nindexes, int nrequested_workers,
- int max_items, int elevel,
- BufferAccessStrategy bstrategy);
+ int vac_work_mem, int max_offset,
+ int elevel, BufferAccessStrategy bstrategy);
extern void parallel_vacuum_end(ParallelVacuumState *pvs, IndexBulkDeleteResult **istats);
-extern VacDeadItems *parallel_vacuum_get_dead_items(ParallelVacuumState *pvs);
+extern TidStore *parallel_vacuum_get_dead_items(ParallelVacuumState *pvs);
extern void parallel_vacuum_bulkdel_all_indexes(ParallelVacuumState *pvs,
long num_table_tuples,
int num_index_scans);
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index 07002fdfbe..537b34b30c 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -207,6 +207,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_PGSTATS_DATA,
LWTRANCHE_LAUNCHER_DSA,
LWTRANCHE_LAUNCHER_HASH,
+ LWTRANCHE_PARALLEL_VACUUM_DSA,
LWTRANCHE_FIRST_USER_DEFINED
} BuiltinTrancheIds;
diff --git a/src/test/regress/expected/cluster.out b/src/test/regress/expected/cluster.out
index 2eec483eaa..e04f50726f 100644
--- a/src/test/regress/expected/cluster.out
+++ b/src/test/regress/expected/cluster.out
@@ -526,7 +526,7 @@ create index cluster_sort on clstr_4 (hundred, thousand, tenthous);
-- ensure we don't use the index in CLUSTER nor the checking SELECTs
set enable_indexscan = off;
-- Use external sort:
-set maintenance_work_mem = '1MB';
+set maintenance_work_mem = '2MB';
cluster clstr_4 using cluster_sort;
select * from
(select hundred, lag(hundred) over () as lhundred,
diff --git a/src/test/regress/expected/create_index.out b/src/test/regress/expected/create_index.out
index acfd9d1f4f..d320ad87dd 100644
--- a/src/test/regress/expected/create_index.out
+++ b/src/test/regress/expected/create_index.out
@@ -1214,7 +1214,7 @@ DROP TABLE unlogged_hash_table;
-- CREATE INDEX hash_ovfl_index ON hash_ovfl_heap USING hash (x int4_ops);
-- Test hash index build tuplesorting. Force hash tuplesort using low
-- maintenance_work_mem setting and fillfactor:
-SET maintenance_work_mem = '1MB';
+SET maintenance_work_mem = '2MB';
CREATE INDEX hash_tuplesort_idx ON tenk1 USING hash (stringu1 name_ops) WITH (fillfactor = 10);
EXPLAIN (COSTS OFF)
SELECT count(*) FROM tenk1 WHERE stringu1 = 'TVAAAA';
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 919d947ec0..66d671a641 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -2041,8 +2041,8 @@ pg_stat_progress_vacuum| SELECT s.pid,
s.param3 AS heap_blks_scanned,
s.param4 AS heap_blks_vacuumed,
s.param5 AS index_vacuum_count,
- s.param6 AS max_dead_tuples,
- s.param7 AS num_dead_tuples
+ s.param6 AS max_dead_tuple_bytes,
+ s.param7 AS dead_tuple_bytes
FROM (pg_stat_get_progress_info('VACUUM'::text) s(pid, datid, relid, param1, param2, param3, param4, param5, param6, param7, param8, param9, param10, param11, param12, param13, param14, param15, param16, param17, param18, param19, param20)
LEFT JOIN pg_database d ON ((s.datid = d.oid)));
pg_stat_recovery_prefetch| SELECT stats_reset,
diff --git a/src/test/regress/sql/cluster.sql b/src/test/regress/sql/cluster.sql
index a4cfaae807..a4cb5b98a5 100644
--- a/src/test/regress/sql/cluster.sql
+++ b/src/test/regress/sql/cluster.sql
@@ -258,7 +258,7 @@ create index cluster_sort on clstr_4 (hundred, thousand, tenthous);
set enable_indexscan = off;
-- Use external sort:
-set maintenance_work_mem = '1MB';
+set maintenance_work_mem = '2MB';
cluster clstr_4 using cluster_sort;
select * from
(select hundred, lag(hundred) over () as lhundred,
diff --git a/src/test/regress/sql/create_index.sql b/src/test/regress/sql/create_index.sql
index d49ce9f300..d6e2471b00 100644
--- a/src/test/regress/sql/create_index.sql
+++ b/src/test/regress/sql/create_index.sql
@@ -367,7 +367,7 @@ DROP TABLE unlogged_hash_table;
-- Test hash index build tuplesorting. Force hash tuplesort using low
-- maintenance_work_mem setting and fillfactor:
-SET maintenance_work_mem = '1MB';
+SET maintenance_work_mem = '2MB';
CREATE INDEX hash_tuplesort_idx ON tenk1 USING hash (stringu1 name_ops) WITH (fillfactor = 10);
EXPLAIN (COSTS OFF)
SELECT count(*) FROM tenk1 WHERE stringu1 = 'TVAAAA';
--
2.31.1
[application/octet-stream] v32-0005-Tool-for-measuring-radix-tree-and-tidstore-perfo.patch (24.7K, ../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/15-v32-0005-Tool-for-measuring-radix-tree-and-tidstore-perfo.patch)
download | inline diff:
From cff1ffa9af592765cf9073291fb1665b09b61d8a Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 16 Sep 2022 11:57:03 +0900
Subject: [PATCH v32 05/18] Tool for measuring radix tree and tidstore
performance
Includes Meson support, but commented out to avoid warnings
XXX: Not for commit
---
contrib/bench_radix_tree/Makefile | 21 +
.../bench_radix_tree--1.0.sql | 88 +++
contrib/bench_radix_tree/bench_radix_tree.c | 747 ++++++++++++++++++
.../bench_radix_tree/bench_radix_tree.control | 6 +
contrib/bench_radix_tree/expected/bench.out | 13 +
contrib/bench_radix_tree/meson.build | 33 +
contrib/bench_radix_tree/sql/bench.sql | 16 +
contrib/meson.build | 1 +
8 files changed, 925 insertions(+)
create mode 100644 contrib/bench_radix_tree/Makefile
create mode 100644 contrib/bench_radix_tree/bench_radix_tree--1.0.sql
create mode 100644 contrib/bench_radix_tree/bench_radix_tree.c
create mode 100644 contrib/bench_radix_tree/bench_radix_tree.control
create mode 100644 contrib/bench_radix_tree/expected/bench.out
create mode 100644 contrib/bench_radix_tree/meson.build
create mode 100644 contrib/bench_radix_tree/sql/bench.sql
diff --git a/contrib/bench_radix_tree/Makefile b/contrib/bench_radix_tree/Makefile
new file mode 100644
index 0000000000..952bb0ceae
--- /dev/null
+++ b/contrib/bench_radix_tree/Makefile
@@ -0,0 +1,21 @@
+# contrib/bench_radix_tree/Makefile
+
+MODULE_big = bench_radix_tree
+OBJS = \
+ bench_radix_tree.o
+
+EXTENSION = bench_radix_tree
+DATA = bench_radix_tree--1.0.sql
+
+REGRESS = bench_fixed_height
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/bench_radix_tree
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/contrib/bench_radix_tree/bench_radix_tree--1.0.sql b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
new file mode 100644
index 0000000000..ad66265e23
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree--1.0.sql
@@ -0,0 +1,88 @@
+/* contrib/bench_radix_tree/bench_radix_tree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION bench_radix_tree" to load this file. \quit
+
+create function bench_shuffle_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_seq_search(
+minblk int4,
+maxblk int4,
+random_block bool DEFAULT false,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT array_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT array_load_ms int8,
+OUT rt_search_ms int8,
+OUT array_serach_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_load_random_int(
+cnt int8,
+OUT mem_allocated int8,
+OUT load_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_search_random_nodes(
+cnt int8,
+filter_str text DEFAULT NULL,
+OUT mem_allocated int8,
+OUT load_ms int8,
+OUT search_ms int8)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C VOLATILE PARALLEL UNSAFE;
+
+create function bench_fixed_height_search(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_load_ms int8,
+OUT rt_search_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_node128_load(
+fanout int4,
+OUT fanout int4,
+OUT nkeys int8,
+OUT rt_mem_allocated int8,
+OUT rt_sparseload_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
+
+create function bench_tidstore_load(
+minblk int4,
+maxblk int4,
+OUT mem_allocated int8,
+OUT load_ms int8,
+OUT iter_ms int8
+)
+returns record
+as 'MODULE_PATHNAME'
+LANGUAGE C STRICT VOLATILE PARALLEL UNSAFE;
diff --git a/contrib/bench_radix_tree/bench_radix_tree.c b/contrib/bench_radix_tree/bench_radix_tree.c
new file mode 100644
index 0000000000..6e5149e2c4
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.c
@@ -0,0 +1,747 @@
+/*-------------------------------------------------------------------------
+ *
+ * bench_radix_tree.c
+ *
+ * Copyright (c) 2016-2022, PostgreSQL Global Development Group
+ *
+ * contrib/bench_radix_tree/bench_radix_tree.c
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/tidstore.h"
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "funcapi.h"
+#include "lib/radixtree.h"
+#include <math.h>
+#include "miscadmin.h"
+#include "storage/lwlock.h"
+#include "utils/builtins.h"
+#include "utils/timestamp.h"
+
+PG_MODULE_MAGIC;
+
+/* run benchmark also for binary-search case? */
+/* #define MEASURE_BINARY_SEARCH 1 */
+
+#define TIDS_PER_BLOCK_FOR_LOAD 30
+#define TIDS_PER_BLOCK_FOR_LOOKUP 50
+
+//#define RT_DEBUG
+#define RT_PREFIX rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE uint64
+// WIP: compiles with warnings because rt_attach is defined but not used
+// #define RT_SHMEM
+#include "lib/radixtree.h"
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+ return tree->ctl->num_keys;
+}
+
+
+PG_FUNCTION_INFO_V1(bench_seq_search);
+PG_FUNCTION_INFO_V1(bench_shuffle_search);
+PG_FUNCTION_INFO_V1(bench_load_random_int);
+PG_FUNCTION_INFO_V1(bench_fixed_height_search);
+PG_FUNCTION_INFO_V1(bench_search_random_nodes);
+PG_FUNCTION_INFO_V1(bench_node128_load);
+PG_FUNCTION_INFO_V1(bench_tidstore_load);
+
+static uint64
+tid_to_key_off(ItemPointer tid, uint32 *off)
+{
+ uint64 upper;
+ uint32 shift = pg_ceil_log2_32(MaxHeapTuplesPerPage);
+ int64 tid_i;
+
+ Assert(ItemPointerGetOffsetNumber(tid) < MaxHeapTuplesPerPage);
+
+ tid_i = ItemPointerGetOffsetNumber(tid);
+ tid_i |= (uint64) ItemPointerGetBlockNumber(tid) << shift;
+
+ /* log(sizeof(uint64) * BITS_PER_BYTE, 2) = log(64, 2) = 6 */
+ *off = tid_i & ((1 << 6) - 1);
+ upper = tid_i >> 6;
+ Assert(*off < (sizeof(uint64) * BITS_PER_BYTE));
+
+ Assert(*off < 64);
+
+ return upper;
+}
+
+static int
+shuffle_randrange(pg_prng_state *state, int lower, int upper)
+{
+ return (int) floor(pg_prng_double(state) * ((upper - lower) + 0.999999)) + lower;
+}
+
+/* Naive Fisher-Yates implementation*/
+static void
+shuffle_itemptrs(ItemPointer itemptr, uint64 nitems)
+{
+ /* reproducability */
+ pg_prng_state state;
+
+ pg_prng_seed(&state, 0);
+
+ for (int i = 0; i < nitems - 1; i++)
+ {
+ int j = shuffle_randrange(&state, i, nitems - 1);
+ ItemPointerData t = itemptr[j];
+
+ itemptr[j] = itemptr[i];
+ itemptr[i] = t;
+ }
+}
+
+static ItemPointer
+generate_tids(BlockNumber minblk, BlockNumber maxblk, int ntids_per_blk, uint64 *ntids_p,
+ bool random_block)
+{
+ ItemPointer tids;
+ uint64 maxitems;
+ uint64 ntids = 0;
+ pg_prng_state state;
+
+ maxitems = (maxblk - minblk + 1) * ntids_per_blk;
+ tids = MemoryContextAllocHuge(TopTransactionContext,
+ sizeof(ItemPointerData) * maxitems);
+
+ if (random_block)
+ pg_prng_seed(&state, 0x9E3779B185EBCA87);
+
+ for (BlockNumber blk = minblk; blk < maxblk; blk++)
+ {
+ if (random_block && !pg_prng_bool(&state))
+ continue;
+
+ for (OffsetNumber off = FirstOffsetNumber;
+ off <= ntids_per_blk; off++)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ ItemPointerSetBlockNumber(&(tids[ntids]), blk);
+ ItemPointerSetOffsetNumber(&(tids[ntids]), off);
+
+ ntids++;
+ }
+ }
+
+ *ntids_p = ntids;
+ return tids;
+}
+
+#ifdef MEASURE_BINARY_SEARCH
+static int
+vac_cmp_itemptr(const void *left, const void *right)
+{
+ BlockNumber lblk,
+ rblk;
+ OffsetNumber loff,
+ roff;
+
+ lblk = ItemPointerGetBlockNumber((ItemPointer) left);
+ rblk = ItemPointerGetBlockNumber((ItemPointer) right);
+
+ if (lblk < rblk)
+ return -1;
+ if (lblk > rblk)
+ return 1;
+
+ loff = ItemPointerGetOffsetNumber((ItemPointer) left);
+ roff = ItemPointerGetOffsetNumber((ItemPointer) right);
+
+ if (loff < roff)
+ return -1;
+ if (loff > roff)
+ return 1;
+
+ return 0;
+}
+#endif
+
+Datum
+bench_tidstore_load(PG_FUNCTION_ARGS)
+{
+ BlockNumber minblk = PG_GETARG_INT32(0);
+ BlockNumber maxblk = PG_GETARG_INT32(1);
+ TidStore *ts;
+ TidStoreIter *iter;
+ TidStoreIterResult *result;
+ OffsetNumber *offs;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_ms;
+ int64 iter_ms;
+ TupleDesc tupdesc;
+ Datum values[3];
+ bool nulls[3] = {false};
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ offs = palloc(sizeof(OffsetNumber) * TIDS_PER_BLOCK_FOR_LOAD);
+ for (int i = 0; i < TIDS_PER_BLOCK_FOR_LOAD; i++)
+ offs[i] = i + 1; /* FirstOffsetNumber is 1 */
+
+ ts = tidstore_create(1 * 1024L * 1024L * 1024L, MaxHeapTuplesPerPage, NULL);
+
+ /* load tids */
+ start_time = GetCurrentTimestamp();
+ for (BlockNumber blkno = minblk; blkno < maxblk; blkno++)
+ tidstore_add_tids(ts, blkno, offs, TIDS_PER_BLOCK_FOR_LOAD);
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_ms = secs * 1000 + usecs / 1000;
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* iterate through tids */
+ iter = tidstore_begin_iterate(ts);
+ start_time = GetCurrentTimestamp();
+ while ((result = tidstore_iterate_next(iter)) != NULL)
+ ;
+ tidstore_end_iterate(iter);
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ iter_ms = secs * 1000 + usecs / 1000;
+
+ values[0] = Int64GetDatum(tidstore_memory_usage(ts));
+ values[1] = Int64GetDatum(load_ms);
+ values[2] = Int64GetDatum(iter_ms);
+
+ tidstore_destroy(ts);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+static Datum
+bench_search(FunctionCallInfo fcinfo, bool shuffle)
+{
+ BlockNumber minblk = PG_GETARG_INT32(0);
+ BlockNumber maxblk = PG_GETARG_INT32(1);
+ bool random_block = PG_GETARG_BOOL(2);
+ rt_radix_tree *rt = NULL;
+ uint64 ntids;
+ uint64 key;
+ uint64 last_key = PG_UINT64_MAX;
+ uint64 val = 0;
+ ItemPointer tids;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_load_ms,
+ rt_search_ms;
+ Datum values[7];
+ bool nulls[7];
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ tids = generate_tids(minblk, maxblk, TIDS_PER_BLOCK_FOR_LOAD, &ntids, random_block);
+
+ /* measure the load time of the radix tree */
+ rt = rt_create(CurrentMemoryContext);
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ uint32 off;
+
+ CHECK_FOR_INTERRUPTS();
+
+ key = tid_to_key_off(tid, &off);
+
+ if (last_key != PG_UINT64_MAX && last_key != key)
+ {
+ rt_set(rt, last_key, &val);
+ val = 0;
+ }
+
+ last_key = key;
+ val |= (uint64) 1 << off;
+ }
+ if (last_key != PG_UINT64_MAX)
+ rt_set(rt, last_key, &val);
+
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_load_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+ if (shuffle)
+ shuffle_itemptrs(tids, ntids);
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ /* meaure the serach time of the radix tree */
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ uint32 off;
+ volatile bool ret; /* prevent calling rt_search from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ key = tid_to_key_off(tid, &off);
+
+ ret = rt_search(rt, key, &val);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_search_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int64GetDatum(rt_num_entries(rt));
+ values[1] = Int64GetDatum(rt_memory_usage(rt));
+ values[2] = Int64GetDatum(sizeof(ItemPointerData) * ntids);
+ values[3] = Int64GetDatum(rt_load_ms);
+ nulls[4] = true; /* ar_load_ms */
+ values[5] = Int64GetDatum(rt_search_ms);
+ nulls[6] = true; /* ar_search_ms */
+
+#ifdef MEASURE_BINARY_SEARCH
+ {
+ ItemPointer itemptrs = NULL;
+
+ int64 ar_load_ms,
+ ar_search_ms;
+
+ /* measure the load time of the array */
+ itemptrs = MemoryContextAllocHuge(CurrentMemoryContext,
+ sizeof(ItemPointerData) * ntids);
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointerSetBlockNumber(&(itemptrs[i]),
+ ItemPointerGetBlockNumber(&(tids[i])));
+ ItemPointerSetOffsetNumber(&(itemptrs[i]),
+ ItemPointerGetOffsetNumber(&(tids[i])));
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ ar_load_ms = secs * 1000 + usecs / 1000;
+
+ /* next, measure the serach time of the array */
+ start_time = GetCurrentTimestamp();
+ for (int i = 0; i < ntids; i++)
+ {
+ ItemPointer tid = &(tids[i]);
+ volatile bool ret; /* prevent calling bsearch from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ ret = bsearch((void *) tid,
+ (void *) itemptrs,
+ ntids,
+ sizeof(ItemPointerData),
+ vac_cmp_itemptr);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ ar_search_ms = secs * 1000 + usecs / 1000;
+
+ /* set the result */
+ nulls[4] = false;
+ values[4] = Int64GetDatum(ar_load_ms);
+ nulls[6] = false;
+ values[6] = Int64GetDatum(ar_search_ms);
+ }
+#endif
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_seq_search(PG_FUNCTION_ARGS)
+{
+ return bench_search(fcinfo, false);
+}
+
+Datum
+bench_shuffle_search(PG_FUNCTION_ARGS)
+{
+ return bench_search(fcinfo, true);
+}
+
+Datum
+bench_load_random_int(PG_FUNCTION_ARGS)
+{
+ uint64 cnt = (uint64) PG_GETARG_INT64(0);
+ rt_radix_tree *rt;
+ pg_prng_state state;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_time_ms;
+ Datum values[2];
+ bool nulls[2];
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ pg_prng_seed(&state, 0);
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 key = pg_prng_uint64(&state);
+
+ rt_set(rt, key, &key);
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_time_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int64GetDatum(rt_memory_usage(rt));
+ values[1] = Int64GetDatum(load_time_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+/* copy of splitmix64() */
+static uint64
+hash64(uint64 x)
+{
+ x ^= x >> 30;
+ x *= UINT64CONST(0xbf58476d1ce4e5b9);
+ x ^= x >> 27;
+ x *= UINT64CONST(0x94d049bb133111eb);
+ x ^= x >> 31;
+ return x;
+}
+
+/* attempts to have a relatively even population of node kinds */
+Datum
+bench_search_random_nodes(PG_FUNCTION_ARGS)
+{
+ uint64 cnt = (uint64) PG_GETARG_INT64(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 load_time_ms;
+ int64 search_time_ms;
+ Datum values[3] = {0};
+ bool nulls[3] = {0};
+ /* from trial and error */
+ uint64 filter = (((uint64) 0x7F<<32) | (0x07<<24) | (0xFF<<16) | 0xFF);
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ if (!PG_ARGISNULL(1))
+ {
+ char *filter_str = text_to_cstring(PG_GETARG_TEXT_P(1));
+
+ if (sscanf(filter_str, "0x%lX", &filter) == 0)
+ elog(ERROR, "invalid filter string %s", filter_str);
+ }
+ elog(NOTICE, "bench with filter 0x%lX", filter);
+
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 hash = hash64(i);
+ uint64 key = hash & filter;
+
+ rt_set(rt, key, &key);
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ load_time_ms = secs * 1000 + usecs / 1000;
+
+ elog(NOTICE, "sleeping for 2 seconds...");
+ pg_usleep(2 * 1000000L);
+
+ start_time = GetCurrentTimestamp();
+ for (uint64 i = 0; i < cnt; i++)
+ {
+ uint64 hash = hash64(i);
+ uint64 key = hash & filter;
+ uint64 val;
+ volatile bool ret; /* prevent calling rt_search from being
+ * optimized out */
+
+ CHECK_FOR_INTERRUPTS();
+
+ ret = rt_search(rt, key, &val);
+ (void) ret;
+ }
+ end_time = GetCurrentTimestamp();
+
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ search_time_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ values[0] = Int64GetDatum(rt_memory_usage(rt));
+ values[1] = Int64GetDatum(load_time_ms);
+ values[2] = Int64GetDatum(search_time_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_fixed_height_search(PG_FUNCTION_ARGS)
+{
+ int fanout = PG_GETARG_INT32(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_load_ms,
+ rt_search_ms;
+ Datum values[5];
+ bool nulls[5];
+
+ /* test boundary between vector and iteration */
+ const int n_keys = 5 * 16 * 16 * 16 * 16;
+ uint64 r,
+ h,
+ i,
+ j,
+ k;
+ uint64 key_id;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ rt = rt_create(CurrentMemoryContext);
+
+ start_time = GetCurrentTimestamp();
+
+ key_id = 0;
+
+ /*
+ * lower nodes have limited fanout, the top is only limited by
+ * bits-per-byte
+ */
+ for (r = 1;; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ for (i = 1; i <= fanout; i++)
+ {
+ for (j = 1; j <= fanout; j++)
+ {
+ for (k = 1; k <= fanout; k++)
+ {
+ uint64 key;
+
+ key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+ CHECK_FOR_INTERRUPTS();
+
+ key_id++;
+ if (key_id > n_keys)
+ goto finish_set;
+
+ rt_set(rt, key, &key_id);
+ }
+ }
+ }
+ }
+ }
+finish_set:
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_load_ms = secs * 1000 + usecs / 1000;
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ /* meaure the search time of the radix tree */
+ start_time = GetCurrentTimestamp();
+
+ key_id = 0;
+ for (r = 1;; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ for (i = 1; i <= fanout; i++)
+ {
+ for (j = 1; j <= fanout; j++)
+ {
+ for (k = 1; k <= fanout; k++)
+ {
+ uint64 key,
+ val;
+
+ key = (r << 32) | (h << 24) | (i << 16) | (j << 8) | (k);
+
+ CHECK_FOR_INTERRUPTS();
+
+ key_id++;
+ if (key_id > n_keys)
+ goto finish_search;
+
+ rt_search(rt, key, &val);
+ }
+ }
+ }
+ }
+ }
+finish_search:
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_search_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int32GetDatum(fanout);
+ values[1] = Int64GetDatum(rt_num_entries(rt));
+ values[2] = Int64GetDatum(rt_memory_usage(rt));
+ values[3] = Int64GetDatum(rt_load_ms);
+ values[4] = Int64GetDatum(rt_search_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+Datum
+bench_node128_load(PG_FUNCTION_ARGS)
+{
+ int fanout = PG_GETARG_INT32(0);
+ rt_radix_tree *rt;
+ TupleDesc tupdesc;
+ TimestampTz start_time,
+ end_time;
+ long secs;
+ int usecs;
+ int64 rt_sparseload_ms;
+ Datum values[5];
+ bool nulls[5];
+
+ uint64 r,
+ h;
+ uint64 key_id;
+
+ /* Build a tuple descriptor for our result type */
+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)
+ elog(ERROR, "return type must be a row type");
+
+ rt = rt_create(CurrentMemoryContext);
+
+ key_id = 0;
+
+ for (r = 1; r <= fanout; r++)
+ {
+ for (h = 1; h <= fanout; h++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (h);
+
+ key_id++;
+ rt_set(rt, key, &key_id);
+ }
+ }
+
+#ifdef RT_DEBUG
+ rt_stats(rt);
+#endif
+
+ /* measure sparse deletion and re-loading */
+ start_time = GetCurrentTimestamp();
+
+ for (int t = 0; t<10000; t++)
+ {
+ /* delete one key in each leaf */
+ for (r = 1; r <= fanout; r++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (fanout);
+
+ rt_delete(rt, key);
+ }
+
+ /* add them all back */
+ key_id = 0;
+ for (r = 1; r <= fanout; r++)
+ {
+ uint64 key;
+
+ key = (r << 8) | (fanout);
+
+ key_id++;
+ rt_set(rt, key, &key_id);
+ }
+ }
+ end_time = GetCurrentTimestamp();
+ TimestampDifference(start_time, end_time, &secs, &usecs);
+ rt_sparseload_ms = secs * 1000 + usecs / 1000;
+
+ MemSet(nulls, false, sizeof(nulls));
+ values[0] = Int32GetDatum(fanout);
+ values[1] = Int64GetDatum(rt_num_entries(rt));
+ values[2] = Int64GetDatum(rt_memory_usage(rt));
+ values[3] = Int64GetDatum(rt_sparseload_ms);
+
+ rt_free(rt);
+ PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls)));
+}
+
+/* to silence warnings about unused iter functions */
+static void pg_attribute_unused()
+stub_iter()
+{
+ rt_radix_tree *rt;
+ rt_iter *iter;
+ uint64 key = 1;
+ uint64 value = 1;
+
+ rt = rt_create(CurrentMemoryContext);
+
+ iter = rt_begin_iterate(rt);
+ rt_iterate_next(iter, &key, &value);
+ rt_end_iterate(iter);
+}
\ No newline at end of file
diff --git a/contrib/bench_radix_tree/bench_radix_tree.control b/contrib/bench_radix_tree/bench_radix_tree.control
new file mode 100644
index 0000000000..1d988e6c9a
--- /dev/null
+++ b/contrib/bench_radix_tree/bench_radix_tree.control
@@ -0,0 +1,6 @@
+# bench_radix_tree extension
+comment = 'benchmark suits for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/bench_radix_tree'
+relocatable = true
+trusted = true
diff --git a/contrib/bench_radix_tree/expected/bench.out b/contrib/bench_radix_tree/expected/bench.out
new file mode 100644
index 0000000000..60c303892e
--- /dev/null
+++ b/contrib/bench_radix_tree/expected/bench.out
@@ -0,0 +1,13 @@
+create extension bench_radix_tree;
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/bench_radix_tree/meson.build b/contrib/bench_radix_tree/meson.build
new file mode 100644
index 0000000000..332c1ae7df
--- /dev/null
+++ b/contrib/bench_radix_tree/meson.build
@@ -0,0 +1,33 @@
+bench_radix_tree_sources = files(
+ 'bench_radix_tree.c',
+)
+
+if host_system == 'windows'
+ bench_radix_tree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'bench_radix_tree',
+ '--FILEDESC', 'bench_radix_tree - performance test code for radix tree',])
+endif
+
+bench_radix_tree = shared_module('bench_radix_tree',
+ bench_radix_tree_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += bench_radix_tree
+
+install_data(
+ 'bench_radix_tree.control',
+ 'bench_radix_tree--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'bench_radix_tree',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'bench_radix_tree',
+ ],
+ },
+}
diff --git a/contrib/bench_radix_tree/sql/bench.sql b/contrib/bench_radix_tree/sql/bench.sql
new file mode 100644
index 0000000000..a46018c9d4
--- /dev/null
+++ b/contrib/bench_radix_tree/sql/bench.sql
@@ -0,0 +1,16 @@
+create extension bench_radix_tree;
+
+\o seq_search.data
+begin;
+select * from bench_seq_search(0, 1000000);
+commit;
+
+\o shuffle_search.data
+begin;
+select * from bench_shuffle_search(0, 1000000);
+commit;
+
+\o random_load.data
+begin;
+select * from bench_load_random_int(10000000);
+commit;
diff --git a/contrib/meson.build b/contrib/meson.build
index bd4a57c43c..421d469f8c 100644
--- a/contrib/meson.build
+++ b/contrib/meson.build
@@ -12,6 +12,7 @@ subdir('amcheck')
subdir('auth_delay')
subdir('auto_explain')
subdir('basic_archive')
+subdir('bench_radix_tree')
subdir('bloom')
subdir('basebackup_to_shell')
subdir('bool_plperl')
--
2.31.1
[application/octet-stream] v32-0004-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch (36.0K, ../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/16-v32-0004-Add-TIDStore-to-store-sets-of-TIDs-ItemPointerDa.patch)
download | inline diff:
From a804e3ebba8733d65497d5e9c3a47b32f175ea1e Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Fri, 23 Dec 2022 23:50:39 +0900
Subject: [PATCH v32 04/18] Add TIDStore, to store sets of TIDs
(ItemPointerData) efficiently.
The TIDStore is designed to store large sets of TIDs efficiently, and
is backed by the radix tree. A TID is encoded into 64-bit key and
value and inserted to the radix tree.
The TIDStore is not used for anything yet, aside from the test code,
but the follow up patch integrates the TIDStore with lazy vacuum,
reducing lazy vacuum memory usage and lifting the 1GB limit on its
size, by storing the list of dead TIDs more efficiently.
This includes a unit test module, in src/test/modules/test_tidstore.
---
doc/src/sgml/monitoring.sgml | 4 +
src/backend/access/common/Makefile | 1 +
src/backend/access/common/meson.build | 1 +
src/backend/access/common/tidstore.c | 681 ++++++++++++++++++
src/backend/storage/lmgr/lwlock.c | 2 +
src/include/access/tidstore.h | 49 ++
src/include/storage/lwlock.h | 1 +
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_tidstore/Makefile | 23 +
.../test_tidstore/expected/test_tidstore.out | 13 +
src/test/modules/test_tidstore/meson.build | 35 +
.../test_tidstore/sql/test_tidstore.sql | 7 +
.../test_tidstore/test_tidstore--1.0.sql | 8 +
.../modules/test_tidstore/test_tidstore.c | 226 ++++++
.../test_tidstore/test_tidstore.control | 4 +
16 files changed, 1057 insertions(+)
create mode 100644 src/backend/access/common/tidstore.c
create mode 100644 src/include/access/tidstore.h
create mode 100644 src/test/modules/test_tidstore/Makefile
create mode 100644 src/test/modules/test_tidstore/expected/test_tidstore.out
create mode 100644 src/test/modules/test_tidstore/meson.build
create mode 100644 src/test/modules/test_tidstore/sql/test_tidstore.sql
create mode 100644 src/test/modules/test_tidstore/test_tidstore--1.0.sql
create mode 100644 src/test/modules/test_tidstore/test_tidstore.c
create mode 100644 src/test/modules/test_tidstore/test_tidstore.control
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index 2903b67170..be4448fe6e 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -2211,6 +2211,10 @@ postgres 27093 0.0 0.0 30096 2752 ? Ss 11:34 0:00 postgres: ser
<entry>Waiting to access a shared TID bitmap during a parallel bitmap
index scan.</entry>
</row>
+ <row>
+ <entry><literal>SharedTidStore</literal></entry>
+ <entry>Waiting to access a shared TID store.</entry>
+ </row>
<row>
<entry><literal>SharedTupleStore</literal></entry>
<entry>Waiting to access a shared tuple store during parallel
diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile
index b9aff0ccfd..67b8cc6108 100644
--- a/src/backend/access/common/Makefile
+++ b/src/backend/access/common/Makefile
@@ -27,6 +27,7 @@ OBJS = \
syncscan.o \
toast_compression.o \
toast_internals.o \
+ tidstore.o \
tupconvert.o \
tupdesc.o
diff --git a/src/backend/access/common/meson.build b/src/backend/access/common/meson.build
index f5ac17b498..fce19c09ce 100644
--- a/src/backend/access/common/meson.build
+++ b/src/backend/access/common/meson.build
@@ -15,6 +15,7 @@ backend_sources += files(
'syncscan.c',
'toast_compression.c',
'toast_internals.c',
+ 'tidstore.c',
'tupconvert.c',
'tupdesc.c',
)
diff --git a/src/backend/access/common/tidstore.c b/src/backend/access/common/tidstore.c
new file mode 100644
index 0000000000..8c05e60d92
--- /dev/null
+++ b/src/backend/access/common/tidstore.c
@@ -0,0 +1,681 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.c
+ * Tid (ItemPointerData) storage implementation.
+ *
+ * This module provides a in-memory data structure to store Tids (ItemPointer).
+ * Internally, a tid is encoded as a pair of 64-bit key and 64-bit value, and
+ * stored in the radix tree.
+ *
+ * A TidStore can be shared among parallel worker processes by passing DSA area
+ * to tidstore_create(). Other backends can attach to the shared TidStore by
+ * tidstore_attach().
+ *
+ * Regarding the concurrency, it basically relies on the concurrency support in
+ * the radix tree, but we acquires the lock on a TidStore in some cases, for
+ * example, when to reset the store and when to access the number tids in the
+ * store (num_tids).
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * IDENTIFICATION
+ * src/backend/access/common/tidstore.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/tidstore.h"
+#include "miscadmin.h"
+#include "port/pg_bitutils.h"
+#include "storage/lwlock.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/*
+ * For encoding purposes, tids are represented as a pair of 64-bit key and
+ * 64-bit value. First, we construct 64-bit unsigned integer by combining
+ * the block number and the offset number. The number of bits used for the
+ * offset number is specified by max_offsets in tidstore_create(). We are
+ * frugal with the bits, because smaller keys could help keeping the radix
+ * tree shallow.
+ *
+ * For example, a tid of heap with 8kB blocks uses the lowest 9 bits for
+ * the offset number and uses the next 32 bits for the block number. That
+ * is, only 41 bits are used:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ *
+ * X = bits used for offset number
+ * Y = bits used for block number
+ * u = unused bit
+ * (high on the left, low on the right)
+ *
+ * 9 bits are enough for the offset number, because MaxHeapTuplesPerPage < 2^9
+ * on 8kB blocks.
+ *
+ * The 64-bit value is the bitmap representation of the lowest 6 bits
+ * (TIDSTORE_VALUE_NBITS) of the integer, and the rest 35 bits are used
+ * as the key:
+ *
+ * uuuuuuuY YYYYYYYY YYYYYYYY YYYYYYYY YYYYYYYX XXXXXXXX
+ * |----| value
+ * |---------------------------------------------| key
+ *
+ * The maximum height of the radix tree is 5 in this case.
+ */
+#define TIDSTORE_VALUE_NBITS 6 /* log(64, 2) */
+#define TIDSTORE_OFFSET_MASK ((1 << TIDSTORE_VALUE_NBITS) - 1)
+
+/* A magic value used to identify our TidStores. */
+#define TIDSTORE_MAGIC 0x826f6a10
+
+#define RT_PREFIX local_rt
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE uint64
+#include "lib/radixtree.h"
+
+#define RT_PREFIX shared_rt
+#define RT_SHMEM
+#define RT_SCOPE static
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_VALUE_TYPE uint64
+#include "lib/radixtree.h"
+
+/* The control object for a TidStore */
+typedef struct TidStoreControl
+{
+ /* the number of tids in the store */
+ int64 num_tids;
+
+ /* These values are never changed after creation */
+ size_t max_bytes; /* the maximum bytes a TidStore can use */
+ int max_offset; /* the maximum offset number */
+ int offset_nbits; /* the number of bits required for an offset
+ * number */
+ int offset_key_nbits; /* the number of bits of an offset number
+ * used in a key */
+
+ /* The below fields are used only in shared case */
+
+ uint32 magic;
+ LWLock lock;
+
+ /* handles for TidStore and radix tree */
+ tidstore_handle handle;
+ shared_rt_handle tree_handle;
+} TidStoreControl;
+
+/* Per-backend state for a TidStore */
+struct TidStore
+{
+ /*
+ * Control object. This is allocated in DSA area 'area' in the shared
+ * case, otherwise in backend-local memory.
+ */
+ TidStoreControl *control;
+
+ /* Storage for Tids. Use either one depending on TidStoreIsShared() */
+ union
+ {
+ local_rt_radix_tree *local;
+ shared_rt_radix_tree *shared;
+ } tree;
+
+ /* DSA area for TidStore if used */
+ dsa_area *area;
+};
+#define TidStoreIsShared(ts) ((ts)->area != NULL)
+
+/* Iterator for TidStore */
+typedef struct TidStoreIter
+{
+ TidStore *ts;
+
+ /* iterator of radix tree. Use either one depending on TidStoreIsShared() */
+ union
+ {
+ shared_rt_iter *shared;
+ local_rt_iter *local;
+ } tree_iter;
+
+ /* we returned all tids? */
+ bool finished;
+
+ /* save for the next iteration */
+ uint64 next_key;
+ uint64 next_val;
+
+ /* output for the caller */
+ TidStoreIterResult result;
+} TidStoreIter;
+
+static void tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val);
+static inline BlockNumber key_get_blkno(TidStore *ts, uint64 key);
+static inline uint64 encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit);
+static inline uint64 tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit);
+
+/*
+ * Create a TidStore. The returned object is allocated in backend-local memory.
+ * The radix tree for storage is allocated in DSA area is 'area' is non-NULL.
+ */
+TidStore *
+tidstore_create(size_t max_bytes, int max_offset, dsa_area *area)
+{
+ TidStore *ts;
+
+ ts = palloc0(sizeof(TidStore));
+
+ /*
+ * Create the radix tree for the main storage.
+ *
+ * Memory consumption depends on the number of stored tids, but also on the
+ * distribution of them, how the radix tree stores, and the memory management
+ * that backed the radix tree. The maximum bytes that a TidStore can
+ * use is specified by the max_bytes in tidstore_create(). We want the total
+ * amount of memory consumption by a TidStore not to exceed the max_bytes.
+ *
+ * In local TidStore cases, the radix tree uses slab allocators for each kind
+ * of node class. The most memory consuming case while adding Tids associated
+ * with one page (i.e. during tidstore_add_tids()) is that we allocate a new
+ * slab block for a new radix tree node, which is approximately 70kB. Therefore,
+ * we deduct 70kB from the max_bytes.
+ *
+ * In shared cases, DSA allocates the memory segments big enough to follow
+ * a geometric series that approximately doubles the total DSA size (see
+ * make_new_segment() in dsa.c). We simulated the how DSA increases segment
+ * size and the simulation revealed, the 75% threshold for the maximum bytes
+ * perfectly works in case where the max_bytes is a power-of-2, and the 60%
+ * threshold works for other cases.
+ */
+ if (area != NULL)
+ {
+ dsa_pointer dp;
+ float ratio = ((max_bytes & (max_bytes - 1)) == 0) ? 0.75 : 0.6;
+
+ ts->tree.shared = shared_rt_create(CurrentMemoryContext, area,
+ LWTRANCHE_SHARED_TIDSTORE);
+
+ dp = dsa_allocate0(area, sizeof(TidStoreControl));
+ ts->control = (TidStoreControl *) dsa_get_address(area, dp);
+ ts->control->max_bytes = (uint64) (max_bytes * ratio);
+ ts->area = area;
+
+ ts->control->magic = TIDSTORE_MAGIC;
+ LWLockInitialize(&ts->control->lock, LWTRANCHE_SHARED_TIDSTORE);
+ ts->control->handle = dp;
+ ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+ }
+ else
+ {
+ ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+ ts->control = (TidStoreControl *) palloc0(sizeof(TidStoreControl));
+ ts->control->max_bytes = max_bytes - (70 * 1024);
+ }
+
+ ts->control->max_offset = max_offset;
+ ts->control->offset_nbits = pg_ceil_log2_32(max_offset);
+
+ if (ts->control->offset_nbits < TIDSTORE_VALUE_NBITS)
+ ts->control->offset_nbits = TIDSTORE_VALUE_NBITS;
+
+ ts->control->offset_key_nbits =
+ ts->control->offset_nbits - TIDSTORE_VALUE_NBITS;
+
+ return ts;
+}
+
+/*
+ * Attach to the shared TidStore using a handle. The returned object is
+ * allocated in backend-local memory using the CurrentMemoryContext.
+ */
+TidStore *
+tidstore_attach(dsa_area *area, tidstore_handle handle)
+{
+ TidStore *ts;
+ dsa_pointer control;
+
+ Assert(area != NULL);
+ Assert(DsaPointerIsValid(handle));
+
+ /* create per-backend state */
+ ts = palloc0(sizeof(TidStore));
+
+ /* Find the control object in shared memory */
+ control = handle;
+
+ /* Set up the TidStore */
+ ts->control = (TidStoreControl *) dsa_get_address(area, control);
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ ts->tree.shared = shared_rt_attach(area, ts->control->tree_handle);
+ ts->area = area;
+
+ return ts;
+}
+
+/*
+ * Detach from a TidStore. This detaches from radix tree and frees the
+ * backend-local resources. The radix tree will continue to exist until
+ * it is either explicitly destroyed, or the area that backs it is returned
+ * to the operating system.
+ */
+void
+tidstore_detach(TidStore *ts)
+{
+ Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+ shared_rt_detach(ts->tree.shared);
+ pfree(ts);
+}
+
+/*
+ * Destroy a TidStore, returning all memory.
+ *
+ * TODO: The caller must be certain that no other backend will attempt to
+ * access the TidStore before calling this function. Other backend must
+ * explicitly call tidstore_detach to free up backend-local memory associated
+ * with the TidStore. The backend that calls tidstore_destroy must not call
+ * tidstore_detach.
+ */
+void
+tidstore_destroy(TidStore *ts)
+{
+ if (TidStoreIsShared(ts))
+ {
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ /*
+ * Vandalize the control block to help catch programming error where
+ * other backends access the memory formerly occupied by this radix
+ * tree.
+ */
+ ts->control->magic = 0;
+ dsa_free(ts->area, ts->control->handle);
+ shared_rt_free(ts->tree.shared);
+ }
+ else
+ {
+ pfree(ts->control);
+ local_rt_free(ts->tree.local);
+ }
+
+ pfree(ts);
+}
+
+/*
+ * Forget all collected Tids. It's similar to tidstore_destroy but we don't free
+ * entire TidStore but recreate only the radix tree storage.
+ */
+void
+tidstore_reset(TidStore *ts)
+{
+ if (TidStoreIsShared(ts))
+ {
+ Assert(ts->control->magic == TIDSTORE_MAGIC);
+
+ LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+ /*
+ * Free the radix tree and return allocated DSA segments to
+ * the operating system.
+ */
+ shared_rt_free(ts->tree.shared);
+ dsa_trim(ts->area);
+
+ /* Recreate the radix tree */
+ ts->tree.shared = shared_rt_create(CurrentMemoryContext, ts->area,
+ LWTRANCHE_SHARED_TIDSTORE);
+
+ /* update the radix tree handle as we recreated it */
+ ts->control->tree_handle = shared_rt_get_handle(ts->tree.shared);
+
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
+
+ LWLockRelease(&ts->control->lock);
+ }
+ else
+ {
+ local_rt_free(ts->tree.local);
+ ts->tree.local = local_rt_create(CurrentMemoryContext);
+
+ /* Reset the statistics */
+ ts->control->num_tids = 0;
+ }
+}
+
+/* Add Tids on a block to TidStore */
+void
+tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets)
+{
+ uint64 *values;
+ uint64 key;
+ uint64 prev_key;
+ uint64 off_bitmap = 0;
+ int idx;
+ const uint64 key_base = ((uint64) blkno) << ts->control->offset_key_nbits;
+ const int nkeys = UINT64CONST(1) << ts->control->offset_key_nbits;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ values = palloc(sizeof(uint64) * nkeys);
+ key = prev_key = key_base;
+
+ for (int i = 0; i < num_offsets; i++)
+ {
+ uint64 off_bit;
+
+ /* encode the tid to a key and partial offset */
+ key = encode_key_off(ts, blkno, offsets[i], &off_bit);
+
+ /* make sure we scanned the line pointer array in order */
+ Assert(key >= prev_key);
+
+ if (key > prev_key)
+ {
+ idx = prev_key - key_base;
+ Assert(idx >= 0 && idx < nkeys);
+
+ /* write out offset bitmap for this key */
+ values[idx] = off_bitmap;
+
+ /* zero out any gaps up to the current key */
+ for (int empty_idx = idx + 1; empty_idx < key - key_base; empty_idx++)
+ values[empty_idx] = 0;
+
+ /* reset for current key -- the current offset will be handled below */
+ off_bitmap = 0;
+ prev_key = key;
+ }
+
+ off_bitmap |= off_bit;
+ }
+
+ /* save the final index for later */
+ idx = key - key_base;
+ /* write out last offset bitmap */
+ values[idx] = off_bitmap;
+
+ if (TidStoreIsShared(ts))
+ LWLockAcquire(&ts->control->lock, LW_EXCLUSIVE);
+
+ /* insert the calculated key-values to the tree */
+ for (int i = 0; i <= idx; i++)
+ {
+ if (values[i])
+ {
+ key = key_base + i;
+
+ if (TidStoreIsShared(ts))
+ shared_rt_set(ts->tree.shared, key, &values[i]);
+ else
+ local_rt_set(ts->tree.local, key, &values[i]);
+ }
+ }
+
+ /* update statistics */
+ ts->control->num_tids += num_offsets;
+
+ if (TidStoreIsShared(ts))
+ LWLockRelease(&ts->control->lock);
+
+ pfree(values);
+}
+
+/* Return true if the given tid is present in the TidStore */
+bool
+tidstore_lookup_tid(TidStore *ts, ItemPointer tid)
+{
+ uint64 key;
+ uint64 val = 0;
+ uint64 off_bit;
+ bool found;
+
+ key = tid_to_key_off(ts, tid, &off_bit);
+
+ if (TidStoreIsShared(ts))
+ found = shared_rt_search(ts->tree.shared, key, &val);
+ else
+ found = local_rt_search(ts->tree.local, key, &val);
+
+ if (!found)
+ return false;
+
+ return (val & off_bit) != 0;
+}
+
+/*
+ * Prepare to iterate through a TidStore. Since the radix tree is locked during the
+ * iteration, so tidstore_end_iterate() needs to called when finished.
+ *
+ * Concurrent updates during the iteration will be blocked when inserting a
+ * key-value to the radix tree.
+ */
+TidStoreIter *
+tidstore_begin_iterate(TidStore *ts)
+{
+ TidStoreIter *iter;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ iter = palloc0(sizeof(TidStoreIter));
+ iter->ts = ts;
+
+ iter->result.blkno = InvalidBlockNumber;
+ iter->result.offsets = palloc(sizeof(OffsetNumber) * ts->control->max_offset);
+
+ if (TidStoreIsShared(ts))
+ iter->tree_iter.shared = shared_rt_begin_iterate(ts->tree.shared);
+ else
+ iter->tree_iter.local = local_rt_begin_iterate(ts->tree.local);
+
+ /* If the TidStore is empty, there is no business */
+ if (tidstore_num_tids(ts) == 0)
+ iter->finished = true;
+
+ return iter;
+}
+
+static inline bool
+tidstore_iter_kv(TidStoreIter *iter, uint64 *key, uint64 *val)
+{
+ if (TidStoreIsShared(iter->ts))
+ return shared_rt_iterate_next(iter->tree_iter.shared, key, val);
+
+ return local_rt_iterate_next(iter->tree_iter.local, key, val);
+}
+
+/*
+ * Scan the TidStore and return a pointer to TidStoreIterResult that has tids
+ * in one block. We return the block numbers in ascending order and the offset
+ * numbers in each result is also sorted in ascending order.
+ */
+TidStoreIterResult *
+tidstore_iterate_next(TidStoreIter *iter)
+{
+ uint64 key;
+ uint64 val;
+ TidStoreIterResult *result = &(iter->result);
+
+ if (iter->finished)
+ return NULL;
+
+ if (BlockNumberIsValid(result->blkno))
+ {
+ /* Process the previously collected key-value */
+ result->num_offsets = 0;
+ tidstore_iter_extract_tids(iter, iter->next_key, iter->next_val);
+ }
+
+ while (tidstore_iter_kv(iter, &key, &val))
+ {
+ BlockNumber blkno;
+
+ blkno = key_get_blkno(iter->ts, key);
+
+ if (BlockNumberIsValid(result->blkno) && result->blkno != blkno)
+ {
+ /*
+ * We got a key-value pair for a different block. So return the
+ * collected tids, and remember the key-value for the next iteration.
+ */
+ iter->next_key = key;
+ iter->next_val = val;
+ return result;
+ }
+
+ /* Collect tids extracted from the key-value pair */
+ tidstore_iter_extract_tids(iter, key, val);
+ }
+
+ iter->finished = true;
+ return result;
+}
+
+/*
+ * Finish an iteration over TidStore. This needs to be called after finishing
+ * or when existing an iteration.
+ */
+void
+tidstore_end_iterate(TidStoreIter *iter)
+{
+ if (TidStoreIsShared(iter->ts))
+ shared_rt_end_iterate(iter->tree_iter.shared);
+ else
+ local_rt_end_iterate(iter->tree_iter.local);
+
+ pfree(iter->result.offsets);
+ pfree(iter);
+}
+
+/* Return the number of tids we collected so far */
+int64
+tidstore_num_tids(TidStore *ts)
+{
+ uint64 num_tids;
+
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ if (!TidStoreIsShared(ts))
+ return ts->control->num_tids;
+
+ LWLockAcquire(&ts->control->lock, LW_SHARED);
+ num_tids = ts->control->num_tids;
+ LWLockRelease(&ts->control->lock);
+
+ return num_tids;
+}
+
+/* Return true if the current memory usage of TidStore exceeds the limit */
+bool
+tidstore_is_full(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ return (tidstore_memory_usage(ts) > ts->control->max_bytes);
+}
+
+/* Return the maximum memory TidStore can use */
+size_t
+tidstore_max_memory(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ return ts->control->max_bytes;
+}
+
+/* Return the memory usage of TidStore */
+size_t
+tidstore_memory_usage(TidStore *ts)
+{
+ Assert(!TidStoreIsShared(ts) || ts->control->magic == TIDSTORE_MAGIC);
+
+ /*
+ * In the shared case, TidStoreControl and radix_tree are backed by the
+ * same DSA area and rt_memory_usage() returns the value including both.
+ * So we don't need to add the size of TidStoreControl separately.
+ */
+ if (TidStoreIsShared(ts))
+ return sizeof(TidStore) + shared_rt_memory_usage(ts->tree.shared);
+
+ return sizeof(TidStore) + sizeof(TidStore) + local_rt_memory_usage(ts->tree.local);
+}
+
+/*
+ * Get a handle that can be used by other processes to attach to this TidStore
+ */
+tidstore_handle
+tidstore_get_handle(TidStore *ts)
+{
+ Assert(TidStoreIsShared(ts) && ts->control->magic == TIDSTORE_MAGIC);
+
+ return ts->control->handle;
+}
+
+/* Extract tids from the given key-value pair */
+static void
+tidstore_iter_extract_tids(TidStoreIter *iter, uint64 key, uint64 val)
+{
+ TidStoreIterResult *result = (&iter->result);
+
+ while (val)
+ {
+ uint64 tid_i;
+ OffsetNumber off;
+
+ tid_i = key << TIDSTORE_VALUE_NBITS;
+ tid_i |= pg_rightmost_one_pos64(val);
+
+ off = tid_i & ((UINT64CONST(1) << iter->ts->control->offset_nbits) - 1);
+
+ Assert(result->num_offsets < iter->ts->control->max_offset);
+ result->offsets[result->num_offsets++] = off;
+
+ /* unset the rightmost bit */
+ val &= ~pg_rightmost_one64(val);
+ }
+
+ result->blkno = key_get_blkno(iter->ts, key);
+}
+
+/* Get block number from the given key */
+static inline BlockNumber
+key_get_blkno(TidStore *ts, uint64 key)
+{
+ return (BlockNumber) (key >> ts->control->offset_key_nbits);
+}
+
+/* Encode a tid to key and offset */
+static inline uint64
+tid_to_key_off(TidStore *ts, ItemPointer tid, uint64 *off_bit)
+{
+ uint32 offset = ItemPointerGetOffsetNumber(tid);
+ BlockNumber block = ItemPointerGetBlockNumber(tid);
+
+ return encode_key_off(ts, block, offset, off_bit);
+}
+
+/* encode a block and offset to a key and partial offset */
+static inline uint64
+encode_key_off(TidStore *ts, BlockNumber block, uint32 offset, uint64 *off_bit)
+{
+ uint64 key;
+ uint64 tid_i;
+ uint32 off_lower;
+
+ off_lower = offset & TIDSTORE_OFFSET_MASK;
+ Assert(off_lower < (sizeof(uint64) * BITS_PER_BYTE));
+
+ *off_bit = UINT64CONST(1) << off_lower;
+ tid_i = offset | ((uint64) block << ts->control->offset_nbits);
+ key = tid_i >> TIDSTORE_VALUE_NBITS;
+
+ return key;
+}
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index d2ec396045..55b3a04097 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -176,6 +176,8 @@ static const char *const BuiltinTrancheNames[] = {
"SharedTupleStore",
/* LWTRANCHE_SHARED_TIDBITMAP: */
"SharedTidBitmap",
+ /* LWTRANCHE_SHARED_TIDSTORE: */
+ "SharedTidStore",
/* LWTRANCHE_PARALLEL_APPEND: */
"ParallelAppend",
/* LWTRANCHE_PER_XACT_PREDICATE_LIST: */
diff --git a/src/include/access/tidstore.h b/src/include/access/tidstore.h
new file mode 100644
index 0000000000..a35a52124a
--- /dev/null
+++ b/src/include/access/tidstore.h
@@ -0,0 +1,49 @@
+/*-------------------------------------------------------------------------
+ *
+ * tidstore.h
+ * Tid storage.
+ *
+ *
+ * Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/access/tidstore.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef TIDSTORE_H
+#define TIDSTORE_H
+
+#include "storage/itemptr.h"
+#include "utils/dsa.h"
+
+typedef dsa_pointer tidstore_handle;
+
+typedef struct TidStore TidStore;
+typedef struct TidStoreIter TidStoreIter;
+
+typedef struct TidStoreIterResult
+{
+ BlockNumber blkno;
+ OffsetNumber *offsets;
+ int num_offsets;
+} TidStoreIterResult;
+
+extern TidStore *tidstore_create(size_t max_bytes, int max_offset, dsa_area *dsa);
+extern TidStore *tidstore_attach(dsa_area *dsa, dsa_pointer handle);
+extern void tidstore_detach(TidStore *ts);
+extern void tidstore_destroy(TidStore *ts);
+extern void tidstore_reset(TidStore *ts);
+extern void tidstore_add_tids(TidStore *ts, BlockNumber blkno, OffsetNumber *offsets,
+ int num_offsets);
+extern bool tidstore_lookup_tid(TidStore *ts, ItemPointer tid);
+extern TidStoreIter * tidstore_begin_iterate(TidStore *ts);
+extern TidStoreIterResult *tidstore_iterate_next(TidStoreIter *iter);
+extern void tidstore_end_iterate(TidStoreIter *iter);
+extern int64 tidstore_num_tids(TidStore *ts);
+extern bool tidstore_is_full(TidStore *ts);
+extern size_t tidstore_max_memory(TidStore *ts);
+extern size_t tidstore_memory_usage(TidStore *ts);
+extern tidstore_handle tidstore_get_handle(TidStore *ts);
+
+#endif /* TIDSTORE_H */
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index d2c7afb8f4..07002fdfbe 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -199,6 +199,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_PER_SESSION_RECORD_TYPMOD,
LWTRANCHE_SHARED_TUPLESTORE,
LWTRANCHE_SHARED_TIDBITMAP,
+ LWTRANCHE_SHARED_TIDSTORE,
LWTRANCHE_PARALLEL_APPEND,
LWTRANCHE_PER_XACT_PREDICATE_LIST,
LWTRANCHE_PGSTATS_DSA,
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 89f42bf9e3..a6ec135430 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
test_rls_hooks \
test_shm_mq \
test_slru \
+ test_tidstore \
unsafe_tests \
worker_spi
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index beaf4080fb..f126ea9f2e 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -31,5 +31,6 @@ subdir('test_regex')
subdir('test_rls_hooks')
subdir('test_shm_mq')
subdir('test_slru')
+subdir('test_tidstore')
subdir('unsafe_tests')
subdir('worker_spi')
diff --git a/src/test/modules/test_tidstore/Makefile b/src/test/modules/test_tidstore/Makefile
new file mode 100644
index 0000000000..dab107d70c
--- /dev/null
+++ b/src/test/modules/test_tidstore/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_tidstore/Makefile
+
+MODULE_big = test_tidstore
+OBJS = \
+ $(WIN32RES) \
+ test_tidstore.o
+PGFILEDESC = "test_tidstore - test code for src/backend/access/common/tidstore.c"
+
+EXTENSION = test_tidstore
+DATA = test_tidstore--1.0.sql
+
+REGRESS = test_tidstore
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_tidstore
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_tidstore/expected/test_tidstore.out b/src/test/modules/test_tidstore/expected/test_tidstore.out
new file mode 100644
index 0000000000..7ff2f9af87
--- /dev/null
+++ b/src/test/modules/test_tidstore/expected/test_tidstore.out
@@ -0,0 +1,13 @@
+CREATE EXTENSION test_tidstore;
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
+NOTICE: testing empty tidstore
+NOTICE: testing basic operations
+ test_tidstore
+---------------
+
+(1 row)
+
diff --git a/src/test/modules/test_tidstore/meson.build b/src/test/modules/test_tidstore/meson.build
new file mode 100644
index 0000000000..31f2da7b61
--- /dev/null
+++ b/src/test/modules/test_tidstore/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_tidstore_sources = files(
+ 'test_tidstore.c',
+)
+
+if host_system == 'windows'
+ test_tidstore_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_tidstore',
+ '--FILEDESC', 'test_tidstore - test code for src/backend/access/common/tidstore.c',])
+endif
+
+test_tidstore = shared_module('test_tidstore',
+ test_tidstore_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += test_tidstore
+
+install_data(
+ 'test_tidstore.control',
+ 'test_tidstore--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'test_tidstore',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'test_tidstore',
+ ],
+ },
+}
diff --git a/src/test/modules/test_tidstore/sql/test_tidstore.sql b/src/test/modules/test_tidstore/sql/test_tidstore.sql
new file mode 100644
index 0000000000..03aea31815
--- /dev/null
+++ b/src/test/modules/test_tidstore/sql/test_tidstore.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_tidstore;
+
+--
+-- All the logic is in the test_tidstore() function. It will throw
+-- an error if something fails.
+--
+SELECT test_tidstore();
diff --git a/src/test/modules/test_tidstore/test_tidstore--1.0.sql b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
new file mode 100644
index 0000000000..47e9149900
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_tidstore/test_tidstore--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_tidstore" to load this file. \quit
+
+CREATE FUNCTION test_tidstore()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_tidstore/test_tidstore.c b/src/test/modules/test_tidstore/test_tidstore.c
new file mode 100644
index 0000000000..9a1217f833
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.c
@@ -0,0 +1,226 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_tidstore.c
+ * Test TidStore data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_tidstore/test_tidstore.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "access/htup_details.h"
+#include "access/tidstore.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "storage/lwlock.h"
+#include "utils/memutils.h"
+
+PG_MODULE_MAGIC;
+
+/* #define TEST_SHARED_TIDSTORE 1 */
+
+#define TEST_TIDSTORE_MAX_BYTES (2 * 1024 * 1024L) /* 2MB */
+
+PG_FUNCTION_INFO_V1(test_tidstore);
+
+static void
+check_tid(TidStore *ts, BlockNumber blkno, OffsetNumber off, bool expect)
+{
+ ItemPointerData tid;
+ bool found;
+
+ ItemPointerSet(&tid, blkno, off);
+
+ found = tidstore_lookup_tid(ts, &tid);
+
+ if (found != expect)
+ elog(ERROR, "lookup TID (%u, %u) returned %d, expected %d",
+ blkno, off, found, expect);
+}
+
+static void
+test_basic(int max_offset)
+{
+#define TEST_TIDSTORE_NUM_BLOCKS 5
+#define TEST_TIDSTORE_NUM_OFFSETS 5
+
+ TidStore *ts;
+ TidStoreIter *iter;
+ TidStoreIterResult *iter_result;
+ BlockNumber blks[TEST_TIDSTORE_NUM_BLOCKS] = {
+ 0, MaxBlockNumber, MaxBlockNumber - 1, 1, MaxBlockNumber / 2,
+ };
+ BlockNumber blks_sorted[TEST_TIDSTORE_NUM_BLOCKS] = {
+ 0, 1, MaxBlockNumber / 2, MaxBlockNumber - 1, MaxBlockNumber
+ };
+ OffsetNumber offs[TEST_TIDSTORE_NUM_OFFSETS];
+ int blk_idx;
+
+#ifdef TEST_SHARED_TIDSTORE
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_tidstore");
+ dsa = dsa_create(tranche_id);
+
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, dsa);
+#else
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, max_offset, NULL);
+#endif
+
+ /* prepare the offset array */
+ offs[0] = FirstOffsetNumber;
+ offs[1] = FirstOffsetNumber + 1;
+ offs[2] = max_offset / 2;
+ offs[3] = max_offset - 1;
+ offs[4] = max_offset;
+
+ /* add tids */
+ for (int i = 0; i < TEST_TIDSTORE_NUM_BLOCKS; i++)
+ tidstore_add_tids(ts, blks[i], offs, TEST_TIDSTORE_NUM_OFFSETS);
+
+ /* lookup test */
+ for (OffsetNumber off = FirstOffsetNumber ; off < max_offset; off++)
+ {
+ bool expect = false;
+ for (int i = 0; i < TEST_TIDSTORE_NUM_OFFSETS; i++)
+ {
+ if (offs[i] == off)
+ {
+ expect = true;
+ break;
+ }
+ }
+
+ check_tid(ts, 0, off, expect);
+ check_tid(ts, 2, off, false);
+ check_tid(ts, MaxBlockNumber - 2, off, false);
+ check_tid(ts, MaxBlockNumber, off, expect);
+ }
+
+ /* test the number of tids */
+ if (tidstore_num_tids(ts) != (TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS))
+ elog(ERROR, "tidstore_num_tids returned " UINT64_FORMAT ", expected %d",
+ tidstore_num_tids(ts),
+ TEST_TIDSTORE_NUM_BLOCKS * TEST_TIDSTORE_NUM_OFFSETS);
+
+ /* iteration test */
+ iter = tidstore_begin_iterate(ts);
+ blk_idx = 0;
+ while ((iter_result = tidstore_iterate_next(iter)) != NULL)
+ {
+ /* check the returned block number */
+ if (blks_sorted[blk_idx] != iter_result->blkno)
+ elog(ERROR, "tidstore_iterate_next returned block number %u, expected %u",
+ iter_result->blkno, blks_sorted[blk_idx]);
+
+ /* check the returned offset numbers */
+ if (TEST_TIDSTORE_NUM_OFFSETS != iter_result->num_offsets)
+ elog(ERROR, "tidstore_iterate_next returned %u offsets, expected %u",
+ iter_result->num_offsets, TEST_TIDSTORE_NUM_OFFSETS);
+
+ for (int i = 0; i < iter_result->num_offsets; i++)
+ {
+ if (offs[i] != iter_result->offsets[i])
+ elog(ERROR, "tidstore_iterate_next returned offset number %u on block %u, expected %u",
+ iter_result->offsets[i], iter_result->blkno, offs[i]);
+ }
+
+ blk_idx++;
+ }
+
+ if (blk_idx != TEST_TIDSTORE_NUM_BLOCKS)
+ elog(ERROR, "tidstore_iterate_next returned %d blocks, expected %d",
+ blk_idx, TEST_TIDSTORE_NUM_BLOCKS);
+
+ /* remove all tids */
+ tidstore_reset(ts);
+
+ /* test the number of tids */
+ if (tidstore_num_tids(ts) != 0)
+ elog(ERROR, "tidstore_num_tids on empty store returned non-zero");
+
+ /* lookup test for empty store */
+ for (OffsetNumber off = FirstOffsetNumber ; off < MaxHeapTuplesPerPage;
+ off++)
+ {
+ check_tid(ts, 0, off, false);
+ check_tid(ts, 2, off, false);
+ check_tid(ts, MaxBlockNumber - 2, off, false);
+ check_tid(ts, MaxBlockNumber, off, false);
+ }
+
+ tidstore_destroy(ts);
+
+#ifdef TEST_SHARED_TIDSTORE
+ dsa_detach(dsa);
+#endif
+}
+
+static void
+test_empty(void)
+{
+ TidStore *ts;
+ TidStoreIter *iter;
+ ItemPointerData tid;
+
+#ifdef TEST_SHARED_TIDSTORE
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_tidstore");
+ dsa = dsa_create(tranche_id);
+
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, dsa);
+#else
+ ts = tidstore_create(TEST_TIDSTORE_MAX_BYTES, MaxHeapTuplesPerPage, NULL);
+#endif
+
+ elog(NOTICE, "testing empty tidstore");
+
+ ItemPointerSet(&tid, 0, FirstOffsetNumber);
+ if (tidstore_lookup_tid(ts, &tid))
+ elog(ERROR, "tidstore_lookup_tid for (0,1) on empty store returned true");
+
+ ItemPointerSet(&tid, MaxBlockNumber, MaxOffsetNumber);
+ if (tidstore_lookup_tid(ts, &tid))
+ elog(ERROR, "tidstore_lookup_tid for (%u,%u) on empty store returned true",
+ MaxBlockNumber, MaxOffsetNumber);
+
+ if (tidstore_num_tids(ts) != 0)
+ elog(ERROR, "tidstore_num_entries on empty store returned non-zero");
+
+ if (tidstore_is_full(ts))
+ elog(ERROR, "tidstore_is_full on empty store returned true");
+
+ iter = tidstore_begin_iterate(ts);
+
+ if (tidstore_iterate_next(iter) != NULL)
+ elog(ERROR, "tidstore_iterate_next on empty store returned TIDs");
+
+ tidstore_end_iterate(iter);
+
+ tidstore_destroy(ts);
+
+#ifdef TEST_SHARED_TIDSTORE
+ dsa_detach(dsa);
+#endif
+}
+
+Datum
+test_tidstore(PG_FUNCTION_ARGS)
+{
+ test_empty();
+
+ elog(NOTICE, "testing basic operations");
+ test_basic(MaxHeapTuplesPerPage);
+ test_basic(10);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_tidstore/test_tidstore.control b/src/test/modules/test_tidstore/test_tidstore.control
new file mode 100644
index 0000000000..9b6bd4638f
--- /dev/null
+++ b/src/test/modules/test_tidstore/test_tidstore.control
@@ -0,0 +1,4 @@
+comment = 'Test code for tidstore'
+default_version = '1.0'
+module_pathname = '$libdir/test_tidstore'
+relocatable = true
--
2.31.1
[application/octet-stream] v32-0002-Move-some-bitmap-logic-out-of-bitmapset.c.patch (6.0K, ../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/17-v32-0002-Move-some-bitmap-logic-out-of-bitmapset.c.patch)
download | inline diff:
From 7fe0c744e052286a8c44716494fe4d644b0e8451 Mon Sep 17 00:00:00 2001
From: John Naylor <[email protected]>
Date: Tue, 6 Dec 2022 13:39:41 +0700
Subject: [PATCH v32 02/18] Move some bitmap logic out of bitmapset.c
Add pg_rightmost_one32/64 functions. This functionality was previously
private to bitmapset.c as the RIGHTMOST_ONE macro. It has practical
use in other contexts, so move to pg_bitutils.h.
Also move the logic for selecting appropriate pg_bitutils functions
based on word size to bitmapset.h for wider visibility and add
appropriate selection for bmw_rightmost_one().
Since the previous macro relied on casting to signedbitmapword,
and the new functions do not, remove that typedef.
Design input and review by Tom Lane
Discussion: https://www.postgresql.org/message-id/CAFBsxsFW2JjTo58jtDB%2B3sZhxMx3t-3evew8%3DAcr%2BGGhC%2BkFaA%40mail.gmail.com
---
src/backend/nodes/bitmapset.c | 34 +-------------------------------
src/include/nodes/bitmapset.h | 16 +++++++++++++--
src/include/port/pg_bitutils.h | 31 +++++++++++++++++++++++++++++
src/tools/pgindent/typedefs.list | 1 -
4 files changed, 46 insertions(+), 36 deletions(-)
diff --git a/src/backend/nodes/bitmapset.c b/src/backend/nodes/bitmapset.c
index 7ba3cf635b..0b2962ed73 100644
--- a/src/backend/nodes/bitmapset.c
+++ b/src/backend/nodes/bitmapset.c
@@ -30,39 +30,7 @@
#define BITMAPSET_SIZE(nwords) \
(offsetof(Bitmapset, words) + (nwords) * sizeof(bitmapword))
-/*----------
- * This is a well-known cute trick for isolating the rightmost one-bit
- * in a word. It assumes two's complement arithmetic. Consider any
- * nonzero value, and focus attention on the rightmost one. The value is
- * then something like
- * xxxxxx10000
- * where x's are unspecified bits. The two's complement negative is formed
- * by inverting all the bits and adding one. Inversion gives
- * yyyyyy01111
- * where each y is the inverse of the corresponding x. Incrementing gives
- * yyyyyy10000
- * and then ANDing with the original value gives
- * 00000010000
- * This works for all cases except original value = zero, where of course
- * we get zero.
- *----------
- */
-#define RIGHTMOST_ONE(x) ((signedbitmapword) (x) & -((signedbitmapword) (x)))
-
-#define HAS_MULTIPLE_ONES(x) ((bitmapword) RIGHTMOST_ONE(x) != (x))
-
-/* Select appropriate bit-twiddling functions for bitmap word size */
-#if BITS_PER_BITMAPWORD == 32
-#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w)
-#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w)
-#define bmw_popcount(w) pg_popcount32(w)
-#elif BITS_PER_BITMAPWORD == 64
-#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w)
-#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w)
-#define bmw_popcount(w) pg_popcount64(w)
-#else
-#error "invalid BITS_PER_BITMAPWORD"
-#endif
+#define HAS_MULTIPLE_ONES(x) (bmw_rightmost_one(x) != (x))
static bool bms_is_empty_internal(const Bitmapset *a);
diff --git a/src/include/nodes/bitmapset.h b/src/include/nodes/bitmapset.h
index 14de6a9ff1..c7e1711147 100644
--- a/src/include/nodes/bitmapset.h
+++ b/src/include/nodes/bitmapset.h
@@ -36,13 +36,11 @@ struct List;
#define BITS_PER_BITMAPWORD 64
typedef uint64 bitmapword; /* must be an unsigned type */
-typedef int64 signedbitmapword; /* must be the matching signed type */
#else
#define BITS_PER_BITMAPWORD 32
typedef uint32 bitmapword; /* must be an unsigned type */
-typedef int32 signedbitmapword; /* must be the matching signed type */
#endif
@@ -73,6 +71,20 @@ typedef enum
BMS_MULTIPLE /* >1 member */
} BMS_Membership;
+/* Select appropriate bit-twiddling functions for bitmap word size */
+#if BITS_PER_BITMAPWORD == 32
+#define bmw_rightmost_one(w) pg_rightmost_one32(w)
+#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w)
+#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w)
+#define bmw_popcount(w) pg_popcount32(w)
+#elif BITS_PER_BITMAPWORD == 64
+#define bmw_rightmost_one(w) pg_rightmost_one64(w)
+#define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w)
+#define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w)
+#define bmw_popcount(w) pg_popcount64(w)
+#else
+#error "invalid BITS_PER_BITMAPWORD"
+#endif
/*
* function prototypes in nodes/bitmapset.c
diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h
index 158ef73a2b..bf7588e075 100644
--- a/src/include/port/pg_bitutils.h
+++ b/src/include/port/pg_bitutils.h
@@ -32,6 +32,37 @@ extern PGDLLIMPORT const uint8 pg_leftmost_one_pos[256];
extern PGDLLIMPORT const uint8 pg_rightmost_one_pos[256];
extern PGDLLIMPORT const uint8 pg_number_of_ones[256];
+/*----------
+ * This is a well-known cute trick for isolating the rightmost one-bit
+ * in a word. It assumes two's complement arithmetic. Consider any
+ * nonzero value, and focus attention on the rightmost one. The value is
+ * then something like
+ * xxxxxx10000
+ * where x's are unspecified bits. The two's complement negative is formed
+ * by inverting all the bits and adding one. Inversion gives
+ * yyyyyy01111
+ * where each y is the inverse of the corresponding x. Incrementing gives
+ * yyyyyy10000
+ * and then ANDing with the original value gives
+ * 00000010000
+ * This works for all cases except original value = zero, where of course
+ * we get zero.
+ *----------
+ */
+static inline uint32
+pg_rightmost_one32(uint32 word)
+{
+ int32 result = (int32) word & -((int32) word);
+ return (uint32) result;
+}
+
+static inline uint64
+pg_rightmost_one64(uint64 word)
+{
+ int64 result = (int64) word & -((int64) word);
+ return (uint64) result;
+}
+
/*
* pg_leftmost_one_pos32
* Returns the position of the most significant set bit in "word",
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index b4058b88c3..fd3d83c781 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3684,7 +3684,6 @@ shmem_request_hook_type
shmem_startup_hook_type
sig_atomic_t
sigjmp_buf
-signedbitmapword
sigset_t
size_t
slist_head
--
2.31.1
[application/octet-stream] v32-0001-Introduce-helper-SIMD-functions-for-small-byte-a.patch (2.9K, ../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/18-v32-0001-Introduce-helper-SIMD-functions-for-small-byte-a.patch)
download | inline diff:
From 51fe658fcecefb2b8c0d826c7d7d6070eb9e878c Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Mon, 24 Oct 2022 14:07:09 +0900
Subject: [PATCH v32 01/18] Introduce helper SIMD functions for small byte
arrays
vector8_min - helper for emulating ">=" semantics
vector8_highbit_mask - used to turn the result of a vector
comparison into a bitmask
Masahiko Sawada
Reviewed by Nathan Bossart, additional adjustments by me
Discussion: https://www.postgresql.org/message-id/CAD21AoDap240WDDdUDE0JMpCmuMMnGajrKrkCRxM7zn9Xk3JRA%40mail.gmail.com
---
src/include/port/simd.h | 47 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)
diff --git a/src/include/port/simd.h b/src/include/port/simd.h
index 1fa6c3bc6c..dfae14e463 100644
--- a/src/include/port/simd.h
+++ b/src/include/port/simd.h
@@ -79,6 +79,7 @@ static inline bool vector8_has_le(const Vector8 v, const uint8 c);
static inline bool vector8_is_highbit_set(const Vector8 v);
#ifndef USE_NO_SIMD
static inline bool vector32_is_highbit_set(const Vector32 v);
+static inline uint32 vector8_highbit_mask(const Vector8 v);
#endif
/* arithmetic operations */
@@ -96,6 +97,7 @@ static inline Vector8 vector8_ssub(const Vector8 v1, const Vector8 v2);
*/
#ifndef USE_NO_SIMD
static inline Vector8 vector8_eq(const Vector8 v1, const Vector8 v2);
+static inline Vector8 vector8_min(const Vector8 v1, const Vector8 v2);
static inline Vector32 vector32_eq(const Vector32 v1, const Vector32 v2);
#endif
@@ -299,6 +301,36 @@ vector32_is_highbit_set(const Vector32 v)
}
#endif /* ! USE_NO_SIMD */
+/*
+ * Return a bitmask formed from the high-bit of each element.
+ */
+#ifndef USE_NO_SIMD
+static inline uint32
+vector8_highbit_mask(const Vector8 v)
+{
+#ifdef USE_SSE2
+ return (uint32) _mm_movemask_epi8(v);
+#elif defined(USE_NEON)
+ /*
+ * Note: There is a faster way to do this, but it returns a uint64 and
+ * and if the caller wanted to extract the bit position using CTZ,
+ * it would have to divide that result by 4.
+ */
+ static const uint8 mask[16] = {
+ 1 << 0, 1 << 1, 1 << 2, 1 << 3,
+ 1 << 4, 1 << 5, 1 << 6, 1 << 7,
+ 1 << 0, 1 << 1, 1 << 2, 1 << 3,
+ 1 << 4, 1 << 5, 1 << 6, 1 << 7,
+ };
+
+ uint8x16_t masked = vandq_u8(vld1q_u8(mask), (uint8x16_t) vshrq_n_s8(v, 7));
+ uint8x16_t maskedhi = vextq_u8(masked, masked, 8);
+
+ return (uint32) vaddvq_u16((uint16x8_t) vzip1q_u8(masked, maskedhi));
+#endif
+}
+#endif /* ! USE_NO_SIMD */
+
/*
* Return the bitwise OR of the inputs
*/
@@ -372,4 +404,19 @@ vector32_eq(const Vector32 v1, const Vector32 v2)
}
#endif /* ! USE_NO_SIMD */
+/*
+ * Given two vectors, return a vector with the minimum element of each.
+ */
+#ifndef USE_NO_SIMD
+static inline Vector8
+vector8_min(const Vector8 v1, const Vector8 v2)
+{
+#ifdef USE_SSE2
+ return _mm_min_epu8(v1, v2);
+#elif defined(USE_NEON)
+ return vminq_u8(v1, v2);
+#endif
+}
+#endif /* ! USE_NO_SIMD */
+
#endif /* SIMD_H */
--
2.31.1
[application/octet-stream] v32-0003-Add-radixtree-template.patch (117.0K, ../CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com/19-v32-0003-Add-radixtree-template.patch)
download | inline diff:
From b88b152cac7c31b49416c4e59e93b3b5f0813759 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Wed, 14 Sep 2022 12:38:51 +0000
Subject: [PATCH v32 03/18] Add radixtree template
WIP: commit message based on template comments
---
src/backend/utils/mmgr/dsa.c | 12 +
src/include/lib/radixtree.h | 2516 +++++++++++++++++
src/include/lib/radixtree_delete_impl.h | 122 +
src/include/lib/radixtree_insert_impl.h | 328 +++
src/include/lib/radixtree_iter_impl.h | 153 +
src/include/lib/radixtree_search_impl.h | 138 +
src/include/utils/dsa.h | 1 +
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_radixtree/.gitignore | 4 +
src/test/modules/test_radixtree/Makefile | 23 +
src/test/modules/test_radixtree/README | 7 +
.../expected/test_radixtree.out | 36 +
src/test/modules/test_radixtree/meson.build | 35 +
.../test_radixtree/sql/test_radixtree.sql | 7 +
.../test_radixtree/test_radixtree--1.0.sql | 8 +
.../modules/test_radixtree/test_radixtree.c | 681 +++++
.../test_radixtree/test_radixtree.control | 4 +
src/tools/pginclude/cpluspluscheck | 6 +
src/tools/pginclude/headerscheck | 6 +
20 files changed, 4089 insertions(+)
create mode 100644 src/include/lib/radixtree.h
create mode 100644 src/include/lib/radixtree_delete_impl.h
create mode 100644 src/include/lib/radixtree_insert_impl.h
create mode 100644 src/include/lib/radixtree_iter_impl.h
create mode 100644 src/include/lib/radixtree_search_impl.h
create mode 100644 src/test/modules/test_radixtree/.gitignore
create mode 100644 src/test/modules/test_radixtree/Makefile
create mode 100644 src/test/modules/test_radixtree/README
create mode 100644 src/test/modules/test_radixtree/expected/test_radixtree.out
create mode 100644 src/test/modules/test_radixtree/meson.build
create mode 100644 src/test/modules/test_radixtree/sql/test_radixtree.sql
create mode 100644 src/test/modules/test_radixtree/test_radixtree--1.0.sql
create mode 100644 src/test/modules/test_radixtree/test_radixtree.c
create mode 100644 src/test/modules/test_radixtree/test_radixtree.control
diff --git a/src/backend/utils/mmgr/dsa.c b/src/backend/utils/mmgr/dsa.c
index f5a62061a3..80555aefff 100644
--- a/src/backend/utils/mmgr/dsa.c
+++ b/src/backend/utils/mmgr/dsa.c
@@ -1024,6 +1024,18 @@ dsa_set_size_limit(dsa_area *area, size_t limit)
LWLockRelease(DSA_AREA_LOCK(area));
}
+size_t
+dsa_get_total_size(dsa_area *area)
+{
+ size_t size;
+
+ LWLockAcquire(DSA_AREA_LOCK(area), LW_SHARED);
+ size = area->control->total_segment_size;
+ LWLockRelease(DSA_AREA_LOCK(area));
+
+ return size;
+}
+
/*
* Aggressively free all spare memory in the hope of returning DSM segments to
* the operating system.
diff --git a/src/include/lib/radixtree.h b/src/include/lib/radixtree.h
new file mode 100644
index 0000000000..e546bd705c
--- /dev/null
+++ b/src/include/lib/radixtree.h
@@ -0,0 +1,2516 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree.h
+ * Template for adaptive radix tree.
+ *
+ * This module employs the idea from the paper "The Adaptive Radix Tree: ARTful
+ * Indexing for Main-Memory Databases" by Viktor Leis, Alfons Kemper, and Thomas
+ * Neumann, 2013. The radix tree uses adaptive node sizes, a small number of node
+ * types, each with a different numbers of elements. Depending on the number of
+ * children, the appropriate node type is used.
+ *
+ * WIP: notes about traditional radix tree trading off span vs height...
+ *
+ * There are two kinds of nodes, inner nodes and leaves. Inner nodes
+ * map partial keys to child pointers.
+ *
+ * The ART paper mentions three ways to implement leaves:
+ *
+ * "- Single-value leaves: The values are stored using an addi-
+ * tional leaf node type which stores one value.
+ * - Multi-value leaves: The values are stored in one of four
+ * different leaf node types, which mirror the structure of
+ * inner nodes, but contain values instead of pointers.
+ * - Combined pointer/value slots: If values fit into point-
+ * ers, no separate node types are necessary. Instead, each
+ * pointer storage location in an inner node can either
+ * store a pointer or a value."
+ *
+ * We chose "multi-value leaves" to avoid the additional pointer traversal
+ * required by "single-value leaves"
+ *
+ * For simplicity, the key is assumed to be 64-bit unsigned integer. The
+ * tree doesn't need to contain paths where the highest bytes of all keys
+ * are zero. That way, the tree's height adapts to the distribution of keys.
+ *
+ * TODO: In the future it might be worthwhile to offer configurability of
+ * leaf implementation for different use cases. Single-values leaves would
+ * give more flexibility in key type, including variable-length keys.
+ *
+ * There are some optimizations not yet implemented, particularly path
+ * compression and lazy path expansion.
+ *
+ * To handle concurrency, we use a single reader-writer lock for the radix
+ * tree. The radix tree is exclusively locked during write operations such
+ * as RT_SET() and RT_DELETE(), and shared locked during read operations
+ * such as RT_SEARCH(). An iteration also holds the shared lock on the radix
+ * tree until it is completed.
+ *
+ * TODO: The current locking mechanism is not optimized for high concurrency
+ * with mixed read-write workloads. In the future it might be worthwhile
+ * to replace it with the Optimistic Lock Coupling or ROWEX mentioned in
+ * the paper "The ART of Practical Synchronization" by the same authors as
+ * the ART paper, 2016.
+ *
+ * WIP: the radix tree nodes don't shrink.
+ *
+ * To generate a radix tree and associated functions for a use case several
+ * macros have to be #define'ed before this file is included. Including
+ * the file #undef's all those, so a new radix tree can be generated
+ * afterwards.
+ * The relevant parameters are:
+ * - RT_PREFIX - prefix for all symbol names generated. A prefix of 'foo'
+ * will result in radix tree type 'foo_radix_tree' and functions like
+ * 'foo_create'/'foo_free' and so forth.
+ * - RT_DECLARE - if defined function prototypes and type declarations are
+ * generated
+ * - RT_DEFINE - if defined function definitions are generated
+ * - RT_SCOPE - in which scope (e.g. extern, static inline) do function
+ * declarations reside
+ * - RT_VALUE_TYPE - the type of the value.
+ *
+ * Optional parameters:
+ * - RT_SHMEM - if defined, the radix tree is created in the DSA area
+ * so that multiple processes can access it simultaneously.
+ * - RT_DEBUG - if defined add stats tracking and debugging functions
+ *
+ * Interface
+ * ---------
+ *
+ * RT_CREATE - Create a new, empty radix tree
+ * RT_FREE - Free the radix tree
+ * RT_SEARCH - Search a key-value pair
+ * RT_SET - Set a key-value pair
+ * RT_BEGIN_ITERATE - Begin iterating through all key-value pairs
+ * RT_ITERATE_NEXT - Return next key-value pair, if any
+ * RT_END_ITER - End iteration
+ * RT_MEMORY_USAGE - Get the memory usage
+ *
+ * Interface for Shared Memory
+ * ---------
+ *
+ * RT_ATTACH - Attach to the radix tree
+ * RT_DETACH - Detach from the radix tree
+ * RT_GET_HANDLE - Return the handle of the radix tree
+ *
+ * Optional Interface
+ * ---------
+ *
+ * RT_DELETE - Delete a key-value pair. Declared/define if RT_USE_DELETE is defined
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/include/lib/radixtree.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+
+#include "lib/stringinfo.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "port/pg_bitutils.h"
+#include "port/simd.h"
+#include "utils/dsa.h"
+#include "utils/memutils.h"
+
+/* helpers */
+#define RT_MAKE_PREFIX(a) CppConcat(a,_)
+#define RT_MAKE_NAME(name) RT_MAKE_NAME_(RT_MAKE_PREFIX(RT_PREFIX),name)
+#define RT_MAKE_NAME_(a,b) CppConcat(a,b)
+
+/* function declarations */
+#define RT_CREATE RT_MAKE_NAME(create)
+#define RT_FREE RT_MAKE_NAME(free)
+#define RT_SEARCH RT_MAKE_NAME(search)
+#ifdef RT_SHMEM
+#define RT_ATTACH RT_MAKE_NAME(attach)
+#define RT_DETACH RT_MAKE_NAME(detach)
+#define RT_GET_HANDLE RT_MAKE_NAME(get_handle)
+#endif
+#define RT_SET RT_MAKE_NAME(set)
+#define RT_BEGIN_ITERATE RT_MAKE_NAME(begin_iterate)
+#define RT_ITERATE_NEXT RT_MAKE_NAME(iterate_next)
+#define RT_END_ITERATE RT_MAKE_NAME(end_iterate)
+#ifdef RT_USE_DELETE
+#define RT_DELETE RT_MAKE_NAME(delete)
+#endif
+#define RT_MEMORY_USAGE RT_MAKE_NAME(memory_usage)
+#ifdef RT_DEBUG
+#define RT_DUMP RT_MAKE_NAME(dump)
+#define RT_DUMP_NODE RT_MAKE_NAME(dump_node)
+#define RT_DUMP_SEARCH RT_MAKE_NAME(dump_search)
+#define RT_STATS RT_MAKE_NAME(stats)
+#endif
+
+/* internal helper functions (no externally visible prototypes) */
+#define RT_NEW_ROOT RT_MAKE_NAME(new_root)
+#define RT_ALLOC_NODE RT_MAKE_NAME(alloc_node)
+#define RT_INIT_NODE RT_MAKE_NAME(init_node)
+#define RT_FREE_NODE RT_MAKE_NAME(free_node)
+#define RT_FREE_RECURSE RT_MAKE_NAME(free_recurse)
+#define RT_EXTEND RT_MAKE_NAME(extend)
+#define RT_SET_EXTEND RT_MAKE_NAME(set_extend)
+#define RT_SWITCH_NODE_KIND RT_MAKE_NAME(grow_node_kind)
+#define RT_COPY_NODE RT_MAKE_NAME(copy_node)
+#define RT_REPLACE_NODE RT_MAKE_NAME(replace_node)
+#define RT_PTR_GET_LOCAL RT_MAKE_NAME(ptr_get_local)
+#define RT_PTR_ALLOC_IS_VALID RT_MAKE_NAME(ptr_stored_is_valid)
+#define RT_NODE_3_SEARCH_EQ RT_MAKE_NAME(node_3_search_eq)
+#define RT_NODE_32_SEARCH_EQ RT_MAKE_NAME(node_32_search_eq)
+#define RT_NODE_3_GET_INSERTPOS RT_MAKE_NAME(node_3_get_insertpos)
+#define RT_NODE_32_GET_INSERTPOS RT_MAKE_NAME(node_32_get_insertpos)
+#define RT_CHUNK_CHILDREN_ARRAY_SHIFT RT_MAKE_NAME(chunk_children_array_shift)
+#define RT_CHUNK_VALUES_ARRAY_SHIFT RT_MAKE_NAME(chunk_values_array_shift)
+#define RT_CHUNK_CHILDREN_ARRAY_DELETE RT_MAKE_NAME(chunk_children_array_delete)
+#define RT_CHUNK_VALUES_ARRAY_DELETE RT_MAKE_NAME(chunk_values_array_delete)
+#define RT_CHUNK_CHILDREN_ARRAY_COPY RT_MAKE_NAME(chunk_children_array_copy)
+#define RT_CHUNK_VALUES_ARRAY_COPY RT_MAKE_NAME(chunk_values_array_copy)
+#define RT_NODE_125_IS_CHUNK_USED RT_MAKE_NAME(node_125_is_chunk_used)
+#define RT_NODE_INNER_125_GET_CHILD RT_MAKE_NAME(node_inner_125_get_child)
+#define RT_NODE_LEAF_125_GET_VALUE RT_MAKE_NAME(node_leaf_125_get_value)
+#define RT_NODE_INNER_256_IS_CHUNK_USED RT_MAKE_NAME(node_inner_256_is_chunk_used)
+#define RT_NODE_LEAF_256_IS_CHUNK_USED RT_MAKE_NAME(node_leaf_256_is_chunk_used)
+#define RT_NODE_INNER_256_GET_CHILD RT_MAKE_NAME(node_inner_256_get_child)
+#define RT_NODE_LEAF_256_GET_VALUE RT_MAKE_NAME(node_leaf_256_get_value)
+#define RT_NODE_INNER_256_SET RT_MAKE_NAME(node_inner_256_set)
+#define RT_NODE_LEAF_256_SET RT_MAKE_NAME(node_leaf_256_set)
+#define RT_NODE_INNER_256_DELETE RT_MAKE_NAME(node_inner_256_delete)
+#define RT_NODE_LEAF_256_DELETE RT_MAKE_NAME(node_leaf_256_delete)
+#define RT_KEY_GET_SHIFT RT_MAKE_NAME(key_get_shift)
+#define RT_SHIFT_GET_MAX_VAL RT_MAKE_NAME(shift_get_max_val)
+#define RT_NODE_SEARCH_INNER RT_MAKE_NAME(node_search_inner)
+#define RT_NODE_SEARCH_LEAF RT_MAKE_NAME(node_search_leaf)
+#define RT_NODE_UPDATE_INNER RT_MAKE_NAME(node_update_inner)
+#define RT_NODE_DELETE_INNER RT_MAKE_NAME(node_delete_inner)
+#define RT_NODE_DELETE_LEAF RT_MAKE_NAME(node_delete_leaf)
+#define RT_NODE_INSERT_INNER RT_MAKE_NAME(node_insert_inner)
+#define RT_NODE_INSERT_LEAF RT_MAKE_NAME(node_insert_leaf)
+#define RT_NODE_INNER_ITERATE_NEXT RT_MAKE_NAME(node_inner_iterate_next)
+#define RT_NODE_LEAF_ITERATE_NEXT RT_MAKE_NAME(node_leaf_iterate_next)
+#define RT_UPDATE_ITER_STACK RT_MAKE_NAME(update_iter_stack)
+#define RT_ITER_UPDATE_KEY RT_MAKE_NAME(iter_update_key)
+#define RT_VERIFY_NODE RT_MAKE_NAME(verify_node)
+
+/* type declarations */
+#define RT_RADIX_TREE RT_MAKE_NAME(radix_tree)
+#define RT_RADIX_TREE_CONTROL RT_MAKE_NAME(radix_tree_control)
+#define RT_ITER RT_MAKE_NAME(iter)
+#ifdef RT_SHMEM
+#define RT_HANDLE RT_MAKE_NAME(handle)
+#endif
+#define RT_NODE RT_MAKE_NAME(node)
+#define RT_NODE_ITER RT_MAKE_NAME(node_iter)
+#define RT_NODE_BASE_3 RT_MAKE_NAME(node_base_3)
+#define RT_NODE_BASE_32 RT_MAKE_NAME(node_base_32)
+#define RT_NODE_BASE_125 RT_MAKE_NAME(node_base_125)
+#define RT_NODE_BASE_256 RT_MAKE_NAME(node_base_256)
+#define RT_NODE_INNER_3 RT_MAKE_NAME(node_inner_3)
+#define RT_NODE_INNER_32 RT_MAKE_NAME(node_inner_32)
+#define RT_NODE_INNER_125 RT_MAKE_NAME(node_inner_125)
+#define RT_NODE_INNER_256 RT_MAKE_NAME(node_inner_256)
+#define RT_NODE_LEAF_3 RT_MAKE_NAME(node_leaf_3)
+#define RT_NODE_LEAF_32 RT_MAKE_NAME(node_leaf_32)
+#define RT_NODE_LEAF_125 RT_MAKE_NAME(node_leaf_125)
+#define RT_NODE_LEAF_256 RT_MAKE_NAME(node_leaf_256)
+#define RT_SIZE_CLASS RT_MAKE_NAME(size_class)
+#define RT_SIZE_CLASS_ELEM RT_MAKE_NAME(size_class_elem)
+#define RT_SIZE_CLASS_INFO RT_MAKE_NAME(size_class_info)
+#define RT_CLASS_3 RT_MAKE_NAME(class_3)
+#define RT_CLASS_32_MIN RT_MAKE_NAME(class_32_min)
+#define RT_CLASS_32_MAX RT_MAKE_NAME(class_32_max)
+#define RT_CLASS_125 RT_MAKE_NAME(class_125)
+#define RT_CLASS_256 RT_MAKE_NAME(class_256)
+
+/* generate forward declarations necessary to use the radix tree */
+#ifdef RT_DECLARE
+
+typedef struct RT_RADIX_TREE RT_RADIX_TREE;
+typedef struct RT_ITER RT_ITER;
+
+#ifdef RT_SHMEM
+typedef dsa_pointer RT_HANDLE;
+#endif
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id);
+RT_SCOPE RT_RADIX_TREE * RT_ATTACH(dsa_area *dsa, dsa_pointer dp);
+RT_SCOPE void RT_DETACH(RT_RADIX_TREE *tree);
+RT_SCOPE RT_HANDLE RT_GET_HANDLE(RT_RADIX_TREE *tree);
+#else
+RT_SCOPE RT_RADIX_TREE * RT_CREATE(MemoryContext ctx);
+#endif
+RT_SCOPE void RT_FREE(RT_RADIX_TREE *tree);
+
+RT_SCOPE bool RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p);
+RT_SCOPE bool RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p);
+#ifdef RT_USE_DELETE
+RT_SCOPE bool RT_DELETE(RT_RADIX_TREE *tree, uint64 key);
+#endif
+
+RT_SCOPE RT_ITER * RT_BEGIN_ITERATE(RT_RADIX_TREE *tree);
+RT_SCOPE bool RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p);
+RT_SCOPE void RT_END_ITERATE(RT_ITER *iter);
+
+RT_SCOPE uint64 RT_MEMORY_USAGE(RT_RADIX_TREE *tree);
+
+#ifdef RT_DEBUG
+RT_SCOPE void RT_DUMP(RT_RADIX_TREE *tree);
+RT_SCOPE void RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key);
+RT_SCOPE void RT_STATS(RT_RADIX_TREE *tree);
+#endif
+
+#endif /* RT_DECLARE */
+
+
+/* generate implementation of the radix tree */
+#ifdef RT_DEFINE
+
+/* The number of bits encoded in one tree level */
+#define RT_NODE_SPAN BITS_PER_BYTE
+
+/* The number of maximum slots in the node */
+#define RT_NODE_MAX_SLOTS (1 << RT_NODE_SPAN)
+
+/* Mask for extracting a chunk from the key */
+#define RT_CHUNK_MASK ((1 << RT_NODE_SPAN) - 1)
+
+/* Maximum shift the radix tree uses */
+#define RT_MAX_SHIFT RT_KEY_GET_SHIFT(UINT64_MAX)
+
+/* Tree level the radix tree uses */
+#define RT_MAX_LEVEL ((sizeof(uint64) * BITS_PER_BYTE) / RT_NODE_SPAN)
+
+/*
+ * Number of bits necessary for isset array in the slot-index node.
+ * Since bitmapword can be 64 bits, the only values that make sense
+ * here are 64 and 128.
+ */
+#define RT_SLOT_IDX_LIMIT (RT_NODE_MAX_SLOTS / 2)
+
+/* Invalid index used in node-125 */
+#define RT_INVALID_SLOT_IDX 0xFF
+
+/* Get a chunk from the key */
+#define RT_GET_KEY_CHUNK(key, shift) ((uint8) (((key) >> (shift)) & RT_CHUNK_MASK))
+
+/* For accessing bitmaps */
+#define RT_BM_IDX(x) ((x) / BITS_PER_BITMAPWORD)
+#define RT_BM_BIT(x) ((x) % BITS_PER_BITMAPWORD)
+
+/*
+ * Node kinds
+ *
+ * The different node kinds are what make the tree "adaptive".
+ *
+ * Each node kind is associated with a different datatype and different
+ * search/set/delete/iterate algorithms adapted for its size. The largest
+ * kind, node256 is basically the same as a traditional radix tree,
+ * and would be most wasteful of memory when sparsely populated. The
+ * smaller nodes expend some additional CPU time to enable a smaller
+ * memory footprint.
+ *
+ * XXX There are 4 node kinds, and this should never be increased,
+ * for several reasons:
+ * 1. With 5 or more kinds, gcc tends to use a jump table for switch
+ * statements.
+ * 2. The 4 kinds can be represented with 2 bits, so we have the option
+ * in the future to tag the node pointer with the kind, even on
+ * platforms with 32-bit pointers. This might speed up node traversal
+ * in trees with highly random node kinds.
+ * 3. We can have multiple size classes per node kind.
+ */
+#define RT_NODE_KIND_3 0x00
+#define RT_NODE_KIND_32 0x01
+#define RT_NODE_KIND_125 0x02
+#define RT_NODE_KIND_256 0x03
+#define RT_NODE_KIND_COUNT 4
+
+/*
+ * Calculate the slab blocksize so that we can allocate at least 32 chunks
+ * from the block.
+ */
+#define RT_SLAB_BLOCK_SIZE(size) \
+ Max((SLAB_DEFAULT_BLOCK_SIZE / (size)) * (size), (size) * 32)
+
+/* Common type for all nodes types */
+typedef struct RT_NODE
+{
+ /*
+ * Number of children. We use uint16 to be able to indicate 256 children
+ * at the fanout of 8.
+ */
+ uint16 count;
+
+ /*
+ * Max capacity for the current size class. Storing this in the
+ * node enables multiple size classes per node kind.
+ * Technically, kinds with a single size class don't need this, so we could
+ * keep this in the individual base types, but the code is simpler this way.
+ * Note: node256 is unique in that it cannot possibly have more than a
+ * single size class, so for that kind we store zero, and uint8 is
+ * sufficient for other kinds.
+ */
+ uint8 fanout;
+
+ /*
+ * Shift indicates which part of the key space is represented by this
+ * node. That is, the key is shifted by 'shift' and the lowest
+ * RT_NODE_SPAN bits are then represented in chunk.
+ */
+ uint8 shift;
+
+ /* Node kind, one per search/set algorithm */
+ uint8 kind;
+} RT_NODE;
+
+
+#define RT_PTR_LOCAL RT_NODE *
+
+#ifdef RT_SHMEM
+#define RT_PTR_ALLOC dsa_pointer
+#else
+#define RT_PTR_ALLOC RT_PTR_LOCAL
+#endif
+
+
+#ifdef RT_SHMEM
+#define RT_INVALID_PTR_ALLOC InvalidDsaPointer
+#else
+#define RT_INVALID_PTR_ALLOC NULL
+#endif
+
+#ifdef RT_SHMEM
+#define RT_LOCK_EXCLUSIVE(tree) LWLockAcquire(&tree->ctl->lock, LW_EXCLUSIVE)
+#define RT_LOCK_SHARED(tree) LWLockAcquire(&tree->ctl->lock, LW_SHARED)
+#define RT_UNLOCK(tree) LWLockRelease(&tree->ctl->lock);
+#else
+#define RT_LOCK_EXCLUSIVE(tree) ((void) 0)
+#define RT_LOCK_SHARED(tree) ((void) 0)
+#define RT_UNLOCK(tree) ((void) 0)
+#endif
+
+/*
+ * Inner nodes and leaf nodes have analogous structure. To distinguish
+ * them at runtime, we take advantage of the fact that the key chunk
+ * is accessed by shifting: Inner tree nodes (shift > 0), store the
+ * pointer to its child node in the slot. In leaf nodes (shift == 0),
+ * the slot contains the value corresponding to the key.
+ */
+#define RT_NODE_IS_LEAF(n) (((RT_PTR_LOCAL) (n))->shift == 0)
+
+#define RT_NODE_MUST_GROW(node) \
+ ((node)->base.n.count == (node)->base.n.fanout)
+
+/*
+ * Base type of each node kinds for leaf and inner nodes.
+ * The base types must be a be able to accommodate the largest size
+ * class for variable-sized node kinds.
+ */
+typedef struct RT_NODE_BASE_3
+{
+ RT_NODE n;
+
+ /* 3 children, for key chunks */
+ uint8 chunks[3];
+} RT_NODE_BASE_3;
+
+typedef struct RT_NODE_BASE_32
+{
+ RT_NODE n;
+
+ /* 32 children, for key chunks */
+ uint8 chunks[32];
+} RT_NODE_BASE_32;
+
+/*
+ * node-125 uses slot_idx array, an array of RT_NODE_MAX_SLOTS length
+ * to store indexes into a second array that contains the values (or
+ * child pointers).
+ */
+typedef struct RT_NODE_BASE_125
+{
+ RT_NODE n;
+
+ /* The index of slots for each fanout */
+ uint8 slot_idxs[RT_NODE_MAX_SLOTS];
+
+ /* bitmap to track which slots are in use */
+ bitmapword isset[RT_BM_IDX(RT_SLOT_IDX_LIMIT)];
+} RT_NODE_BASE_125;
+
+typedef struct RT_NODE_BASE_256
+{
+ RT_NODE n;
+} RT_NODE_BASE_256;
+
+/*
+ * Inner and leaf nodes.
+ *
+ * These are separate because the value type might be different than
+ * something fitting into a pointer-width type.
+ */
+typedef struct RT_NODE_INNER_3
+{
+ RT_NODE_BASE_3 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_3;
+
+typedef struct RT_NODE_LEAF_3
+{
+ RT_NODE_BASE_3 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_3;
+
+typedef struct RT_NODE_INNER_32
+{
+ RT_NODE_BASE_32 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_32;
+
+typedef struct RT_NODE_LEAF_32
+{
+ RT_NODE_BASE_32 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_32;
+
+typedef struct RT_NODE_INNER_125
+{
+ RT_NODE_BASE_125 base;
+
+ /* number of children depends on size class */
+ RT_PTR_ALLOC children[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_INNER_125;
+
+typedef struct RT_NODE_LEAF_125
+{
+ RT_NODE_BASE_125 base;
+
+ /* number of values depends on size class */
+ RT_VALUE_TYPE values[FLEXIBLE_ARRAY_MEMBER];
+} RT_NODE_LEAF_125;
+
+/*
+ * node-256 is the largest node type. This node has an array
+ * for directly storing values (or child pointers in inner nodes).
+ * Unlike other node kinds, it's array size is by definition
+ * fixed.
+ */
+typedef struct RT_NODE_INNER_256
+{
+ RT_NODE_BASE_256 base;
+
+ /* Slots for 256 children */
+ RT_PTR_ALLOC children[RT_NODE_MAX_SLOTS];
+} RT_NODE_INNER_256;
+
+typedef struct RT_NODE_LEAF_256
+{
+ RT_NODE_BASE_256 base;
+
+ /*
+ * Unlike with inner256, zero is a valid value here, so we use a
+ * bitmap to track which slots are in use.
+ */
+ bitmapword isset[RT_BM_IDX(RT_NODE_MAX_SLOTS)];
+
+ /* Slots for 256 values */
+ RT_VALUE_TYPE values[RT_NODE_MAX_SLOTS];
+} RT_NODE_LEAF_256;
+
+/*
+ * Node size classes
+ *
+ * Nodes of different kinds necessarily belong to different size classes.
+ * The main innovation in our implementation compared to the ART paper
+ * is decoupling the notion of size class from kind.
+ *
+ * The size classes within a given node kind have the same underlying
+ * type, but a variable number of children/values. This is possible
+ * because the base type contains small fixed data structures that
+ * work the same way regardless of how full the node is. We store the
+ * node's allocated capacity in the "fanout" member of RT_NODE, to allow
+ * runtime introspection.
+ *
+ * Growing from one node kind to another requires special code for each
+ * case, but growing from one size class to another within the same kind
+ * is basically just allocate + memcpy.
+ *
+ * The size classes have been chosen so that inner nodes on platforms
+ * with 64-bit pointers (and leaf nodes when using a 64-bit key) are
+ * equal to or slightly smaller than some DSA size class.
+ */
+typedef enum RT_SIZE_CLASS
+{
+ RT_CLASS_3 = 0,
+ RT_CLASS_32_MIN,
+ RT_CLASS_32_MAX,
+ RT_CLASS_125,
+ RT_CLASS_256
+} RT_SIZE_CLASS;
+
+/* Information for each size class */
+typedef struct RT_SIZE_CLASS_ELEM
+{
+ const char *name;
+ int fanout;
+
+ /* slab chunk size */
+ Size inner_size;
+ Size leaf_size;
+} RT_SIZE_CLASS_ELEM;
+
+static const RT_SIZE_CLASS_ELEM RT_SIZE_CLASS_INFO[] = {
+ [RT_CLASS_3] = {
+ .name = "radix tree node 3",
+ .fanout = 3,
+ .inner_size = sizeof(RT_NODE_INNER_3) + 3 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_3) + 3 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_32_MIN] = {
+ .name = "radix tree node 15",
+ .fanout = 15,
+ .inner_size = sizeof(RT_NODE_INNER_32) + 15 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_32) + 15 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_32_MAX] = {
+ .name = "radix tree node 32",
+ .fanout = 32,
+ .inner_size = sizeof(RT_NODE_INNER_32) + 32 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_32) + 32 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_125] = {
+ .name = "radix tree node 125",
+ .fanout = 125,
+ .inner_size = sizeof(RT_NODE_INNER_125) + 125 * sizeof(RT_PTR_ALLOC),
+ .leaf_size = sizeof(RT_NODE_LEAF_125) + 125 * sizeof(RT_VALUE_TYPE),
+ },
+ [RT_CLASS_256] = {
+ .name = "radix tree node 256",
+ .fanout = 256,
+ .inner_size = sizeof(RT_NODE_INNER_256),
+ .leaf_size = sizeof(RT_NODE_LEAF_256),
+ },
+};
+
+#define RT_SIZE_CLASS_COUNT lengthof(RT_SIZE_CLASS_INFO)
+
+#ifdef RT_SHMEM
+/* A magic value used to identify our radix tree */
+#define RT_RADIX_TREE_MAGIC 0x54A48167
+#endif
+
+/* Contains the actual tree and ancillary info */
+// WIP: this name is a bit strange
+typedef struct RT_RADIX_TREE_CONTROL
+{
+#ifdef RT_SHMEM
+ RT_HANDLE handle;
+ uint32 magic;
+ LWLock lock;
+#endif
+
+ RT_PTR_ALLOC root;
+ uint64 max_val;
+ uint64 num_keys;
+
+ /* statistics */
+#ifdef RT_DEBUG
+ int32 cnt[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE_CONTROL;
+
+/* Entry point for allocating and accessing the tree */
+typedef struct RT_RADIX_TREE
+{
+ MemoryContext context;
+
+ /* pointing to either local memory or DSA */
+ RT_RADIX_TREE_CONTROL *ctl;
+
+#ifdef RT_SHMEM
+ dsa_area *dsa;
+#else
+ MemoryContextData *inner_slabs[RT_SIZE_CLASS_COUNT];
+ MemoryContextData *leaf_slabs[RT_SIZE_CLASS_COUNT];
+#endif
+} RT_RADIX_TREE;
+
+/*
+ * Iteration support.
+ *
+ * Iterating the radix tree returns each pair of key and value in the ascending
+ * order of the key. To support this, the we iterate nodes of each level.
+ *
+ * RT_NODE_ITER struct is used to track the iteration within a node.
+ *
+ * RT_ITER is the struct for iteration of the radix tree, and uses RT_NODE_ITER
+ * in order to track the iteration of each level. During iteration, we also
+ * construct the key whenever updating the node iteration information, e.g., when
+ * advancing the current index within the node or when moving to the next node
+ * at the same level.
+ *
+ * XXX: Currently we allow only one process to do iteration. Therefore, rt_node_iter
+ * has the local pointers to nodes, rather than RT_PTR_ALLOC.
+ * We need either a safeguard to disallow other processes to begin the iteration
+ * while one process is doing or to allow multiple processes to do the iteration.
+ */
+typedef struct RT_NODE_ITER
+{
+ RT_PTR_LOCAL node; /* current node being iterated */
+ int current_idx; /* current position. -1 for initial value */
+} RT_NODE_ITER;
+
+typedef struct RT_ITER
+{
+ RT_RADIX_TREE *tree;
+
+ /* Track the iteration on nodes of each level */
+ RT_NODE_ITER stack[RT_MAX_LEVEL];
+ int stack_len;
+
+ /* The key is constructed during iteration */
+ uint64 key;
+} RT_ITER;
+
+
+static void RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_PTR_ALLOC child);
+static bool RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_VALUE_TYPE *value_p);
+
+/* verification (available only with assertion) */
+static void RT_VERIFY_NODE(RT_PTR_LOCAL node);
+
+/* Get the local address of an allocated node */
+static inline RT_PTR_LOCAL
+RT_PTR_GET_LOCAL(RT_RADIX_TREE *tree, RT_PTR_ALLOC node)
+{
+#ifdef RT_SHMEM
+ return dsa_get_address(tree->dsa, (dsa_pointer) node);
+#else
+ return node;
+#endif
+}
+
+static inline bool
+RT_PTR_ALLOC_IS_VALID(RT_PTR_ALLOC ptr)
+{
+#ifdef RT_SHMEM
+ return DsaPointerIsValid(ptr);
+#else
+ return PointerIsValid(ptr);
+#endif
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_3_SEARCH_EQ(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+ int idx = -1;
+
+ for (int i = 0; i < node->n.count; i++)
+ {
+ if (node->chunks[i] == chunk)
+ {
+ idx = i;
+ break;
+ }
+ }
+
+ return idx;
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_3_GET_INSERTPOS(RT_NODE_BASE_3 *node, uint8 chunk)
+{
+ int idx;
+
+ for (idx = 0; idx < node->n.count; idx++)
+ {
+ if (node->chunks[idx] >= chunk)
+ break;
+ }
+
+ return idx;
+}
+
+/*
+ * Return index of the first element in the node's chunk array that equals
+ * 'chunk'. Return -1 if there is no such element.
+ */
+static inline int
+RT_NODE_32_SEARCH_EQ(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+ int count = node->n.count;
+#ifndef USE_NO_SIMD
+ Vector8 spread_chunk;
+ Vector8 haystack1;
+ Vector8 haystack2;
+ Vector8 cmp1;
+ Vector8 cmp2;
+ uint32 bitfield;
+ int index_simd = -1;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+ int index = -1;
+
+ for (int i = 0; i < count; i++)
+ {
+ if (node->chunks[i] == chunk)
+ {
+ index = i;
+ break;
+ }
+ }
+#endif
+
+#ifndef USE_NO_SIMD
+ /* replicate the search key */
+ spread_chunk = vector8_broadcast(chunk);
+
+ /* compare to all 32 keys stored in the node */
+ vector8_load(&haystack1, &node->chunks[0]);
+ vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+ cmp1 = vector8_eq(spread_chunk, haystack1);
+ cmp2 = vector8_eq(spread_chunk, haystack2);
+
+ /* convert comparison to a bitfield */
+ bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+
+ /* mask off invalid entries */
+ bitfield &= ((UINT64CONST(1) << count) - 1);
+
+ /* convert bitfield to index by counting trailing zeros */
+ if (bitfield)
+ index_simd = pg_rightmost_one_pos32(bitfield);
+
+ Assert(index_simd == index);
+ return index_simd;
+#else
+ return index;
+#endif
+}
+
+/*
+ * Return index of the chunk and slot arrays for inserting into the node,
+ * such that the chunk array remains ordered.
+ */
+static inline int
+RT_NODE_32_GET_INSERTPOS(RT_NODE_BASE_32 *node, uint8 chunk)
+{
+ int count = node->n.count;
+#ifndef USE_NO_SIMD
+ Vector8 spread_chunk;
+ Vector8 haystack1;
+ Vector8 haystack2;
+ Vector8 cmp1;
+ Vector8 cmp2;
+ Vector8 min1;
+ Vector8 min2;
+ uint32 bitfield;
+ int index_simd;
+#endif
+
+#if defined(USE_NO_SIMD) || defined(USE_ASSERT_CHECKING)
+ int index;
+
+ for (index = 0; index < count; index++)
+ {
+ /*
+ * This is coded with '>=' to match what we can do with SIMD,
+ * with an assert to keep us honest.
+ */
+ if (node->chunks[index] >= chunk)
+ {
+ Assert(node->chunks[index] != chunk);
+ break;
+ }
+ }
+#endif
+
+#ifndef USE_NO_SIMD
+ /*
+ * This is a bit more complicated than RT_NODE_32_SEARCH_EQ(), because
+ * no unsigned uint8 comparison instruction exists, at least for SSE2. So
+ * we need to play some trickery using vector8_min() to effectively get
+ * >=. There'll never be any equal elements in current uses, but that's
+ * what we get here...
+ */
+ spread_chunk = vector8_broadcast(chunk);
+ vector8_load(&haystack1, &node->chunks[0]);
+ vector8_load(&haystack2, &node->chunks[sizeof(Vector8)]);
+ min1 = vector8_min(spread_chunk, haystack1);
+ min2 = vector8_min(spread_chunk, haystack2);
+ cmp1 = vector8_eq(spread_chunk, min1);
+ cmp2 = vector8_eq(spread_chunk, min2);
+ bitfield = vector8_highbit_mask(cmp1) | (vector8_highbit_mask(cmp2) << sizeof(Vector8));
+ bitfield &= ((UINT64CONST(1) << count) - 1);
+
+ if (bitfield)
+ index_simd = pg_rightmost_one_pos32(bitfield);
+ else
+ index_simd = count;
+
+ Assert(index_simd == index);
+ return index_simd;
+#else
+ return index;
+#endif
+}
+
+
+/*
+ * Functions to manipulate both chunks array and children/values array.
+ * These are used for node-3 and node-32.
+ */
+
+/* Shift the elements right at 'idx' by one */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_SHIFT(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+ memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+ memmove(&(children[idx + 1]), &(children[idx]), sizeof(RT_PTR_ALLOC) * (count - idx));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_SHIFT(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+ memmove(&(chunks[idx + 1]), &(chunks[idx]), sizeof(uint8) * (count - idx));
+ memmove(&(values[idx + 1]), &(values[idx]), sizeof(RT_VALUE_TYPE) * (count - idx));
+}
+
+/* Delete the element at 'idx' */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_DELETE(uint8 *chunks, RT_PTR_ALLOC *children, int count, int idx)
+{
+ memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+ memmove(&(children[idx]), &(children[idx + 1]), sizeof(RT_PTR_ALLOC) * (count - idx - 1));
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_DELETE(uint8 *chunks, RT_VALUE_TYPE *values, int count, int idx)
+{
+ memmove(&(chunks[idx]), &(chunks[idx + 1]), sizeof(uint8) * (count - idx - 1));
+ memmove(&(values[idx]), &(values[idx + 1]), sizeof(RT_VALUE_TYPE) * (count - idx - 1));
+}
+
+/* Copy both chunks and children/values arrays */
+static inline void
+RT_CHUNK_CHILDREN_ARRAY_COPY(uint8 *src_chunks, RT_PTR_ALLOC *src_children,
+ uint8 *dst_chunks, RT_PTR_ALLOC *dst_children)
+{
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const Size chunk_size = sizeof(uint8) * fanout;
+ const Size children_size = sizeof(RT_PTR_ALLOC) * fanout;
+
+ memcpy(dst_chunks, src_chunks, chunk_size);
+ memcpy(dst_children, src_children, children_size);
+}
+
+static inline void
+RT_CHUNK_VALUES_ARRAY_COPY(uint8 *src_chunks, RT_VALUE_TYPE *src_values,
+ uint8 *dst_chunks, RT_VALUE_TYPE *dst_values)
+{
+ const int fanout = RT_SIZE_CLASS_INFO[RT_CLASS_3].fanout;
+ const Size chunk_size = sizeof(uint8) * fanout;
+ const Size values_size = sizeof(RT_VALUE_TYPE) * fanout;
+
+ memcpy(dst_chunks, src_chunks, chunk_size);
+ memcpy(dst_values, src_values, values_size);
+}
+
+/* Functions to manipulate inner and leaf node-125 */
+
+/* Does the given chunk in the node has the value? */
+static inline bool
+RT_NODE_125_IS_CHUNK_USED(RT_NODE_BASE_125 *node, uint8 chunk)
+{
+ return node->slot_idxs[chunk] != RT_INVALID_SLOT_IDX;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_125_GET_CHILD(RT_NODE_INNER_125 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ return node->children[node->base.slot_idxs[chunk]];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_125_GET_VALUE(RT_NODE_LEAF_125 *node, uint8 chunk)
+{
+ Assert(RT_NODE_IS_LEAF(node));
+ Assert(((RT_NODE_BASE_125 *) node)->slot_idxs[chunk] != RT_INVALID_SLOT_IDX);
+ return node->values[node->base.slot_idxs[chunk]];
+}
+
+/* Functions to manipulate inner and leaf node-256 */
+
+/* Return true if the slot corresponding to the given chunk is in use */
+static inline bool
+RT_NODE_INNER_256_IS_CHUNK_USED(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ return node->children[chunk] != RT_INVALID_PTR_ALLOC;
+}
+
+static inline bool
+RT_NODE_LEAF_256_IS_CHUNK_USED(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ return (node->isset[idx] & ((bitmapword) 1 << bitnum)) != 0;
+}
+
+static inline RT_PTR_ALLOC
+RT_NODE_INNER_256_GET_CHILD(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ Assert(RT_NODE_INNER_256_IS_CHUNK_USED(node, chunk));
+ return node->children[chunk];
+}
+
+static inline RT_VALUE_TYPE
+RT_NODE_LEAF_256_GET_VALUE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ Assert(RT_NODE_IS_LEAF(node));
+ Assert(RT_NODE_LEAF_256_IS_CHUNK_USED(node, chunk));
+ return node->values[chunk];
+}
+
+/* Set the child in the node-256 */
+static inline void
+RT_NODE_INNER_256_SET(RT_NODE_INNER_256 *node, uint8 chunk, RT_PTR_ALLOC child)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ node->children[chunk] = child;
+}
+
+/* Set the value in the node-256 */
+static inline void
+RT_NODE_LEAF_256_SET(RT_NODE_LEAF_256 *node, uint8 chunk, RT_VALUE_TYPE value)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ node->isset[idx] |= ((bitmapword) 1 << bitnum);
+ node->values[chunk] = value;
+}
+
+/* Set the slot at the given chunk position */
+static inline void
+RT_NODE_INNER_256_DELETE(RT_NODE_INNER_256 *node, uint8 chunk)
+{
+ Assert(!RT_NODE_IS_LEAF(node));
+ node->children[chunk] = RT_INVALID_PTR_ALLOC;
+}
+
+static inline void
+RT_NODE_LEAF_256_DELETE(RT_NODE_LEAF_256 *node, uint8 chunk)
+{
+ int idx = RT_BM_IDX(chunk);
+ int bitnum = RT_BM_BIT(chunk);
+
+ Assert(RT_NODE_IS_LEAF(node));
+ node->isset[idx] &= ~((bitmapword) 1 << bitnum);
+}
+
+/*
+ * Return the largest shift that will allowing storing the given key.
+ */
+static inline int
+RT_KEY_GET_SHIFT(uint64 key)
+{
+ if (key == 0)
+ return 0;
+ else
+ return (pg_leftmost_one_pos64(key) / RT_NODE_SPAN) * RT_NODE_SPAN;
+}
+
+/*
+ * Return the max value that can be stored in the tree with the given shift.
+ */
+static uint64
+RT_SHIFT_GET_MAX_VAL(int shift)
+{
+ if (shift == RT_MAX_SHIFT)
+ return UINT64_MAX;
+
+ return (UINT64CONST(1) << (shift + RT_NODE_SPAN)) - 1;
+}
+
+/*
+ * Allocate a new node with the given node kind.
+ */
+static RT_PTR_ALLOC
+RT_ALLOC_NODE(RT_RADIX_TREE *tree, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+ RT_PTR_ALLOC allocnode;
+ size_t allocsize;
+
+ if (is_leaf)
+ allocsize = RT_SIZE_CLASS_INFO[size_class].leaf_size;
+ else
+ allocsize = RT_SIZE_CLASS_INFO[size_class].inner_size;
+
+#ifdef RT_SHMEM
+ allocnode = dsa_allocate(tree->dsa, allocsize);
+#else
+ if (is_leaf)
+ allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->leaf_slabs[size_class],
+ allocsize);
+ else
+ allocnode = (RT_PTR_ALLOC) MemoryContextAlloc(tree->inner_slabs[size_class],
+ allocsize);
+#endif
+
+#ifdef RT_DEBUG
+ /* update the statistics */
+ tree->ctl->cnt[size_class]++;
+#endif
+
+ return allocnode;
+}
+
+/* Initialize the node contents */
+static inline void
+RT_INIT_NODE(RT_PTR_LOCAL node, uint8 kind, RT_SIZE_CLASS size_class, bool is_leaf)
+{
+ if (is_leaf)
+ MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].leaf_size);
+ else
+ MemSet(node, 0, RT_SIZE_CLASS_INFO[size_class].inner_size);
+
+ node->kind = kind;
+
+ if (kind == RT_NODE_KIND_256)
+ /* See comment for the RT_NODE type */
+ Assert(node->fanout == 0);
+ else
+ node->fanout = RT_SIZE_CLASS_INFO[size_class].fanout;
+
+ /* Initialize slot_idxs to invalid values */
+ if (kind == RT_NODE_KIND_125)
+ {
+ RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+
+ memset(n125->slot_idxs, RT_INVALID_SLOT_IDX, sizeof(n125->slot_idxs));
+ }
+}
+
+/*
+ * Create a new node as the root. Subordinate nodes will be created during
+ * the insertion.
+ */
+static pg_noinline void
+RT_NEW_ROOT(RT_RADIX_TREE *tree, uint64 key)
+{
+ int shift = RT_KEY_GET_SHIFT(key);
+ bool is_leaf = shift == 0;
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(newnode, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ newnode->shift = shift;
+ tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(shift);
+ tree->ctl->root = allocnode;
+}
+
+static inline void
+RT_COPY_NODE(RT_PTR_LOCAL newnode, RT_PTR_LOCAL oldnode)
+{
+ newnode->shift = oldnode->shift;
+ newnode->count = oldnode->count;
+}
+
+/*
+ * Given a new allocated node and an old node, initalize the new
+ * node with the necessary fields and return its local pointer.
+ */
+static inline RT_PTR_LOCAL
+RT_SWITCH_NODE_KIND(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, RT_PTR_LOCAL node,
+ uint8 new_kind, uint8 new_class, bool is_leaf)
+{
+ RT_PTR_LOCAL newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(newnode, new_kind, new_class, is_leaf);
+ RT_COPY_NODE(newnode, node);
+
+ return newnode;
+}
+
+/* Free the given node */
+static void
+RT_FREE_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode)
+{
+ /* If we're deleting the root node, make the tree empty */
+ if (tree->ctl->root == allocnode)
+ {
+ tree->ctl->root = RT_INVALID_PTR_ALLOC;
+ tree->ctl->max_val = 0;
+ }
+
+#ifdef RT_DEBUG
+ {
+ int i;
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+ /* update the statistics */
+ for (i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ if (node->fanout == RT_SIZE_CLASS_INFO[i].fanout)
+ break;
+ }
+
+ /* fanout of node256 is intentionally 0 */
+ if (i == RT_SIZE_CLASS_COUNT)
+ i = RT_CLASS_256;
+
+ tree->ctl->cnt[i]--;
+ Assert(tree->ctl->cnt[i] >= 0);
+ }
+#endif
+
+#ifdef RT_SHMEM
+ dsa_free(tree->dsa, allocnode);
+#else
+ pfree(allocnode);
+#endif
+}
+
+/* Update the parent's pointer when growing a node */
+static inline void
+RT_NODE_UPDATE_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC new_child)
+{
+#define RT_ACTION_UPDATE
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+#undef RT_ACTION_UPDATE
+}
+
+/*
+ * Replace old_child with new_child, and free the old one.
+ */
+static inline void
+RT_REPLACE_NODE(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent,
+ RT_PTR_ALLOC stored_old_child, RT_PTR_LOCAL old_child,
+ RT_PTR_ALLOC new_child, uint64 key)
+{
+#ifdef USE_ASSERT_CHECKING
+ RT_PTR_LOCAL new = RT_PTR_GET_LOCAL(tree, new_child);
+
+ Assert(old_child->shift == new->shift);
+ Assert(old_child->count == new->count);
+#endif
+
+ if (parent == old_child)
+ {
+ /* Replace the root node with the new larger node */
+ tree->ctl->root = new_child;
+ }
+ else
+ RT_NODE_UPDATE_INNER(parent, key, new_child);
+
+ RT_FREE_NODE(tree, stored_old_child);
+}
+
+/*
+ * The radix tree doesn't have sufficient height. Extend the radix tree so
+ * it can store the key.
+ */
+static pg_noinline void
+RT_EXTEND(RT_RADIX_TREE *tree, uint64 key)
+{
+ int target_shift;
+ RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+ int shift = root->shift + RT_NODE_SPAN;
+
+ target_shift = RT_KEY_GET_SHIFT(key);
+
+ /* Grow tree from 'shift' to 'target_shift' */
+ while (shift <= target_shift)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL node;
+ RT_NODE_INNER_3 *n3;
+
+ allocnode = RT_ALLOC_NODE(tree, RT_CLASS_3, true);
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ RT_INIT_NODE(node, RT_NODE_KIND_3, RT_CLASS_3, true);
+ node->shift = shift;
+ node->count = 1;
+
+ n3 = (RT_NODE_INNER_3 *) node;
+ n3->base.chunks[0] = 0;
+ n3->children[0] = tree->ctl->root;
+
+ /* Update the root */
+ tree->ctl->root = allocnode;
+
+ shift += RT_NODE_SPAN;
+ }
+
+ tree->ctl->max_val = RT_SHIFT_GET_MAX_VAL(target_shift);
+}
+
+/*
+ * The radix tree doesn't have inner and leaf nodes for given key-value pair.
+ * Insert inner and leaf nodes from 'node' to bottom.
+ */
+static pg_noinline void
+RT_SET_EXTEND(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p, RT_PTR_LOCAL parent,
+ RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node)
+{
+ int shift = node->shift;
+
+ Assert(RT_PTR_GET_LOCAL(tree, stored_node) == node);
+
+ while (shift >= RT_NODE_SPAN)
+ {
+ RT_PTR_ALLOC allocchild;
+ RT_PTR_LOCAL newchild;
+ int newshift = shift - RT_NODE_SPAN;
+ bool is_leaf = newshift == 0;
+
+ allocchild = RT_ALLOC_NODE(tree, RT_CLASS_3, is_leaf);
+ newchild = RT_PTR_GET_LOCAL(tree, allocchild);
+ RT_INIT_NODE(newchild, RT_NODE_KIND_3, RT_CLASS_3, is_leaf);
+ newchild->shift = newshift;
+ RT_NODE_INSERT_INNER(tree, parent, stored_node, node, key, allocchild);
+
+ parent = node;
+ node = newchild;
+ stored_node = allocchild;
+ shift -= RT_NODE_SPAN;
+ }
+
+ RT_NODE_INSERT_LEAF(tree, parent, stored_node, node, key, value_p);
+ tree->ctl->num_keys++;
+}
+
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the child
+ * pointer is set to child_p.
+ */
+static inline bool
+RT_NODE_SEARCH_INNER(RT_PTR_LOCAL node, uint64 key, RT_PTR_ALLOC *child_p)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Return true if the key is found, otherwise return false. On success, the pointer
+ * to the value is set to value_p.
+ */
+static inline bool
+RT_NODE_SEARCH_LEAF(RT_PTR_LOCAL node, uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_search_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Search for the child pointer corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_INNER(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Search for the value corresponding to 'key' in the given node.
+ *
+ * Delete the node and return true if the key is found, otherwise return false.
+ */
+static inline bool
+RT_NODE_DELETE_LEAF(RT_PTR_LOCAL node, uint64 key)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_delete_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+#endif
+
+/*
+ * Insert "child" into "node".
+ *
+ * "parent" is the parent of "node", so the grandparent of the child.
+ * If the node we're inserting into needs to grow, we update the parent's
+ * child pointer with the pointer to the new larger node.
+ */
+static void
+RT_NODE_INSERT_INNER(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_PTR_ALLOC child)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/* Like RT_NODE_INSERT_INNER, but for leaf nodes */
+static bool
+RT_NODE_INSERT_LEAF(RT_RADIX_TREE *tree, RT_PTR_LOCAL parent, RT_PTR_ALLOC stored_node, RT_PTR_LOCAL node,
+ uint64 key, RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_insert_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Create the radix tree in the given memory context and return it.
+ */
+RT_SCOPE RT_RADIX_TREE *
+#ifdef RT_SHMEM
+RT_CREATE(MemoryContext ctx, dsa_area *dsa, int tranche_id)
+#else
+RT_CREATE(MemoryContext ctx)
+#endif
+{
+ RT_RADIX_TREE *tree;
+ MemoryContext old_ctx;
+#ifdef RT_SHMEM
+ dsa_pointer dp;
+#endif
+
+ old_ctx = MemoryContextSwitchTo(ctx);
+
+ tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+ tree->context = ctx;
+
+#ifdef RT_SHMEM
+ tree->dsa = dsa;
+ dp = dsa_allocate0(dsa, sizeof(RT_RADIX_TREE_CONTROL));
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, dp);
+ tree->ctl->handle = dp;
+ tree->ctl->magic = RT_RADIX_TREE_MAGIC;
+ LWLockInitialize(&tree->ctl->lock, tranche_id);
+#else
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) palloc0(sizeof(RT_RADIX_TREE_CONTROL));
+
+ /* Create a slab context for each size class */
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ RT_SIZE_CLASS_ELEM size_class = RT_SIZE_CLASS_INFO[i];
+ size_t inner_blocksize = RT_SLAB_BLOCK_SIZE(size_class.inner_size);
+ size_t leaf_blocksize = RT_SLAB_BLOCK_SIZE(size_class.leaf_size);
+
+ tree->inner_slabs[i] = SlabContextCreate(ctx,
+ size_class.name,
+ inner_blocksize,
+ size_class.inner_size);
+ tree->leaf_slabs[i] = SlabContextCreate(ctx,
+ size_class.name,
+ leaf_blocksize,
+ size_class.leaf_size);
+ }
+#endif
+
+ tree->ctl->root = RT_INVALID_PTR_ALLOC;
+
+ MemoryContextSwitchTo(old_ctx);
+
+ return tree;
+}
+
+#ifdef RT_SHMEM
+RT_SCOPE RT_RADIX_TREE *
+RT_ATTACH(dsa_area *dsa, RT_HANDLE handle)
+{
+ RT_RADIX_TREE *tree;
+ dsa_pointer control;
+
+ tree = (RT_RADIX_TREE *) palloc0(sizeof(RT_RADIX_TREE));
+
+ /* Find the control object in shard memory */
+ control = handle;
+
+ tree->dsa = dsa;
+ tree->ctl = (RT_RADIX_TREE_CONTROL *) dsa_get_address(dsa, control);
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+ return tree;
+}
+
+RT_SCOPE void
+RT_DETACH(RT_RADIX_TREE *tree)
+{
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ pfree(tree);
+}
+
+RT_SCOPE RT_HANDLE
+RT_GET_HANDLE(RT_RADIX_TREE *tree)
+{
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ return tree->ctl->handle;
+}
+
+/*
+ * Recursively free all nodes allocated to the DSA area.
+ */
+static void
+RT_FREE_RECURSE(RT_RADIX_TREE *tree, RT_PTR_ALLOC ptr)
+{
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, ptr);
+
+ check_stack_depth();
+ CHECK_FOR_INTERRUPTS();
+
+ /* The leaf node doesn't have child pointers */
+ if (RT_NODE_IS_LEAF(node))
+ {
+ dsa_free(tree->dsa, ptr);
+ return;
+ }
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+ for (int i = 0; i < n3->base.n.count; i++)
+ RT_FREE_RECURSE(tree, n3->children[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+ for (int i = 0; i < n32->base.n.count; i++)
+ RT_FREE_RECURSE(tree, n32->children[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) node;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+ continue;
+
+ RT_FREE_RECURSE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+ }
+
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ RT_FREE_RECURSE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+ }
+
+ break;
+ }
+ }
+
+ /* Free the inner node */
+ dsa_free(tree->dsa, ptr);
+}
+#endif
+
+/*
+ * Free the given radix tree.
+ */
+RT_SCOPE void
+RT_FREE(RT_RADIX_TREE *tree)
+{
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+
+ /* Free all memory used for radix tree nodes */
+ if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ RT_FREE_RECURSE(tree, tree->ctl->root);
+
+ /*
+ * Vandalize the control block to help catch programming error where
+ * other backends access the memory formerly occupied by this radix tree.
+ */
+ tree->ctl->magic = 0;
+ dsa_free(tree->dsa, tree->ctl->handle);
+#else
+ pfree(tree->ctl);
+
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ MemoryContextDelete(tree->inner_slabs[i]);
+ MemoryContextDelete(tree->leaf_slabs[i]);
+ }
+#endif
+
+ pfree(tree);
+}
+
+/*
+ * Set key to value. If the entry already exists, we update its value to 'value'
+ * and return true. Returns false if entry doesn't yet exist.
+ */
+RT_SCOPE bool
+RT_SET(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+ int shift;
+ bool updated;
+ RT_PTR_LOCAL parent;
+ RT_PTR_ALLOC stored_child;
+ RT_PTR_LOCAL child;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ RT_LOCK_EXCLUSIVE(tree);
+
+ /* Empty tree, create the root */
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ RT_NEW_ROOT(tree, key);
+
+ /* Extend the tree if necessary */
+ if (key > tree->ctl->max_val)
+ RT_EXTEND(tree, key);
+
+ stored_child = tree->ctl->root;
+ parent = RT_PTR_GET_LOCAL(tree, stored_child);
+ shift = parent->shift;
+
+ /* Descend the tree until we reach a leaf node */
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC new_child = RT_INVALID_PTR_ALLOC;
+
+ child = RT_PTR_GET_LOCAL(tree, stored_child);
+
+ if (RT_NODE_IS_LEAF(child))
+ break;
+
+ if (!RT_NODE_SEARCH_INNER(child, key, &new_child))
+ {
+ RT_SET_EXTEND(tree, key, value_p, parent, stored_child, child);
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ parent = child;
+ stored_child = new_child;
+ shift -= RT_NODE_SPAN;
+ }
+
+ updated = RT_NODE_INSERT_LEAF(tree, parent, stored_child, child, key, value_p);
+
+ /* Update the statistics */
+ if (!updated)
+ tree->ctl->num_keys++;
+
+ RT_UNLOCK(tree);
+ return updated;
+}
+
+/*
+ * Search the given key in the radix tree. Return true if there is the key,
+ * otherwise return false. On success, we set the value to *value_p so it must
+ * not be NULL.
+ */
+RT_SCOPE bool
+RT_SEARCH(RT_RADIX_TREE *tree, uint64 key, RT_VALUE_TYPE *value_p)
+{
+ RT_PTR_LOCAL node;
+ int shift;
+ bool found;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+ Assert(value_p != NULL);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ node = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+ shift = node->shift;
+
+ /* Descend the tree until a leaf node */
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+ if (RT_NODE_IS_LEAF(node))
+ break;
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ node = RT_PTR_GET_LOCAL(tree, child);
+ shift -= RT_NODE_SPAN;
+ }
+
+ found = RT_NODE_SEARCH_LEAF(node, key, value_p);
+
+ RT_UNLOCK(tree);
+ return found;
+}
+
+#ifdef RT_USE_DELETE
+/*
+ * Delete the given key from the radix tree. Return true if the key is found (and
+ * deleted), otherwise do nothing and return false.
+ */
+RT_SCOPE bool
+RT_DELETE(RT_RADIX_TREE *tree, uint64 key)
+{
+ RT_PTR_LOCAL node;
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_ALLOC stack[RT_MAX_LEVEL] = {0};
+ int shift;
+ int level;
+ bool deleted;
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ RT_LOCK_EXCLUSIVE(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root) || key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ /*
+ * Descend the tree to search the key while building a stack of nodes we
+ * visited.
+ */
+ allocnode = tree->ctl->root;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift = node->shift;
+ level = -1;
+ while (shift > 0)
+ {
+ RT_PTR_ALLOC child = RT_INVALID_PTR_ALLOC;
+
+ /* Push the current node to the stack */
+ stack[++level] = allocnode;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ {
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ allocnode = child;
+ shift -= RT_NODE_SPAN;
+ }
+
+ /* Delete the key from the leaf node if exists */
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ deleted = RT_NODE_DELETE_LEAF(node, key);
+
+ if (!deleted)
+ {
+ /* no key is found in the leaf node */
+ RT_UNLOCK(tree);
+ return false;
+ }
+
+ /* Found the key to delete. Update the statistics */
+ tree->ctl->num_keys--;
+
+ /*
+ * Return if the leaf node still has keys and we don't need to delete the
+ * node.
+ */
+ if (node->count > 0)
+ {
+ RT_UNLOCK(tree);
+ return true;
+ }
+
+ /* Free the empty leaf node */
+ RT_FREE_NODE(tree, allocnode);
+
+ /* Delete the key in inner nodes recursively */
+ while (level >= 0)
+ {
+ allocnode = stack[level--];
+
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ deleted = RT_NODE_DELETE_INNER(node, key);
+ Assert(deleted);
+
+ /* If the node didn't become empty, we stop deleting the key */
+ if (node->count > 0)
+ break;
+
+ /* The node became empty */
+ RT_FREE_NODE(tree, allocnode);
+ }
+
+ RT_UNLOCK(tree);
+ return true;
+}
+#endif
+
+static inline void
+RT_ITER_UPDATE_KEY(RT_ITER *iter, uint8 chunk, uint8 shift)
+{
+ iter->key &= ~(((uint64) RT_CHUNK_MASK) << shift);
+ iter->key |= (((uint64) chunk) << shift);
+}
+
+/*
+ * Advance the slot in the inner node. Return the child if exists, otherwise
+ * null.
+ */
+static inline RT_PTR_LOCAL
+RT_NODE_INNER_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter)
+{
+#define RT_NODE_LEVEL_INNER
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_INNER
+}
+
+/*
+ * Advance the slot in the leaf node. On success, return true and the value
+ * is set to value_p, otherwise return false.
+ */
+static inline bool
+RT_NODE_LEAF_ITERATE_NEXT(RT_ITER *iter, RT_NODE_ITER *node_iter,
+ RT_VALUE_TYPE *value_p)
+{
+#define RT_NODE_LEVEL_LEAF
+#include "lib/radixtree_iter_impl.h"
+#undef RT_NODE_LEVEL_LEAF
+}
+
+/*
+ * Update each node_iter for inner nodes in the iterator node stack.
+ */
+static void
+RT_UPDATE_ITER_STACK(RT_ITER *iter, RT_PTR_LOCAL from_node, int from)
+{
+ int level = from;
+ RT_PTR_LOCAL node = from_node;
+
+ for (;;)
+ {
+ RT_NODE_ITER *node_iter = &(iter->stack[level--]);
+
+ node_iter->node = node;
+ node_iter->current_idx = -1;
+
+ /* We don't advance the leaf node iterator here */
+ if (RT_NODE_IS_LEAF(node))
+ return;
+
+ /* Advance to the next slot in the inner node */
+ node = RT_NODE_INNER_ITERATE_NEXT(iter, node_iter);
+
+ /* We must find the first children in the node */
+ Assert(node);
+ }
+}
+
+/*
+ * Create and return the iterator for the given radix tree.
+ *
+ * The radix tree is locked in shared mode during the iteration, so
+ * RT_END_ITERATE needs to be called when finished to release the lock.
+ */
+RT_SCOPE RT_ITER *
+RT_BEGIN_ITERATE(RT_RADIX_TREE *tree)
+{
+ MemoryContext old_ctx;
+ RT_ITER *iter;
+ RT_PTR_LOCAL root;
+ int top_level;
+
+ old_ctx = MemoryContextSwitchTo(tree->context);
+
+ iter = (RT_ITER *) palloc0(sizeof(RT_ITER));
+ iter->tree = tree;
+
+ RT_LOCK_SHARED(tree);
+
+ /* empty tree */
+ if (!iter->tree->ctl->root)
+ return iter;
+
+ root = RT_PTR_GET_LOCAL(tree, iter->tree->ctl->root);
+ top_level = root->shift / RT_NODE_SPAN;
+ iter->stack_len = top_level;
+
+ /*
+ * Descend to the left most leaf node from the root. The key is being
+ * constructed while descending to the leaf.
+ */
+ RT_UPDATE_ITER_STACK(iter, root, top_level);
+
+ MemoryContextSwitchTo(old_ctx);
+
+ return iter;
+}
+
+/*
+ * Return true with setting key_p and value_p if there is next key. Otherwise
+ * return false.
+ */
+RT_SCOPE bool
+RT_ITERATE_NEXT(RT_ITER *iter, uint64 *key_p, RT_VALUE_TYPE *value_p)
+{
+ /* Empty tree */
+ if (!iter->tree->ctl->root)
+ return false;
+
+ for (;;)
+ {
+ RT_PTR_LOCAL child = NULL;
+ RT_VALUE_TYPE value;
+ int level;
+ bool found;
+
+ /* Advance the leaf node iterator to get next key-value pair */
+ found = RT_NODE_LEAF_ITERATE_NEXT(iter, &(iter->stack[0]), &value);
+
+ if (found)
+ {
+ *key_p = iter->key;
+ *value_p = value;
+ return true;
+ }
+
+ /*
+ * We've visited all values in the leaf node, so advance inner node
+ * iterators from the level=1 until we find the next child node.
+ */
+ for (level = 1; level <= iter->stack_len; level++)
+ {
+ child = RT_NODE_INNER_ITERATE_NEXT(iter, &(iter->stack[level]));
+
+ if (child)
+ break;
+ }
+
+ /* the iteration finished */
+ if (!child)
+ return false;
+
+ /*
+ * Set the node to the node iterator and update the iterator stack
+ * from this node.
+ */
+ RT_UPDATE_ITER_STACK(iter, child, level - 1);
+
+ /* Node iterators are updated, so try again from the leaf */
+ }
+
+ return false;
+}
+
+/*
+ * Terminate the iteration and release the lock.
+ *
+ * This function needs to be called after finishing or when exiting an
+ * iteration.
+ */
+RT_SCOPE void
+RT_END_ITERATE(RT_ITER *iter)
+{
+#ifdef RT_SHMEM
+ Assert(LWLockHeldByMe(&iter->tree->ctl->lock));
+#endif
+
+ RT_UNLOCK(iter->tree);
+ pfree(iter);
+}
+
+/*
+ * Return the statistics of the amount of memory used by the radix tree.
+ */
+RT_SCOPE uint64
+RT_MEMORY_USAGE(RT_RADIX_TREE *tree)
+{
+ Size total = 0;
+
+ RT_LOCK_SHARED(tree);
+
+#ifdef RT_SHMEM
+ Assert(tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+ total = dsa_get_total_size(tree->dsa);
+#else
+ for (int i = 0; i < RT_SIZE_CLASS_COUNT; i++)
+ {
+ total += MemoryContextMemAllocated(tree->inner_slabs[i], true);
+ total += MemoryContextMemAllocated(tree->leaf_slabs[i], true);
+ }
+#endif
+
+ RT_UNLOCK(tree);
+ return total;
+}
+
+/*
+ * Verify the radix tree node.
+ */
+static void
+RT_VERIFY_NODE(RT_PTR_LOCAL node)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(node->count >= 0);
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE_BASE_3 *n3 = (RT_NODE_BASE_3 *) node;
+
+ for (int i = 1; i < n3->n.count; i++)
+ Assert(n3->chunks[i - 1] < n3->chunks[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE_BASE_32 *n32 = (RT_NODE_BASE_32 *) node;
+
+ for (int i = 1; i < n32->n.count; i++)
+ Assert(n32->chunks[i - 1] < n32->chunks[i]);
+
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_BASE_125 *n125 = (RT_NODE_BASE_125 *) node;
+ int cnt = 0;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ uint8 slot = n125->slot_idxs[i];
+ int idx = RT_BM_IDX(slot);
+ int bitnum = RT_BM_BIT(slot);
+
+ if (!RT_NODE_125_IS_CHUNK_USED(n125, i))
+ continue;
+
+ /* Check if the corresponding slot is used */
+ Assert(slot < node->fanout);
+ Assert((n125->isset[idx] & ((bitmapword) 1 << bitnum)) != 0);
+
+ cnt++;
+ }
+
+ Assert(n125->n.count == cnt);
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+ int cnt = 0;
+
+ for (int i = 0; i < RT_BM_IDX(RT_NODE_MAX_SLOTS); i++)
+ cnt += bmw_popcount(n256->isset[i]);
+
+ /* Check if the number of used chunk matches */
+ Assert(n256->base.n.count == cnt);
+
+ break;
+ }
+ }
+ }
+#endif
+}
+
+/***************** DEBUG FUNCTIONS *****************/
+#ifdef RT_DEBUG
+
+#define RT_UINT64_FORMAT_HEX "%" INT64_MODIFIER "X"
+
+RT_SCOPE void
+RT_STATS(RT_RADIX_TREE *tree)
+{
+ RT_LOCK_SHARED(tree);
+
+ fprintf(stderr, "max_val = " UINT64_FORMAT "\n", tree->ctl->max_val);
+ fprintf(stderr, "num_keys = " UINT64_FORMAT "\n", tree->ctl->num_keys);
+
+#ifdef RT_SHMEM
+ fprintf(stderr, "handle = " UINT64_FORMAT "\n", tree->ctl->handle);
+#endif
+
+ if (RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_PTR_LOCAL root = RT_PTR_GET_LOCAL(tree, tree->ctl->root);
+
+ fprintf(stderr, "height = %d, n3 = %u, n15 = %u, n32 = %u, n125 = %u, n256 = %u\n",
+ root->shift / RT_NODE_SPAN,
+ tree->ctl->cnt[RT_CLASS_3],
+ tree->ctl->cnt[RT_CLASS_32_MIN],
+ tree->ctl->cnt[RT_CLASS_32_MAX],
+ tree->ctl->cnt[RT_CLASS_125],
+ tree->ctl->cnt[RT_CLASS_256]);
+ }
+
+ RT_UNLOCK(tree);
+}
+
+static void
+RT_DUMP_NODE(RT_RADIX_TREE *tree, RT_PTR_ALLOC allocnode, int level,
+ bool recurse, StringInfo buf)
+{
+ RT_PTR_LOCAL node = RT_PTR_GET_LOCAL(tree, allocnode);
+ StringInfoData spaces;
+
+ initStringInfo(&spaces);
+ appendStringInfoSpaces(&spaces, (level * 4) + 1);
+
+ appendStringInfo(buf, "%s%s[%s] kind %d, fanout %d, count %u, shift %u:\n",
+ spaces.data,
+ level == 0 ? "" : "-> ",
+ RT_NODE_IS_LEAF(node) ? "LEAF" : "INNR",
+ (node->kind == RT_NODE_KIND_3) ? 3 :
+ (node->kind == RT_NODE_KIND_32) ? 32 :
+ (node->kind == RT_NODE_KIND_125) ? 125 : 256,
+ node->fanout == 0 ? 256 : node->fanout,
+ node->count, node->shift);
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ for (int i = 0; i < node->count; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_3 *n3 = (RT_NODE_LEAF_3 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+ spaces.data, i, n3->base.chunks[i]);
+ }
+ else
+ {
+ RT_NODE_INNER_3 *n3 = (RT_NODE_INNER_3 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X",
+ spaces.data, i, n3->base.chunks[i]);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, n3->children[i], level + 1,
+ recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ for (int i = 0; i < node->count; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_32 *n32 = (RT_NODE_LEAF_32 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X\n",
+ spaces.data, i, n32->base.chunks[i]);
+ }
+ else
+ {
+ RT_NODE_INNER_32 *n32 = (RT_NODE_INNER_32 *) node;
+
+ appendStringInfo(buf, "%schunk[%d] 0x%X",
+ spaces.data, i, n32->base.chunks[i]);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, n32->children[i], level + 1,
+ recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE_BASE_125 *b125 = (RT_NODE_BASE_125 *) node;
+ char *sep = "";
+
+ appendStringInfo(buf, "%sslot_idxs: ", spaces.data);
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+ continue;
+
+ appendStringInfo(buf, "%s[%d]=%d ",
+ sep, i, b125->slot_idxs[i]);
+ sep = ",";
+ }
+
+ appendStringInfo(buf, "\n%sisset-bitmap: ", spaces.data);
+ for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+ appendStringInfo(buf, "%X ", ((uint8 *) b125->isset)[i]);
+ appendStringInfo(buf, "\n");
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(b125, i))
+ continue;
+
+ if (RT_NODE_IS_LEAF(node))
+ appendStringInfo(buf, "%schunk 0x%X\n",
+ spaces.data, i);
+ else
+ {
+ RT_NODE_INNER_125 *n125 = (RT_NODE_INNER_125 *) b125;
+
+ appendStringInfo(buf, "%schunk 0x%X",
+ spaces.data, i);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, RT_NODE_INNER_125_GET_CHILD(n125, i),
+ level + 1, recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+ appendStringInfo(buf, "%sisset-bitmap: ", spaces.data);
+ for (int i = 0; i < (RT_SLOT_IDX_LIMIT / BITS_PER_BYTE); i++)
+ appendStringInfo(buf, "%X ", ((uint8 *) n256->isset)[i]);
+ appendStringInfo(buf, "\n");
+ }
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_NODE_LEAF_256 *n256 = (RT_NODE_LEAF_256 *) node;
+
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ appendStringInfo(buf, "%schunk 0x%X\n",
+ spaces.data, i);
+ }
+ else
+ {
+ RT_NODE_INNER_256 *n256 = (RT_NODE_INNER_256 *) node;
+
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+ continue;
+
+ appendStringInfo(buf, "%schunk 0x%X",
+ spaces.data, i);
+
+ if (recurse)
+ {
+ appendStringInfo(buf, "\n");
+ RT_DUMP_NODE(tree, RT_NODE_INNER_256_GET_CHILD(n256, i),
+ level + 1, recurse, buf);
+ }
+ else
+ appendStringInfo(buf, " (skipped)\n");
+ }
+ }
+ break;
+ }
+ }
+}
+
+RT_SCOPE void
+RT_DUMP_SEARCH(RT_RADIX_TREE *tree, uint64 key)
+{
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL node;
+ StringInfoData buf;
+ int shift;
+ int level = 0;
+
+ RT_STATS(tree);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "empty tree\n");
+ return;
+ }
+
+ if (key > tree->ctl->max_val)
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "key " UINT64_FORMAT "(0x" RT_UINT64_FORMAT_HEX ") is larger than max val\n",
+ key, key);
+ return;
+ }
+
+ initStringInfo(&buf);
+ allocnode = tree->ctl->root;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift = node->shift;
+ while (shift >= 0)
+ {
+ RT_PTR_ALLOC child;
+
+ RT_DUMP_NODE(tree, allocnode, level, false, &buf);
+
+ if (RT_NODE_IS_LEAF(node))
+ {
+ RT_VALUE_TYPE dummy;
+
+ /* We reached at a leaf node, find the corresponding slot */
+ RT_NODE_SEARCH_LEAF(node, key, &dummy);
+
+ break;
+ }
+
+ if (!RT_NODE_SEARCH_INNER(node, key, &child))
+ break;
+
+ allocnode = child;
+ node = RT_PTR_GET_LOCAL(tree, allocnode);
+ shift -= RT_NODE_SPAN;
+ level++;
+ }
+ RT_UNLOCK(tree);
+
+ fprintf(stderr, "%s", buf.data);
+}
+
+RT_SCOPE void
+RT_DUMP(RT_RADIX_TREE *tree)
+{
+ StringInfoData buf;
+
+ RT_STATS(tree);
+
+ RT_LOCK_SHARED(tree);
+
+ if (!RT_PTR_ALLOC_IS_VALID(tree->ctl->root))
+ {
+ RT_UNLOCK(tree);
+ fprintf(stderr, "empty tree\n");
+ return;
+ }
+
+ initStringInfo(&buf);
+
+ RT_DUMP_NODE(tree, tree->ctl->root, 0, true, &buf);
+ RT_UNLOCK(tree);
+
+ fprintf(stderr, "%s",buf.data);
+}
+#endif
+
+#endif /* RT_DEFINE */
+
+
+/* undefine external parameters, so next radix tree can be defined */
+#undef RT_PREFIX
+#undef RT_SCOPE
+#undef RT_DECLARE
+#undef RT_DEFINE
+#undef RT_VALUE_TYPE
+
+/* locally declared macros */
+#undef RT_MAKE_PREFIX
+#undef RT_MAKE_NAME
+#undef RT_MAKE_NAME_
+#undef RT_NODE_SPAN
+#undef RT_NODE_MAX_SLOTS
+#undef RT_CHUNK_MASK
+#undef RT_MAX_SHIFT
+#undef RT_MAX_LEVEL
+#undef RT_GET_KEY_CHUNK
+#undef RT_BM_IDX
+#undef RT_BM_BIT
+#undef RT_LOCK_EXCLUSIVE
+#undef RT_LOCK_SHARED
+#undef RT_UNLOCK
+#undef RT_NODE_IS_LEAF
+#undef RT_NODE_MUST_GROW
+#undef RT_NODE_KIND_COUNT
+#undef RT_SIZE_CLASS_COUNT
+#undef RT_SLOT_IDX_LIMIT
+#undef RT_INVALID_SLOT_IDX
+#undef RT_SLAB_BLOCK_SIZE
+#undef RT_RADIX_TREE_MAGIC
+#undef RT_UINT64_FORMAT_HEX
+
+/* type declarations */
+#undef RT_RADIX_TREE
+#undef RT_RADIX_TREE_CONTROL
+#undef RT_PTR_LOCAL
+#undef RT_PTR_ALLOC
+#undef RT_INVALID_PTR_ALLOC
+#undef RT_HANDLE
+#undef RT_ITER
+#undef RT_NODE
+#undef RT_NODE_ITER
+#undef RT_NODE_KIND_3
+#undef RT_NODE_KIND_32
+#undef RT_NODE_KIND_125
+#undef RT_NODE_KIND_256
+#undef RT_NODE_BASE_3
+#undef RT_NODE_BASE_32
+#undef RT_NODE_BASE_125
+#undef RT_NODE_BASE_256
+#undef RT_NODE_INNER_3
+#undef RT_NODE_INNER_32
+#undef RT_NODE_INNER_125
+#undef RT_NODE_INNER_256
+#undef RT_NODE_LEAF_3
+#undef RT_NODE_LEAF_32
+#undef RT_NODE_LEAF_125
+#undef RT_NODE_LEAF_256
+#undef RT_SIZE_CLASS
+#undef RT_SIZE_CLASS_ELEM
+#undef RT_SIZE_CLASS_INFO
+#undef RT_CLASS_3
+#undef RT_CLASS_32_MIN
+#undef RT_CLASS_32_MAX
+#undef RT_CLASS_125
+#undef RT_CLASS_256
+
+/* function declarations */
+#undef RT_CREATE
+#undef RT_FREE
+#undef RT_ATTACH
+#undef RT_DETACH
+#undef RT_GET_HANDLE
+#undef RT_SEARCH
+#undef RT_SET
+#undef RT_BEGIN_ITERATE
+#undef RT_ITERATE_NEXT
+#undef RT_END_ITERATE
+#undef RT_USE_DELETE
+#undef RT_DELETE
+#undef RT_MEMORY_USAGE
+#undef RT_DUMP
+#undef RT_DUMP_NODE
+#undef RT_DUMP_SEARCH
+#undef RT_STATS
+
+/* internal helper functions */
+#undef RT_NEW_ROOT
+#undef RT_ALLOC_NODE
+#undef RT_INIT_NODE
+#undef RT_FREE_NODE
+#undef RT_FREE_RECURSE
+#undef RT_EXTEND
+#undef RT_SET_EXTEND
+#undef RT_SWITCH_NODE_KIND
+#undef RT_COPY_NODE
+#undef RT_REPLACE_NODE
+#undef RT_PTR_GET_LOCAL
+#undef RT_PTR_ALLOC_IS_VALID
+#undef RT_NODE_3_SEARCH_EQ
+#undef RT_NODE_32_SEARCH_EQ
+#undef RT_NODE_3_GET_INSERTPOS
+#undef RT_NODE_32_GET_INSERTPOS
+#undef RT_CHUNK_CHILDREN_ARRAY_SHIFT
+#undef RT_CHUNK_VALUES_ARRAY_SHIFT
+#undef RT_CHUNK_CHILDREN_ARRAY_DELETE
+#undef RT_CHUNK_VALUES_ARRAY_DELETE
+#undef RT_CHUNK_CHILDREN_ARRAY_COPY
+#undef RT_CHUNK_VALUES_ARRAY_COPY
+#undef RT_NODE_125_IS_CHUNK_USED
+#undef RT_NODE_INNER_125_GET_CHILD
+#undef RT_NODE_LEAF_125_GET_VALUE
+#undef RT_NODE_INNER_256_IS_CHUNK_USED
+#undef RT_NODE_LEAF_256_IS_CHUNK_USED
+#undef RT_NODE_INNER_256_GET_CHILD
+#undef RT_NODE_LEAF_256_GET_VALUE
+#undef RT_NODE_INNER_256_SET
+#undef RT_NODE_LEAF_256_SET
+#undef RT_NODE_INNER_256_DELETE
+#undef RT_NODE_LEAF_256_DELETE
+#undef RT_KEY_GET_SHIFT
+#undef RT_SHIFT_GET_MAX_VAL
+#undef RT_NODE_SEARCH_INNER
+#undef RT_NODE_SEARCH_LEAF
+#undef RT_NODE_UPDATE_INNER
+#undef RT_NODE_DELETE_INNER
+#undef RT_NODE_DELETE_LEAF
+#undef RT_NODE_INSERT_INNER
+#undef RT_NODE_INSERT_LEAF
+#undef RT_NODE_INNER_ITERATE_NEXT
+#undef RT_NODE_LEAF_ITERATE_NEXT
+#undef RT_UPDATE_ITER_STACK
+#undef RT_ITER_UPDATE_KEY
+#undef RT_VERIFY_NODE
+
+#undef RT_DEBUG
diff --git a/src/include/lib/radixtree_delete_impl.h b/src/include/lib/radixtree_delete_impl.h
new file mode 100644
index 0000000000..5f6dda1f12
--- /dev/null
+++ b/src/include/lib/radixtree_delete_impl.h
@@ -0,0 +1,122 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_delete_impl.h
+ * Common implementation for deletion in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ * TODO: Shrink nodes when deletion would allow them to fit in a smaller
+ * size class.
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_delete_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_DELETE(n3->base.chunks, n3->values,
+ n3->base.n.count, idx);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_DELETE(n3->base.chunks, n3->children,
+ n3->base.n.count, idx);
+#endif
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_DELETE(n32->base.chunks, n32->values,
+ n32->base.n.count, idx);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_DELETE(n32->base.chunks, n32->children,
+ n32->base.n.count, idx);
+#endif
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+ int idx;
+ int bitnum;
+
+ if (slotpos == RT_INVALID_SLOT_IDX)
+ return false;
+
+ idx = RT_BM_IDX(slotpos);
+ bitnum = RT_BM_BIT(slotpos);
+ n125->base.isset[idx] &= ~((bitmapword) 1 << bitnum);
+ n125->base.slot_idxs[chunk] = RT_INVALID_SLOT_IDX;
+
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_DELETE(n256, chunk);
+#else
+ RT_NODE_INNER_256_DELETE(n256, chunk);
+#endif
+ break;
+ }
+ }
+
+ /* update statistics */
+ node->count--;
+
+ return true;
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_insert_impl.h b/src/include/lib/radixtree_insert_impl.h
new file mode 100644
index 0000000000..d56e58dcac
--- /dev/null
+++ b/src/include/lib/radixtree_insert_impl.h
@@ -0,0 +1,328 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_insert_impl.h
+ * Common implementation for insertion in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_insert_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ const bool is_leaf = true;
+ bool chunk_exists = false;
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+ const bool is_leaf = false;
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ int idx = RT_NODE_3_SEARCH_EQ(&n3->base, chunk);
+
+ if (idx != -1)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n3->values[idx] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n3)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE32_TYPE *new32;
+ const uint8 new_kind = RT_NODE_KIND_32;
+ const RT_SIZE_CLASS new_class = RT_CLASS_32_MIN;
+
+ /* grow node from 3 to 32 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_COPY(n3->base.chunks, n3->values,
+ new32->base.chunks, new32->values);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_COPY(n3->base.chunks, n3->children,
+ new32->base.chunks, new32->children);
+#endif
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int insertpos = RT_NODE_3_GET_INSERTPOS(&n3->base, chunk);
+ int count = n3->base.n.count;
+
+ /* shift chunks and children */
+ if (insertpos < count)
+ {
+ Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_SHIFT(n3->base.chunks, n3->values,
+ count, insertpos);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_SHIFT(n3->base.chunks, n3->children,
+ count, insertpos);
+#endif
+ }
+
+ n3->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+ n3->values[insertpos] = *value_p;
+#else
+ n3->children[insertpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_32:
+ {
+ const RT_SIZE_CLASS_ELEM class32_max = RT_SIZE_CLASS_INFO[RT_CLASS_32_MAX];
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ int idx = RT_NODE_32_SEARCH_EQ(&n32->base, chunk);
+
+ if (idx != -1)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n32->values[idx] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n32)) &&
+ n32->base.n.fanout < class32_max.fanout)
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ const RT_SIZE_CLASS_ELEM class32_min = RT_SIZE_CLASS_INFO[RT_CLASS_32_MIN];
+ const RT_SIZE_CLASS new_class = RT_CLASS_32_MAX;
+
+ Assert(n32->base.n.fanout == class32_min.fanout);
+
+ /* grow to the next size class of this kind */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_PTR_GET_LOCAL(tree, allocnode);
+ n32 = (RT_NODE32_TYPE *) newnode;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ memcpy(newnode, node, class32_min.leaf_size);
+#else
+ memcpy(newnode, node, class32_min.inner_size);
+#endif
+ newnode->fanout = class32_max.fanout;
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+
+ if (unlikely(RT_NODE_MUST_GROW(n32)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE125_TYPE *new125;
+ const uint8 new_kind = RT_NODE_KIND_125;
+ const RT_SIZE_CLASS new_class = RT_CLASS_125;
+
+ Assert(n32->base.n.fanout == class32_max.fanout);
+
+ /* grow node from 32 to 125 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new125 = (RT_NODE125_TYPE *) newnode;
+
+ for (int i = 0; i < class32_max.fanout; i++)
+ {
+ new125->base.slot_idxs[n32->base.chunks[i]] = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ new125->values[i] = n32->values[i];
+#else
+ new125->children[i] = n32->children[i];
+#endif
+ }
+
+ /*
+ * Since we just copied a dense array, we can set the bits
+ * using a single store, provided the length of that array
+ * is at most the number of bits in a bitmapword.
+ */
+ Assert(class32_max.fanout <= sizeof(bitmapword) * BITS_PER_BYTE);
+ new125->base.isset[0] = (bitmapword) (((uint64) 1 << class32_max.fanout) - 1);
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int insertpos = RT_NODE_32_GET_INSERTPOS(&n32->base, chunk);
+ int count = n32->base.n.count;
+
+ if (insertpos < count)
+ {
+ Assert(count > 0);
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_CHUNK_VALUES_ARRAY_SHIFT(n32->base.chunks, n32->values,
+ count, insertpos);
+#else
+ RT_CHUNK_CHILDREN_ARRAY_SHIFT(n32->base.chunks, n32->children,
+ count, insertpos);
+#endif
+ }
+
+ n32->base.chunks[insertpos] = chunk;
+#ifdef RT_NODE_LEVEL_LEAF
+ n32->values[insertpos] = *value_p;
+#else
+ n32->children[insertpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos;
+ int cnt = 0;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ slotpos = n125->base.slot_idxs[chunk];
+ if (slotpos != RT_INVALID_SLOT_IDX)
+ {
+ /* found the existing chunk */
+ chunk_exists = true;
+ n125->values[slotpos] = *value_p;
+ break;
+ }
+#endif
+ if (unlikely(RT_NODE_MUST_GROW(n125)))
+ {
+ RT_PTR_ALLOC allocnode;
+ RT_PTR_LOCAL newnode;
+ RT_NODE256_TYPE *new256;
+ const uint8 new_kind = RT_NODE_KIND_256;
+ const RT_SIZE_CLASS new_class = RT_CLASS_256;
+
+ /* grow node from 125 to 256 */
+ allocnode = RT_ALLOC_NODE(tree, new_class, is_leaf);
+ newnode = RT_SWITCH_NODE_KIND(tree, allocnode, node, new_kind, new_class, is_leaf);
+ new256 = (RT_NODE256_TYPE *) newnode;
+
+ for (int i = 0; i < RT_NODE_MAX_SLOTS && cnt < n125->base.n.count; i++)
+ {
+ if (!RT_NODE_125_IS_CHUNK_USED(&n125->base, i))
+ continue;
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_NODE_LEAF_256_SET(new256, i, RT_NODE_LEAF_125_GET_VALUE(n125, i));
+#else
+ RT_NODE_INNER_256_SET(new256, i, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+ cnt++;
+ }
+
+ RT_REPLACE_NODE(tree, parent, stored_node, node, allocnode, key);
+ node = newnode;
+ }
+ else
+ {
+ int idx;
+ bitmapword inverse;
+
+ /* get the first word with at least one bit not set */
+ for (idx = 0; idx < RT_BM_IDX(RT_SLOT_IDX_LIMIT); idx++)
+ {
+ if (n125->base.isset[idx] < ~((bitmapword) 0))
+ break;
+ }
+
+ /* To get the first unset bit in X, get the first set bit in ~X */
+ inverse = ~(n125->base.isset[idx]);
+ slotpos = idx * BITS_PER_BITMAPWORD;
+ slotpos += bmw_rightmost_one_pos(inverse);
+ Assert(slotpos < node->fanout);
+
+ /* mark the slot used */
+ n125->base.isset[idx] |= bmw_rightmost_one(inverse);
+ n125->base.slot_idxs[chunk] = slotpos;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ n125->values[slotpos] = *value_p;
+#else
+ n125->children[slotpos] = child;
+#endif
+ break;
+ }
+ }
+ /* FALLTHROUGH */
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ chunk_exists = RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk);
+ Assert(chunk_exists || node->count < RT_NODE_MAX_SLOTS);
+ RT_NODE_LEAF_256_SET(n256, chunk, *value_p);
+#else
+ Assert(node->count < RT_NODE_MAX_SLOTS);
+ RT_NODE_INNER_256_SET(n256, chunk, child);
+#endif
+ break;
+ }
+ }
+
+ /* Update statistics */
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!chunk_exists)
+ node->count++;
+#else
+ node->count++;
+#endif
+
+ /*
+ * Done. Finally, verify the chunk and value is inserted or replaced
+ * properly in the node.
+ */
+ RT_VERIFY_NODE(node);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ return chunk_exists;
+#else
+ return;
+#endif
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_iter_impl.h b/src/include/lib/radixtree_iter_impl.h
new file mode 100644
index 0000000000..98c78eb237
--- /dev/null
+++ b/src/include/lib/radixtree_iter_impl.h
@@ -0,0 +1,153 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_iter_impl.h
+ * Common implementation for iteration in leaf and inner nodes.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_iter_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ bool found = false;
+ uint8 key_chunk;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ RT_VALUE_TYPE value;
+
+ Assert(RT_NODE_IS_LEAF(node_iter->node));
+#else
+ RT_PTR_LOCAL child = NULL;
+
+ Assert(!RT_NODE_IS_LEAF(node_iter->node));
+#endif
+
+#ifdef RT_SHMEM
+ Assert(iter->tree->ctl->magic == RT_RADIX_TREE_MAGIC);
+#endif
+
+ switch (node_iter->node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node_iter->node;
+
+ node_iter->current_idx++;
+ if (node_iter->current_idx >= n3->base.n.count)
+ break;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = n3->values[node_iter->current_idx];
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, n3->children[node_iter->current_idx]);
+#endif
+ key_chunk = n3->base.chunks[node_iter->current_idx];
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node_iter->node;
+
+ node_iter->current_idx++;
+ if (node_iter->current_idx >= n32->base.n.count)
+ break;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ value = n32->values[node_iter->current_idx];
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, n32->children[node_iter->current_idx]);
+#endif
+ key_chunk = n32->base.chunks[node_iter->current_idx];
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node_iter->node;
+ int i;
+
+ for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ {
+ if (RT_NODE_125_IS_CHUNK_USED((RT_NODE_BASE_125 *) n125, i))
+ break;
+ }
+
+ if (i >= RT_NODE_MAX_SLOTS)
+ break;
+
+ node_iter->current_idx = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = RT_NODE_LEAF_125_GET_VALUE(n125, i);
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_125_GET_CHILD(n125, i));
+#endif
+ key_chunk = i;
+ found = true;
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node_iter->node;
+ int i;
+
+ for (i = node_iter->current_idx + 1; i < RT_NODE_MAX_SLOTS; i++)
+ {
+#ifdef RT_NODE_LEVEL_LEAF
+ if (RT_NODE_LEAF_256_IS_CHUNK_USED(n256, i))
+#else
+ if (RT_NODE_INNER_256_IS_CHUNK_USED(n256, i))
+#endif
+ break;
+ }
+
+ if (i >= RT_NODE_MAX_SLOTS)
+ break;
+
+ node_iter->current_idx = i;
+#ifdef RT_NODE_LEVEL_LEAF
+ value = RT_NODE_LEAF_256_GET_VALUE(n256, i);
+#else
+ child = RT_PTR_GET_LOCAL(iter->tree, RT_NODE_INNER_256_GET_CHILD(n256, i));
+#endif
+ key_chunk = i;
+ found = true;
+ break;
+ }
+ }
+
+ if (found)
+ {
+ RT_ITER_UPDATE_KEY(iter, key_chunk, node_iter->node->shift);
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = value;
+#endif
+ }
+
+#ifdef RT_NODE_LEVEL_LEAF
+ return found;
+#else
+ return child;
+#endif
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/lib/radixtree_search_impl.h b/src/include/lib/radixtree_search_impl.h
new file mode 100644
index 0000000000..a8925c75d0
--- /dev/null
+++ b/src/include/lib/radixtree_search_impl.h
@@ -0,0 +1,138 @@
+/*-------------------------------------------------------------------------
+ *
+ * radixtree_search_impl.h
+ * Common implementation for search in leaf and inner nodes, plus
+ * update for inner nodes only.
+ *
+ * Note: There is deliberately no #include guard here
+ *
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * src/include/lib/radixtree_search_impl.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#if defined(RT_NODE_LEVEL_INNER)
+#define RT_NODE3_TYPE RT_NODE_INNER_3
+#define RT_NODE32_TYPE RT_NODE_INNER_32
+#define RT_NODE125_TYPE RT_NODE_INNER_125
+#define RT_NODE256_TYPE RT_NODE_INNER_256
+#elif defined(RT_NODE_LEVEL_LEAF)
+#define RT_NODE3_TYPE RT_NODE_LEAF_3
+#define RT_NODE32_TYPE RT_NODE_LEAF_32
+#define RT_NODE125_TYPE RT_NODE_LEAF_125
+#define RT_NODE256_TYPE RT_NODE_LEAF_256
+#else
+#error node level must be either inner or leaf
+#endif
+
+ uint8 chunk = RT_GET_KEY_CHUNK(key, node->shift);
+
+#ifdef RT_NODE_LEVEL_LEAF
+ Assert(value_p != NULL);
+ Assert(RT_NODE_IS_LEAF(node));
+#else
+#ifndef RT_ACTION_UPDATE
+ Assert(child_p != NULL);
+#endif
+ Assert(!RT_NODE_IS_LEAF(node));
+#endif
+
+ switch (node->kind)
+ {
+ case RT_NODE_KIND_3:
+ {
+ RT_NODE3_TYPE *n3 = (RT_NODE3_TYPE *) node;
+ int idx = RT_NODE_3_SEARCH_EQ((RT_NODE_BASE_3 *) n3, chunk);
+
+#ifdef RT_ACTION_UPDATE
+ Assert(idx >= 0);
+ n3->children[idx] = new_child;
+#else
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n3->values[idx];
+#else
+ *child_p = n3->children[idx];
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_32:
+ {
+ RT_NODE32_TYPE *n32 = (RT_NODE32_TYPE *) node;
+ int idx = RT_NODE_32_SEARCH_EQ((RT_NODE_BASE_32 *) n32, chunk);
+
+#ifdef RT_ACTION_UPDATE
+ Assert(idx >= 0);
+ n32->children[idx] = new_child;
+#else
+ if (idx < 0)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = n32->values[idx];
+#else
+ *child_p = n32->children[idx];
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_125:
+ {
+ RT_NODE125_TYPE *n125 = (RT_NODE125_TYPE *) node;
+ int slotpos = n125->base.slot_idxs[chunk];
+
+#ifdef RT_ACTION_UPDATE
+ Assert(slotpos != RT_INVALID_SLOT_IDX);
+ n125->children[slotpos] = new_child;
+#else
+ if (slotpos == RT_INVALID_SLOT_IDX)
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_125_GET_VALUE(n125, chunk);
+#else
+ *child_p = RT_NODE_INNER_125_GET_CHILD(n125, chunk);
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ case RT_NODE_KIND_256:
+ {
+ RT_NODE256_TYPE *n256 = (RT_NODE256_TYPE *) node;
+
+#ifdef RT_ACTION_UPDATE
+ RT_NODE_INNER_256_SET(n256, chunk, new_child);
+#else
+#ifdef RT_NODE_LEVEL_LEAF
+ if (!RT_NODE_LEAF_256_IS_CHUNK_USED(n256, chunk))
+#else
+ if (!RT_NODE_INNER_256_IS_CHUNK_USED(n256, chunk))
+#endif
+ return false;
+
+#ifdef RT_NODE_LEVEL_LEAF
+ *value_p = RT_NODE_LEAF_256_GET_VALUE(n256, chunk);
+#else
+ *child_p = RT_NODE_INNER_256_GET_CHILD(n256, chunk);
+#endif
+#endif /* RT_ACTION_UPDATE */
+ break;
+ }
+ }
+
+#ifdef RT_ACTION_UPDATE
+ return;
+#else
+ return true;
+#endif /* RT_ACTION_UPDATE */
+
+#undef RT_NODE3_TYPE
+#undef RT_NODE32_TYPE
+#undef RT_NODE125_TYPE
+#undef RT_NODE256_TYPE
diff --git a/src/include/utils/dsa.h b/src/include/utils/dsa.h
index 3ce4ee300a..2af215484f 100644
--- a/src/include/utils/dsa.h
+++ b/src/include/utils/dsa.h
@@ -121,6 +121,7 @@ extern dsa_handle dsa_get_handle(dsa_area *area);
extern dsa_pointer dsa_allocate_extended(dsa_area *area, size_t size, int flags);
extern void dsa_free(dsa_area *area, dsa_pointer dp);
extern void *dsa_get_address(dsa_area *area, dsa_pointer dp);
+extern size_t dsa_get_total_size(dsa_area *area);
extern void dsa_trim(dsa_area *area);
extern void dsa_dump(dsa_area *area);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 79e3033ec2..89f42bf9e3 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -28,6 +28,7 @@ SUBDIRS = \
test_pg_db_role_setting \
test_pg_dump \
test_predtest \
+ test_radixtree \
test_rbtree \
test_regex \
test_rls_hooks \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index dcb82ed68f..beaf4080fb 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -25,6 +25,7 @@ subdir('test_parser')
subdir('test_pg_db_role_setting')
subdir('test_pg_dump')
subdir('test_predtest')
+subdir('test_radixtree')
subdir('test_rbtree')
subdir('test_regex')
subdir('test_rls_hooks')
diff --git a/src/test/modules/test_radixtree/.gitignore b/src/test/modules/test_radixtree/.gitignore
new file mode 100644
index 0000000000..5dcb3ff972
--- /dev/null
+++ b/src/test/modules/test_radixtree/.gitignore
@@ -0,0 +1,4 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
diff --git a/src/test/modules/test_radixtree/Makefile b/src/test/modules/test_radixtree/Makefile
new file mode 100644
index 0000000000..da06b93da3
--- /dev/null
+++ b/src/test/modules/test_radixtree/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_radixtree/Makefile
+
+MODULE_big = test_radixtree
+OBJS = \
+ $(WIN32RES) \
+ test_radixtree.o
+PGFILEDESC = "test_radixtree - test code for src/backend/lib/radixtree.c"
+
+EXTENSION = test_radixtree
+DATA = test_radixtree--1.0.sql
+
+REGRESS = test_radixtree
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_radixtree
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_radixtree/README b/src/test/modules/test_radixtree/README
new file mode 100644
index 0000000000..a8b271869a
--- /dev/null
+++ b/src/test/modules/test_radixtree/README
@@ -0,0 +1,7 @@
+test_integerset contains unit tests for testing the integer set implementation
+in src/backend/lib/integerset.c.
+
+The tests verify the correctness of the implementation, but they can also be
+used as a micro-benchmark. If you set the 'intset_test_stats' flag in
+test_integerset.c, the tests will print extra information about execution time
+and memory usage.
diff --git a/src/test/modules/test_radixtree/expected/test_radixtree.out b/src/test/modules/test_radixtree/expected/test_radixtree.out
new file mode 100644
index 0000000000..ce645cb8b5
--- /dev/null
+++ b/src/test/modules/test_radixtree/expected/test_radixtree.out
@@ -0,0 +1,36 @@
+CREATE EXTENSION test_radixtree;
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
+NOTICE: testing basic operations with leaf node 4
+NOTICE: testing basic operations with inner node 4
+NOTICE: testing basic operations with leaf node 32
+NOTICE: testing basic operations with inner node 32
+NOTICE: testing basic operations with leaf node 125
+NOTICE: testing basic operations with inner node 125
+NOTICE: testing basic operations with leaf node 256
+NOTICE: testing basic operations with inner node 256
+NOTICE: testing radix tree node types with shift "0"
+NOTICE: testing radix tree node types with shift "8"
+NOTICE: testing radix tree node types with shift "16"
+NOTICE: testing radix tree node types with shift "24"
+NOTICE: testing radix tree node types with shift "32"
+NOTICE: testing radix tree node types with shift "40"
+NOTICE: testing radix tree node types with shift "48"
+NOTICE: testing radix tree node types with shift "56"
+NOTICE: testing radix tree with pattern "all ones"
+NOTICE: testing radix tree with pattern "alternating bits"
+NOTICE: testing radix tree with pattern "clusters of ten"
+NOTICE: testing radix tree with pattern "clusters of hundred"
+NOTICE: testing radix tree with pattern "one-every-64k"
+NOTICE: testing radix tree with pattern "sparse"
+NOTICE: testing radix tree with pattern "single values, distance > 2^32"
+NOTICE: testing radix tree with pattern "clusters, distance > 2^32"
+NOTICE: testing radix tree with pattern "clusters, distance > 2^60"
+ test_radixtree
+----------------
+
+(1 row)
+
diff --git a/src/test/modules/test_radixtree/meson.build b/src/test/modules/test_radixtree/meson.build
new file mode 100644
index 0000000000..6add06bbdb
--- /dev/null
+++ b/src/test/modules/test_radixtree/meson.build
@@ -0,0 +1,35 @@
+# FIXME: prevent install during main install, but not during test :/
+
+test_radixtree_sources = files(
+ 'test_radixtree.c',
+)
+
+if host_system == 'windows'
+ test_radixtree_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
+ '--NAME', 'test_radixtree',
+ '--FILEDESC', 'test_radixtree - test code for src/include//lib/radixtree.h',])
+endif
+
+test_radixtree = shared_module('test_radixtree',
+ test_radixtree_sources,
+ link_with: pgport_srv,
+ kwargs: pg_mod_args,
+)
+testprep_targets += test_radixtree
+
+install_data(
+ 'test_radixtree.control',
+ 'test_radixtree--1.0.sql',
+ kwargs: contrib_data_args,
+)
+
+tests += {
+ 'name': 'test_radixtree',
+ 'sd': meson.current_source_dir(),
+ 'bd': meson.current_build_dir(),
+ 'regress': {
+ 'sql': [
+ 'test_radixtree',
+ ],
+ },
+}
diff --git a/src/test/modules/test_radixtree/sql/test_radixtree.sql b/src/test/modules/test_radixtree/sql/test_radixtree.sql
new file mode 100644
index 0000000000..41ece5e9f5
--- /dev/null
+++ b/src/test/modules/test_radixtree/sql/test_radixtree.sql
@@ -0,0 +1,7 @@
+CREATE EXTENSION test_radixtree;
+
+--
+-- All the logic is in the test_radixtree() function. It will throw
+-- an error if something fails.
+--
+SELECT test_radixtree();
diff --git a/src/test/modules/test_radixtree/test_radixtree--1.0.sql b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
new file mode 100644
index 0000000000..074a5a7ea7
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree--1.0.sql
@@ -0,0 +1,8 @@
+/* src/test/modules/test_radixtree/test_radixtree--1.0.sql */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION test_radixtree" to load this file. \quit
+
+CREATE FUNCTION test_radixtree()
+RETURNS pg_catalog.void STRICT
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_radixtree/test_radixtree.c b/src/test/modules/test_radixtree/test_radixtree.c
new file mode 100644
index 0000000000..afe53382f3
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.c
@@ -0,0 +1,681 @@
+/*--------------------------------------------------------------------------
+ *
+ * test_radixtree.c
+ * Test radixtree set data structure.
+ *
+ * Copyright (c) 2023, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_radixtree/test_radixtree.c
+ *
+ * -------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include "common/pg_prng.h"
+#include "fmgr.h"
+#include "miscadmin.h"
+#include "nodes/bitmapset.h"
+#include "storage/block.h"
+#include "storage/itemptr.h"
+#include "storage/lwlock.h"
+#include "utils/memutils.h"
+#include "utils/timestamp.h"
+
+#define UINT64_HEX_FORMAT "%" INT64_MODIFIER "X"
+
+/*
+ * The tests pass with uint32, but build with warnings because the string
+ * format expects uint64.
+ */
+typedef uint64 TestValueType;
+
+/*
+ * If you enable this, the "pattern" tests will print information about
+ * how long populating, probing, and iterating the test set takes, and
+ * how much memory the test set consumed. That can be used as
+ * micro-benchmark of various operations and input patterns (you might
+ * want to increase the number of values used in each of the test, if
+ * you do that, to reduce noise).
+ *
+ * The information is printed to the server's stderr, mostly because
+ * that's where MemoryContextStats() output goes.
+ */
+static const bool rt_test_stats = false;
+
+static int rt_node_kind_fanouts[] = {
+ 0,
+ 4, /* RT_NODE_KIND_4 */
+ 32, /* RT_NODE_KIND_32 */
+ 125, /* RT_NODE_KIND_125 */
+ 256 /* RT_NODE_KIND_256 */
+};
+/*
+ * A struct to define a pattern of integers, for use with the test_pattern()
+ * function.
+ */
+typedef struct
+{
+ char *test_name; /* short name of the test, for humans */
+ char *pattern_str; /* a bit pattern */
+ uint64 spacing; /* pattern repeats at this interval */
+ uint64 num_values; /* number of integers to set in total */
+} test_spec;
+
+/* Test patterns borrowed from test_integerset.c */
+static const test_spec test_specs[] = {
+ {
+ "all ones", "1111111111",
+ 10, 1000000
+ },
+ {
+ "alternating bits", "0101010101",
+ 10, 1000000
+ },
+ {
+ "clusters of ten", "1111111111",
+ 10000, 1000000
+ },
+ {
+ "clusters of hundred",
+ "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111",
+ 10000, 1000000
+ },
+ {
+ "one-every-64k", "1",
+ 65536, 1000000
+ },
+ {
+ "sparse", "100000000000000000000000000000001",
+ 10000000, 1000000
+ },
+ {
+ "single values, distance > 2^32", "1",
+ UINT64CONST(10000000000), 100000
+ },
+ {
+ "clusters, distance > 2^32", "10101010",
+ UINT64CONST(10000000000), 1000000
+ },
+ {
+ "clusters, distance > 2^60", "10101010",
+ UINT64CONST(2000000000000000000),
+ 23 /* can't be much higher than this, or we
+ * overflow uint64 */
+ }
+};
+
+/* define the radix tree implementation to test */
+#define RT_PREFIX rt
+#define RT_SCOPE
+#define RT_DECLARE
+#define RT_DEFINE
+#define RT_USE_DELETE
+#define RT_VALUE_TYPE TestValueType
+/* #define RT_SHMEM */
+#include "lib/radixtree.h"
+
+
+/*
+ * Return the number of keys in the radix tree.
+ */
+static uint64
+rt_num_entries(rt_radix_tree *tree)
+{
+ return tree->ctl->num_keys;
+}
+
+PG_MODULE_MAGIC;
+
+PG_FUNCTION_INFO_V1(test_radixtree);
+
+static void
+test_empty(void)
+{
+ rt_radix_tree *radixtree;
+ rt_iter *iter;
+ TestValueType dummy;
+ uint64 key;
+ TestValueType val;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
+ dsa = dsa_create(tranche_id);
+
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ if (rt_search(radixtree, 0, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_search(radixtree, 1, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_search(radixtree, PG_UINT64_MAX, &dummy))
+ elog(ERROR, "rt_search on empty tree returned true");
+
+ if (rt_delete(radixtree, 0))
+ elog(ERROR, "rt_delete on empty tree returned true");
+
+ if (rt_num_entries(radixtree) != 0)
+ elog(ERROR, "rt_num_entries on empty tree return non-zero");
+
+ iter = rt_begin_iterate(radixtree);
+
+ if (rt_iterate_next(iter, &key, &val))
+ elog(ERROR, "rt_itereate_next on empty tree returned true");
+
+ rt_end_iterate(iter);
+
+ rt_free(radixtree);
+
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+static void
+test_basic(int children, bool test_inner)
+{
+ rt_radix_tree *radixtree;
+ uint64 *keys;
+ int shift = test_inner ? 8 : 0;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing basic operations with %s node %d",
+ test_inner ? "inner" : "leaf", children);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ /* prepare keys in order like 1, 32, 2, 31, 2, ... */
+ keys = palloc(sizeof(uint64) * children);
+ for (int i = 0; i < children; i++)
+ {
+ if (i % 2 == 0)
+ keys[i] = (uint64) ((i / 2) + 1) << shift;
+ else
+ keys[i] = (uint64) (children - (i / 2)) << shift;
+ }
+
+ /* insert keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (rt_set(radixtree, keys[i], (TestValueType*) &keys[i]))
+ elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+ }
+
+ /* look up keys */
+ for (int i = 0; i < children; i++)
+ {
+ TestValueType value;
+
+ if (!rt_search(radixtree, keys[i], &value))
+ elog(ERROR, "could not find key 0x" UINT64_HEX_FORMAT, keys[i]);
+ if (value != (TestValueType) keys[i])
+ elog(ERROR, "rt_search returned 0x" UINT64_HEX_FORMAT ", expected " UINT64_HEX_FORMAT,
+ value, (TestValueType) keys[i]);
+ }
+
+ /* update keys */
+ for (int i = 0; i < children; i++)
+ {
+ TestValueType update = keys[i] + 1;
+ if (!rt_set(radixtree, keys[i], (TestValueType*) &update))
+ elog(ERROR, "could not update key 0x" UINT64_HEX_FORMAT, keys[i]);
+ }
+
+ /* repeat deleting and inserting keys */
+ for (int i = 0; i < children; i++)
+ {
+ if (!rt_delete(radixtree, keys[i]))
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, keys[i]);
+ if (rt_set(radixtree, keys[i], (TestValueType*) &keys[i]))
+ elog(ERROR, "new inserted key 0x" UINT64_HEX_FORMAT " is found ", keys[i]);
+ }
+
+ pfree(keys);
+ rt_free(radixtree);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Check if keys from start to end with the shift exist in the tree.
+ */
+static void
+check_search_on_node(rt_radix_tree *radixtree, uint8 shift, int start, int end,
+ int incr)
+{
+ for (int i = start; i < end; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ TestValueType val;
+
+ if (!rt_search(radixtree, key, &val))
+ elog(ERROR, "key 0x" UINT64_HEX_FORMAT " is not found on node-%d",
+ key, end);
+ if (val != (TestValueType) key)
+ elog(ERROR, "rt_search with key 0x" UINT64_HEX_FORMAT " returns 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+ key, val, key);
+ }
+}
+
+static void
+test_node_types_insert(rt_radix_tree *radixtree, uint8 shift, bool insert_asc)
+{
+ uint64 num_entries;
+ int ninserted = 0;
+ int start = insert_asc ? 0 : 256;
+ int incr = insert_asc ? 1 : -1;
+ int end = insert_asc ? 256 : 0;
+ int node_kind_idx = 1;
+
+ for (int i = start; i != end; i += incr)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_set(radixtree, key, (TestValueType*) &key);
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " is found", key);
+
+ /*
+ * After filling all slots in each node type, check if the values
+ * are stored properly.
+ */
+ if (ninserted == rt_node_kind_fanouts[node_kind_idx] - 1)
+ {
+ int check_start = insert_asc
+ ? rt_node_kind_fanouts[node_kind_idx - 1]
+ : rt_node_kind_fanouts[node_kind_idx];
+ int check_end = insert_asc
+ ? rt_node_kind_fanouts[node_kind_idx]
+ : rt_node_kind_fanouts[node_kind_idx - 1];
+
+ check_search_on_node(radixtree, shift, check_start, check_end, incr);
+ node_kind_idx++;
+ }
+
+ ninserted++;
+ }
+
+ num_entries = rt_num_entries(radixtree);
+
+ if (num_entries != 256)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+static void
+test_node_types_delete(rt_radix_tree *radixtree, uint8 shift)
+{
+ uint64 num_entries;
+
+ for (int i = 0; i < 256; i++)
+ {
+ uint64 key = ((uint64) i << shift);
+ bool found;
+
+ found = rt_delete(radixtree, key);
+
+ if (!found)
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, key);
+ }
+
+ num_entries = rt_num_entries(radixtree);
+
+ /* The tree must be empty */
+ if (num_entries != 0)
+ elog(ERROR,
+ "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT,
+ num_entries, UINT64CONST(256));
+}
+
+/*
+ * Test for inserting and deleting key-value pairs to each node type at the given shift
+ * level.
+ */
+static void
+test_node_types(uint8 shift)
+{
+ rt_radix_tree *radixtree;
+
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing radix tree node types with shift \"%d\"", shift);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(CurrentMemoryContext, dsa, tranche_id);
+#else
+ radixtree = rt_create(CurrentMemoryContext);
+#endif
+
+ /*
+ * Insert and search entries for every node type at the 'shift' level,
+ * then delete all entries to make it empty, and insert and search entries
+ * again.
+ */
+ test_node_types_insert(radixtree, shift, true);
+ test_node_types_delete(radixtree, shift);
+ test_node_types_insert(radixtree, shift, false);
+
+ rt_free(radixtree);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+/*
+ * Test with a repeating pattern, defined by the 'spec'.
+ */
+static void
+test_pattern(const test_spec * spec)
+{
+ rt_radix_tree *radixtree;
+ rt_iter *iter;
+ MemoryContext radixtree_ctx;
+ TimestampTz starttime;
+ TimestampTz endtime;
+ uint64 n;
+ uint64 last_int;
+ uint64 ndeleted;
+ uint64 nbefore;
+ uint64 nafter;
+ int patternlen;
+ uint64 *pattern_values;
+ uint64 pattern_num_values;
+#ifdef RT_SHMEM
+ int tranche_id = LWLockNewTrancheId();
+ dsa_area *dsa;
+
+ LWLockRegisterTranche(tranche_id, "test_radix_tree");
+ dsa = dsa_create(tranche_id);
+#endif
+
+ elog(NOTICE, "testing radix tree with pattern \"%s\"", spec->test_name);
+ if (rt_test_stats)
+ fprintf(stderr, "-----\ntesting radix tree with pattern \"%s\"\n", spec->test_name);
+
+ /* Pre-process the pattern, creating an array of integers from it. */
+ patternlen = strlen(spec->pattern_str);
+ pattern_values = palloc(patternlen * sizeof(uint64));
+ pattern_num_values = 0;
+ for (int i = 0; i < patternlen; i++)
+ {
+ if (spec->pattern_str[i] == '1')
+ pattern_values[pattern_num_values++] = i;
+ }
+
+ /*
+ * Allocate the radix tree.
+ *
+ * Allocate it in a separate memory context, so that we can print its
+ * memory usage easily.
+ */
+ radixtree_ctx = AllocSetContextCreate(CurrentMemoryContext,
+ "radixtree test",
+ ALLOCSET_SMALL_SIZES);
+ MemoryContextSetIdentifier(radixtree_ctx, spec->test_name);
+
+#ifdef RT_SHMEM
+ radixtree = rt_create(radixtree_ctx, dsa, tranche_id);
+#else
+ radixtree = rt_create(radixtree_ctx);
+#endif
+
+
+ /*
+ * Add values to the set.
+ */
+ starttime = GetCurrentTimestamp();
+
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ uint64 x = 0;
+
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ bool found;
+
+ x = last_int + pattern_values[i];
+
+ found = rt_set(radixtree, x, (TestValueType*) &x);
+
+ if (found)
+ elog(ERROR, "newly inserted key 0x" UINT64_HEX_FORMAT " found", x);
+
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+
+ endtime = GetCurrentTimestamp();
+
+ if (rt_test_stats)
+ fprintf(stderr, "added " UINT64_FORMAT " values in %d ms\n",
+ spec->num_values, (int) (endtime - starttime) / 1000);
+
+ /*
+ * Print stats on the amount of memory used.
+ *
+ * We print the usage reported by rt_memory_usage(), as well as the stats
+ * from the memory context. They should be in the same ballpark, but it's
+ * hard to automate testing that, so if you're making changes to the
+ * implementation, just observe that manually.
+ */
+ if (rt_test_stats)
+ {
+ uint64 mem_usage;
+
+ /*
+ * Also print memory usage as reported by rt_memory_usage(). It
+ * should be in the same ballpark as the usage reported by
+ * MemoryContextStats().
+ */
+ mem_usage = rt_memory_usage(radixtree);
+ fprintf(stderr, "rt_memory_usage() reported " UINT64_FORMAT " (%0.2f bytes / integer)\n",
+ mem_usage, (double) mem_usage / spec->num_values);
+
+ MemoryContextStats(radixtree_ctx);
+ }
+
+ /* Check that rt_num_entries works */
+ n = rt_num_entries(radixtree);
+ if (n != spec->num_values)
+ elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT, n, spec->num_values);
+
+ /*
+ * Test random-access probes with rt_search()
+ */
+ starttime = GetCurrentTimestamp();
+
+ for (n = 0; n < 100000; n++)
+ {
+ bool found;
+ bool expected;
+ uint64 x;
+ TestValueType v;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit values that are actually in the set.
+ */
+ x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+ /* Do we expect this value to be present in the set? */
+ if (x >= last_int)
+ expected = false;
+ else
+ {
+ uint64 idx = x % spec->spacing;
+
+ if (idx >= patternlen)
+ expected = false;
+ else if (spec->pattern_str[idx] == '1')
+ expected = true;
+ else
+ expected = false;
+ }
+
+ /* Is it present according to rt_search() ? */
+ found = rt_search(radixtree, x, &v);
+
+ if (found != expected)
+ elog(ERROR, "mismatch at 0x" UINT64_HEX_FORMAT ": %d vs %d", x, found, expected);
+ if (found && (v != (TestValueType) x))
+ elog(ERROR, "found 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT,
+ v, x);
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "probed " UINT64_FORMAT " values in %d ms\n",
+ n, (int) (endtime - starttime) / 1000);
+
+ /*
+ * Test iterator
+ */
+ starttime = GetCurrentTimestamp();
+
+ iter = rt_begin_iterate(radixtree);
+ n = 0;
+ last_int = 0;
+ while (n < spec->num_values)
+ {
+ for (int i = 0; i < pattern_num_values && n < spec->num_values; i++)
+ {
+ uint64 expected = last_int + pattern_values[i];
+ uint64 x;
+ TestValueType val;
+
+ if (!rt_iterate_next(iter, &x, &val))
+ break;
+
+ if (x != expected)
+ elog(ERROR,
+ "iterate returned wrong key; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d",
+ x, expected, i);
+ if (val != (TestValueType) expected)
+ elog(ERROR,
+ "iterate returned wrong value; got 0x" UINT64_HEX_FORMAT ", expected 0x" UINT64_HEX_FORMAT " at %d", x, expected, i);
+ n++;
+ }
+ last_int += spec->spacing;
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "iterated " UINT64_FORMAT " values in %d ms\n",
+ n, (int) (endtime - starttime) / 1000);
+
+ rt_end_iterate(iter);
+
+ if (n < spec->num_values)
+ elog(ERROR, "iterator stopped short after " UINT64_FORMAT " entries, expected " UINT64_FORMAT, n, spec->num_values);
+ if (n > spec->num_values)
+ elog(ERROR, "iterator returned " UINT64_FORMAT " entries, " UINT64_FORMAT " was expected", n, spec->num_values);
+
+ /*
+ * Test random-access probes with rt_delete()
+ */
+ starttime = GetCurrentTimestamp();
+
+ nbefore = rt_num_entries(radixtree);
+ ndeleted = 0;
+ for (n = 0; n < 1; n++)
+ {
+ bool found;
+ uint64 x;
+ TestValueType v;
+
+ /*
+ * Pick next value to probe at random. We limit the probes to the
+ * last integer that we added to the set, plus an arbitrary constant
+ * (1000). There's no point in probing the whole 0 - 2^64 range, if
+ * only a small part of the integer space is used. We would very
+ * rarely hit values that are actually in the set.
+ */
+ x = pg_prng_uint64_range(&pg_global_prng_state, 0, last_int + 1000);
+
+ /* Is it present according to rt_search() ? */
+ found = rt_search(radixtree, x, &v);
+
+ if (!found)
+ continue;
+
+ /* If the key is found, delete it and check again */
+ if (!rt_delete(radixtree, x))
+ elog(ERROR, "could not delete key 0x" UINT64_HEX_FORMAT, x);
+ if (rt_search(radixtree, x, &v))
+ elog(ERROR, "found deleted key 0x" UINT64_HEX_FORMAT, x);
+ if (rt_delete(radixtree, x))
+ elog(ERROR, "deleted already-deleted key 0x" UINT64_HEX_FORMAT, x);
+
+ ndeleted++;
+ }
+ endtime = GetCurrentTimestamp();
+ if (rt_test_stats)
+ fprintf(stderr, "deleted " UINT64_FORMAT " values in %d ms\n",
+ ndeleted, (int) (endtime - starttime) / 1000);
+
+ nafter = rt_num_entries(radixtree);
+
+ /* Check that rt_num_entries works */
+ if ((nbefore - ndeleted) != nafter)
+ elog(ERROR, "rt_num_entries returned " UINT64_FORMAT ", expected " UINT64_FORMAT "after " UINT64_FORMAT " deletion",
+ nafter, (nbefore - ndeleted), ndeleted);
+
+ rt_free(radixtree);
+ MemoryContextDelete(radixtree_ctx);
+#ifdef RT_SHMEM
+ dsa_detach(dsa);
+#endif
+}
+
+Datum
+test_radixtree(PG_FUNCTION_ARGS)
+{
+ test_empty();
+
+ for (int i = 1; i < lengthof(rt_node_kind_fanouts); i++)
+ {
+ test_basic(rt_node_kind_fanouts[i], false);
+ test_basic(rt_node_kind_fanouts[i], true);
+ }
+
+ for (int shift = 0; shift <= (64 - 8); shift += 8)
+ test_node_types(shift);
+
+ /* Test different test patterns, with lots of entries */
+ for (int i = 0; i < lengthof(test_specs); i++)
+ test_pattern(&test_specs[i]);
+
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_radixtree/test_radixtree.control b/src/test/modules/test_radixtree/test_radixtree.control
new file mode 100644
index 0000000000..e53f2a3e0c
--- /dev/null
+++ b/src/test/modules/test_radixtree/test_radixtree.control
@@ -0,0 +1,4 @@
+comment = 'Test code for radix tree'
+default_version = '1.0'
+module_pathname = '$libdir/test_radixtree'
+relocatable = true
diff --git a/src/tools/pginclude/cpluspluscheck b/src/tools/pginclude/cpluspluscheck
index 4e09c4686b..202bf1c04e 100755
--- a/src/tools/pginclude/cpluspluscheck
+++ b/src/tools/pginclude/cpluspluscheck
@@ -101,6 +101,12 @@ do
test "$f" = src/include/nodes/nodetags.h && continue
test "$f" = src/backend/nodes/nodetags.h && continue
+ # radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+ test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+ test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+ test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+ test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
# These files are not meant to be included standalone, because
# they contain lists that might have multiple use-cases.
test "$f" = src/include/access/rmgrlist.h && continue
diff --git a/src/tools/pginclude/headerscheck b/src/tools/pginclude/headerscheck
index 8dee1b5670..133313255c 100755
--- a/src/tools/pginclude/headerscheck
+++ b/src/tools/pginclude/headerscheck
@@ -96,6 +96,12 @@ do
test "$f" = src/include/nodes/nodetags.h && continue
test "$f" = src/backend/nodes/nodetags.h && continue
+ # radixtree_*_impl.h cannot be included standalone: they are just code fragments.
+ test "$f" = src/include/lib/radixtree_delete_impl.h && continue
+ test "$f" = src/include/lib/radixtree_insert_impl.h && continue
+ test "$f" = src/include/lib/radixtree_iter_impl.h && continue
+ test "$f" = src/include/lib/radixtree_search_impl.h && continue
+
# These files are not meant to be included standalone, because
# they contain lists that might have multiple use-cases.
test "$f" = src/include/access/rmgrlist.h && continue
--
2.31.1
view thread (59+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: [PoC] Improve dead tuple storage for lazy vacuum
In-Reply-To: <CAD21AoCw9RPvEtYQxLCT6jO6RXF0_j8d8M-b+jaLERa1aifeCQ@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox