public inbox for [email protected]
help / color / mirror / Atom feedFrom: Bharath Rupireddy <[email protected]>
To: Andres Freund <[email protected]>
Cc: Dilip Kumar <[email protected]>
Cc: Luc Vlaming <[email protected]>
Cc: Justin Pryzby <[email protected]>
Cc: PostgreSQL-development <[email protected]>
Cc: Paul Guo <[email protected]>
Cc: Jeff Davis <[email protected]>
Cc: Michael Paquier <[email protected]>
Cc: Matthias van de Meent <[email protected]>
Subject: Re: New Table Access Methods for Multi and Single Inserts
Date: Tue, 1 Aug 2023 22:00:00 +0530
Message-ID: <CALj2ACX5UMWVFdrRNUE0KDrg54WV1cumBXwcETXhrPc1ibKAQA@mail.gmail.com> (raw)
In-Reply-To: <[email protected]>
References: <CALj2ACXdrOmB6Na9amHWZHKvRT3Z0nwTRsCwoMT-npOBtmXLXg@mail.gmail.com>
<[email protected]>
On Sun, Jun 4, 2023 at 4:08 AM Andres Freund <[email protected]> wrote:
>
> Hi,
>
> This patch was referenced in a discussion at pgcon, so I thought I'd give it a
> look, even though Bharat said that he won't have time to drive it forward...
Thanks. Finally, I started to spend time on this. Just curious - may
I know the discussion in/for which this patch is referenced? What was
the motive? Is it captured somewhere?
> On 2021-04-19 10:21:36 +0530, Bharath Rupireddy wrote:
> > + .tuple_insert_begin = heap_insert_begin,
> > + .tuple_insert_v2 = heap_insert_v2,
> > + .multi_insert_v2 = heap_multi_insert_v2,
> > + .multi_insert_flush = heap_multi_insert_flush,
> > + .tuple_insert_end = heap_insert_end,
>
> I don't think we should have multiple callback for the insertion APIs in
> tableam.h. I think it'd be good to continue supporting the old table_*()
> functions, but supporting multiple insert APIs in each AM doesn't make much
> sense to me.
I named these new functions XXX_v2 for compatibility reasons. Because,
it's quite possible for external modules to use existing
table_tuple_insert, table_multi_insert functions. If we were to change
the existing insert tableams, all the external modules using them
would have to change their code, is that okay?
> > +/*
> > + * GetTupleSize - Compute the tuple size given a table slot.
> > +inline Size
>
> I think this embeds too much knowledge of the set of slot types in core
> code. I don't see why it's needed either?
The heapam multi-insert implementation needs to know the tuple size
from the slot to decide whether or not to flush the tuples from the
buffers. I couldn't find a direct way then to know the tuple size from
the slot, so added that helper function. With a better understanding
now, I think we can rely on the memory allocated for TupleTableSlot's
tts_mcxt. While this works for the materialized slots passed in to the
insert functions, for non-materialized slots the flushing decision can
be solely on the number of tuples stored in the buffers. Another way
is to add a get_tuple_size callback to TupleTableSlotOps and let the
tuple slot providers give us the tuple size.
> > diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
> > index 414b6b4d57..2a1470a7b6 100644
> > --- a/src/include/access/tableam.h
> > +++ b/src/include/access/tableam.h
> > @@ -229,6 +229,32 @@ typedef struct TM_IndexDeleteOp
> > TM_IndexStatus *status;
> > } TM_IndexDeleteOp;
> >
> > +/* Holds table insert state. */
> > +typedef struct TableInsertState
>
> I suspect we should design it to be usable for updates and deletes in the
> future, and thus name it TableModifyState.
There are different parameters that insert/update/delete would want to
pass across in the state. So, having Table{Insert/Update/Delete}State
may be a better idea than having the unneeded variables lying around
or having a union and state_type as INSERT/UPDATE/DELETE, no? Do you
have a different thought here?
> I think we should instead have a generic TableModifyState, which each AM then
> embeds into an AM specific AM state. Forcing two very related structs to be
> allocated separately doesn't seem wise in this case.
The v7 patches have largely changed the way these options and
parameters are passed, please have a look.
> > +{
> > + Relation rel;
> > + /* Bulk insert state if requested, otherwise NULL. */
> > + struct BulkInsertStateData *bistate;
> > + CommandId cid;
>
> Hm - I'm not sure it's a good idea to force the cid to be the same for all
> inserts done via one TableInsertState.
If required, someone can always pass a new CID before every
tuple_insert_v2/tuple_multi_insert_v2 call via TableInsertState. Isn't
it sufficient?
> > @@ -1430,6 +1473,50 @@ table_multi_insert(Relation rel, TupleTableSlot **slots, int nslots,
> > cid, options, bistate);
> > }
> >
> > +static inline TableInsertState*
> > +table_insert_begin(Relation rel, CommandId cid, int options,
> > + bool alloc_bistate, bool is_multi)
>
> Why have alloc_bistate and options?
"alloc_bistate" is for the caller to specify if they need a bulk
insert state or not. "options" is for the caller to specify if they
need table_tuple_insert performance options such as
TABLE_INSERT_SKIP_FSM, TABLE_INSERT_FROZEN, TABLE_INSERT_NO_LOGICAL.
The v7 patches have changed the way these options and parameters are
passed, please have a look.
> > +static inline void
> > +table_insert_end(TableInsertState *state)
> > +{
> > + /* Deallocate bulk insert state here, since it's AM independent. */
> > + if (state->bistate)
> > + FreeBulkInsertState(state->bistate);
> > +
> > + state->rel->rd_tableam->tuple_insert_end(state);
> > +}
>
> Seems like the order in here should be swapped?
Right. It looks like BulkInsertState is for heapam, it really doesn't
have to be in table_XXX functions, hence it all the way down to
heap_insert_XXX functions.
I'm attaching the v7 patch set with the above review comments
addressed. My initial idea behind these new insert APIs was the
ability to re-use the multi insert code in COPY for CTAS and REFRESH
MATERIALIZED VIEW. I'm open to more thoughts here.
The v7 patches have largely changed the way state structure (heapam
specific things are moved all the way down to heapam.c) is defined,
the parameters are passed, and simplified the multi insert logic a
lot.
0001 - introduces new single and multi insert table AM and heapam
implementation of the new AM.
0002 - optimizes CREATE TABLE AS to use the new multi inserts table AM
making it faster by 2.13X or 53%.
0003 - optimizes REFRESH MATERIALIZED VIEW to use the new multi
inserts table AM making it faster by 1.52X or 34%.
0004 - uses the new multi inserts table AM for COPY FROM - I'm yet to
spend time on this, I'll share the patch when ready.
Thoughts?
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] v7-0001-New-table-AMs-for-single-and-multi-inserts.patch (13.7K, 2-v7-0001-New-table-AMs-for-single-and-multi-inserts.patch)
download | inline diff:
From 6803736e5695ab0ef06d263e9ba260db02d3b80c Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Tue, 1 Aug 2023 09:38:47 +0000
Subject: [PATCH v7] New table AMs for single and multi inserts
---
src/backend/access/heap/heapam.c | 180 +++++++++++++++++++++++
src/backend/access/heap/heapam_handler.c | 6 +
src/include/access/heapam.h | 45 ++++++
src/include/access/tableam.h | 107 ++++++++++++++
4 files changed, 338 insertions(+)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index 7ed72abe59..ba4347026a 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -68,6 +68,7 @@
#include "utils/datum.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
+#include "utils/memutils.h"
#include "utils/relcache.h"
#include "utils/snapmgr.h"
#include "utils/spccache.h"
@@ -75,6 +76,7 @@
static HeapTuple heap_prepare_insert(Relation relation, HeapTuple tup,
TransactionId xid, CommandId cid, int options);
+static void heap_multi_insert_flush(TableInsertState *state);
static XLogRecPtr log_heap_update(Relation reln, Buffer oldbuf,
Buffer newbuf, HeapTuple oldtup,
HeapTuple newtup, HeapTuple old_key_tuple,
@@ -2443,6 +2445,184 @@ heap_multi_insert(Relation relation, TupleTableSlot **slots, int ntuples,
pgstat_count_heap_insert(relation, ntuples);
}
+/*
+ * Initialize state required for an insert a single tuple or multiple tuples
+ * into a heap.
+ */
+TableInsertState *
+heap_insert_begin(Relation rel, CommandId cid, int table_am_flags,
+ int table_insert_flags)
+{
+ TableInsertState *tistate;
+
+ tistate = (TableInsertState *) palloc0(sizeof(TableInsertState));
+ tistate->rel = rel;
+ tistate->cid = cid;
+ tistate->table_am_flags = table_am_flags;
+ tistate->table_insert_flags = table_insert_flags;
+
+ if ((table_am_flags & TABLEAM_USE_MULTI_INSERTS) != 0 ||
+ (table_am_flags & TABLEAM_USE_BULKWRITE_BUFFER_ACCESS_STRATEGY))
+ {
+ tistate->table_am_data =
+ (HeapInsertState *) palloc0(sizeof(HeapInsertState));
+ }
+
+ if ((table_am_flags & TABLEAM_USE_MULTI_INSERTS) != 0)
+ {
+ ((HeapInsertState *) tistate->table_am_data)->mistate =
+ (HeapMultiInsertState *) palloc0(sizeof(HeapMultiInsertState));
+
+ ((HeapInsertState *) tistate->table_am_data)->mistate->slots =
+ palloc0(sizeof(TupleTableSlot *) * HEAP_MAX_BUFFERED_SLOTS);
+
+ ((HeapInsertState *) tistate->table_am_data)->mistate->context =
+ AllocSetContextCreate(CurrentMemoryContext,
+ "heap_multi_insert_v2 memory context",
+ ALLOCSET_DEFAULT_SIZES);
+ }
+
+ if ((table_am_flags & TABLEAM_USE_BULKWRITE_BUFFER_ACCESS_STRATEGY) != 0)
+ ((HeapInsertState *) tistate->table_am_data)->bistate = GetBulkInsertState();
+
+ return tistate;
+}
+
+/*
+ * Insert a single tuple into a heap.
+ */
+void
+heap_insert_v2(TableInsertState *state, TupleTableSlot *slot)
+{
+ bool shouldFree = true;
+ HeapTuple tuple = ExecFetchSlotHeapTuple(slot, true, &shouldFree);
+ BulkInsertState bistate = NULL;
+
+ /* Update tuple with table oid */
+ slot->tts_tableOid = RelationGetRelid(state->rel);
+ tuple->t_tableOid = slot->tts_tableOid;
+
+ if (state->table_am_data != NULL &&
+ ((HeapInsertState *) state->table_am_data)->bistate != NULL)
+ {
+ bistate = ((HeapInsertState *) state->table_am_data)->bistate;
+ }
+
+ /* Perform insertion, and copy the resulting ItemPointer */
+ heap_insert(state->rel, tuple, state->cid, state->table_insert_flags,
+ bistate);
+ ItemPointerCopy(&tuple->t_self, &slot->tts_tid);
+
+ if (shouldFree)
+ pfree(tuple);
+}
+
+/*
+ * Store passed-in tuple into in-memory buffered slots. When full, insert
+ * multiple tuples from the buffers into heap.
+ */
+void
+heap_multi_insert_v2(TableInsertState *state, TupleTableSlot *slot)
+{
+ TupleTableSlot *dstslot;
+ HeapMultiInsertState *mistate;
+
+ Assert(state->table_am_data != NULL &&
+ ((HeapInsertState *) state->table_am_data)->mistate != NULL);
+
+ mistate = ((HeapInsertState *) state->table_am_data)->mistate;
+ dstslot = mistate->slots[mistate->cur_slots];
+
+ if (dstslot == NULL)
+ {
+ dstslot = table_slot_create(state->rel, NULL);
+ mistate->slots[mistate->cur_slots] = dstslot;
+ }
+
+ ExecClearTuple(dstslot);
+ ExecCopySlot(dstslot, slot);
+ mistate->cur_slots++;
+
+ /*
+ * When passed-in slot is already materialized, memory allocated in slot's
+ * memory context is a close approximation for us to track the required
+ * space for the tuple in slot.
+ *
+ * For non-materialized slots, the flushing decision happens solely on the
+ * number of tuples stored in the buffer.
+ */
+ if (TTS_SHOULDFREE(slot))
+ mistate->cur_size += MemoryContextMemAllocated(slot->tts_mcxt, false);
+
+ if (mistate->cur_slots >= HEAP_MAX_BUFFERED_SLOTS ||
+ mistate->cur_size >= HEAP_MAX_BUFFERED_BYTES)
+ heap_multi_insert_flush(state);
+}
+
+/*
+ * Clean up state used to insert a single or multiple tuples into a heap.
+ */
+void
+heap_insert_end(TableInsertState *state)
+{
+ if (state->table_am_data != NULL &&
+ ((HeapInsertState *) state->table_am_data)->mistate != NULL)
+ {
+ HeapMultiInsertState *mistate =
+ ((HeapInsertState *) state->table_am_data)->mistate;
+
+ /* Insert remaining tuples from multi-insert buffers */
+ if (mistate->cur_slots > 0 || mistate->cur_size > 0)
+ heap_multi_insert_flush(state);
+
+ MemoryContextDelete(mistate->context);
+
+ for (int i = 0; i < HEAP_MAX_BUFFERED_SLOTS && mistate->slots[i] != NULL; i++)
+ ExecDropSingleTupleTableSlot(mistate->slots[i]);
+
+ pfree(mistate);
+ ((HeapInsertState *) state->table_am_data)->mistate = NULL;
+ }
+
+ if (state->table_am_data != NULL &&
+ ((HeapInsertState *) state->table_am_data)->bistate != NULL)
+ {
+ FreeBulkInsertState(((HeapInsertState *) state->table_am_data)->bistate);
+ }
+
+ pfree(state->table_am_data);
+ state->table_am_data = NULL;
+ pfree(state);
+}
+
+/*
+ * Insert multiple tuples from in-memory buffered slots into heap.
+ */
+static void
+heap_multi_insert_flush(TableInsertState *state)
+{
+ HeapMultiInsertState *mistate;
+ BulkInsertState bistate = NULL;
+ MemoryContext oldcontext;
+
+ mistate = ((HeapInsertState *) state->table_am_data)->mistate;
+
+ if (state->table_am_data != NULL &&
+ ((HeapInsertState *) state->table_am_data)->bistate != NULL)
+ {
+ bistate = ((HeapInsertState *) state->table_am_data)->bistate;
+ }
+
+ oldcontext = MemoryContextSwitchTo(mistate->context);
+ heap_multi_insert(state->rel, mistate->slots, mistate->cur_slots,
+ state->cid, state->table_insert_flags, bistate);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(mistate->context);
+
+ mistate->cur_slots = 0;
+ mistate->cur_size = 0;
+}
+
/*
* simple_heap_insert - insert a tuple
*
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 5a17112c91..6f144d88dd 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2568,6 +2568,12 @@ static const TableAmRoutine heapam_methods = {
.tuple_insert_speculative = heapam_tuple_insert_speculative,
.tuple_complete_speculative = heapam_tuple_complete_speculative,
.multi_insert = heap_multi_insert,
+
+ .tuple_insert_begin = heap_insert_begin,
+ .tuple_insert_v2 = heap_insert_v2,
+ .tuple_multi_insert_v2 = heap_multi_insert_v2,
+ .tuple_insert_end = heap_insert_end,
+
.tuple_delete = heapam_tuple_delete,
.tuple_update = heapam_tuple_update,
.tuple_lock = heapam_tuple_lock,
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index faf5026519..a1ea26cbd6 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -191,6 +191,40 @@ typedef struct HeapPageFreeze
} HeapPageFreeze;
+/*
+ * Maximum number of slots that multi-insert buffers can hold.
+ *
+ * Caution: Don't make this too big, as we could end up with this many tuples
+ * stored in multi insert buffer. For instance, increasing this can cause
+ * quadratic growth in memory requirements during copies into partitioned
+ * tables with a large number of partitions.
+ */
+#define HEAP_MAX_BUFFERED_SLOTS 1000
+
+/* Maximum size of all tuples that multi-insert buffers can hold */
+#define HEAP_MAX_BUFFERED_BYTES 65535
+
+typedef struct HeapMultiInsertState
+{
+ /* Memory context to use for flushing multi-insert buffers */
+ MemoryContext context;
+
+ /* Array of buffered slots */
+ TupleTableSlot **slots;
+
+ /* Number of slots that multi-insert buffers currently hold */
+ int cur_slots;
+
+ /* Size of all tuples that multi-insert buffers currently hold */
+ Size cur_size;
+} HeapMultiInsertState;
+
+typedef struct HeapInsertState
+{
+ struct BulkInsertStateData *bistate;
+ HeapMultiInsertState *mistate;
+} HeapInsertState;
+
/* ----------------
* function prototypes for heap access method
*
@@ -241,6 +275,17 @@ extern void heap_insert(Relation relation, HeapTuple tup, CommandId cid,
extern void heap_multi_insert(Relation relation, struct TupleTableSlot **slots,
int ntuples, CommandId cid, int options,
BulkInsertState bistate);
+
+extern TableInsertState* heap_insert_begin(Relation rel,
+ CommandId cid,
+ int table_am_flags,
+ int table_insert_flags);
+extern void heap_insert_v2(TableInsertState *state,
+ TupleTableSlot *slot);
+extern void heap_multi_insert_v2(TableInsertState *state,
+ TupleTableSlot *slot);
+extern void heap_insert_end(TableInsertState *state);
+
extern TM_Result heap_delete(Relation relation, ItemPointer tid,
CommandId cid, Snapshot crosscheck, bool wait,
struct TM_FailureData *tmfd, bool changingPart);
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index 230bc39cc0..5ea3eeee8a 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -247,6 +247,35 @@ typedef struct TM_IndexDeleteOp
TM_IndexStatus *status;
} TM_IndexDeleteOp;
+/* Use multi (buffer multiple tuples and insert them at once) inserts */
+#define TABLEAM_USE_MULTI_INSERTS 0x000001
+
+/* Use BAS_BULKWRITE buffer access strategy */
+#define TABLEAM_USE_BULKWRITE_BUFFER_ACCESS_STRATEGY 0x000002
+
+
+/* Holds table insert state. */
+typedef struct TableInsertState
+{
+ /* Table AM-agnostic data starts here */
+ Relation rel; /* Target relation */
+
+ /*
+ * Command ID for this insertion. If required, change this for each pass of
+ * insert functions.
+ */
+ CommandId cid;
+
+ /* Table AM options (TABLEAM_XXX macros) */
+ int table_am_flags;
+
+ /* table_tuple_insert performance options (TABLE_INSERT_XXX macros) */
+ int table_insert_flags;
+
+ /* Table AM specific data starts here */
+ void *table_am_data;
+} TableInsertState;
+
/* "options" flag bits for table_tuple_insert */
/* TABLE_INSERT_SKIP_WAL was 0x0001; RelationNeedsWAL() now governs */
#define TABLE_INSERT_SKIP_FSM 0x0002
@@ -522,6 +551,19 @@ typedef struct TableAmRoutine
void (*multi_insert) (Relation rel, TupleTableSlot **slots, int nslots,
CommandId cid, int options, struct BulkInsertStateData *bistate);
+ TableInsertState *(*tuple_insert_begin) (Relation rel,
+ CommandId cid,
+ int table_am_flags,
+ int table_insert_flags);
+
+ void (*tuple_insert_v2) (TableInsertState *state,
+ TupleTableSlot *slot);
+
+ void (*tuple_multi_insert_v2) (TableInsertState *state,
+ TupleTableSlot *slot);
+
+ void (*tuple_insert_end) (TableInsertState *state);
+
/* see table_tuple_delete() for reference about parameters */
TM_Result (*tuple_delete) (Relation rel,
ItemPointer tid,
@@ -1456,6 +1498,71 @@ table_multi_insert(Relation rel, TupleTableSlot **slots, int nslots,
cid, options, bistate);
}
+static inline TableInsertState *
+table_insert_begin(Relation rel, CommandId cid, int table_am_flags,
+ int table_insert_flags)
+{
+ /* XXX: Really it doesn't have to be an optional callback */
+ if (rel->rd_tableam && rel->rd_tableam->tuple_insert_begin)
+ {
+ return rel->rd_tableam->tuple_insert_begin(rel, cid, table_am_flags,
+ table_insert_flags);
+ }
+ else
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("table_insert_begin access method is not implemented for relation \"%s\"",
+ RelationGetRelationName(rel)));
+}
+
+static inline void
+table_tuple_insert_v2(TableInsertState *state, TupleTableSlot *slot)
+{
+ /* XXX: Really it doesn't have to be an optional callback */
+ if (state->rel->rd_tableam &&
+ state->rel->rd_tableam->tuple_insert_begin)
+ {
+ return state->rel->rd_tableam->tuple_insert_v2(state, slot);
+ }
+ else
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("table_tuple_insert_v2 access method is not implemented for relation \"%s\"",
+ RelationGetRelationName(state->rel)));
+}
+
+static inline void
+table_multi_insert_v2(TableInsertState *state, TupleTableSlot *slot)
+{
+ /* XXX: Really it doesn't have to be an optional callback */
+ if (state->rel->rd_tableam &&
+ state->rel->rd_tableam->tuple_insert_begin)
+ {
+ return state->rel->rd_tableam->tuple_multi_insert_v2(state, slot);
+ }
+ else
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("table_multi_insert_v2 access method is not implemented for relation \"%s\"",
+ RelationGetRelationName(state->rel)));
+}
+
+static inline void
+table_insert_end(TableInsertState *state)
+{
+ /* XXX: Really it doesn't have to be an optional callback */
+ if (state->rel->rd_tableam &&
+ state->rel->rd_tableam->tuple_insert_begin)
+ {
+ return state->rel->rd_tableam->tuple_insert_end(state);
+ }
+ else
+ ereport(ERROR,
+ errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("table_insert_end access method is not implemented for relation \"%s\"",
+ RelationGetRelationName(state->rel)));
+}
+
/*
* Delete a tuple.
*
--
2.34.1
[application/octet-stream] v7-0002-Optimize-CTAS-with-multi-inserts.patch (2.8K, 3-v7-0002-Optimize-CTAS-with-multi-inserts.patch)
download | inline diff:
From d999b3e11272f44b8cbe75fb4d5d4c518ab43324 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Tue, 1 Aug 2023 09:37:22 +0000
Subject: [PATCH v7] Optimize CTAS with multi inserts
---
src/backend/commands/createas.c | 30 ++++++++++++++----------------
1 file changed, 14 insertions(+), 16 deletions(-)
diff --git a/src/backend/commands/createas.c b/src/backend/commands/createas.c
index e91920ca14..ac30906288 100644
--- a/src/backend/commands/createas.c
+++ b/src/backend/commands/createas.c
@@ -58,9 +58,7 @@ typedef struct
/* These fields are filled by intorel_startup: */
Relation rel; /* relation to write to */
ObjectAddress reladdr; /* address of rel, for ExecCreateTableAs */
- CommandId output_cid; /* cmin to insert in output tuples */
- int ti_options; /* table_tuple_insert performance options */
- BulkInsertState bistate; /* bulk insert state */
+ TableInsertState *ti_state; /* table insert state */
} DR_intorel;
/* utility functions for CTAS definition creation */
@@ -557,17 +555,24 @@ intorel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
*/
myState->rel = intoRelationDesc;
myState->reladdr = intoRelationAddr;
- myState->output_cid = GetCurrentCommandId(true);
- myState->ti_options = TABLE_INSERT_SKIP_FSM;
/*
* If WITH NO DATA is specified, there is no need to set up the state for
* bulk inserts as there are no tuples to insert.
*/
if (!into->skipData)
- myState->bistate = GetBulkInsertState();
+ {
+ int table_am_flags = TABLEAM_USE_MULTI_INSERTS |
+ TABLEAM_USE_BULKWRITE_BUFFER_ACCESS_STRATEGY;
+ int table_insert_flags = TABLE_INSERT_SKIP_FSM;
+
+ myState->ti_state = table_insert_begin(intoRelationDesc,
+ GetCurrentCommandId(true),
+ table_am_flags,
+ table_insert_flags);
+ }
else
- myState->bistate = NULL;
+ myState->ti_state = NULL;
/*
* Valid smgr_targblock implies something already wrote to the relation.
@@ -595,11 +600,7 @@ intorel_receive(TupleTableSlot *slot, DestReceiver *self)
* would not be cheap either. This also doesn't allow accessing per-AM
* data (say a tuple's xmin), but since we don't do that here...
*/
- table_tuple_insert(myState->rel,
- slot,
- myState->output_cid,
- myState->ti_options,
- myState->bistate);
+ table_multi_insert_v2(myState->ti_state, slot);
}
/* We know this is a newly created relation, so there are no indexes */
@@ -617,10 +618,7 @@ intorel_shutdown(DestReceiver *self)
IntoClause *into = myState->into;
if (!into->skipData)
- {
- FreeBulkInsertState(myState->bistate);
- table_finish_bulk_insert(myState->rel, myState->ti_options);
- }
+ table_insert_end(myState->ti_state);
/* close rel, but keep lock until commit */
table_close(myState->rel, NoLock);
--
2.34.1
[application/octet-stream] v7-0003-Optimize-RMV-with-multi-inserts.patch (3.1K, 4-v7-0003-Optimize-RMV-with-multi-inserts.patch)
download | inline diff:
From ceb8f43687dd0baf86fac3a096d98134e06bcec8 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Tue, 1 Aug 2023 09:50:20 +0000
Subject: [PATCH v7] Optimize RMV with multi inserts
---
src/backend/commands/matview.c | 36 +++++++++++++---------------------
1 file changed, 14 insertions(+), 22 deletions(-)
diff --git a/src/backend/commands/matview.c b/src/backend/commands/matview.c
index ac2e74fa3f..c7ab2d1f44 100644
--- a/src/backend/commands/matview.c
+++ b/src/backend/commands/matview.c
@@ -52,10 +52,7 @@ typedef struct
DestReceiver pub; /* publicly-known function pointers */
Oid transientoid; /* OID of new heap into which to store */
/* These fields are filled by transientrel_startup: */
- Relation transientrel; /* relation to write to */
- CommandId output_cid; /* cmin to insert in output tuples */
- int ti_options; /* table_tuple_insert performance options */
- BulkInsertState bistate; /* bulk insert state */
+ TableInsertState *ti_state; /* table insert state */
} DR_transientrel;
static int matview_maintenance_depth = 0;
@@ -454,16 +451,18 @@ transientrel_startup(DestReceiver *self, int operation, TupleDesc typeinfo)
{
DR_transientrel *myState = (DR_transientrel *) self;
Relation transientrel;
+ int table_am_flags = TABLEAM_USE_MULTI_INSERTS |
+ TABLEAM_USE_BULKWRITE_BUFFER_ACCESS_STRATEGY;
+ int table_insert_flags = TABLE_INSERT_SKIP_FSM |
+ TABLE_INSERT_FROZEN;
transientrel = table_open(myState->transientoid, NoLock);
- /*
- * Fill private fields of myState for use by later routines
- */
- myState->transientrel = transientrel;
- myState->output_cid = GetCurrentCommandId(true);
- myState->ti_options = TABLE_INSERT_SKIP_FSM | TABLE_INSERT_FROZEN;
- myState->bistate = GetBulkInsertState();
+ /* Fill private fields of myState for use by later routines */
+ myState->ti_state = table_insert_begin(transientrel,
+ GetCurrentCommandId(true),
+ table_am_flags,
+ table_insert_flags);
/*
* Valid smgr_targblock implies something already wrote to the relation.
@@ -488,12 +487,7 @@ transientrel_receive(TupleTableSlot *slot, DestReceiver *self)
* cheap either. This also doesn't allow accessing per-AM data (say a
* tuple's xmin), but since we don't do that here...
*/
-
- table_tuple_insert(myState->transientrel,
- slot,
- myState->output_cid,
- myState->ti_options,
- myState->bistate);
+ table_multi_insert_v2(myState->ti_state, slot);
/* We know this is a newly created relation, so there are no indexes */
@@ -507,14 +501,12 @@ static void
transientrel_shutdown(DestReceiver *self)
{
DR_transientrel *myState = (DR_transientrel *) self;
+ Relation transientrel = myState->ti_state->rel;
- FreeBulkInsertState(myState->bistate);
-
- table_finish_bulk_insert(myState->transientrel, myState->ti_options);
+ table_insert_end(myState->ti_state);
/* close transientrel, but keep lock until commit */
- table_close(myState->transientrel, NoLock);
- myState->transientrel = NULL;
+ table_close(transientrel, NoLock);
}
/*
--
2.34.1
view thread (10+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: New Table Access Methods for Multi and Single Inserts
In-Reply-To: <CALj2ACX5UMWVFdrRNUE0KDrg54WV1cumBXwcETXhrPc1ibKAQA@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox