agora inbox for [email protected]help / color / mirror / Atom feed
[PATCH 6/7] Tuple routing for partitioned tables. 12+ messages / 3 participants [nested] [flat]
* [PATCH 6/7] Tuple routing for partitioned tables. @ 2016-07-27 06:47 amit <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: amit @ 2016-07-27 06:47 UTC (permalink / raw) Both COPY FROM and INSERT are covered by this commit. Routing to foreing partitions is not supported at the moment. To implement tuple-routing, introduce a PartitionDispatch data structure. Each partitioned table in a partition tree gets one and contains info such as a pointer to its partition descriptor, partition key execution state, global sequence numbers of its leaf partitions and a way to link to the PartitionDispatch objects of any of its partitions that are partitioned themselves. Starting with the PartitionDispatch object of the root partitioned table and a tuple to route, one can get the global sequence number of the leaf partition that the tuple gets routed to, if one exists. --- src/backend/catalog/partition.c | 342 +++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 154 ++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 58 ++++++- src/backend/executor/nodeModifyTable.c | 130 ++++++++++++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 11 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 8 + src/test/regress/expected/insert.out | 55 +++++ src/test/regress/sql/insert.sql | 27 +++ 11 files changed, 794 insertions(+), 6 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index f439c43..83dc151 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -113,6 +113,28 @@ typedef struct PartitionRangeBound bool lower; /* this is the lower (vs upper) bound */ } PartitionRangeBound; +/*----------------------- + * PartitionDispatch - information about one partitioned table in a partition + * hiearchy required to route a tuple to one of its partitions + * + * relid OID of the table + * key Partition key information of the table + * keystate Execution state required for expressions in the partition key + * partdesc Partition descriptor of the table + * indexes Array with partdesc->nparts members (for details on what + * individual members represent, see how they are set in + * RelationGetPartitionDispatchInfo()) + *----------------------- + */ +typedef struct PartitionDispatchData +{ + Oid relid; + PartitionKey key; + List *keystate; /* list of ExprState */ + PartitionDesc partdesc; + int *indexes; +} PartitionDispatchData; + static int32 qsort_partition_list_value_cmp(const void *a, const void *b, void *arg); static int32 qsort_partition_rbound_cmp(const void *a, const void *b, void *arg); @@ -126,12 +148,22 @@ static PartitionRangeBound *make_one_range_bound(PartitionKey key, int index, Li static int32 partition_rbound_cmp(PartitionKey key, Datum *datums1, RangeDatumContent *content1, bool lower1, PartitionRangeBound *b2); +static int32 partition_rbound_datum_cmp(PartitionKey key, + Datum *rb_datums, RangeDatumContent *rb_content, + Datum *tuple_datums); static int32 partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, int offset, void *probe, bool probe_is_bound); static int partition_bound_bsearch(PartitionKey key, PartitionBoundInfo boundinfo, void *probe, bool probe_is_bound, bool *is_equal); +/* Support get_partition_for_tuple() */ +static void FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); + /* * RelationBuildPartitionDesc * Form rel's partition descriptor @@ -895,6 +927,115 @@ RelationGetPartitionQual(Relation rel, bool recurse) return generate_partition_qual(rel, recurse); } +/* Turn an array of OIDs with N elements into a list */ +#define OID_ARRAY_TO_LIST(arr, N, list) \ + do\ + {\ + int i;\ + for (i = 0; i < (N); i++)\ + (list) = lappend_oid((list), (arr)[i]);\ + } while(0) + +/* + * RelationGetPartitionDispatchInfo + * Returns information necessary to route tuples down a partition tree + * + * All the partitions will be locked with lockmode, unless it is NoLock. + * A list of the OIDs of all the leaf partition of rel is returned in + * *leaf_part_oids. + */ +PartitionDispatch * +RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + List **leaf_part_oids) +{ + PartitionDesc rootpartdesc = RelationGetPartitionDesc(rel); + PartitionDispatchData **pd; + List *all_parts = NIL, + *parted_rels = NIL; + ListCell *lc; + int i, + k, + num_parted; + + /* + * Lock partitions and collect OIDs of the partitioned ones to prepare + * their PartitionDispatch objects. + * + * Cannot use find_all_inheritors() here, because then the order of OIDs + * in parted_rels list would be unknown, which does not help because down + * below, we assign indexes within individual PartitionDispatch in an + * order that's predetermined (determined by the order of OIDs in + * individual partition descriptors). + */ + parted_rels = lappend_oid(parted_rels, RelationGetRelid(rel)); + num_parted = 1; + OID_ARRAY_TO_LIST(rootpartdesc->oids, rootpartdesc->nparts, all_parts); + foreach(lc, all_parts) + { + Relation partrel = heap_open(lfirst_oid(lc), lockmode); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + + /* + * If this partition is a partitined table, add its children to to the + * end of the list, so that they are processed as well. + */ + if (partdesc) + { + num_parted++; + parted_rels = lappend_oid(parted_rels, lfirst_oid(lc)); + OID_ARRAY_TO_LIST(partdesc->oids, partdesc->nparts, all_parts); + } + + heap_close(partrel, NoLock); + } + + /* Generate PartitionDispatch objects for all partitioned tables */ + pd = (PartitionDispatchData **) palloc(num_parted * + sizeof(PartitionDispatchData *)); + *leaf_part_oids = NIL; + i = k = 0; + foreach(lc, parted_rels) + { + /* We locked all partitions above */ + Relation partrel = heap_open(lfirst_oid(lc), NoLock); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + int j, + m; + + pd[i] = (PartitionDispatch) palloc(sizeof(PartitionDispatchData)); + pd[i]->relid = RelationGetRelid(partrel); + pd[i]->key = RelationGetPartitionKey(partrel); + pd[i]->keystate = NIL; + pd[i]->partdesc = partdesc; + pd[i]->indexes = (int *) palloc(partdesc->nparts * sizeof(int)); + heap_close(partrel, NoLock); + + m = 0; + for (j = 0; j < partdesc->nparts; j++) + { + Oid partrelid = partdesc->oids[j]; + + if (get_rel_relkind(partrelid) != RELKIND_PARTITIONED_TABLE) + { + *leaf_part_oids = lappend_oid(*leaf_part_oids, partrelid); + pd[i]->indexes[j] = k++; + } + else + { + /* + * We can assign indexes this way because of the way + * parted_rels has been generated. + */ + pd[i]->indexes[j] = -(i + 1 + m); + m++; + } + } + i++; + } + + return pd; +} + /* Module-local functions */ /* @@ -1346,6 +1487,172 @@ generate_partition_qual(Relation rel, bool recurse) return result; } +/* ---------------- + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for the partition key + * of a tuple. + * + * pkinfo partition key execution info + * slot Heap tuple from which to extract partition key + * estate executor state for evaluating any partition key + * expressions (must be non-NULL) + * values Array of partition key Datums (output area) + * isnull Array of is-null indicators (output area) + * + * the ecxt_scantuple slot of estate's per-tuple expr context must point to + * the heap tuple passed in. + * ---------------- + */ +static void +FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pd->key->partexprs != NIL && pd->keystate == NIL) + { + /* Check caller has set up context correctly */ + Assert(estate != NULL && + GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + + /* First time through, set up expression evaluation state */ + pd->keystate = (List *) ExecPrepareExpr((Expr *) pd->key->partexprs, + estate); + } + + partexpr_item = list_head(pd->keystate); + for (i = 0; i < pd->key->partnatts; i++) + { + AttrNumber keycol = pd->key->partattrs[i]; + Datum datum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + datum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + datum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = datum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Finds a leaf partition for tuple contained in *slot + * + * Returned value is the sequence number of the leaf partition thus found, + * or -1 if no leaf partition is found for the tuple. *failed_at is set + * to the OID of the partitioned table whose partition was not found in + * the latter case. + */ +int +get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + PartitionDispatch parent; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int cur_offset, + cur_index; + int i; + + /* start with the root partitioned table */ + parent = pd[0]; + while(true) + { + PartitionKey key = parent->key; + PartitionDesc partdesc = parent->partdesc; + + /* Quick exit */ + if (partdesc->nparts == 0) + { + *failed_at = parent->relid; + return -1; + } + + /* Extract partition key from tuple */ + FormPartitionKeyDatum(parent, slot, estate, values, isnull); + + if (key->strategy == PARTITION_STRATEGY_RANGE) + { + /* Disallow nulls in the range partition key of the tuple */ + for (i = 0; i < key->partnatts; i++) + if (isnull[i]) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key of row contains null"))); + } + + if (partdesc->boundinfo->has_null && isnull[0]) + /* Tuple maps to the null-accepting list partition */ + cur_index = partdesc->boundinfo->null_index; + else + { + /* Else bsearch in partdesc->boundinfo */ + bool equal = false; + + cur_offset = partition_bound_bsearch(key, partdesc->boundinfo, + values, false, &equal); + switch (key->strategy) + { + case PARTITION_STRATEGY_LIST: + if (cur_offset >= 0 && equal) + cur_index = partdesc->boundinfo->indexes[cur_offset]; + else + cur_index = -1; + break; + + case PARTITION_STRATEGY_RANGE: + /* + * Offset returned is such that the bound at offset is + * found to be less or equal with the tuple. So, the + * bound at offset+1 would be the upper bound. + */ + cur_index = partdesc->boundinfo->indexes[cur_offset+1]; + break; + } + } + + /* + * cur_index < 0 means we failed to find a partition of this parent. + * cur_index >= 0 means we either found the leaf partition, or the + * next parent to find a partition of. + */ + if (cur_index < 0) + { + *failed_at = parent->relid; + return -1; + } + else if (parent->indexes[cur_index] < 0) + parent = pd[-parent->indexes[cur_index]]; + else + break; + } + + return parent->indexes[cur_index]; +} + /* * qsort_partition_list_value_cmp * @@ -1478,6 +1785,36 @@ partition_rbound_cmp(PartitionKey key, } /* + * partition_rbound_datum_cmp + * + * Return whether range bound (specified in rb_datums, rb_content, and + * rb_lower) <=, =, >= partition key of tuple (tuple_datums) + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, + Datum *rb_datums, RangeDatumContent *rb_content, + Datum *tuple_datums) +{ + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (rb_content[i] != RANGE_DATUM_FINITE) + return rb_content[i] == RANGE_DATUM_NEG_INF ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + rb_datums[i], + tuple_datums[i])); + if (cmpval != 0) + break; + } + + return cmpval; +} + +/* * partition_bound_cmp * * Return whether the bound at offset in boundinfo is <=, =, >= the argument @@ -1516,7 +1853,10 @@ partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, bound_datums, content, lower, (PartitionRangeBound *) probe); } - + else + cmpval = partition_rbound_datum_cmp(key, + bound_datums, content, + (Datum *) probe); break; } } diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..3f64f3a 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -161,6 +161,10 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionDispatch *partition_dispatch_info; + int num_partitions; + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1401,67 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + List *leaf_parts; + ListCell *cell; + int i, + num_leaf_parts; + ResultRelInfo *leaf_part_rri; + + /* Get the tuple-routing information and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + num_leaf_parts = list_length(leaf_parts); + cstate->partition_dispatch_info = pd; + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) palloc(num_leaf_parts * + sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation partrel; + + /* + * All partitions locked above; will be closed after CopyFrom is + * finished. + */ + partrel = heap_open(lfirst_oid(cell), NoLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(partrel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + partrel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(tupDesc, RelationGetDescr(partrel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(partrel), + gettext_noop("could not convert row type")); + leaf_part_rri++; + i++; + } + } } else { @@ -2255,6 +2320,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2347,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2458,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2486,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->partition_dispatch_info != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2442,7 +2511,11 @@ CopyFrom(CopyState cstate) values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2567,59 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition to heap_insert the tuple into */ + if (cstate->partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + cstate->partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < cstate->num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding + * to the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = cstate->partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* + * For ExecInsertIndexTuples() to work on the partition's indexes + */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the + * partition rowtype. + */ + map = cstate->partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, true); + } + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2553,7 +2679,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2704,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2723,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); @@ -2614,6 +2748,20 @@ CopyFrom(CopyState cstate) ExecCloseIndices(resultRelInfo); + /* Close all partitions and indices thereof */ + if (cstate->partition_dispatch_info) + { + int i; + + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + } + FreeExecutorState(estate); /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 71ca739..abfb46b 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1322,6 +1322,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index c7a6347..54fb771 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2990,3 +2995,52 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +/* + * ExecFindPartition -- Find a leaf partition in the partition tree rooted + * at parent, for the heap tuple contained in *slot + * + * estate must be non-NULL; we'll need it to compute any expressions in the + * partition key(s) + * + * If no leaf partition is found, this routine errors out with the appropriate + * error message, else it returns the leaf partition sequence number returned + * by get_partition_for_tuple() unchanged. + */ +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, + TupleTableSlot *slot, EState *estate) +{ + int result; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + result = get_partition_for_tuple(pd, slot, estate, &failed_at); + if (result < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return result; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 6eccfb7..ceb5d44 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,56 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + /* Determine the partition to heap_insert the tuple into */ + if (mtstate->mt_partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, true); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +562,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1622,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1713,69 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + /* Form the partition node tree and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + mtstate->mt_partition_dispatch_info = pd; + num_leaf_parts = list_length(leaf_parts); + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2093,15 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + /* * Free the exprcontext */ diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 36f8c54..88380ba 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -806,8 +806,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 70d8325..f76c5d9 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -36,6 +38,7 @@ typedef struct PartitionDescData } PartitionDescData; typedef struct PartitionDescData *PartitionDesc; +typedef struct PartitionDispatchData *PartitionDispatch; extern void RelationBuildPartitionDesc(Relation relation); extern bool partition_bounds_equal(PartitionKey key, @@ -45,4 +48,12 @@ extern void check_new_partition_bound(char *relname, Relation parent, Node *boun extern Oid get_partition_parent(Oid relid); extern List *get_qual_from_partbound(Relation rel, Relation parent, Node *bound); extern List *RelationGetPartitionQual(Relation rel, bool recurse); + +/* For tuple routing */ +extern PartitionDispatch *RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + List **leaf_part_oids); +extern int get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..b4d09f9 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..606cb21 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,13 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionDispatchData **mt_partition_dispatch_info; + /* Tuple-routing support info */ + int mt_num_partitions; /* Number of members in the + * following arrays */ + ResultRelInfo *mt_partitions; /* Per partition result relation */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per partition tuple conversion map */ } ModifyTableState; /* ---------------- diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 2e78fd9..561cefa 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -230,6 +230,61 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_ee_ff1 values ('ff', 1); insert into part_ee_ff2 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +-- fail (partition key (b+0) is null) +insert into range_parted values ('a'); +ERROR: range partition key of row contains null +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------+---+---- + part1 | a | 1 + part1 | a | 1 + part2 | a | 10 + part3 | b | 1 + part4 | b | 10 + part4 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_ee_ff not found in both cases) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_ee_ff values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_ee_ff values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +-------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff1 | ff | 1 + part_ee_ff1 | EE | 1 + part_ee_ff2 | ff | 11 + part_ee_ff2 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index eb92364..846bb58 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,33 @@ insert into part_ee_ff1 values ('cc', 1); insert into part_ee_ff1 values ('ff', 1); insert into part_ee_ff2 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +-- fail (partition key (b+0) is null) +insert into range_parted values ('a'); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_ee_ff not found in both cases) +insert into list_parted values ('EE', 0); +insert into part_ee_ff values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_ee_ff values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------AEEF86BC59A941B2E83AF069 Content-Type: text/x-diff; name="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-19.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-19"; filename*1=".patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH 6/7] Tuple routing for partitioned tables. @ 2016-07-27 06:47 amit <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: amit @ 2016-07-27 06:47 UTC (permalink / raw) Both COPY FROM and INSERT are covered by this commit. Routing to foreing partitions is not supported at the moment. To implement tuple-routing, introduce a PartitionDispatch data structure. Each partitioned table in a partition tree gets one and contains info such as a pointer to its partition descriptor, partition key execution state, global sequence numbers of its leaf partitions and a way to link to the PartitionDispatch objects of any of its partitions that are partitioned themselves. Starting with the PartitionDispatch object of the root partitioned table and a tuple to route, one can get the global sequence number of the leaf partition that the tuple gets routed to, if one exists. --- src/backend/catalog/partition.c | 328 ++++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 171 ++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 58 +++++- src/backend/executor/nodeModifyTable.c | 147 +++++++++++++++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 35 ++++ src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 10 + src/test/regress/expected/insert.out | 55 ++++++ src/test/regress/sql/insert.sql | 27 +++ 11 files changed, 840 insertions(+), 6 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index 0a4f95fc3f..b2cb742264 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -129,6 +129,9 @@ static PartitionRangeBound *make_one_range_bound(PartitionKey key, int index, static int32 partition_rbound_cmp(PartitionKey key, Datum *datums1, RangeDatumContent *content1, bool lower1, PartitionRangeBound *b2); +static int32 partition_rbound_datum_cmp(PartitionKey key, + Datum *rb_datums, RangeDatumContent *rb_content, + Datum *tuple_datums); static int32 partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, @@ -137,6 +140,13 @@ static int partition_bound_bsearch(PartitionKey key, PartitionBoundInfo boundinfo, void *probe, bool probe_is_bound, bool *is_equal); +/* Support get_partition_for_tuple() */ +static void FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); + /* * RelationBuildPartitionDesc * Form rel's partition descriptor @@ -916,6 +926,119 @@ RelationGetPartitionQual(Relation rel, bool recurse) return generate_partition_qual(rel, recurse); } +/* Turn an array of OIDs with N elements into a list */ +#define OID_ARRAY_TO_LIST(arr, N, list) \ + do\ + {\ + int i;\ + for (i = 0; i < (N); i++)\ + (list) = lappend_oid((list), (arr)[i]);\ + } while(0) + +/* + * RelationGetPartitionDispatchInfo + * Returns information necessary to route tuples down a partition tree + * + * All the partitions will be locked with lockmode, unless it is NoLock. + * A list of the OIDs of all the leaf partition of rel is returned in + * *leaf_part_oids. + */ +PartitionDispatch * +RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + int *num_parted, List **leaf_part_oids) +{ + PartitionDesc rootpartdesc = RelationGetPartitionDesc(rel); + PartitionDispatchData **pd; + List *all_parts = NIL, + *parted_rels; + ListCell *lc; + int i, + k; + + /* + * Lock partitions and make a list of the partitioned ones to prepare + * their PartitionDispatch objects below. + * + * Cannot use find_all_inheritors() here, because then the order of OIDs + * in parted_rels list would be unknown, which does not help, because we + * we assign indexes within individual PartitionDispatch in an order that + * is predetermined (determined by the order of OIDs in individual + * partition descriptors). + */ + *num_parted = 1; + parted_rels = list_make1(rel); + OID_ARRAY_TO_LIST(rootpartdesc->oids, rootpartdesc->nparts, all_parts); + foreach(lc, all_parts) + { + Relation partrel = heap_open(lfirst_oid(lc), lockmode); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + + /* + * If this partition is a partitioned table, add its children to the + * end of the list, so that they are processed as well. + */ + if (partdesc) + { + (*num_parted)++; + parted_rels = lappend(parted_rels, partrel); + OID_ARRAY_TO_LIST(partdesc->oids, partdesc->nparts, all_parts); + } + else + heap_close(partrel, NoLock); + + /* + * We keep the partitioned ones open until we're done using the + * information being collected here (for example, see + * ExecEndModifyTable). + */ + } + + /* Generate PartitionDispatch objects for all partitioned tables */ + pd = (PartitionDispatchData **) palloc(*num_parted * + sizeof(PartitionDispatchData *)); + *leaf_part_oids = NIL; + i = k = 0; + foreach(lc, parted_rels) + { + Relation partrel = lfirst(lc); + PartitionKey partkey = RelationGetPartitionKey(partrel); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + int j, + m; + + pd[i] = (PartitionDispatch) palloc(sizeof(PartitionDispatchData)); + pd[i]->reldesc = partrel; + pd[i]->key = partkey; + pd[i]->keystate = NIL; + pd[i]->partdesc = partdesc; + pd[i]->indexes = (int *) palloc(partdesc->nparts * sizeof(int)); + + m = 0; + for (j = 0; j < partdesc->nparts; j++) + { + Oid partrelid = partdesc->oids[j]; + + if (get_rel_relkind(partrelid) != RELKIND_PARTITIONED_TABLE) + { + *leaf_part_oids = lappend_oid(*leaf_part_oids, partrelid); + pd[i]->indexes[j] = k++; + } + else + { + /* + * We can assign indexes this way because of the way + * parted_rels has been generated. + */ + pd[i]->indexes[j] = -(i + 1 + m); + m++; + } + } + i++; + } + + return pd; +} + /* Module-local functions */ /* @@ -1367,6 +1490,176 @@ generate_partition_qual(Relation rel, bool recurse) return result; } +/* ---------------- + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for the partition key + * of a tuple. + * + * pkinfo partition key execution info + * slot Heap tuple from which to extract partition key + * estate executor state for evaluating any partition key + * expressions (must be non-NULL) + * values Array of partition key Datums (output area) + * isnull Array of is-null indicators (output area) + * + * the ecxt_scantuple slot of estate's per-tuple expr context must point to + * the heap tuple passed in. + * ---------------- + */ +static void +FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pd->key->partexprs != NIL && pd->keystate == NIL) + { + /* Check caller has set up context correctly */ + Assert(estate != NULL && + GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + + /* First time through, set up expression evaluation state */ + pd->keystate = (List *) ExecPrepareExpr((Expr *) pd->key->partexprs, + estate); + } + + partexpr_item = list_head(pd->keystate); + for (i = 0; i < pd->key->partnatts; i++) + { + AttrNumber keycol = pd->key->partattrs[i]; + Datum datum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + datum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + datum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = datum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Finds a leaf partition for tuple contained in *slot + * + * Returned value is the sequence number of the leaf partition thus found, + * or -1 if no leaf partition is found for the tuple. *failed_at is set + * to the OID of the partitioned table whose partition was not found in + * the latter case. + */ +int +get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + PartitionDispatch parent; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int cur_offset, + cur_index; + int i; + + /* start with the root partitioned table */ + parent = pd[0]; + while(true) + { + PartitionKey key = parent->key; + PartitionDesc partdesc = parent->partdesc; + + /* Quick exit */ + if (partdesc->nparts == 0) + { + *failed_at = RelationGetRelid(parent->reldesc); + return -1; + } + + /* Extract partition key from tuple */ + FormPartitionKeyDatum(parent, slot, estate, values, isnull); + + if (key->strategy == PARTITION_STRATEGY_RANGE) + { + /* Disallow nulls in the range partition key of the tuple */ + for (i = 0; i < key->partnatts; i++) + if (isnull[i]) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key of row contains null"))); + } + + if (partdesc->boundinfo->has_null && isnull[0]) + /* Tuple maps to the null-accepting list partition */ + cur_index = partdesc->boundinfo->null_index; + else + { + /* Else bsearch in partdesc->boundinfo */ + bool equal = false; + + cur_offset = partition_bound_bsearch(key, partdesc->boundinfo, + values, false, &equal); + switch (key->strategy) + { + case PARTITION_STRATEGY_LIST: + if (cur_offset >= 0 && equal) + cur_index = partdesc->boundinfo->indexes[cur_offset]; + else + cur_index = -1; + break; + + case PARTITION_STRATEGY_RANGE: + /* + * Offset returned is such that the bound at offset is + * found to be less or equal with the tuple. So, the + * bound at offset+1 would be the upper bound. + */ + cur_index = partdesc->boundinfo->indexes[cur_offset+1]; + break; + + default: + elog(ERROR, "unexpected partition strategy: %d", + (int) key->strategy); + } + } + + /* + * cur_index < 0 means we failed to find a partition of this parent. + * cur_index >= 0 means we either found the leaf partition, or the + * next parent to find a partition of. + */ + if (cur_index < 0) + { + *failed_at = RelationGetRelid(parent->reldesc); + return -1; + } + else if (parent->indexes[cur_index] < 0) + parent = pd[-parent->indexes[cur_index]]; + else + break; + } + + return parent->indexes[cur_index]; +} + /* * qsort_partition_list_value_cmp * @@ -1499,6 +1792,36 @@ partition_rbound_cmp(PartitionKey key, } /* + * partition_rbound_datum_cmp + * + * Return whether range bound (specified in rb_datums, rb_content, and + * rb_lower) <=, =, >= partition key of tuple (tuple_datums) + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, + Datum *rb_datums, RangeDatumContent *rb_content, + Datum *tuple_datums) +{ + int i; + int32 cmpval = -1; + + for (i = 0; i < key->partnatts; i++) + { + if (rb_content[i] != RANGE_DATUM_FINITE) + return rb_content[i] == RANGE_DATUM_NEG_INF ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + rb_datums[i], + tuple_datums[i])); + if (cmpval != 0) + break; + } + + return cmpval; +} + +/* * partition_bound_cmp * * Return whether the bound at offset in boundinfo is <=, =, >= the argument @@ -1537,7 +1860,10 @@ partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, bound_datums, content, lower, (PartitionRangeBound *) probe); } - + else + cmpval = partition_rbound_datum_cmp(key, + bound_datums, content, + (Datum *) probe); break; } diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index de9e29c81e..3f8d748a82 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -161,6 +161,11 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionDispatch *partition_dispatch_info; + int num_dispatch; + int num_partitions; + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1402,71 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + List *leaf_parts; + ListCell *cell; + int i, + num_parted, + num_leaf_parts; + ResultRelInfo *leaf_part_rri; + + /* Get the tuple-routing information and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &num_parted, &leaf_parts); + num_leaf_parts = list_length(leaf_parts); + cstate->partition_dispatch_info = pd; + cstate->num_dispatch = num_parted; + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) palloc(num_leaf_parts * + sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation partrel; + + /* + * We locked all the partitions above including the leaf + * partitions. Note that each of the relations in + * cstate->partitions will be closed by CopyFrom() after + * it's finished with its processing. + */ + partrel = heap_open(lfirst_oid(cell), NoLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(partrel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + partrel, + 1, /* dummy */ + false, /* no partition constraint check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(tupDesc, RelationGetDescr(partrel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(partrel), + gettext_noop("could not convert row type")); + leaf_part_rri++; + i++; + } + } } else { @@ -2255,6 +2325,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2352,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2463,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2491,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->partition_dispatch_info != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2442,7 +2516,11 @@ CopyFrom(CopyState cstate) values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2572,59 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition to heap_insert the tuple into */ + if (cstate->partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + cstate->partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < cstate->num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding + * to the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = cstate->partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* + * For ExecInsertIndexTuples() to work on the partition's indexes + */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the + * partition rowtype. + */ + map = cstate->partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, true); + } + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2553,7 +2684,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2709,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2728,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); @@ -2614,6 +2753,32 @@ CopyFrom(CopyState cstate) ExecCloseIndices(resultRelInfo); + /* Close all the partitioned tables, leaf partitions, and their indices */ + if (cstate->partition_dispatch_info) + { + int i; + + /* + * Remember cstate->partition_dispatch_info[0] corresponds to the root + * partitioned table, which we must not try to close, because it is + * the main target table of COPY that will be closed eventually by + * DoCopy(). + */ + for (i = 1; i < cstate->num_dispatch; i++) + { + PartitionDispatch pd = cstate->partition_dispatch_info[i]; + + heap_close(pd->reldesc, NoLock); + } + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + } + FreeExecutorState(estate); /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8a803233ca..c77b216d4f 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1322,6 +1322,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index 21b18b82b9..0f47c7e010 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2991,3 +2996,52 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +/* + * ExecFindPartition -- Find a leaf partition in the partition tree rooted + * at parent, for the heap tuple contained in *slot + * + * estate must be non-NULL; we'll need it to compute any expressions in the + * partition key(s) + * + * If no leaf partition is found, this routine errors out with the appropriate + * error message, else it returns the leaf partition sequence number returned + * by get_partition_for_tuple() unchanged. + */ +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, + TupleTableSlot *slot, EState *estate) +{ + int result; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + result = get_partition_for_tuple(pd, slot, estate, &failed_at); + if (result < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return result; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 6eccfb7cec..c0b58d1841 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,56 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + /* Determine the partition to heap_insert the tuple into */ + if (mtstate->mt_partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, true); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +562,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1622,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1713,75 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + int i, + j, + num_parted, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + /* Form the partition node tree and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &num_parted, &leaf_parts); + mtstate->mt_partition_dispatch_info = pd; + mtstate->mt_num_dispatch = num_parted; + num_leaf_parts = list_length(leaf_parts); + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid partrelid = lfirst_oid(cell); + Relation partrel; + + /* + * We locked all the partitions above including the leaf + * partitions. Note that each of the relations in + * mtstate->mt_partitions will be closed by ExecEndModifyTable(). + */ + partrel = heap_open(partrelid, NoLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(partrel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + partrel, + 1, /* dummy */ + false, /* no partition constraint checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (partrel->rd_rel->relhasindex && operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(partrel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(partrel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2099,26 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all the partitioned tables, leaf partitions, and their indices + * + * Remember node->mt_partition_dispatch_info[0] corresponds to the root + * partitioned table, which we must not try to close, because it is the + * main target table of the query that will be closed by ExecEndPlan(). + */ + for (i = 1; i < node->mt_num_dispatch; i++) + { + PartitionDispatch pd = node->mt_partition_dispatch_info[i]; + + heap_close(pd->reldesc, NoLock); + } + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + /* * Free the exprcontext */ diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 1a541788eb..7364346167 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -806,8 +806,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 70d8325137..21effbf87b 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -37,6 +39,30 @@ typedef struct PartitionDescData typedef struct PartitionDescData *PartitionDesc; +/*----------------------- + * PartitionDispatch - information about one partitioned table in a partition + * hiearchy required to route a tuple to one of its partitions + * + * reldesc Relation descriptor of the table + * key Partition key information of the table + * keystate Execution state required for expressions in the partition key + * partdesc Partition descriptor of the table + * indexes Array with partdesc->nparts members (for details on what + * individual members represent, see how they are set in + * RelationGetPartitionDispatchInfo()) + *----------------------- + */ +typedef struct PartitionDispatchData +{ + Relation reldesc; + PartitionKey key; + List *keystate; /* list of ExprState */ + PartitionDesc partdesc; + int *indexes; +} PartitionDispatchData; + +typedef struct PartitionDispatchData *PartitionDispatch; + extern void RelationBuildPartitionDesc(Relation relation); extern bool partition_bounds_equal(PartitionKey key, PartitionBoundInfo p1, PartitionBoundInfo p2); @@ -45,4 +71,13 @@ extern void check_new_partition_bound(char *relname, Relation parent, Node *boun extern Oid get_partition_parent(Oid relid); extern List *get_qual_from_partbound(Relation rel, Relation parent, Node *bound); extern List *RelationGetPartitionQual(Relation rel, bool recurse); + +/* For tuple routing */ +extern PartitionDispatch *RelationGetPartitionDispatchInfo(Relation rel, + int lockmode, int *num_parted, + List **leaf_part_oids); +extern int get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276be53..b4d09f9564 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index df2dec3a2c..1de5c8196d 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,15 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionDispatchData **mt_partition_dispatch_info; + /* Tuple-routing support info */ + int mt_num_dispatch; /* Number of entries in the above + * array */ + int mt_num_partitions; /* Number of members in the + * following arrays */ + ResultRelInfo *mt_partitions; /* Per partition result relation */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per partition tuple conversion map */ } ModifyTableState; /* ---------------- diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 2e78fd9fc1..561cefa3c4 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -230,6 +230,61 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_ee_ff1 values ('ff', 1); insert into part_ee_ff2 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +-- fail (partition key (b+0) is null) +insert into range_parted values ('a'); +ERROR: range partition key of row contains null +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------+---+---- + part1 | a | 1 + part1 | a | 1 + part2 | a | 10 + part3 | b | 1 + part4 | b | 10 + part4 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_ee_ff not found in both cases) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_ee_ff values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_ee_ff values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +-------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff1 | ff | 1 + part_ee_ff1 | EE | 1 + part_ee_ff2 | ff | 11 + part_ee_ff2 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index eb923646c3..846bb5897a 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,33 @@ insert into part_ee_ff1 values ('cc', 1); insert into part_ee_ff1 values ('ff', 1); insert into part_ee_ff2 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +-- fail (partition key (b+0) is null) +insert into range_parted values ('a'); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_ee_ff not found in both cases) +insert into list_parted values ('EE', 0); +insert into part_ee_ff values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_ee_ff values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 2.11.0 --------------E1933196A000B12EC95A8B6E Content-Type: text/x-diff; name="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-20.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-20"; filename*1=".patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH 6/7] Tuple routing for partitioned tables. @ 2016-07-27 06:47 amit <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: amit @ 2016-07-27 06:47 UTC (permalink / raw) Both COPY FROM and INSERT are covered by this commit. Routing to foreing partitions is not supported at the moment. To implement tuple-routing, introduce a PartitionDispatch data structure. Each partitioned table in a partition tree gets one and contains info such as a pointer to its partition descriptor, partition key execution state, global sequence numbers of its leaf partitions and a way to link to the PartitionDispatch objects of any of its partitions that are partitioned themselves. Starting with the PartitionDispatch object of the root partitioned table and a tuple to route, one can get the global sequence number of the leaf partition that the tuple gets routed to, if one exists. --- src/backend/catalog/partition.c | 342 ++++++++++++++++++++++++++++++++ src/backend/commands/copy.c | 151 ++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 58 ++++++- src/backend/executor/nodeModifyTable.c | 130 ++++++++++++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 11 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 8 + src/test/regress/expected/insert.out | 52 +++++ src/test/regress/sql/insert.sql | 25 +++ 11 files changed, 787 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index ba22453..643a258 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -109,6 +109,28 @@ typedef struct PartitionRangeBound bool lower; /* this is the lower (vs upper) bound */ } PartitionRangeBound; +/*----------------------- + * PartitionDispatch - information about one partitioned table in a partition + * hiearchy required to route a tuple to one of its partitions + * + * relid OID of the table + * key Partition key information of the table + * keystate Execution state required for expressions in the partition key + * partdesc Partition descriptor of the table + * indexes Array with partdesc->nparts members (for details on what + * individual members represent, see how they are set in + * RelationGetPartitionDispatchInfo()) + *----------------------- + */ +typedef struct PartitionDispatchData +{ + Oid relid; + PartitionKey key; + List *keystate; /* list of ExprState */ + PartitionDesc partdesc; + int *indexes; +} PartitionDispatchData; + static int32 qsort_partition_list_value_cmp(const void *a, const void *b, void *arg); static int32 qsort_partition_rbound_cmp(const void *a, const void *b, void *arg); @@ -122,12 +144,21 @@ static PartitionRangeBound *make_one_range_bound(PartitionKey key, int index, Li static int32 partition_rbound_cmp(PartitionKey key, Datum *datums1, bool *inf1, bool lower1, PartitionRangeBound *b2); +static int32 partition_rbound_datum_cmp(PartitionKey key, Datum *rb_datums, bool *rb_inf, + Datum *tuple_datums); static int32 partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, int offset, void *probe, bool probe_is_bound); static int partition_bound_bsearch(PartitionKey key, PartitionBoundInfo boundinfo, void *probe, bool probe_is_bound, bool *is_equal); +/* Support get_partition_for_tuple() */ +static void FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); + /* * RelationBuildPartitionDesc * Form rel's partition descriptor @@ -888,6 +919,115 @@ RelationGetPartitionQual(Relation rel, bool recurse) return generate_partition_qual(rel, recurse); } +/* Turn an array of OIDs with N elements into a list */ +#define OID_ARRAY_TO_LIST(arr, N, list) \ + do\ + {\ + int i;\ + for (i = 0; i < (N); i++)\ + (list) = lappend_oid((list), (arr)[i]);\ + } while(0) + +/* + * RelationGetPartitionDispatchInfo + * Returns information necessary to route tuples down a partition tree + * + * All the partitions will be locked with lockmode, unless it is NoLock. + * A list of the OIDs of all the leaf partition of rel is returned in + * *leaf_part_oids. + */ +PartitionDispatch * +RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + List **leaf_part_oids) +{ + PartitionDesc rootpartdesc = RelationGetPartitionDesc(rel); + PartitionDispatchData **pd; + List *all_parts = NIL, + *parted_rels = NIL; + ListCell *lc; + int i, + k, + num_parted; + + /* + * Lock partitions and collect OIDs of the partitioned ones to prepare + * their PartitionDispatch objects. + * + * Cannot use find_all_inheritors() here, because then the order of OIDs + * in parted_rels list would be unknown, which does not help because down + * below, we assign indexes within individual PartitionDispatch in an + * order that's predetermined (determined by the order of OIDs in + * individual partition descriptors). + */ + parted_rels = lappend_oid(parted_rels, RelationGetRelid(rel)); + num_parted = 1; + OID_ARRAY_TO_LIST(rootpartdesc->oids, rootpartdesc->nparts, all_parts); + foreach(lc, all_parts) + { + Relation partrel = heap_open(lfirst_oid(lc), lockmode); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + + /* + * If this partition is a partitined table, add its children to to the + * end of the list, so that they are processed as well. + */ + if (partdesc) + { + num_parted++; + parted_rels = lappend_oid(parted_rels, lfirst_oid(lc)); + OID_ARRAY_TO_LIST(partdesc->oids, partdesc->nparts, all_parts); + } + + heap_close(partrel, NoLock); + } + + /* Generate PartitionDispatch objects for all partitioned tables */ + pd = (PartitionDispatchData **) palloc(num_parted * + sizeof(PartitionDispatchData *)); + *leaf_part_oids = NIL; + i = k = 0; + foreach(lc, parted_rels) + { + /* We locked all partitions above */ + Relation partrel = heap_open(lfirst_oid(lc), NoLock); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + int j, + m; + + pd[i] = (PartitionDispatch) palloc(sizeof(PartitionDispatchData)); + pd[i]->relid = RelationGetRelid(partrel); + pd[i]->key = RelationGetPartitionKey(partrel); + pd[i]->keystate = NIL; + pd[i]->partdesc = partdesc; + pd[i]->indexes = (int *) palloc(partdesc->nparts * sizeof(int)); + heap_close(partrel, NoLock); + + m = 0; + for (j = 0; j < partdesc->nparts; j++) + { + Oid partrelid = partdesc->oids[j]; + + if (get_rel_relkind(partrelid) != RELKIND_PARTITIONED_TABLE) + { + *leaf_part_oids = lappend_oid(*leaf_part_oids, partrelid); + pd[i]->indexes[j] = k++; + } + else + { + /* + * We can assign indexes this way because of the way + * parted_rels has been generated. + */ + pd[i]->indexes[j] = -(i + 1 + m); + m++; + } + } + i++; + } + + return pd; +} + /* Module-local functions */ /* @@ -1328,6 +1468,172 @@ generate_partition_qual(Relation rel, bool recurse) return result; } +/* ---------------- + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for the partition key + * of a tuple. + * + * pkinfo partition key execution info + * slot Heap tuple from which to extract partition key + * estate executor state for evaluating any partition key + * expressions (must be non-NULL) + * values Array of partition key Datums (output area) + * isnull Array of is-null indicators (output area) + * + * the ecxt_scantuple slot of estate's per-tuple expr context must point to + * the heap tuple passed in. + * ---------------- + */ +static void +FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pd->key->partexprs != NIL && pd->keystate == NIL) + { + /* Check caller has set up context correctly */ + Assert(estate != NULL && + GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + + /* First time through, set up expression evaluation state */ + pd->keystate = (List *) ExecPrepareExpr((Expr *) pd->key->partexprs, + estate); + } + + partexpr_item = list_head(pd->keystate); + for (i = 0; i < pd->key->partnatts; i++) + { + AttrNumber keycol = pd->key->partattrs[i]; + Datum datum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + datum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + datum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = datum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Finds a leaf partition for tuple contained in *slot + * + * Returned value is the sequence number of the leaf partition thus found, + * or -1 if no leaf partition is found for the tuple. *failed_at is set + * to the OID of the partitioned table whose partition was not found in + * the latter case. + */ +int +get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + PartitionDispatch parent; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int cur_offset, + cur_index; + int i; + + /* start with the root partitioned table */ + parent = pd[0]; + while(true) + { + PartitionKey key = parent->key; + PartitionDesc partdesc = parent->partdesc; + + /* Quick exit */ + if (partdesc->nparts == 0) + { + *failed_at = parent->relid; + return -1; + } + + /* Extract partition key from tuple */ + FormPartitionKeyDatum(parent, slot, estate, values, isnull); + + if (key->strategy == PARTITION_STRATEGY_RANGE) + { + /* Disallow nulls in the range partition key of the tuple */ + for (i = 0; i < key->partnatts; i++) + if (isnull[i]) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key of row contains null"))); + } + + if (partdesc->boundinfo->has_null && isnull[0]) + /* Tuple maps to the null-accepting list partition */ + cur_index = partdesc->boundinfo->null_index; + else + { + /* Else bsearch in partdesc->boundinfo */ + bool equal = false; + + cur_offset = partition_bound_bsearch(key, partdesc->boundinfo, + values, false, &equal); + switch (key->strategy) + { + case PARTITION_STRATEGY_LIST: + if (cur_offset >= 0 && equal) + cur_index = partdesc->boundinfo->indexes[cur_offset]; + else + cur_index = -1; + break; + + case PARTITION_STRATEGY_RANGE: + /* + * Offset returned is such that the bound at offset is + * found to be less or equal with the tuple. So, the + * bound at offset+1 would be the upper bound. + */ + cur_index = partdesc->boundinfo->indexes[cur_offset+1]; + break; + } + } + + /* + * cur_index < 0 means we failed to find a partition of this parent. + * cur_index >= 0 means we either found the leaf partition, or the + * next parent to find a partition of. + */ + if (cur_index < 0) + { + *failed_at = parent->relid; + return -1; + } + else if (parent->indexes[cur_index] < 0) + parent = pd[-parent->indexes[cur_index]]; + else + break; + } + + return parent->indexes[cur_index]; +} + /* List partition related support functions */ /* @@ -1461,6 +1767,35 @@ partition_rbound_cmp(PartitionKey key, } /* + * partition_rbound_datum_cmp + * + * Return whether range bound (specified in datums, finite, and lower) is + * <=, =, >= partition key of tuple (tuple_datums) + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, Datum *rb_datums, bool *rb_inf, + Datum *tuple_datums) +{ + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (rb_inf[i]) + return rb_inf[i] == RANGE_DATUM_NEG_INF ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + rb_datums[i], + tuple_datums[i])); + if (cmpval != 0) + break; + } + + return cmpval; +} + +/* * partition_bound_cmp * * Return whether the bound at offset in boundinfo is <=, =, >= the argument @@ -1502,6 +1837,13 @@ partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, cmpval = partition_rbound_cmp(key, datums1, inf1, lower1, b2); } + else + { + Datum *datums2 = (Datum *) probe; + + cmpval = partition_rbound_datum_cmp(key, datums1, inf1, + datums2); + } } break; } diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..7d76ead 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -161,6 +161,10 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionDispatch *partition_dispatch_info; + int num_partitions; + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1401,67 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + List *leaf_parts; + ListCell *cell; + int i, + num_leaf_parts; + ResultRelInfo *leaf_part_rri; + + /* Get the tuple-routing information and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + num_leaf_parts = list_length(leaf_parts); + cstate->partition_dispatch_info = pd; + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) palloc(num_leaf_parts * + sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation partrel; + + /* + * All partitions locked above; will be closed after CopyFrom is + * finished. + */ + partrel = heap_open(lfirst_oid(cell), NoLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(partrel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + partrel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(tupDesc, RelationGetDescr(partrel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(partrel), + gettext_noop("could not convert row type")); + leaf_part_rri++; + i++; + } + } } else { @@ -2255,6 +2320,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2347,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2458,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2486,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->partition_dispatch_info != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2442,7 +2511,11 @@ CopyFrom(CopyState cstate) values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2567,56 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition to heap_insert the tuple into */ + if (cstate->partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + cstate->partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < cstate->num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding + * to the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = cstate->partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* + * For ExecInsertIndexTuples() to work on the partition's indexes + */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the + * partition rowtype. + */ + map = cstate->partition_tupconv_maps[leaf_part_index]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2553,7 +2676,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2701,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2720,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); @@ -2614,6 +2745,20 @@ CopyFrom(CopyState cstate) ExecCloseIndices(resultRelInfo); + /* Close all partitions and indices thereof */ + if (cstate->partition_dispatch_info) + { + int i; + + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + } + FreeExecutorState(estate); /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index b27da1d..be08eff 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1293,6 +1293,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index c7a6347..54fb771 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2990,3 +2995,52 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +/* + * ExecFindPartition -- Find a leaf partition in the partition tree rooted + * at parent, for the heap tuple contained in *slot + * + * estate must be non-NULL; we'll need it to compute any expressions in the + * partition key(s) + * + * If no leaf partition is found, this routine errors out with the appropriate + * error message, else it returns the leaf partition sequence number returned + * by get_partition_for_tuple() unchanged. + */ +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, + TupleTableSlot *slot, EState *estate) +{ + int result; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + result = get_partition_for_tuple(pd, slot, estate, &failed_at); + if (result < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return result; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..aa1d8b9 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,56 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + /* Determine the partition to heap_insert the tuple into */ + if (mtstate->mt_partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, false); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +562,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1622,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1713,69 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + /* Form the partition node tree and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + mtstate->mt_partition_dispatch_info = pd; + num_leaf_parts = list_length(leaf_parts); + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2093,15 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + /* * Free the exprcontext */ diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 70d8325..f76c5d9 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -36,6 +38,7 @@ typedef struct PartitionDescData } PartitionDescData; typedef struct PartitionDescData *PartitionDesc; +typedef struct PartitionDispatchData *PartitionDispatch; extern void RelationBuildPartitionDesc(Relation relation); extern bool partition_bounds_equal(PartitionKey key, @@ -45,4 +48,12 @@ extern void check_new_partition_bound(char *relname, Relation parent, Node *boun extern Oid get_partition_parent(Oid relid); extern List *get_qual_from_partbound(Relation rel, Relation parent, Node *bound); extern List *RelationGetPartitionQual(Relation rel, bool recurse); + +/* For tuple routing */ +extern PartitionDispatch *RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + List **leaf_part_oids); +extern int get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..b4d09f9 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..606cb21 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,13 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionDispatchData **mt_partition_dispatch_info; + /* Tuple-routing support info */ + int mt_num_partitions; /* Number of members in the + * following arrays */ + ResultRelInfo *mt_partitions; /* Per partition result relation */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per partition tuple conversion map */ } ModifyTableState; /* ---------------- diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 9ae6b09..d5dcb59 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -227,6 +227,58 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index b6e821e..fbd30d9 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,31 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------F7EE68175C11124DB3D461AC Content-Type: text/x-diff; name="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-17.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-17"; filename*1=".patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH 6/7] Tuple routing for partitioned tables. @ 2016-07-27 06:47 amit <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: amit @ 2016-07-27 06:47 UTC (permalink / raw) Both COPY FROM and INSERT are covered by this commit. Routing to foreing partitions is not supported at the moment. To implement tuple-routing, introduce a PartitionDispatch data structure. Each partitioned table in a partition tree gets one and contains info such as a pointer to its partition descriptor, partition key execution state, global sequence numbers of its leaf partitions and a way to link to the PartitionDispatch objects of any of its partitions that are partitioned themselves. Starting with the PartitionDispatch object of the root partitioned table and a tuple to route, one can get the global sequence number of the leaf partition that the tuple gets routed to, if one exists. --- src/backend/catalog/partition.c | 342 +++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 151 ++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 58 ++++++- src/backend/executor/nodeModifyTable.c | 130 ++++++++++++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 11 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 8 + src/test/regress/expected/insert.out | 52 +++++ src/test/regress/sql/insert.sql | 25 +++ 11 files changed, 786 insertions(+), 6 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index 197fcef..710829a 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -113,6 +113,28 @@ typedef struct PartitionRangeBound bool lower; /* this is the lower (vs upper) bound */ } PartitionRangeBound; +/*----------------------- + * PartitionDispatch - information about one partitioned table in a partition + * hiearchy required to route a tuple to one of its partitions + * + * relid OID of the table + * key Partition key information of the table + * keystate Execution state required for expressions in the partition key + * partdesc Partition descriptor of the table + * indexes Array with partdesc->nparts members (for details on what + * individual members represent, see how they are set in + * RelationGetPartitionDispatchInfo()) + *----------------------- + */ +typedef struct PartitionDispatchData +{ + Oid relid; + PartitionKey key; + List *keystate; /* list of ExprState */ + PartitionDesc partdesc; + int *indexes; +} PartitionDispatchData; + static int32 qsort_partition_list_value_cmp(const void *a, const void *b, void *arg); static int32 qsort_partition_rbound_cmp(const void *a, const void *b, void *arg); @@ -126,12 +148,22 @@ static PartitionRangeBound *make_one_range_bound(PartitionKey key, int index, Li static int32 partition_rbound_cmp(PartitionKey key, Datum *datums1, RangeDatumContent *content1, bool lower1, PartitionRangeBound *b2); +static int32 partition_rbound_datum_cmp(PartitionKey key, + Datum *rb_datums, RangeDatumContent *rb_content, + Datum *tuple_datums); static int32 partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, int offset, void *probe, bool probe_is_bound); static int partition_bound_bsearch(PartitionKey key, PartitionBoundInfo boundinfo, void *probe, bool probe_is_bound, bool *is_equal); +/* Support get_partition_for_tuple() */ +static void FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); + /* * RelationBuildPartitionDesc * Form rel's partition descriptor @@ -895,6 +927,115 @@ RelationGetPartitionQual(Relation rel, bool recurse) return generate_partition_qual(rel, recurse); } +/* Turn an array of OIDs with N elements into a list */ +#define OID_ARRAY_TO_LIST(arr, N, list) \ + do\ + {\ + int i;\ + for (i = 0; i < (N); i++)\ + (list) = lappend_oid((list), (arr)[i]);\ + } while(0) + +/* + * RelationGetPartitionDispatchInfo + * Returns information necessary to route tuples down a partition tree + * + * All the partitions will be locked with lockmode, unless it is NoLock. + * A list of the OIDs of all the leaf partition of rel is returned in + * *leaf_part_oids. + */ +PartitionDispatch * +RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + List **leaf_part_oids) +{ + PartitionDesc rootpartdesc = RelationGetPartitionDesc(rel); + PartitionDispatchData **pd; + List *all_parts = NIL, + *parted_rels = NIL; + ListCell *lc; + int i, + k, + num_parted; + + /* + * Lock partitions and collect OIDs of the partitioned ones to prepare + * their PartitionDispatch objects. + * + * Cannot use find_all_inheritors() here, because then the order of OIDs + * in parted_rels list would be unknown, which does not help because down + * below, we assign indexes within individual PartitionDispatch in an + * order that's predetermined (determined by the order of OIDs in + * individual partition descriptors). + */ + parted_rels = lappend_oid(parted_rels, RelationGetRelid(rel)); + num_parted = 1; + OID_ARRAY_TO_LIST(rootpartdesc->oids, rootpartdesc->nparts, all_parts); + foreach(lc, all_parts) + { + Relation partrel = heap_open(lfirst_oid(lc), lockmode); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + + /* + * If this partition is a partitined table, add its children to to the + * end of the list, so that they are processed as well. + */ + if (partdesc) + { + num_parted++; + parted_rels = lappend_oid(parted_rels, lfirst_oid(lc)); + OID_ARRAY_TO_LIST(partdesc->oids, partdesc->nparts, all_parts); + } + + heap_close(partrel, NoLock); + } + + /* Generate PartitionDispatch objects for all partitioned tables */ + pd = (PartitionDispatchData **) palloc(num_parted * + sizeof(PartitionDispatchData *)); + *leaf_part_oids = NIL; + i = k = 0; + foreach(lc, parted_rels) + { + /* We locked all partitions above */ + Relation partrel = heap_open(lfirst_oid(lc), NoLock); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + int j, + m; + + pd[i] = (PartitionDispatch) palloc(sizeof(PartitionDispatchData)); + pd[i]->relid = RelationGetRelid(partrel); + pd[i]->key = RelationGetPartitionKey(partrel); + pd[i]->keystate = NIL; + pd[i]->partdesc = partdesc; + pd[i]->indexes = (int *) palloc(partdesc->nparts * sizeof(int)); + heap_close(partrel, NoLock); + + m = 0; + for (j = 0; j < partdesc->nparts; j++) + { + Oid partrelid = partdesc->oids[j]; + + if (get_rel_relkind(partrelid) != RELKIND_PARTITIONED_TABLE) + { + *leaf_part_oids = lappend_oid(*leaf_part_oids, partrelid); + pd[i]->indexes[j] = k++; + } + else + { + /* + * We can assign indexes this way because of the way + * parted_rels has been generated. + */ + pd[i]->indexes[j] = -(i + 1 + m); + m++; + } + } + i++; + } + + return pd; +} + /* Module-local functions */ /* @@ -1329,6 +1470,172 @@ generate_partition_qual(Relation rel, bool recurse) return result; } +/* ---------------- + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for the partition key + * of a tuple. + * + * pkinfo partition key execution info + * slot Heap tuple from which to extract partition key + * estate executor state for evaluating any partition key + * expressions (must be non-NULL) + * values Array of partition key Datums (output area) + * isnull Array of is-null indicators (output area) + * + * the ecxt_scantuple slot of estate's per-tuple expr context must point to + * the heap tuple passed in. + * ---------------- + */ +static void +FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pd->key->partexprs != NIL && pd->keystate == NIL) + { + /* Check caller has set up context correctly */ + Assert(estate != NULL && + GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + + /* First time through, set up expression evaluation state */ + pd->keystate = (List *) ExecPrepareExpr((Expr *) pd->key->partexprs, + estate); + } + + partexpr_item = list_head(pd->keystate); + for (i = 0; i < pd->key->partnatts; i++) + { + AttrNumber keycol = pd->key->partattrs[i]; + Datum datum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + datum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + datum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = datum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Finds a leaf partition for tuple contained in *slot + * + * Returned value is the sequence number of the leaf partition thus found, + * or -1 if no leaf partition is found for the tuple. *failed_at is set + * to the OID of the partitioned table whose partition was not found in + * the latter case. + */ +int +get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + PartitionDispatch parent; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int cur_offset, + cur_index; + int i; + + /* start with the root partitioned table */ + parent = pd[0]; + while(true) + { + PartitionKey key = parent->key; + PartitionDesc partdesc = parent->partdesc; + + /* Quick exit */ + if (partdesc->nparts == 0) + { + *failed_at = parent->relid; + return -1; + } + + /* Extract partition key from tuple */ + FormPartitionKeyDatum(parent, slot, estate, values, isnull); + + if (key->strategy == PARTITION_STRATEGY_RANGE) + { + /* Disallow nulls in the range partition key of the tuple */ + for (i = 0; i < key->partnatts; i++) + if (isnull[i]) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key of row contains null"))); + } + + if (partdesc->boundinfo->has_null && isnull[0]) + /* Tuple maps to the null-accepting list partition */ + cur_index = partdesc->boundinfo->null_index; + else + { + /* Else bsearch in partdesc->boundinfo */ + bool equal = false; + + cur_offset = partition_bound_bsearch(key, partdesc->boundinfo, + values, false, &equal); + switch (key->strategy) + { + case PARTITION_STRATEGY_LIST: + if (cur_offset >= 0 && equal) + cur_index = partdesc->boundinfo->indexes[cur_offset]; + else + cur_index = -1; + break; + + case PARTITION_STRATEGY_RANGE: + /* + * Offset returned is such that the bound at offset is + * found to be less or equal with the tuple. So, the + * bound at offset+1 would be the upper bound. + */ + cur_index = partdesc->boundinfo->indexes[cur_offset+1]; + break; + } + } + + /* + * cur_index < 0 means we failed to find a partition of this parent. + * cur_index >= 0 means we either found the leaf partition, or the + * next parent to find a partition of. + */ + if (cur_index < 0) + { + *failed_at = parent->relid; + return -1; + } + else if (parent->indexes[cur_index] < 0) + parent = pd[-parent->indexes[cur_index]]; + else + break; + } + + return parent->indexes[cur_index]; +} + /* * qsort_partition_list_value_cmp * @@ -1461,6 +1768,36 @@ partition_rbound_cmp(PartitionKey key, } /* + * partition_rbound_datum_cmp + * + * Return whether range bound (specified in rb_datums, rb_content, and + * rb_lower) <=, =, >= partition key of tuple (tuple_datums) + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, + Datum *rb_datums, RangeDatumContent *rb_content, + Datum *tuple_datums) +{ + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (rb_content[i] != RANGE_DATUM_FINITE) + return rb_content[i] == RANGE_DATUM_NEG_INF ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + rb_datums[i], + tuple_datums[i])); + if (cmpval != 0) + break; + } + + return cmpval; +} + +/* * partition_bound_cmp * * Return whether the bound at offset in boundinfo is <=, =, >= the argument @@ -1499,7 +1836,10 @@ partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, bound_datums, content, lower, (PartitionRangeBound *) probe); } - + else + cmpval = partition_rbound_datum_cmp(key, + bound_datums, content, + (Datum *) probe); break; } } diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..7d76ead 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -161,6 +161,10 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionDispatch *partition_dispatch_info; + int num_partitions; + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1401,67 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + List *leaf_parts; + ListCell *cell; + int i, + num_leaf_parts; + ResultRelInfo *leaf_part_rri; + + /* Get the tuple-routing information and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + num_leaf_parts = list_length(leaf_parts); + cstate->partition_dispatch_info = pd; + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) palloc(num_leaf_parts * + sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation partrel; + + /* + * All partitions locked above; will be closed after CopyFrom is + * finished. + */ + partrel = heap_open(lfirst_oid(cell), NoLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(partrel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + partrel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(tupDesc, RelationGetDescr(partrel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(partrel), + gettext_noop("could not convert row type")); + leaf_part_rri++; + i++; + } + } } else { @@ -2255,6 +2320,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2347,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2458,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2486,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->partition_dispatch_info != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2442,7 +2511,11 @@ CopyFrom(CopyState cstate) values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2567,56 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition to heap_insert the tuple into */ + if (cstate->partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + cstate->partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < cstate->num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding + * to the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = cstate->partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* + * For ExecInsertIndexTuples() to work on the partition's indexes + */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the + * partition rowtype. + */ + map = cstate->partition_tupconv_maps[leaf_part_index]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2553,7 +2676,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2701,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2720,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); @@ -2614,6 +2745,20 @@ CopyFrom(CopyState cstate) ExecCloseIndices(resultRelInfo); + /* Close all partitions and indices thereof */ + if (cstate->partition_dispatch_info) + { + int i; + + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + } + FreeExecutorState(estate); /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 99c9d10..8a8eb01 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1322,6 +1322,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index c7a6347..54fb771 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2990,3 +2995,52 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +/* + * ExecFindPartition -- Find a leaf partition in the partition tree rooted + * at parent, for the heap tuple contained in *slot + * + * estate must be non-NULL; we'll need it to compute any expressions in the + * partition key(s) + * + * If no leaf partition is found, this routine errors out with the appropriate + * error message, else it returns the leaf partition sequence number returned + * by get_partition_for_tuple() unchanged. + */ +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, + TupleTableSlot *slot, EState *estate) +{ + int result; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + result = get_partition_for_tuple(pd, slot, estate, &failed_at); + if (result < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return result; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 6eccfb7..87a04aa 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,56 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + /* Determine the partition to heap_insert the tuple into */ + if (mtstate->mt_partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, false); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +562,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1622,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1713,69 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + /* Form the partition node tree and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + mtstate->mt_partition_dispatch_info = pd; + num_leaf_parts = list_length(leaf_parts); + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2093,15 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + /* * Free the exprcontext */ diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 36f8c54..88380ba 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -806,8 +806,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 70d8325..f76c5d9 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -36,6 +38,7 @@ typedef struct PartitionDescData } PartitionDescData; typedef struct PartitionDescData *PartitionDesc; +typedef struct PartitionDispatchData *PartitionDispatch; extern void RelationBuildPartitionDesc(Relation relation); extern bool partition_bounds_equal(PartitionKey key, @@ -45,4 +48,12 @@ extern void check_new_partition_bound(char *relname, Relation parent, Node *boun extern Oid get_partition_parent(Oid relid); extern List *get_qual_from_partbound(Relation rel, Relation parent, Node *bound); extern List *RelationGetPartitionQual(Relation rel, bool recurse); + +/* For tuple routing */ +extern PartitionDispatch *RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + List **leaf_part_oids); +extern int get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..b4d09f9 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..606cb21 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,13 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionDispatchData **mt_partition_dispatch_info; + /* Tuple-routing support info */ + int mt_num_partitions; /* Number of members in the + * following arrays */ + ResultRelInfo *mt_partitions; /* Per partition result relation */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per partition tuple conversion map */ } ModifyTableState; /* ---------------- diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index cc9f957..9706034 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -223,6 +223,58 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index 3a9430e..afecb74 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -137,6 +137,31 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------96825311778D254EC2B71375 Content-Type: text/x-diff; name="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-18.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-18"; filename*1=".patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH 7/8] Tuple routing for partitioned tables. @ 2016-07-27 07:59 amit <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: amit @ 2016-07-27 07:59 UTC (permalink / raw) Both COPY FROM and INSERT. --- src/backend/catalog/partition.c | 327 ++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 203 +++++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 49 +++++- src/backend/executor/nodeModifyTable.c | 156 +++++++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 1 + src/backend/optimizer/plan/createplan.c | 77 +++++++ src/backend/optimizer/util/plancat.c | 13 ++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 7 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 10 + src/include/nodes/plannodes.h | 1 + src/include/optimizer/plancat.h | 1 + src/test/regress/expected/insert.out | 52 +++++ src/test/regress/sql/insert.sql | 25 +++ 18 files changed, 932 insertions(+), 7 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index a5800fb..e7e0c7c 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -182,6 +182,18 @@ static List *generate_partition_qual(Relation rel, bool recurse); static PartitionTreeNode GetPartitionTreeNodeRecurse(Relation rel, int offset); static int get_leaf_partition_count(PartitionTreeNode ptnode); +/* Support get_partition_for_tuple() */ +static PartitionKeyExecInfo *BuildPartitionKeyExecInfo(Relation rel); +static void FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); +static int list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull); +static int range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum *tuple); + /* List partition related support functions */ static bool equal_list_info(PartitionKey key, PartitionListInfo *l1, PartitionListInfo *l2); @@ -195,6 +207,8 @@ static PartitionRangeBound *copy_range_bound(PartitionKey key, PartitionRangeBou static bool equal_range_info(PartitionKey key, PartitionRangeInfo *r1, PartitionRangeInfo *r2); static int32 partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg); +static int32 partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg); static bool partition_rbound_eq(PartitionKey key, PartitionRangeBound *b1, PartitionRangeBound *b2); typedef int32 (*partition_rbound_bsearch_cmp_fn) (PartitionKey, @@ -1438,7 +1452,7 @@ GetPartitionTreeNodeRecurse(Relation rel, int offset) /* First build our own node */ parent = (PartitionTreeNode) palloc0(sizeof(PartitionTreeNodeData)); - parent->pkinfo = NULL; + parent->pkinfo = BuildPartitionKeyExecInfo(rel); parent->pdesc = RelationGetPartitionDesc(rel); parent->relid = RelationGetRelid(rel); parent->offset = offset; @@ -1522,6 +1536,283 @@ get_leaf_partition_count(PartitionTreeNode ptnode) return result; } +/* + * BuildPartitionKeyExecInfo + * Construct a list of PartitionKeyExecInfo records for an open + * relation + * + * PartitionKeyExecInfo stores the information about the partition key + * that's needed when inserting tuples into a partitioned table; especially, + * partition key expression state if there are any expression columns in + * the partition key. Normally we build a PartitionKeyExecInfo for a + * partitioned table just once per command, and then use it for (potentially) + * many tuples. + * + */ +static PartitionKeyExecInfo * +BuildPartitionKeyExecInfo(Relation rel) +{ + PartitionKeyExecInfo *pkinfo; + + pkinfo = (PartitionKeyExecInfo *) palloc0(sizeof(PartitionKeyExecInfo)); + pkinfo->pi_Key = RelationGetPartitionKey(rel); + pkinfo->pi_ExpressionState = NIL; + + return pkinfo; +} + +/* + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for partition key columns + */ +static void +FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pkinfo->pi_Key->partexprs != NIL && pkinfo->pi_ExpressionState == NIL) + { + /* First time through, set up expression evaluation state */ + pkinfo->pi_ExpressionState = (List *) + ExecPrepareExpr((Expr *) pkinfo->pi_Key->partexprs, + estate); + /* Check caller has set up context correctly */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + } + + partexpr_item = list_head(pkinfo->pi_ExpressionState); + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + { + AttrNumber keycol = pkinfo->pi_Key->partattrs[i]; + Datum pkDatum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + pkDatum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + pkDatum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = pkDatum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Recursively finds the "leaf" partition for tuple + * + * Returns -1 if no partition is found and sets *failed_at to the OID of + * the partitioned table whose partition was not found. + */ +int +get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + Relation partRel; + PartitionKeyExecInfo *pkinfo = ptnode->pkinfo; + PartitionTreeNode node; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int i; + int index; + + /* Guard against stack overflow due to overly deep partition tree */ + check_stack_depth(); + + if (ptnode->pdesc->nparts == 0) + { + *failed_at = ptnode->relid; + return -1; + } + + /* Extract partition key from tuple */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + FormPartitionKeyDatum(pkinfo, slot, estate, values, isnull); + + /* Disallow nulls, if range partition key */ + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + if (isnull[i] && pkinfo->pi_Key->strategy == PARTITION_STRATEGY_RANGE) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key contains null"))); + + switch (pkinfo->pi_Key->strategy) + { + case PARTITION_STRATEGY_LIST: + index = list_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values[0], isnull[0]); + break; + + case PARTITION_STRATEGY_RANGE: + index = range_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values); + break; + } + + /* No partition found at this level */ + if (index < 0) + { + *failed_at = ptnode->relid; + return index; + } + + partRel = heap_open(ptnode->pdesc->oids[index], NoLock); + + /* Don't recurse if the index'th partition is a leaf partition. */ + if (partRel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) + { + PartitionTreeNode prev; + + /* + * Index returned above considers only this parent (of which partRel + * is a partition). We however want to return the index across the + * the whole partition tree. If partRel is the leftmost partition + * of parent (index = 0) or if none of the left siblings are + * partitioned tables, we can simply return parent's offset plus + * partRel's index as the result. Otherwise, we find the node + * corresponding to the rightmost partitioned table among partRel's + * left siblings and use the information therein. + */ + prev = node = ptnode->downlink; + if (node && node->index < index) + { + /* + * Find the partition tree node such that its index value is the + * greatest value less than the above returned index. + */ + while (node) + { + if (node->index > index) + { + node = prev; + break; + } + + prev = node; + node = node->next; + } + + if (!node) + node = prev; + Assert (node != NULL); + + index = node->offset + node->num_leaf_parts + + (index - node->index - 1); + } + else + /* + * The easy case where all of partRel's left siblings (if any) are + * leaf partitions. + */ + index = ptnode->offset + index; + + heap_close(partRel, NoLock); + return index; + } + + heap_close(partRel, NoLock); + + /* + * Need to recurse as the selected partition is a partitioned table + * itself. Locate the corresponding PartitionTreeNode to pass it down. + */ + node = ptnode->downlink; + while (node->next != NULL && node->index != index) + node = node->next; + Assert (node != NULL); + + return get_partition_for_tuple(node, slot, estate, failed_at); +} + +/* + * list_partition_for_tuple + * Find the list partition for a tuple (arg 'value' contains the + * list partition key of the original tuple) + * + * Returns -1 if none found. + */ +static int +list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull) +{ + PartitionListInfo listinfo; + int found; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_LIST); + listinfo = pdesc->boundinfo->bounds.lists; + + if (isnull && listinfo.has_null) + return listinfo.null_index; + else if (!isnull) + { + found = partition_list_values_bsearch(key, + listinfo.values, + listinfo.nvalues, + value); + if (found >= 0) + return listinfo.indexes[found]; + } + + /* Control reaches here if isnull and !listinfo->has_null */ + return -1; +} + +/* + * range_partition_for_tuple + * Get the index of the range partition for a tuple (arg 'tuple' + * actually contains the range partition key of the original + * tuple) + * + * Returns -1 if none found. + */ +static int +range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, Datum *tuple) +{ + int offset; + PartitionRangeInfo rangeinfo; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_RANGE); + rangeinfo = pdesc->boundinfo->bounds.ranges; + + offset = partition_rbound_bsearch(key, + rangeinfo.bounds, rangeinfo.nbounds, + tuple, partition_rbound_datum_cmp, + true, NULL); + + /* + * Offset returned is such that the bound at offset is found to be less + * or equal with the tuple. That is, the tuple belongs to the partition + * with the rangeinfo.bounds[offset] as the lower bound and + * rangeinfo.bounds[offset+1] as the upper bound, provided the latter is + * indeed an upper (!lower) bound. If it turns out otherwise, the + * corresponding index will be -1, which means no valid partition exists. + */ + return rangeinfo.indexes[offset+1]; +} + /* List partition related support functions */ /* @@ -1767,6 +2058,40 @@ partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg) } /* + * Return whether bound <=, =, >= partition key of tuple + * + * The 3rd argument is void * so that it can be used with + * partition_rbound_bsearch() + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg) +{ + Datum *datums1 = bound->datums, + *datums2 = (Datum *) arg; + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (bound->infinite[i]) + return bound->lower ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + datums1[i], datums2[i])); + if (cmpval != 0) + break; + } + + /* If datums are equal and this is an upper bound, tuple > bound */ + if (cmpval == 0 && !bound->lower) + return -1; + + return cmpval; +} + +/* * Return whether two range bounds are equal simply by comparing datums */ static bool diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..6fb376f 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -30,6 +30,7 @@ #include "commands/defrem.h" #include "commands/trigger.h" #include "executor/executor.h" +#include "foreign/fdwapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -161,6 +162,11 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionTreeNode ptnode; /* partition descriptor node tree */ + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; + List *partition_fdw_priv_lists; + int num_partitions; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1403,94 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + List *leaf_parts; + ListCell *cell; + int i; + int num_leaf_parts; + ResultRelInfo *leaf_part_rri; + PlannerInfo *root = makeNode(PlannerInfo); /* mostly dummy */ + Query *parse = makeNode(Query); /* ditto */ + ModifyTable *plan = makeNode(ModifyTable); /* ditto */ + RangeTblEntry *fdw_rte = makeNode(RangeTblEntry); /* ditto */ + List *fdw_private_lists = NIL; + + cstate->ptnode = RelationGetPartitionTreeNode(rel); + leaf_parts = get_leaf_partition_oids(cstate->ptnode); + num_leaf_parts = list_length(leaf_parts); + + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + /* For use below, iff a partition found to be a foreign table */ + plan->operation = CMD_INSERT; + plan->plans = list_make1(makeNode(Result)); + fdw_rte->rtekind = RTE_RELATION; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + parse->rtable = list_make1(fdw_rte); + root->parse = parse; + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation part_rel; + + part_rel = heap_open(lfirst_oid(cell), RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + /* Special dance for foreign tables */ + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + fdw_rte->relid = RelationGetRelid(part_rel); + fdw_private = leaf_part_rri->ri_FdwRoutine->PlanForeignModify(root, + plan, + 1, + 0); + fdw_private_lists = lappend(fdw_private_lists, fdw_private); + } + + if (!equalTupleDescs(tupDesc, RelationGetDescr(part_rel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + + cstate->partition_fdw_priv_lists = fdw_private_lists; + pfree(fdw_rte); + pfree(plan); + pfree(parse); + pfree(root); + } } else { @@ -1692,6 +1786,8 @@ ClosePipeToProgram(CopyState cstate) static void EndCopy(CopyState cstate) { + int i; + if (cstate->is_program) { ClosePipeToProgram(cstate); @@ -1705,6 +1801,23 @@ EndCopy(CopyState cstate) cstate->filename))); } + /* Close all partitions and indices thereof */ + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + /* XXX - EState not handy here to pass to EndForeignModify() */ + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(NULL, resultRelInfo); + + if (cstate->partition_tupconv_maps[i]) + pfree(cstate->partition_tupconv_maps[i]); + } + MemoryContextDelete(cstate->copycontext); pfree(cstate); } @@ -2255,6 +2368,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2395,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2506,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2534,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->ptnode != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2439,10 +2556,46 @@ CopyFrom(CopyState cstate) */ ExecBSInsertTriggers(estate, resultRelInfo); + /* Initialize FDW partition insert plans */ + if (cstate->ptnode) + { + int i, + j; + List *fdw_private_lists = cstate->partition_fdw_priv_lists; + ModifyTableState *mtstate = makeNode(ModifyTableState); + ResultRelInfo *leaf_part_rri; + + /* Mostly dummy containing enough state for BeginForeignModify */ + mtstate->ps.state = estate; + mtstate->operation = CMD_INSERT; + + j = 0; + leaf_part_rri = cstate->partitions; + for (i = 0; i < cstate->num_partitions; i++) + { + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + Assert(fdw_private_lists); + fdw_private = list_nth(fdw_private_lists, j++); + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, 0); + } + leaf_part_rri++; + } + } + values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2647,31 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition */ + saved_resultRelInfo = resultRelInfo; + if (cstate->ptnode) + { + int i_leaf_partition; + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + cstate->ptnode, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < cstate->num_partitions); + + resultRelInfo = cstate->partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = cstate->partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2523,7 +2701,16 @@ CopyFrom(CopyState cstate) resultRelInfo->ri_PartitionCheck) ExecConstraints(resultRelInfo, slot, estate); - if (useHeapMultiInsert) + if (resultRelInfo->ri_FdwRoutine) + { + resultRelInfo->ri_FdwRoutine->ExecForeignInsert(estate, + resultRelInfo, + slot, + NULL); + /* AFTER ROW INSERT Triggers */ + ExecARInsertTriggers(estate, resultRelInfo, tuple, NIL); + } + else if (useHeapMultiInsert) { /* Add this tuple to the tuple buffer */ if (nBufferedTuples == 0) @@ -2553,7 +2740,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2765,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2784,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 3b72ae3..6478211 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1293,6 +1293,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index ea3f59a..b5125a8 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2996,3 +3001,43 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionTreeNode ptnode, + TupleTableSlot *slot, EState *estate) +{ + int i_leaf_partition; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + i_leaf_partition = get_partition_for_tuple(ptnode, slot, estate, + &failed_at); + + if (i_leaf_partition < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return i_leaf_partition; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..44801b8 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,45 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + if (mtstate->mt_partition_tree_root) + { + int i_leaf_partition; + TupleConversionMap *map; + + /* + * Get the index of leaf partitions that we'll use to get the correct + * ResultRelInfo from mtstate->mt_partitions[]. + */ + i_leaf_partition = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_tree_root, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + i_leaf_partition; + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[i_leaf_partition]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, false); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +551,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1611,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1702,98 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + mtstate->mt_partition_tree_root = RelationGetPartitionTreeNode(rel); + leaf_parts = get_leaf_partition_oids(mtstate->mt_partition_tree_root); + num_leaf_parts = list_length(leaf_parts); + + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (leaf_part_rri->ri_FdwRoutine) + { + ListCell *lc; + List *fdw_private; + int k; + + /* + * There are as many fdw_private's in fdwPrivLists as there + * are FDW partitions, but we must find the intended for the + * this foreign table. + */ + k = 0; + foreach(lc, node->fdwPartitionOids) + { + if (lfirst_oid(lc) == ftoid) + break; + k++; + } + + Assert(k < num_leaf_parts); + fdw_private = (List *) list_nth(node->fdwPrivLists, k); + Assert(fdw_private != NIL); + + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, + eflags); + j++; + } + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2111,23 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(node->ps.state, + resultRelInfo); + + if (node->mt_partition_tupconv_maps[i]) + pfree(node->mt_partition_tupconv_maps[i]); + } + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 28d0036..470dee7 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -188,6 +188,7 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(withCheckOptionLists); COPY_NODE_FIELD(returningLists); COPY_NODE_FIELD(fdwPrivLists); + COPY_NODE_FIELD(fdwPartitionOids); COPY_BITMAPSET_FIELD(fdwDirectModifyPlans); COPY_NODE_FIELD(rowMarks); COPY_SCALAR_FIELD(epqParam); diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 0d858f5..5c8eced 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -340,6 +340,7 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(withCheckOptionLists); WRITE_NODE_FIELD(returningLists); WRITE_NODE_FIELD(fdwPrivLists); + WRITE_NODE_FIELD(fdwPartitionOids); WRITE_BITMAPSET_FIELD(fdwDirectModifyPlans); WRITE_NODE_FIELD(rowMarks); WRITE_INT_FIELD(epqParam); diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c587d4e..ddd78c7 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -1495,6 +1495,7 @@ _readModifyTable(void) READ_NODE_FIELD(withCheckOptionLists); READ_NODE_FIELD(returningLists); READ_NODE_FIELD(fdwPrivLists); + READ_NODE_FIELD(fdwPartitionOids); READ_BITMAPSET_FIELD(fdwDirectModifyPlans); READ_NODE_FIELD(rowMarks); READ_INT_FIELD(epqParam); diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index ad49674..29f40fe 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -22,6 +22,7 @@ #include "access/stratnum.h" #include "access/sysattr.h" #include "catalog/pg_class.h" +#include "catalog/pg_inherits_fn.h" #include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/extensible.h" @@ -6159,6 +6160,82 @@ make_modifytable(PlannerInfo *root, node->fdwPrivLists = fdw_private_list; node->fdwDirectModifyPlans = direct_modify_plans; + /* Collect insert plans for all FDW-managed partitions */ + if (node->operation == CMD_INSERT) + { + RangeTblEntry *rte, + **saved_simple_rte_array; + List *partition_oids, + *fdw_partition_oids; + + Assert(list_length(resultRelations) == 1); + rte = rt_fetch(linitial_int(resultRelations), root->parse->rtable); + Assert(rte->rtekind == RTE_RELATION); + + if (rte->relkind != RELKIND_PARTITIONED_TABLE) + return node; + + partition_oids = find_all_inheritors(rte->relid, NoLock, NULL); + + /* Discard any previous content which is useless anyway */ + fdw_private_list = NIL; + fdw_partition_oids = NIL; + + /* + * To force the FDW driver fetch the intended RTE, we need to temporarily + * switch root->simple_rte_array to the one holding only that RTE at a + * designated index, for every foreign table. + */ + saved_simple_rte_array = root->simple_rte_array; + root->simple_rte_array = (RangeTblEntry **) + palloc0(2 * sizeof(RangeTblEntry *)); + foreach(lc, partition_oids) + { + Oid myoid = lfirst_oid(lc); + FdwRoutine *fdwroutine; + List *fdw_private; + + /* + * We are only interested in foreign tables. Note that this will + * also eliminate any partitioned tables since foreign tables can + * only ever be leaf partitions. + */ + if (!oid_is_foreign_table(myoid)) + continue; + + fdw_partition_oids = lappend_oid(fdw_partition_oids, myoid); + + fdwroutine = GetFdwRoutineByRelId(myoid); + if (fdwroutine && fdwroutine->PlanForeignModify) + { + RangeTblEntry *fdw_rte; + + fdw_rte = copyObject(rte); + fdw_rte->relid = myoid; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + + /* + * Assumes PlanForeignModify() uses planner_rt_fetch(), also, + * see the above comment. + */ + root->simple_rte_array[1] = fdw_rte; + + fdw_private = fdwroutine->PlanForeignModify(root, node, 1, 0); + pfree(fdw_rte); + } + else + fdw_private = NIL; + + fdw_private_list = lappend(fdw_private_list, fdw_private); + } + + pfree(root->simple_rte_array); + root->simple_rte_array = saved_simple_rte_array; + + node->fdwPrivLists = fdw_private_list; + node->fdwPartitionOids = fdw_partition_oids; + } + return node; } diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index a2cbf14..e85ca0a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1715,3 +1715,16 @@ has_row_triggers(PlannerInfo *root, Index rti, CmdType event) heap_close(relation, NoLock); return result; } + +bool +oid_is_foreign_table(Oid relid) +{ + Relation rel; + char relkind; + + rel = heap_open(relid, NoLock); + relkind = rel->rd_rel->relkind; + heap_close(rel, NoLock); + + return relkind == RELKIND_FOREIGN_TABLE; +} diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 057b1e3..86cc598 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -50,4 +52,9 @@ extern List *RelationGetPartitionQual(Relation rel, bool recurse); /* For tuple routing */ extern PartitionTreeNode RelationGetPartitionTreeNode(Relation rel); extern List *get_leaf_partition_oids(PartitionTreeNode ptnode); + +extern int get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..c62946f 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..ce01008 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,15 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionTreeNodeData *mt_partition_tree_root; + /* Partition descriptor node tree */ + ResultRelInfo *mt_partitions; /* Per leaf partition target + * relations */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per leaf partition + * tuple conversion map */ + int mt_num_partitions; /* Number of leaf partition target + * relations in the above array */ } ModifyTableState; /* ---------------- diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index e2fbc7d..d82222c 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -189,6 +189,7 @@ typedef struct ModifyTable List *returningLists; /* per-target-table RETURNING tlists */ List *fdwPrivLists; /* per-target-table FDW private data lists */ Bitmapset *fdwDirectModifyPlans; /* indices of FDW DM plans */ + List *fdwPartitionOids; /* OIDs of FDW-managed partition */ List *rowMarks; /* PlanRowMarks (non-locking only) */ int epqParam; /* ID of Param for EvalPlanQual re-eval */ OnConflictAction onConflictAction; /* ON CONFLICT action */ diff --git a/src/include/optimizer/plancat.h b/src/include/optimizer/plancat.h index 125274e..fac606c 100644 --- a/src/include/optimizer/plancat.h +++ b/src/include/optimizer/plancat.h @@ -56,5 +56,6 @@ extern Selectivity join_selectivity(PlannerInfo *root, SpecialJoinInfo *sjinfo); extern bool has_row_triggers(PlannerInfo *root, Index rti, CmdType event); +extern bool oid_is_foreign_table(Oid relid); #endif /* PLANCAT_H */ diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 9ae6b09..d5dcb59 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -227,6 +227,58 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index b6e821e..fbd30d9 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,31 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------30A9D66B3BEA4523792AC3CB Content-Type: text/x-diff; name="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-14.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-14"; filename*1=".patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH 7/8] Tuple routing for partitioned tables. @ 2016-07-27 07:59 amit <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: amit @ 2016-07-27 07:59 UTC (permalink / raw) Both COPY FROM and INSERT are covered by this commit. Routing to foreing partitions is not supported at the moment. --- src/backend/catalog/partition.c | 289 +++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 151 ++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 58 ++++++- src/backend/executor/nodeModifyTable.c | 130 ++++++++++++++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 6 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 8 + src/test/regress/expected/insert.out | 52 ++++++ src/test/regress/sql/insert.sql | 25 +++ 11 files changed, 728 insertions(+), 6 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index 03c7fa1..d758500 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -161,6 +161,18 @@ static Oid get_partition_operator(PartitionKey key, int col, StrategyNumber stra static List *generate_partition_qual(Relation rel, bool recurse); +/* Support get_partition_for_tuple() */ +static PartitionKeyExecInfo *BuildPartitionKeyExecInfo(Relation rel); +static void FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); +static int list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull); +static int range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum *tuple); + /* List partition related support functions */ static bool equal_list_info(PartitionKey key, PartitionListInfo *l1, PartitionListInfo *l2); @@ -174,6 +186,8 @@ static PartitionRangeBound *copy_range_bound(PartitionKey key, PartitionRangeBou static bool equal_range_info(PartitionKey key, PartitionRangeInfo *r1, PartitionRangeInfo *r2); static int32 partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg); +static int32 partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg); static bool partition_rbound_eq(PartitionKey key, PartitionRangeBound *b1, PartitionRangeBound *b2); typedef int32 (*partition_rbound_bsearch_cmp_fn) (PartitionKey, @@ -988,7 +1002,7 @@ RelationGetPartitionDispatchInfo(Relation rel, int lockmode, pd[i] = (PartitionDispatch) palloc(sizeof(PartitionDispatchData)); pd[i]->relid = RelationGetRelid(partrel); - pd[i]->pkinfo = NULL; + pd[i]->pkinfo = BuildPartitionKeyExecInfo(partrel); pd[i]->partdesc = partdesc; pd[i]->indexes = (int *) palloc(partdesc->nparts * sizeof(int)); heap_close(partrel, NoLock); @@ -1459,6 +1473,245 @@ generate_partition_qual(Relation rel, bool recurse) return result; } +/* + * BuildPartitionKeyExecInfo + * Construct a list of PartitionKeyExecInfo records for an open + * relation + * + * PartitionKeyExecInfo stores the information about the partition key + * that's needed when inserting tuples into a partitioned table; especially, + * partition key expression state if there are any expression columns in + * the partition key. Normally we build a PartitionKeyExecInfo for a + * partitioned table just once per command, and then use it for (potentially) + * many tuples. + * + */ +static PartitionKeyExecInfo * +BuildPartitionKeyExecInfo(Relation rel) +{ + PartitionKeyExecInfo *pkinfo; + + pkinfo = (PartitionKeyExecInfo *) palloc0(sizeof(PartitionKeyExecInfo)); + pkinfo->pi_Key = RelationGetPartitionKey(rel); + pkinfo->pi_ExpressionState = NIL; + + return pkinfo; +} + +/* ---------------- + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for the partition key + * of a tuple. + * + * pkinfo partition key execution info + * slot Heap tuple from which to extract partition key + * estate executor state for evaluating any partition key + * expressions (must be non-NULL) + * values Array of partition key Datums (output area) + * isnull Array of is-null indicators (output area) + * + * the ecxt_scantuple slot of estate's per-tuple expr context must point to + * the heap tuple passed in. + * ---------------- + */ +static void +FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pkinfo->pi_Key->partexprs != NIL && pkinfo->pi_ExpressionState == NIL) + { + /* Check caller has set up context correctly */ + Assert(estate != NULL && + GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + + /* First time through, set up expression evaluation state */ + pkinfo->pi_ExpressionState = (List *) + ExecPrepareExpr((Expr *) pkinfo->pi_Key->partexprs, + estate); + } + + partexpr_item = list_head(pkinfo->pi_ExpressionState); + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + { + AttrNumber keycol = pkinfo->pi_Key->partattrs[i]; + Datum pkDatum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + pkDatum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + pkDatum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = pkDatum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Finds a leaf partition for tuple contained in *slot + * + * Returned value is the sequence number of the leaf partition thus found, + * or -1 if no leaf partition is found for the tuple. *failed_at is set + * to the OID of the partitioned table whose partition was not found in + * the latter case. + */ +int +get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + PartitionDispatch parent; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int cur_idx; + int i; + + /* start with the root partitioned table */ + parent = pd[0]; + while(1) + { + PartitionDesc partdesc = parent->partdesc; + PartitionKeyExecInfo *pkinfo = parent->pkinfo; + + /* Quick exit */ + if (partdesc->nparts == 0) + { + *failed_at = parent->relid; + return -1; + } + + /* Extract partition key from tuple */ + FormPartitionKeyDatum(pkinfo, slot, estate, values, isnull); + + switch (pkinfo->pi_Key->strategy) + { + case PARTITION_STRATEGY_LIST: + cur_idx = list_partition_for_tuple(pkinfo->pi_Key, partdesc, + values[0], isnull[0]); + break; + + case PARTITION_STRATEGY_RANGE: + /* Disallow nulls in the partition key of the tuple */ + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + if (isnull[i]) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key of row contains null"))); + + cur_idx = range_partition_for_tuple(pkinfo->pi_Key, partdesc, + values); + break; + } + + /* + * cur_idx < 0 means we failed to find a partition of this parent. + * cur_idx >= 0 means we either found the leaf partition we have been + * looking for, or the next parent to find a partition of. + */ + if (cur_idx < 0) + { + *failed_at = parent->relid; + return -1; + } + else if (parent->indexes[cur_idx] < 0) + parent = pd[-parent->indexes[cur_idx]]; + else + break; + } + + return parent->indexes[cur_idx]; +} + +/* + * list_partition_for_tuple + * Find the list partition for a tuple (arg 'value' contains the + * list partition key of the original tuple) + * + * Returns -1 if none found. + */ +static int +list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull) +{ + PartitionListInfo listinfo; + int found; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_LIST); + listinfo = pdesc->boundinfo->bounds.lists; + + if (isnull && listinfo.has_null) + return listinfo.null_index; + else if (!isnull) + { + found = partition_list_values_bsearch(key, + listinfo.values, + listinfo.nvalues, + value); + if (found >= 0) + return listinfo.indexes[found]; + } + + /* Control reaches here if isnull and !listinfo->has_null */ + return -1; +} + +/* + * range_partition_for_tuple + * Get the index of the range partition for a tuple (arg 'tuple' + * actually contains the range partition key of the original + * tuple) + * + * Returns -1 if none found. + */ +static int +range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, Datum *tuple) +{ + int offset; + PartitionRangeInfo rangeinfo; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_RANGE); + rangeinfo = pdesc->boundinfo->bounds.ranges; + + offset = partition_rbound_bsearch(key, + rangeinfo.bounds, rangeinfo.nbounds, + tuple, partition_rbound_datum_cmp, + true, NULL); + + /* + * Offset returned is such that the bound at offset is found to be less + * or equal with the tuple. That is, the tuple belongs to the partition + * with the rangeinfo.bounds[offset] as the lower bound and + * rangeinfo.bounds[offset+1] as the upper bound, provided the latter is + * indeed an upper (!lower) bound. If it turns out otherwise, the + * corresponding index will be -1, which means no valid partition exists. + */ + return rangeinfo.indexes[offset+1]; +} + /* List partition related support functions */ /* @@ -1704,6 +1957,40 @@ partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg) } /* + * Return whether bound <=, =, >= partition key of tuple + * + * The 3rd argument is void * so that it can be used with + * partition_rbound_bsearch() + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg) +{ + Datum *datums1 = bound->datums, + *datums2 = (Datum *) arg; + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (bound->infinite[i]) + return bound->lower ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + datums1[i], datums2[i])); + if (cmpval != 0) + break; + } + + /* If datums are equal and this is an upper bound, tuple > bound */ + if (cmpval == 0 && !bound->lower) + return -1; + + return cmpval; +} + +/* * Return whether two range bounds are equal simply by comparing datums */ static bool diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..7d76ead 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -161,6 +161,10 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionDispatch *partition_dispatch_info; + int num_partitions; + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1401,67 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + List *leaf_parts; + ListCell *cell; + int i, + num_leaf_parts; + ResultRelInfo *leaf_part_rri; + + /* Get the tuple-routing information and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + num_leaf_parts = list_length(leaf_parts); + cstate->partition_dispatch_info = pd; + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) palloc(num_leaf_parts * + sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation partrel; + + /* + * All partitions locked above; will be closed after CopyFrom is + * finished. + */ + partrel = heap_open(lfirst_oid(cell), NoLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(partrel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + partrel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(tupDesc, RelationGetDescr(partrel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(partrel), + gettext_noop("could not convert row type")); + leaf_part_rri++; + i++; + } + } } else { @@ -2255,6 +2320,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2347,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2458,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2486,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->partition_dispatch_info != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2442,7 +2511,11 @@ CopyFrom(CopyState cstate) values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2567,56 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition to heap_insert the tuple into */ + if (cstate->partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + cstate->partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < cstate->num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding + * to the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = cstate->partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* + * For ExecInsertIndexTuples() to work on the partition's indexes + */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the + * partition rowtype. + */ + map = cstate->partition_tupconv_maps[leaf_part_index]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2553,7 +2676,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2701,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2720,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); @@ -2614,6 +2745,20 @@ CopyFrom(CopyState cstate) ExecCloseIndices(resultRelInfo); + /* Close all partitions and indices thereof */ + if (cstate->partition_dispatch_info) + { + int i; + + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + } + FreeExecutorState(estate); /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 3b72ae3..6478211 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1293,6 +1293,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index c7a6347..54fb771 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2990,3 +2995,52 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +/* + * ExecFindPartition -- Find a leaf partition in the partition tree rooted + * at parent, for the heap tuple contained in *slot + * + * estate must be non-NULL; we'll need it to compute any expressions in the + * partition key(s) + * + * If no leaf partition is found, this routine errors out with the appropriate + * error message, else it returns the leaf partition sequence number returned + * by get_partition_for_tuple() unchanged. + */ +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, + TupleTableSlot *slot, EState *estate) +{ + int result; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + result = get_partition_for_tuple(pd, slot, estate, &failed_at); + if (result < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return result; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..aa1d8b9 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,56 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + /* Determine the partition to heap_insert the tuple into */ + if (mtstate->mt_partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, false); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +562,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1622,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1713,69 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + /* Form the partition node tree and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + mtstate->mt_partition_dispatch_info = pd; + num_leaf_parts = list_length(leaf_parts); + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2093,15 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + /* * Free the exprcontext */ diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index fb43cc1..1d231a3 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -50,4 +52,8 @@ extern List *RelationGetPartitionQual(Relation rel, bool recurse); /* For tuple routing */ extern PartitionDispatch *RelationGetPartitionDispatchInfo(Relation rel, int lockmode, List **leaf_part_oids); +extern int get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..b4d09f9 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..606cb21 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,13 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionDispatchData **mt_partition_dispatch_info; + /* Tuple-routing support info */ + int mt_num_partitions; /* Number of members in the + * following arrays */ + ResultRelInfo *mt_partitions; /* Per partition result relation */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per partition tuple conversion map */ } ModifyTableState; /* ---------------- diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 9ae6b09..d5dcb59 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -227,6 +227,58 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index b6e821e..fbd30d9 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,31 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------38C0D572AC793CAA7276C062 Content-Type: text/x-diff; name="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-16.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-16"; filename*1=".patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH 7/8] Tuple routing for partitioned tables. @ 2016-07-27 07:59 amit <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: amit @ 2016-07-27 07:59 UTC (permalink / raw) Both COPY FROM and INSERT. --- src/backend/catalog/partition.c | 317 ++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 222 +++++++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 59 ++++++- src/backend/executor/nodeModifyTable.c | 160 ++++++++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 1 + src/backend/optimizer/plan/createplan.c | 76 ++++++++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 7 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 10 + src/include/nodes/plannodes.h | 1 + src/test/regress/expected/insert.out | 52 +++++ src/test/regress/sql/insert.sql | 25 +++ 16 files changed, 940 insertions(+), 7 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index 7bc4b15..4e253c1 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -185,6 +185,18 @@ static List *generate_partition_qual(Relation rel, bool recurse); static PartitionTreeNode GetPartitionTreeNodeRecurse(Relation rel, int offset, int lockmode); static int get_leaf_partition_count(PartitionTreeNode parent); +/* Support get_partition_for_tuple() */ +static PartitionKeyExecInfo *BuildPartitionKeyExecInfo(Relation rel); +static void FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); +static int list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull); +static int range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum *tuple); + /* List partition related support functions */ static bool equal_list_info(PartitionKey key, PartitionListInfo *l1, PartitionListInfo *l2); @@ -198,6 +210,8 @@ static PartitionRangeBound *copy_range_bound(PartitionKey key, PartitionRangeBou static bool equal_range_info(PartitionKey key, PartitionRangeInfo *r1, PartitionRangeInfo *r2); static int32 partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg); +static int32 partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg); static bool partition_rbound_eq(PartitionKey key, PartitionRangeBound *b1, PartitionRangeBound *b2); typedef int32 (*partition_rbound_bsearch_cmp_fn) (PartitionKey, @@ -1443,7 +1457,7 @@ GetPartitionTreeNodeRecurse(Relation rel, int offset, int lockmode) /* First build our own node */ parent = (PartitionTreeNode) palloc0(sizeof(PartitionTreeNodeData)); - parent->pkinfo = NULL; + parent->pkinfo = BuildPartitionKeyExecInfo(rel); parent->pdesc = RelationGetPartitionDesc(rel); parent->relid = RelationGetRelid(rel); parent->offset = offset; @@ -1527,6 +1541,273 @@ get_leaf_partition_count(PartitionTreeNode parent) return result; } +/* + * BuildPartitionKeyExecInfo + * Construct a list of PartitionKeyExecInfo records for an open + * relation + * + * PartitionKeyExecInfo stores the information about the partition key + * that's needed when inserting tuples into a partitioned table; especially, + * partition key expression state if there are any expression columns in + * the partition key. Normally we build a PartitionKeyExecInfo for a + * partitioned table just once per command, and then use it for (potentially) + * many tuples. + * + */ +static PartitionKeyExecInfo * +BuildPartitionKeyExecInfo(Relation rel) +{ + PartitionKeyExecInfo *pkinfo; + + pkinfo = (PartitionKeyExecInfo *) palloc0(sizeof(PartitionKeyExecInfo)); + pkinfo->pi_Key = RelationGetPartitionKey(rel); + pkinfo->pi_ExpressionState = NIL; + + return pkinfo; +} + +/* ---------------- + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for the partition key + * of a tuple. + * + * pkinfo partition key execution info + * slot Heap tuple from which to extract partition key + * estate executor state for evaluating any partition key + * expressions (must be non-NULL) + * values Array of partition key Datums (output area) + * isnull Array of is-null indicators (output area) + * + * the ecxt_scantuple slot of estate's per-tuple expr context must point to + * the heap tuple passed in. + * ---------------- + */ +static void +FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pkinfo->pi_Key->partexprs != NIL && pkinfo->pi_ExpressionState == NIL) + { + /* Check caller has set up context correctly */ + Assert(estate != NULL && + GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + + /* First time through, set up expression evaluation state */ + pkinfo->pi_ExpressionState = (List *) + ExecPrepareExpr((Expr *) pkinfo->pi_Key->partexprs, + estate); + } + + partexpr_item = list_head(pkinfo->pi_ExpressionState); + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + { + AttrNumber keycol = pkinfo->pi_Key->partattrs[i]; + Datum pkDatum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + pkDatum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + pkDatum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = pkDatum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Recursively finds the "leaf" partition for tuple + * + * parent is the root of the partition tree through which we are trying + * to route the tuple contained in *slot. + * + * Returned value is the sequence number of the leaf partition thus found, + * or -1 if no leaf partition is found for the tuple. *failed_at is set + * to the OID of the partitioned table whose partition was not found in + * the latter case. + */ +int +get_partition_for_tuple(PartitionTreeNode parent, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + PartitionKeyExecInfo *pkinfo = parent->pkinfo; + PartitionTreeNode node, + prev; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int i; + int cur_idx; + + /* Guard against stack overflow due to overly deep partition tree */ + check_stack_depth(); + + if (parent->pdesc->nparts == 0) + { + *failed_at = parent->relid; + return -1; + } + + /* Extract partition key from tuple */ + FormPartitionKeyDatum(pkinfo, slot, estate, values, isnull); + + switch (pkinfo->pi_Key->strategy) + { + case PARTITION_STRATEGY_LIST: + cur_idx = list_partition_for_tuple(pkinfo->pi_Key, parent->pdesc, + values[0], isnull[0]); + break; + + case PARTITION_STRATEGY_RANGE: + /* Disallow nulls in the partition key of the tuple */ + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + if (isnull[i]) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key of row contains null"))); + + cur_idx = range_partition_for_tuple(pkinfo->pi_Key, parent->pdesc, + values); + break; + } + + /* No partition found at this level */ + if (cur_idx < 0) + { + *failed_at = parent->relid; + return cur_idx; + } + + /* + * The partition we found may either be a leaf partition or partitioned + * itself. In the latter case, we need to recurse by finding the proper + * node (ie, such that node->index == cur_idx) to pass down. + */ + prev = parent; + node = parent->downlink; + while (node != NULL) + { + if (node->index >= cur_idx) + break; + + prev = node; + node = node->next; + } + + /* + * If we couldn't find a node, that means cur_idx is actually a leaf + * partition. In the simpler case where all of the left siblings are leaf + * partitions themselves, we can get the correct index to return by adding + * cur_idx to the parent node's offset. Otherwise, we need to compensate + * for the leaf partitions in the partition subtrees of all the left + * siblings that are partitioned, which, fortunately it suffices to look + * at the rightmost such node. + */ + Assert(prev != NULL); + if (!node || cur_idx < node->index) + return prev == parent ? prev->offset + cur_idx + : prev->offset + prev->num_leaf_parts + + (cur_idx - prev->index - 1); + + /* + * Must recurse because it turns out that the partition at cur_idx is + * partitioned itself + */ + Assert (node != NULL && node->index == cur_idx); + + return get_partition_for_tuple(node, slot, estate, failed_at); +} + +/* + * list_partition_for_tuple + * Find the list partition for a tuple (arg 'value' contains the + * list partition key of the original tuple) + * + * Returns -1 if none found. + */ +static int +list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull) +{ + PartitionListInfo listinfo; + int found; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_LIST); + listinfo = pdesc->boundinfo->bounds.lists; + + if (isnull && listinfo.has_null) + return listinfo.null_index; + else if (!isnull) + { + found = partition_list_values_bsearch(key, + listinfo.values, + listinfo.nvalues, + value); + if (found >= 0) + return listinfo.indexes[found]; + } + + /* Control reaches here if isnull and !listinfo->has_null */ + return -1; +} + +/* + * range_partition_for_tuple + * Get the index of the range partition for a tuple (arg 'tuple' + * actually contains the range partition key of the original + * tuple) + * + * Returns -1 if none found. + */ +static int +range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, Datum *tuple) +{ + int offset; + PartitionRangeInfo rangeinfo; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_RANGE); + rangeinfo = pdesc->boundinfo->bounds.ranges; + + offset = partition_rbound_bsearch(key, + rangeinfo.bounds, rangeinfo.nbounds, + tuple, partition_rbound_datum_cmp, + true, NULL); + + /* + * Offset returned is such that the bound at offset is found to be less + * or equal with the tuple. That is, the tuple belongs to the partition + * with the rangeinfo.bounds[offset] as the lower bound and + * rangeinfo.bounds[offset+1] as the upper bound, provided the latter is + * indeed an upper (!lower) bound. If it turns out otherwise, the + * corresponding index will be -1, which means no valid partition exists. + */ + return rangeinfo.indexes[offset+1]; +} + /* List partition related support functions */ /* @@ -1772,6 +2053,40 @@ partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg) } /* + * Return whether bound <=, =, >= partition key of tuple + * + * The 3rd argument is void * so that it can be used with + * partition_rbound_bsearch() + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg) +{ + Datum *datums1 = bound->datums, + *datums2 = (Datum *) arg; + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (bound->infinite[i]) + return bound->lower ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + datums1[i], datums2[i])); + if (cmpval != 0) + break; + } + + /* If datums are equal and this is an upper bound, tuple > bound */ + if (cmpval == 0 && !bound->lower) + return -1; + + return cmpval; +} + +/* * Return whether two range bounds are equal simply by comparing datums */ static bool diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..cb0ca0e 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -30,6 +30,7 @@ #include "commands/defrem.h" #include "commands/trigger.h" #include "executor/executor.h" +#include "foreign/fdwapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -161,6 +162,11 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionTreeNode partition_tree; /* partition node tree */ + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; + List *partition_fdw_priv_lists; + int num_partitions; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1403,91 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + List *leaf_parts; + ListCell *cell; + int i; + int num_leaf_parts; + ResultRelInfo *leaf_part_rri; + PlannerInfo *root = makeNode(PlannerInfo); /* mostly dummy */ + ModifyTable *plan = makeNode(ModifyTable); /* ditto */ + RangeTblEntry *fdw_rte = makeNode(RangeTblEntry); /* ditto */ + List *fdw_private_lists = NIL; + + /* Form the partition node tree and lock partitions */ + cstate->partition_tree = RelationGetPartitionTreeNode(rel, + RowExclusiveLock); + leaf_parts = get_leaf_partition_oids(cstate->partition_tree); + num_leaf_parts = list_length(leaf_parts); + + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + /* For use below, iff a partition found to be a foreign table */ + plan->operation = CMD_INSERT; + plan->plans = list_make1(makeNode(Result)); + fdw_rte->rtekind = RTE_RELATION; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + root->parse = makeNode(Query); + root->parse->rtable = list_make1(fdw_rte); + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation part_rel; + + part_rel = heap_open(lfirst_oid(cell), RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + /* Special dance for foreign tables */ + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + fdw_rte->relid = RelationGetRelid(part_rel); + fdw_private = leaf_part_rri->ri_FdwRoutine->PlanForeignModify(root, + plan, + 1, + 0); + fdw_private_lists = lappend(fdw_private_lists, fdw_private); + } + + if (!equalTupleDescs(tupDesc, RelationGetDescr(part_rel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + + cstate->partition_fdw_priv_lists = fdw_private_lists; + } } else { @@ -2255,6 +2346,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2373,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2484,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2512,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->partition_tree != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2439,10 +2534,46 @@ CopyFrom(CopyState cstate) */ ExecBSInsertTriggers(estate, resultRelInfo); + /* Initialize FDW partition insert plans */ + if (cstate->partition_tree) + { + int i, + j; + List *fdw_private_lists = cstate->partition_fdw_priv_lists; + ModifyTableState *mtstate = makeNode(ModifyTableState); + ResultRelInfo *leaf_part_rri; + + /* Mostly dummy containing enough state for BeginForeignModify */ + mtstate->ps.state = estate; + mtstate->operation = CMD_INSERT; + + j = 0; + leaf_part_rri = cstate->partitions; + for (i = 0; i < cstate->num_partitions; i++) + { + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + Assert(fdw_private_lists); + fdw_private = list_nth(fdw_private_lists, j++); + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, 0); + } + leaf_part_rri++; + } + } + values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2625,50 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition to heap_insert the tuple into */ + if (cstate->partition_tree) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + cstate->partition_tree, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < cstate->num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding + * to the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = cstate->partitions + leaf_part_index; + + /* + * For ExecInsertIndexTuples() to work on the partition's indexes + */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the + * partition rowtype. + */ + map = cstate->partition_tupconv_maps[leaf_part_index]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2523,7 +2698,16 @@ CopyFrom(CopyState cstate) resultRelInfo->ri_PartitionCheck) ExecConstraints(resultRelInfo, slot, estate); - if (useHeapMultiInsert) + if (resultRelInfo->ri_FdwRoutine) + { + resultRelInfo->ri_FdwRoutine->ExecForeignInsert(estate, + resultRelInfo, + slot, + NULL); + /* AFTER ROW INSERT Triggers */ + ExecARInsertTriggers(estate, resultRelInfo, tuple, NIL); + } + else if (useHeapMultiInsert) { /* Add this tuple to the tuple buffer */ if (nBufferedTuples == 0) @@ -2553,7 +2737,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2762,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2781,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); @@ -2614,6 +2806,28 @@ CopyFrom(CopyState cstate) ExecCloseIndices(resultRelInfo); + /* + * Close all partitions and indices thereof, also clean up any + * state of FDW modify plans. + */ + if (cstate->partition_tree) + { + int i; + + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(estate, + resultRelInfo); + } + } + FreeExecutorState(estate); /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 3b72ae3..6478211 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1293,6 +1293,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index ea3f59a..3b4dd38 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2996,3 +3001,53 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +/* + * ExecFindPartition -- Find a leaf partition in the partition tree rooted + * at parent, for the heap tuple contained in *slot + * + * estate must be non-NULL; we'll need it to compute any expressions in the + * partition key(s) + * + * If no leaf partition is found, this routine errors out with the appropriate + * error message, else it returns the leaf partition sequence number returned + * by get_partition_for_tuple() unchanged. + */ +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionTreeNode parent, + TupleTableSlot *slot, EState *estate) +{ + int result; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + result = get_partition_for_tuple(parent, slot, estate, &failed_at); + + if (result < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return result; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..3523a2d 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,50 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + /* Determine the partition to heap_insert the tuple into */ + if (mtstate->mt_partition_tree) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_tree, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + leaf_part_index; + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, false); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +556,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1616,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1707,100 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + /* Form the partition node tree and lock partitions */ + mtstate->mt_partition_tree = RelationGetPartitionTreeNode(rel, + RowExclusiveLock); + leaf_parts = get_leaf_partition_oids(mtstate->mt_partition_tree); + num_leaf_parts = list_length(leaf_parts); + + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (leaf_part_rri->ri_FdwRoutine) + { + ListCell *lc; + List *fdw_private; + int k; + + /* + * There are as many fdw_private's in fdwPrivLists as there + * are FDW partitions, but we must find the intended for the + * this foreign table. + */ + k = 0; + foreach(lc, node->fdwPartitionOids) + { + if (lfirst_oid(lc) == ftoid) + break; + k++; + } + + Assert(k < num_leaf_parts); + fdw_private = (List *) list_nth(node->fdwPrivLists, k); + Assert(fdw_private != NIL); + + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, + eflags); + j++; + } + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2118,20 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(node->ps.state, + resultRelInfo); + } + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 28d0036..470dee7 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -188,6 +188,7 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(withCheckOptionLists); COPY_NODE_FIELD(returningLists); COPY_NODE_FIELD(fdwPrivLists); + COPY_NODE_FIELD(fdwPartitionOids); COPY_BITMAPSET_FIELD(fdwDirectModifyPlans); COPY_NODE_FIELD(rowMarks); COPY_SCALAR_FIELD(epqParam); diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 0d858f5..5c8eced 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -340,6 +340,7 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(withCheckOptionLists); WRITE_NODE_FIELD(returningLists); WRITE_NODE_FIELD(fdwPrivLists); + WRITE_NODE_FIELD(fdwPartitionOids); WRITE_BITMAPSET_FIELD(fdwDirectModifyPlans); WRITE_NODE_FIELD(rowMarks); WRITE_INT_FIELD(epqParam); diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c587d4e..ddd78c7 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -1495,6 +1495,7 @@ _readModifyTable(void) READ_NODE_FIELD(withCheckOptionLists); READ_NODE_FIELD(returningLists); READ_NODE_FIELD(fdwPrivLists); + READ_NODE_FIELD(fdwPartitionOids); READ_BITMAPSET_FIELD(fdwDirectModifyPlans); READ_NODE_FIELD(rowMarks); READ_INT_FIELD(epqParam); diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index ad49674..caf2f44 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -22,6 +22,7 @@ #include "access/stratnum.h" #include "access/sysattr.h" #include "catalog/pg_class.h" +#include "catalog/pg_inherits_fn.h" #include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/extensible.h" @@ -6159,6 +6160,81 @@ make_modifytable(PlannerInfo *root, node->fdwPrivLists = fdw_private_list; node->fdwDirectModifyPlans = direct_modify_plans; + /* Collect insert plans for all FDW-managed partitions */ + if (node->operation == CMD_INSERT) + { + RangeTblEntry *rte, + **saved_simple_rte_array; + List *partition_oids, + *fdw_partition_oids; + + Assert(list_length(resultRelations) == 1); + rte = rt_fetch(linitial_int(resultRelations), root->parse->rtable); + Assert(rte->rtekind == RTE_RELATION); + + if (rte->relkind != RELKIND_PARTITIONED_TABLE) + return node; + + partition_oids = find_all_inheritors(rte->relid, RowExclusiveLock, + NULL); + + /* Discard any previous content which is useless anyway */ + fdw_private_list = NIL; + fdw_partition_oids = NIL; + + /* + * To force the FDW driver fetch the intended RTE, we need to temporarily + * switch root->simple_rte_array to the one holding only that RTE at a + * designated index, for every foreign table. + */ + saved_simple_rte_array = root->simple_rte_array; + root->simple_rte_array = (RangeTblEntry **) + palloc0(2 * sizeof(RangeTblEntry *)); + foreach(lc, partition_oids) + { + Oid myoid = lfirst_oid(lc); + FdwRoutine *fdwroutine; + List *fdw_private; + + /* + * We are only interested in foreign tables. Note that this will + * also eliminate any partitioned tables since foreign tables can + * only ever be leaf partitions. + */ + if (get_rel_relkind(myoid) != RELKIND_FOREIGN_TABLE) + continue; + + fdw_partition_oids = lappend_oid(fdw_partition_oids, myoid); + + fdwroutine = GetFdwRoutineByRelId(myoid); + if (fdwroutine && fdwroutine->PlanForeignModify) + { + RangeTblEntry *fdw_rte; + + fdw_rte = copyObject(rte); + fdw_rte->relid = myoid; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + + /* + * Assumes PlanForeignModify() uses planner_rt_fetch(), also, + * see the above comment. + */ + root->simple_rte_array[1] = fdw_rte; + + fdw_private = fdwroutine->PlanForeignModify(root, node, 1, 0); + } + else + fdw_private = NIL; + + fdw_private_list = lappend(fdw_private_list, fdw_private); + } + + root->simple_rte_array = saved_simple_rte_array; + + node->fdwPrivLists = fdw_private_list; + node->fdwPartitionOids = fdw_partition_oids; + } + return node; } diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 672183b..e1ccaf8 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -50,4 +52,9 @@ extern List *RelationGetPartitionQual(Relation rel, bool recurse); /* For tuple routing */ extern PartitionTreeNode RelationGetPartitionTreeNode(Relation rel, int lockmode); extern List *get_leaf_partition_oids(PartitionTreeNode parent); + +extern int get_partition_for_tuple(PartitionTreeNode parent, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..fadc402 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionTreeNode parent, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..06675bc 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,15 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionTreeNodeData *mt_partition_tree; + /* Partition descriptor node tree */ + ResultRelInfo *mt_partitions; /* Per leaf partition target + * relations */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per leaf partition + * tuple conversion map */ + int mt_num_partitions; /* Number of leaf partition target + * relations in the above array */ } ModifyTableState; /* ---------------- diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index e2fbc7d..d82222c 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -189,6 +189,7 @@ typedef struct ModifyTable List *returningLists; /* per-target-table RETURNING tlists */ List *fdwPrivLists; /* per-target-table FDW private data lists */ Bitmapset *fdwDirectModifyPlans; /* indices of FDW DM plans */ + List *fdwPartitionOids; /* OIDs of FDW-managed partition */ List *rowMarks; /* PlanRowMarks (non-locking only) */ int epqParam; /* ID of Param for EvalPlanQual re-eval */ OnConflictAction onConflictAction; /* ON CONFLICT action */ diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 9ae6b09..d5dcb59 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -227,6 +227,58 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index b6e821e..fbd30d9 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,31 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------1C84B3F970C18697850BA485 Content-Type: text/x-diff; name="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-15.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-15"; filename*1=".patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH 7/8] Tuple routing for partitioned tables. @ 2016-07-27 07:59 amit <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: amit @ 2016-07-27 07:59 UTC (permalink / raw) Both COPY FROM and INSERT. --- src/backend/catalog/partition.c | 327 ++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 205 +++++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 47 +++++- src/backend/executor/nodeModifyTable.c | 142 +++++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 1 + src/backend/optimizer/plan/createplan.c | 77 +++++++ src/backend/optimizer/util/plancat.c | 13 ++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 7 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 10 + src/include/nodes/plannodes.h | 1 + src/include/optimizer/plancat.h | 1 + src/test/regress/expected/insert.out | 59 ++++++- src/test/regress/sql/insert.sql | 28 +++ 18 files changed, 926 insertions(+), 9 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index ae9e945..373e9e5 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -181,6 +181,18 @@ static List *generate_partition_qual(Relation rel, bool recurse); static PartitionTreeNode GetPartitionTreeNodeRecurse(Relation rel, int offset); static int get_leaf_partition_count(PartitionTreeNode ptnode); +/* Support get_partition_for_tuple() */ +static PartitionKeyExecInfo *BuildPartitionKeyExecInfo(Relation rel); +static void FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); +static int list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull); +static int range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum *tuple); + /* List partition related support functions */ static bool equal_list_info(PartitionKey key, PartitionListInfo *l1, PartitionListInfo *l2); @@ -194,6 +206,8 @@ static PartitionRangeBound *copy_range_bound(PartitionKey key, PartitionRangeBou static bool equal_range_info(PartitionKey key, PartitionRangeInfo *r1, PartitionRangeInfo *r2); static int32 partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg); +static int32 partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg); static bool partition_rbound_eq(PartitionKey key, PartitionRangeBound *b1, PartitionRangeBound *b2); typedef int32 (*partition_rbound_bsearch_cmp_fn) (PartitionKey, @@ -1442,7 +1456,7 @@ GetPartitionTreeNodeRecurse(Relation rel, int offset) /* First build our own node */ parent = (PartitionTreeNode) palloc0(sizeof(PartitionTreeNodeData)); - parent->pkinfo = NULL; + parent->pkinfo = BuildPartitionKeyExecInfo(rel); parent->pdesc = RelationGetPartitionDesc(rel); parent->relid = RelationGetRelid(rel); parent->offset = offset; @@ -1526,6 +1540,284 @@ get_leaf_partition_count(PartitionTreeNode ptnode) return result; } +/* + * BuildPartitionKeyExecInfo + * Construct a list of PartitionKeyExecInfo records for an open + * relation + * + * PartitionKeyExecInfo stores the information about the partition key + * that's needed when inserting tuples into a partitioned table; especially, + * partition key expression state if there are any expression columns in + * the partition key. Normally we build a PartitionKeyExecInfo for a + * partitioned table just once per command, and then use it for (potentially) + * many tuples. + * + */ +static PartitionKeyExecInfo * +BuildPartitionKeyExecInfo(Relation rel) +{ + PartitionKeyExecInfo *pkinfo; + + pkinfo = (PartitionKeyExecInfo *) palloc0(sizeof(PartitionKeyExecInfo)); + pkinfo->pi_Key = RelationGetPartitionKey(rel); + pkinfo->pi_ExpressionState = NIL; + + return pkinfo; +} + +/* + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for partition key columns + */ +static void +FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pkinfo->pi_Key->partexprs != NIL && pkinfo->pi_ExpressionState == NIL) + { + /* First time through, set up expression evaluation state */ + pkinfo->pi_ExpressionState = (List *) + ExecPrepareExpr((Expr *) pkinfo->pi_Key->partexprs, + estate); + /* Check caller has set up context correctly */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + } + + partexpr_item = list_head(pkinfo->pi_ExpressionState); + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + { + AttrNumber keycol = pkinfo->pi_Key->partattrs[i]; + Datum pkDatum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + pkDatum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + pkDatum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = pkDatum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Recursively finds the "leaf" partition for tuple + * + * Returns -1 if no partition is found and sets *failed_at to the OID of + * the partitioned table whose partition was not found. + */ +int +get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + Relation partRel; + PartitionKeyExecInfo *pkinfo = ptnode->pkinfo; + PartitionTreeNode node; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int i; + int index; + + /* Guard against stack overflow due to overly deep partition tree */ + check_stack_depth(); + + if (ptnode->pdesc->nparts == 0) + { + *failed_at = ptnode->relid; + return -1; + } + + /* Extract partition key from tuple */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + FormPartitionKeyDatum(pkinfo, slot, estate, values, isnull); + + /* Disallow nulls, if range partition key */ + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + if (isnull[i] && pkinfo->pi_Key->strategy == PARTITION_STRATEGY_RANGE) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key contains null"))); + + switch (pkinfo->pi_Key->strategy) + { + case PARTITION_STRATEGY_LIST: + index = list_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values[0], isnull[0]); + break; + + case PARTITION_STRATEGY_RANGE: + index = range_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values); + break; + } + + /* No partition found at this level */ + if (index < 0) + { + *failed_at = ptnode->relid; + return index; + } + + partRel = heap_open(ptnode->pdesc->oids[index], NoLock); + + /* Don't recurse if the index'th partition is a leaf partition. */ + if (partRel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) + { + PartitionTreeNode prev; + + /* + * Index returned above is the array index within pdesc->parts[] of + * the parent rel, however, we want to return the leaf partition index + * across the whole partition tree. Note that some partitions within + * pdesc->parts[] may be partitioned themselves and hence stand for + * the leaf partitions in their partition subtrees. We would need to + * skip past the indexes of leaf partitions of all such partition + * subtrees if they are to left of the above returned index. In fact, + * finding the PartitionTreeNode of the rightmost subtree is enough + * since its offset counts the leaf partitions on its left including + * those of partition subtrees to its left. + */ + prev = node = ptnode->downlink; + if (node && node->index < index) + { + /* + * Find the partition tree node such that its index value is the + * greatest value less than the above returned index. + */ + while (node) + { + if (node->index > index) + { + node = prev; + break; + } + + prev = node; + node = node->next; + } + + if (!node) + node = prev; + Assert (node != NULL); + + index = node->offset + node->num_leaf_parts + + (index - node->index - 1); + } + else + /* + * The easy case where we don't have any partition subtree to the + * left of the index. + */ + index = ptnode->offset + index; + + heap_close(partRel, NoLock); + return index; + } + + heap_close(partRel, NoLock); + + /* + * Need to perform recursion as the selected partition is partitioned + * itself. Locate the PartitionTreeNode corresponding to the partition + * passing it down. + */ + node = ptnode->downlink; + while (node->next != NULL && node->index != index) + node = node->next; + Assert (node != NULL); + + return get_partition_for_tuple(node, slot, estate, failed_at); +} + +/* + * list_partition_for_tuple + * Find the list partition for a tuple + * + * Returns -1 if none found. + */ +static int +list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull) +{ + PartitionListInfo listinfo; + int found; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_LIST); + listinfo = pdesc->boundinfo->bounds.lists; + + if (isnull && listinfo.has_null) + return listinfo.null_index; + else if (!isnull) + { + found = partition_list_values_bsearch(key, + listinfo.values, + listinfo.nvalues, + value); + if (found >= 0) + return listinfo.indexes[found]; + } + + /* Control reaches here if isnull and !listinfo->has_null */ + return -1; +} + +/* + * range_partition_for_tuple + * Search the range partition for a range key ('values') + * + * Returns -1 if none found. + */ +static int +range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, Datum *tuple) +{ + int offset; + PartitionRangeInfo rangeinfo; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_RANGE); + rangeinfo = pdesc->boundinfo->bounds.ranges; + + offset = partition_rbound_bsearch(key, + rangeinfo.bounds, rangeinfo.nbounds, + tuple, partition_rbound_datum_cmp, + true, NULL); + + /* + * Offset returned is such that the bound at offset is found to be + * less or equal with the tuple. That is, the tuple belongs to the + * partition with the rangeinfo.bounds[offset] as the lower bound and + * rangeinfo.bounds[offset+1] as the upper bound, provided the latter + * is indeed marked !lower (that is, it's an upper bound). If it turns + * out that it is a lower bound then the corresponding index will be -1, + * which means no valid partition exists. + */ + return rangeinfo.indexes[offset+1]; +} + /* List partition related support functions */ /* @@ -1787,6 +2079,39 @@ partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg) } /* + * Return whether the passed in range bound <=, =, >= tuple specified in arg + * + * The 2nd argument is void * so that it can be used with + * partition_rbound_bsearch() + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg) +{ + Datum *datums1 = bound->datums, + *datums2 = (Datum *) arg; + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (bound->infinite[i]) + return bound->lower ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + datums1[i], datums2[i])); + if (cmpval != 0) + break; + } + + if (cmpval == 0 && !bound->inclusive) + return bound->lower ? 1 : -1; + + return cmpval; +} + +/* * Are two (consecutive) range bounds equal without distinguishing lower * and upper? */ diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 77d4dcb..91e4d12 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -30,6 +30,7 @@ #include "commands/defrem.h" #include "commands/trigger.h" #include "executor/executor.h" +#include "foreign/fdwapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -161,6 +162,11 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionTreeNode ptnode; /* partition descriptor node tree */ + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; + List *partition_fdw_priv_lists; + int num_partitions; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1403,94 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + List *leaf_part_oids; + ListCell *cell; + int i; + int num_leaf_parts; + ResultRelInfo *leaf_rel_rri; + PlannerInfo *root = makeNode(PlannerInfo); /* mostly dummy */ + Query *parse = makeNode(Query); /* ditto */ + ModifyTable *plan = makeNode(ModifyTable); /* ditto */ + RangeTblEntry *fdw_rte = makeNode(RangeTblEntry); /* ditto */ + List *fdw_private_lists = NIL; + + cstate->ptnode = RelationGetPartitionTreeNode(rel); + leaf_part_oids = get_leaf_partition_oids_v2(cstate->ptnode); + num_leaf_parts = list_length(leaf_part_oids); + + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + /* For use below, iff a partition found to be a foreign table */ + plan->operation = CMD_INSERT; + plan->plans = list_make1(makeNode(Result)); + fdw_rte->rtekind = RTE_RELATION; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + parse->rtable = list_make1(fdw_rte); + root->parse = parse; + + leaf_rel_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_part_oids) + { + Relation leaf_rel; + + leaf_rel = heap_open(lfirst_oid(cell), RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(leaf_rel, CMD_INSERT); + + InitResultRelInfo(leaf_rel_rri, + leaf_rel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_rel_rri, false); + + /* Special dance for foreign tables */ + if (leaf_rel_rri->ri_FdwRoutine) + { + List *fdw_private; + + fdw_rte->relid = RelationGetRelid(leaf_rel); + fdw_private = leaf_rel_rri->ri_FdwRoutine->PlanForeignModify(root, + plan, + 1, + 0); + fdw_private_lists = lappend(fdw_private_lists, fdw_private); + } + + if (!equalTupleDescs(tupDesc, RelationGetDescr(leaf_rel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(leaf_rel), + gettext_noop("could not convert row type")); + + leaf_rel_rri++; + i++; + } + + cstate->partition_fdw_priv_lists = fdw_private_lists; + pfree(fdw_rte); + pfree(plan); + pfree(parse); + pfree(root); + } } else { @@ -1692,6 +1786,8 @@ ClosePipeToProgram(CopyState cstate) static void EndCopy(CopyState cstate) { + int i; + if (cstate->is_program) { ClosePipeToProgram(cstate); @@ -1705,6 +1801,23 @@ EndCopy(CopyState cstate) cstate->filename))); } + /* Close all partitions and indices thereof */ + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + /* XXX - EState not handy here to pass to EndForeignModify() */ + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(NULL, resultRelInfo); + + if (cstate->partition_tupconv_maps[i]) + pfree(cstate->partition_tupconv_maps[i]); + } + MemoryContextDelete(cstate->copycontext); pfree(cstate); } @@ -2255,6 +2368,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2275,7 +2389,8 @@ CopyFrom(CopyState cstate) Assert(cstate->rel); - if (cstate->rel->rd_rel->relkind != RELKIND_RELATION) + if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) { if (cstate->rel->rd_rel->relkind == RELKIND_VIEW) ereport(ERROR, @@ -2383,6 +2498,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2410,6 +2526,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->ptnode != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2431,10 +2548,46 @@ CopyFrom(CopyState cstate) */ ExecBSInsertTriggers(estate, resultRelInfo); + /* Initialize FDW partition insert plans */ + if (cstate->ptnode) + { + int i, + j; + List *fdw_private_lists = cstate->partition_fdw_priv_lists; + ModifyTableState *mtstate = makeNode(ModifyTableState); + ResultRelInfo *leaf_part_rri; + + /* Mostly dummy containing enough state for BeginForeignModify */ + mtstate->ps.state = estate; + mtstate->operation = CMD_INSERT; + + j = 0; + leaf_part_rri = cstate->partitions; + for (i = 0; i < cstate->num_partitions; i++) + { + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + Assert(fdw_private_lists); + fdw_private = list_nth(fdw_private_lists, j++); + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, 0); + } + leaf_part_rri++; + } + } + values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2486,6 +2639,31 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition */ + saved_resultRelInfo = resultRelInfo; + if (cstate->ptnode) + { + int i_leaf_partition; + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + cstate->ptnode, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < cstate->num_partitions); + + resultRelInfo = cstate->partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = cstate->partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2506,7 +2684,16 @@ CopyFrom(CopyState cstate) if (cstate->rel->rd_att->constr || resultRelInfo->ri_PartitionCheck) ExecConstraints(resultRelInfo, slot, estate); - if (useHeapMultiInsert) + if (resultRelInfo->ri_FdwRoutine) + { + resultRelInfo->ri_FdwRoutine->ExecForeignInsert(estate, + resultRelInfo, + slot, + NULL); + /* AFTER ROW INSERT Triggers */ + ExecARInsertTriggers(estate, resultRelInfo, tuple, NIL); + } + else if (useHeapMultiInsert) { /* Add this tuple to the tuple buffer */ if (nBufferedTuples == 0) @@ -2536,7 +2723,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, + tuple, mycid, hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self), @@ -2556,6 +2744,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2569,7 +2763,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 395e116..685b184 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1294,6 +1294,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index ea3f59a..62bf8b7 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2996,3 +3001,41 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionTreeNode ptnode, + TupleTableSlot *slot, EState *estate) +{ + int i_leaf_partition; + Oid failed_at; + + i_leaf_partition = get_partition_for_tuple(ptnode, slot, estate, + &failed_at); + + if (i_leaf_partition < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return i_leaf_partition; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..d0a5306 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,31 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + saved_resultRelInfo = resultRelInfo; + + if (mtstate->mt_partition_tree_root) + { + int i_leaf_partition; + ExprContext *econtext = GetPerTupleExprContext(estate); + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_tree_root, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < mtstate->mt_num_partitions); + + resultRelInfo = mtstate->mt_partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = mtstate->mt_partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +537,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1597,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1688,98 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + int i, + j, + num_leaf_parts; + List *leaf_part_oids; + ListCell *cell; + ResultRelInfo *leaf_rel_rri; + + mtstate->mt_partition_tree_root = RelationGetPartitionTreeNode(rel); + leaf_part_oids = get_leaf_partition_oids_v2(mtstate->mt_partition_tree_root); + num_leaf_parts = list_length(leaf_part_oids); + + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_rel_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_part_oids) + { + Oid ftoid = lfirst_oid(cell); + Relation leaf_rel; + + leaf_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(leaf_rel, CMD_INSERT); + + InitResultRelInfo(leaf_rel_rri, + leaf_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_rel_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_rel_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_rel_rri, false); + + if (leaf_rel_rri->ri_FdwRoutine) + { + ListCell *lc; + List *fdw_private; + int k; + + /* + * There are as many fdw_private's in fdwPrivLists as there + * are FDW partitions, but we must find the intended for the + * this foreign table. + */ + k = 0; + foreach(lc, node->fdwPartitionOids) + { + if (lfirst_oid(lc) == ftoid) + break; + k++; + } + + Assert(k < num_leaf_parts); + fdw_private = (List *) list_nth(node->fdwPrivLists, k); + Assert(fdw_private != NIL); + + leaf_rel_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_rel_rri, + fdw_private, + 0, + eflags); + j++; + } + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(leaf_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(leaf_rel), + gettext_noop("could not convert row type")); + + leaf_rel_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2097,23 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(node->ps.state, + resultRelInfo); + + if (node->mt_partition_tupconv_maps[i]) + pfree(node->mt_partition_tupconv_maps[i]); + } + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 28d0036..470dee7 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -188,6 +188,7 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(withCheckOptionLists); COPY_NODE_FIELD(returningLists); COPY_NODE_FIELD(fdwPrivLists); + COPY_NODE_FIELD(fdwPartitionOids); COPY_BITMAPSET_FIELD(fdwDirectModifyPlans); COPY_NODE_FIELD(rowMarks); COPY_SCALAR_FIELD(epqParam); diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 0d858f5..5c8eced 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -340,6 +340,7 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(withCheckOptionLists); WRITE_NODE_FIELD(returningLists); WRITE_NODE_FIELD(fdwPrivLists); + WRITE_NODE_FIELD(fdwPartitionOids); WRITE_BITMAPSET_FIELD(fdwDirectModifyPlans); WRITE_NODE_FIELD(rowMarks); WRITE_INT_FIELD(epqParam); diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c587d4e..ddd78c7 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -1495,6 +1495,7 @@ _readModifyTable(void) READ_NODE_FIELD(withCheckOptionLists); READ_NODE_FIELD(returningLists); READ_NODE_FIELD(fdwPrivLists); + READ_NODE_FIELD(fdwPartitionOids); READ_BITMAPSET_FIELD(fdwDirectModifyPlans); READ_NODE_FIELD(rowMarks); READ_INT_FIELD(epqParam); diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index ad49674..29f40fe 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -22,6 +22,7 @@ #include "access/stratnum.h" #include "access/sysattr.h" #include "catalog/pg_class.h" +#include "catalog/pg_inherits_fn.h" #include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/extensible.h" @@ -6159,6 +6160,82 @@ make_modifytable(PlannerInfo *root, node->fdwPrivLists = fdw_private_list; node->fdwDirectModifyPlans = direct_modify_plans; + /* Collect insert plans for all FDW-managed partitions */ + if (node->operation == CMD_INSERT) + { + RangeTblEntry *rte, + **saved_simple_rte_array; + List *partition_oids, + *fdw_partition_oids; + + Assert(list_length(resultRelations) == 1); + rte = rt_fetch(linitial_int(resultRelations), root->parse->rtable); + Assert(rte->rtekind == RTE_RELATION); + + if (rte->relkind != RELKIND_PARTITIONED_TABLE) + return node; + + partition_oids = find_all_inheritors(rte->relid, NoLock, NULL); + + /* Discard any previous content which is useless anyway */ + fdw_private_list = NIL; + fdw_partition_oids = NIL; + + /* + * To force the FDW driver fetch the intended RTE, we need to temporarily + * switch root->simple_rte_array to the one holding only that RTE at a + * designated index, for every foreign table. + */ + saved_simple_rte_array = root->simple_rte_array; + root->simple_rte_array = (RangeTblEntry **) + palloc0(2 * sizeof(RangeTblEntry *)); + foreach(lc, partition_oids) + { + Oid myoid = lfirst_oid(lc); + FdwRoutine *fdwroutine; + List *fdw_private; + + /* + * We are only interested in foreign tables. Note that this will + * also eliminate any partitioned tables since foreign tables can + * only ever be leaf partitions. + */ + if (!oid_is_foreign_table(myoid)) + continue; + + fdw_partition_oids = lappend_oid(fdw_partition_oids, myoid); + + fdwroutine = GetFdwRoutineByRelId(myoid); + if (fdwroutine && fdwroutine->PlanForeignModify) + { + RangeTblEntry *fdw_rte; + + fdw_rte = copyObject(rte); + fdw_rte->relid = myoid; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + + /* + * Assumes PlanForeignModify() uses planner_rt_fetch(), also, + * see the above comment. + */ + root->simple_rte_array[1] = fdw_rte; + + fdw_private = fdwroutine->PlanForeignModify(root, node, 1, 0); + pfree(fdw_rte); + } + else + fdw_private = NIL; + + fdw_private_list = lappend(fdw_private_list, fdw_private); + } + + pfree(root->simple_rte_array); + root->simple_rte_array = saved_simple_rte_array; + + node->fdwPrivLists = fdw_private_list; + node->fdwPartitionOids = fdw_partition_oids; + } + return node; } diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index a2cbf14..e85ca0a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1715,3 +1715,16 @@ has_row_triggers(PlannerInfo *root, Index rti, CmdType event) heap_close(relation, NoLock); return result; } + +bool +oid_is_foreign_table(Oid relid) +{ + Relation rel; + char relkind; + + rel = heap_open(relid, NoLock); + relkind = rel->rd_rel->relkind; + heap_close(rel, NoLock); + + return relkind == RELKIND_FOREIGN_TABLE; +} diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index ab6d3a8..09727cf 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -56,4 +58,9 @@ extern List *RelationGetPartitionQual(Relation rel, bool recurse); /* For tuple routing */ extern PartitionTreeNode RelationGetPartitionTreeNode(Relation rel); extern List *get_leaf_partition_oids_v2(PartitionTreeNode ptnode); + +extern int get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..c62946f 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..ce01008 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,15 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionTreeNodeData *mt_partition_tree_root; + /* Partition descriptor node tree */ + ResultRelInfo *mt_partitions; /* Per leaf partition target + * relations */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per leaf partition + * tuple conversion map */ + int mt_num_partitions; /* Number of leaf partition target + * relations in the above array */ } ModifyTableState; /* ---------------- diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index e2fbc7d..d82222c 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -189,6 +189,7 @@ typedef struct ModifyTable List *returningLists; /* per-target-table RETURNING tlists */ List *fdwPrivLists; /* per-target-table FDW private data lists */ Bitmapset *fdwDirectModifyPlans; /* indices of FDW DM plans */ + List *fdwPartitionOids; /* OIDs of FDW-managed partition */ List *rowMarks; /* PlanRowMarks (non-locking only) */ int epqParam; /* ID of Param for EvalPlanQual re-eval */ OnConflictAction onConflictAction; /* ON CONFLICT action */ diff --git a/src/include/optimizer/plancat.h b/src/include/optimizer/plancat.h index 125274e..fac606c 100644 --- a/src/include/optimizer/plancat.h +++ b/src/include/optimizer/plancat.h @@ -56,5 +56,6 @@ extern Selectivity join_selectivity(PlannerInfo *root, SpecialJoinInfo *sjinfo); extern bool has_row_triggers(PlannerInfo *root, Index rti, CmdType event); +extern bool oid_is_foreign_table(Oid relid); #endif /* PLANCAT_H */ diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 5d58d23..619336b 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -222,6 +222,62 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- fail (no list partition defined which accepts nulls) +insert into list_parted (b) values (1); +ERROR: no partition of relation "list_parted" found for row +DETAIL: Failing row contains (null, 1). +create table part_nulls partition of list_parted for values in (null); +-- ok +insert into list_parted (b) values (1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_nulls | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(7 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects @@ -230,9 +286,10 @@ drop cascades to table part_a_10_a_20 drop cascades to table part_b_1_b_10 drop cascades to table part_b_10_b_20 drop table list_parted cascade; -NOTICE: drop cascades to 5 other objects +NOTICE: drop cascades to 6 other objects DETAIL: drop cascades to table part_aa_bb drop cascades to table part_cc_dd drop cascades to table part_ee_ff drop cascades to table part_ee_ff_1_10 drop cascades to table part_ee_ff_10_20 +drop cascades to table part_nulls diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index 5aedeef..210fab4 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -137,6 +137,34 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- fail (no list partition defined which accepts nulls) +insert into list_parted (b) values (1); +create table part_nulls partition of list_parted for values in (null); +-- ok +insert into list_parted (b) values (1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------B737FB31608A9E2BB1F071B4 Content-Type: text/x-diff; name="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-12.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-12"; filename*1=".patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH 7/8] Tuple routing for partitioned tables. @ 2016-07-27 07:59 amit <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: amit @ 2016-07-27 07:59 UTC (permalink / raw) Both COPY FROM and INSERT. --- src/backend/catalog/partition.c | 327 ++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 205 +++++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 47 +++++- src/backend/executor/nodeModifyTable.c | 142 +++++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 1 + src/backend/optimizer/plan/createplan.c | 77 +++++++ src/backend/optimizer/util/plancat.c | 13 ++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 7 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 10 + src/include/nodes/plannodes.h | 1 + src/include/optimizer/plancat.h | 1 + src/test/regress/expected/insert.out | 59 ++++++- src/test/regress/sql/insert.sql | 28 +++ 18 files changed, 926 insertions(+), 9 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index 69c3e52..9cba603 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -181,6 +181,18 @@ static List *generate_partition_qual(Relation rel, bool recurse); static PartitionTreeNode GetPartitionTreeNodeRecurse(Relation rel, int offset); static int get_leaf_partition_count(PartitionTreeNode ptnode); +/* Support get_partition_for_tuple() */ +static PartitionKeyExecInfo *BuildPartitionKeyExecInfo(Relation rel); +static void FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); +static int list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull); +static int range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum *tuple); + /* List partition related support functions */ static bool equal_list_info(PartitionKey key, PartitionListInfo *l1, PartitionListInfo *l2); @@ -195,6 +207,8 @@ static PartitionRangeBound *copy_range_bound(PartitionKey key, PartitionRangeBou static bool equal_range_info(PartitionKey key, PartitionRangeInfo *r1, PartitionRangeInfo *r2); static int32 partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg); +static int32 partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg); static bool partition_rbound_eq(PartitionKey key, PartitionRangeBound *b1, PartitionRangeBound *b2); typedef int32 (*partition_rbound_bsearch_cmp_fn) (PartitionKey, @@ -1449,7 +1463,7 @@ GetPartitionTreeNodeRecurse(Relation rel, int offset) /* First build our own node */ parent = (PartitionTreeNode) palloc0(sizeof(PartitionTreeNodeData)); - parent->pkinfo = NULL; + parent->pkinfo = BuildPartitionKeyExecInfo(rel); parent->pdesc = RelationGetPartitionDesc(rel); parent->relid = RelationGetRelid(rel); parent->offset = offset; @@ -1533,6 +1547,284 @@ get_leaf_partition_count(PartitionTreeNode ptnode) return result; } +/* + * BuildPartitionKeyExecInfo + * Construct a list of PartitionKeyExecInfo records for an open + * relation + * + * PartitionKeyExecInfo stores the information about the partition key + * that's needed when inserting tuples into a partitioned table; especially, + * partition key expression state if there are any expression columns in + * the partition key. Normally we build a PartitionKeyExecInfo for a + * partitioned table just once per command, and then use it for (potentially) + * many tuples. + * + */ +static PartitionKeyExecInfo * +BuildPartitionKeyExecInfo(Relation rel) +{ + PartitionKeyExecInfo *pkinfo; + + pkinfo = (PartitionKeyExecInfo *) palloc0(sizeof(PartitionKeyExecInfo)); + pkinfo->pi_Key = RelationGetPartitionKey(rel); + pkinfo->pi_ExpressionState = NIL; + + return pkinfo; +} + +/* + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for partition key columns + */ +static void +FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pkinfo->pi_Key->partexprs != NIL && pkinfo->pi_ExpressionState == NIL) + { + /* First time through, set up expression evaluation state */ + pkinfo->pi_ExpressionState = (List *) + ExecPrepareExpr((Expr *) pkinfo->pi_Key->partexprs, + estate); + /* Check caller has set up context correctly */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + } + + partexpr_item = list_head(pkinfo->pi_ExpressionState); + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + { + AttrNumber keycol = pkinfo->pi_Key->partattrs[i]; + Datum pkDatum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + pkDatum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + pkDatum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = pkDatum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Recursively finds the "leaf" partition for tuple + * + * Returns -1 if no partition is found and sets *failed_at to the OID of + * the partitioned table whose partition was not found. + */ +int +get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + Relation partRel; + PartitionKeyExecInfo *pkinfo = ptnode->pkinfo; + PartitionTreeNode node; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int i; + int index; + + /* Guard against stack overflow due to overly deep partition tree */ + check_stack_depth(); + + if (ptnode->pdesc->nparts == 0) + { + *failed_at = ptnode->relid; + return -1; + } + + /* Extract partition key from tuple */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + FormPartitionKeyDatum(pkinfo, slot, estate, values, isnull); + + /* Disallow nulls, if range partition key */ + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + if (isnull[i] && pkinfo->pi_Key->strategy == PARTITION_STRATEGY_RANGE) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key contains null"))); + + switch (pkinfo->pi_Key->strategy) + { + case PARTITION_STRATEGY_LIST: + index = list_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values[0], isnull[0]); + break; + + case PARTITION_STRATEGY_RANGE: + index = range_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values); + break; + } + + /* No partition found at this level */ + if (index < 0) + { + *failed_at = ptnode->relid; + return index; + } + + partRel = heap_open(ptnode->pdesc->oids[index], NoLock); + + /* Don't recurse if the index'th partition is a leaf partition. */ + if (partRel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) + { + PartitionTreeNode prev; + + /* + * Index returned above is the array index within pdesc->parts[] of + * the parent rel, however, we want to return the leaf partition index + * across the whole partition tree. Note that some partitions within + * pdesc->parts[] may be partitioned themselves and hence stand for + * the leaf partitions in their partition subtrees. We would need to + * skip past the indexes of leaf partitions of all such partition + * subtrees if they are to left of the above returned index. In fact, + * finding the PartitionTreeNode of the rightmost subtree is enough + * since its offset counts the leaf partitions on its left including + * those of partition subtrees to its left. + */ + prev = node = ptnode->downlink; + if (node && node->index < index) + { + /* + * Find the partition tree node such that its index value is the + * greatest value less than the above returned index. + */ + while (node) + { + if (node->index > index) + { + node = prev; + break; + } + + prev = node; + node = node->next; + } + + if (!node) + node = prev; + Assert (node != NULL); + + index = node->offset + node->num_leaf_parts + + (index - node->index - 1); + } + else + /* + * The easy case where we don't have any partition subtree to the + * left of the index. + */ + index = ptnode->offset + index; + + heap_close(partRel, NoLock); + return index; + } + + heap_close(partRel, NoLock); + + /* + * Need to perform recursion as the selected partition is partitioned + * itself. Locate the PartitionTreeNode corresponding to the partition + * passing it down. + */ + node = ptnode->downlink; + while (node->next != NULL && node->index != index) + node = node->next; + Assert (node != NULL); + + return get_partition_for_tuple(node, slot, estate, failed_at); +} + +/* + * list_partition_for_tuple + * Find the list partition for a tuple + * + * Returns -1 if none found. + */ +static int +list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull) +{ + PartitionListInfo listinfo; + int found; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_LIST); + listinfo = pdesc->boundinfo->bounds.lists; + + if (isnull && listinfo.has_null) + return listinfo.null_index; + else if (!isnull) + { + found = partition_list_values_bsearch(key, + listinfo.values, + listinfo.nvalues, + value); + if (found >= 0) + return listinfo.indexes[found]; + } + + /* Control reaches here if isnull and !listinfo->has_null */ + return -1; +} + +/* + * range_partition_for_tuple + * Search the range partition for a range key ('values') + * + * Returns -1 if none found. + */ +static int +range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, Datum *tuple) +{ + int offset; + PartitionRangeInfo rangeinfo; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_RANGE); + rangeinfo = pdesc->boundinfo->bounds.ranges; + + offset = partition_rbound_bsearch(key, + rangeinfo.bounds, rangeinfo.nbounds, + tuple, partition_rbound_datum_cmp, + true, NULL); + + /* + * Offset returned is such that the bound at offset is found to be + * less or equal with the tuple. That is, the tuple belongs to the + * partition with the rangeinfo.bounds[offset] as the lower bound and + * rangeinfo.bounds[offset+1] as the upper bound, provided the latter + * is indeed marked !lower (that is, it's an upper bound). If it turns + * out that it is a lower bound then the corresponding index will be -1, + * which means no valid partition exists. + */ + return rangeinfo.indexes[offset+1]; +} + /* List partition related support functions */ /* @@ -1798,6 +2090,39 @@ partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg) } /* + * Return whether the passed in range bound <=, =, >= tuple specified in arg + * + * The 2nd argument is void * so that it can be used with + * partition_rbound_bsearch() + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg) +{ + Datum *datums1 = bound->datums, + *datums2 = (Datum *) arg; + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (bound->infinite[i]) + return bound->lower ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + datums1[i], datums2[i])); + if (cmpval != 0) + break; + } + + if (cmpval == 0 && !bound->inclusive) + return bound->lower ? 1 : -1; + + return cmpval; +} + +/* * Are two (consecutive) range bounds equal without distinguishing lower * and upper? */ diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 77d4dcb..91e4d12 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -30,6 +30,7 @@ #include "commands/defrem.h" #include "commands/trigger.h" #include "executor/executor.h" +#include "foreign/fdwapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -161,6 +162,11 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionTreeNode ptnode; /* partition descriptor node tree */ + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; + List *partition_fdw_priv_lists; + int num_partitions; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1403,94 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + List *leaf_part_oids; + ListCell *cell; + int i; + int num_leaf_parts; + ResultRelInfo *leaf_rel_rri; + PlannerInfo *root = makeNode(PlannerInfo); /* mostly dummy */ + Query *parse = makeNode(Query); /* ditto */ + ModifyTable *plan = makeNode(ModifyTable); /* ditto */ + RangeTblEntry *fdw_rte = makeNode(RangeTblEntry); /* ditto */ + List *fdw_private_lists = NIL; + + cstate->ptnode = RelationGetPartitionTreeNode(rel); + leaf_part_oids = get_leaf_partition_oids_v2(cstate->ptnode); + num_leaf_parts = list_length(leaf_part_oids); + + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + /* For use below, iff a partition found to be a foreign table */ + plan->operation = CMD_INSERT; + plan->plans = list_make1(makeNode(Result)); + fdw_rte->rtekind = RTE_RELATION; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + parse->rtable = list_make1(fdw_rte); + root->parse = parse; + + leaf_rel_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_part_oids) + { + Relation leaf_rel; + + leaf_rel = heap_open(lfirst_oid(cell), RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(leaf_rel, CMD_INSERT); + + InitResultRelInfo(leaf_rel_rri, + leaf_rel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_rel_rri, false); + + /* Special dance for foreign tables */ + if (leaf_rel_rri->ri_FdwRoutine) + { + List *fdw_private; + + fdw_rte->relid = RelationGetRelid(leaf_rel); + fdw_private = leaf_rel_rri->ri_FdwRoutine->PlanForeignModify(root, + plan, + 1, + 0); + fdw_private_lists = lappend(fdw_private_lists, fdw_private); + } + + if (!equalTupleDescs(tupDesc, RelationGetDescr(leaf_rel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(leaf_rel), + gettext_noop("could not convert row type")); + + leaf_rel_rri++; + i++; + } + + cstate->partition_fdw_priv_lists = fdw_private_lists; + pfree(fdw_rte); + pfree(plan); + pfree(parse); + pfree(root); + } } else { @@ -1692,6 +1786,8 @@ ClosePipeToProgram(CopyState cstate) static void EndCopy(CopyState cstate) { + int i; + if (cstate->is_program) { ClosePipeToProgram(cstate); @@ -1705,6 +1801,23 @@ EndCopy(CopyState cstate) cstate->filename))); } + /* Close all partitions and indices thereof */ + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + /* XXX - EState not handy here to pass to EndForeignModify() */ + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(NULL, resultRelInfo); + + if (cstate->partition_tupconv_maps[i]) + pfree(cstate->partition_tupconv_maps[i]); + } + MemoryContextDelete(cstate->copycontext); pfree(cstate); } @@ -2255,6 +2368,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2275,7 +2389,8 @@ CopyFrom(CopyState cstate) Assert(cstate->rel); - if (cstate->rel->rd_rel->relkind != RELKIND_RELATION) + if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) { if (cstate->rel->rd_rel->relkind == RELKIND_VIEW) ereport(ERROR, @@ -2383,6 +2498,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2410,6 +2526,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->ptnode != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2431,10 +2548,46 @@ CopyFrom(CopyState cstate) */ ExecBSInsertTriggers(estate, resultRelInfo); + /* Initialize FDW partition insert plans */ + if (cstate->ptnode) + { + int i, + j; + List *fdw_private_lists = cstate->partition_fdw_priv_lists; + ModifyTableState *mtstate = makeNode(ModifyTableState); + ResultRelInfo *leaf_part_rri; + + /* Mostly dummy containing enough state for BeginForeignModify */ + mtstate->ps.state = estate; + mtstate->operation = CMD_INSERT; + + j = 0; + leaf_part_rri = cstate->partitions; + for (i = 0; i < cstate->num_partitions; i++) + { + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + Assert(fdw_private_lists); + fdw_private = list_nth(fdw_private_lists, j++); + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, 0); + } + leaf_part_rri++; + } + } + values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2486,6 +2639,31 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition */ + saved_resultRelInfo = resultRelInfo; + if (cstate->ptnode) + { + int i_leaf_partition; + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + cstate->ptnode, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < cstate->num_partitions); + + resultRelInfo = cstate->partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = cstate->partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2506,7 +2684,16 @@ CopyFrom(CopyState cstate) if (cstate->rel->rd_att->constr || resultRelInfo->ri_PartitionCheck) ExecConstraints(resultRelInfo, slot, estate); - if (useHeapMultiInsert) + if (resultRelInfo->ri_FdwRoutine) + { + resultRelInfo->ri_FdwRoutine->ExecForeignInsert(estate, + resultRelInfo, + slot, + NULL); + /* AFTER ROW INSERT Triggers */ + ExecARInsertTriggers(estate, resultRelInfo, tuple, NIL); + } + else if (useHeapMultiInsert) { /* Add this tuple to the tuple buffer */ if (nBufferedTuples == 0) @@ -2536,7 +2723,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, + tuple, mycid, hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self), @@ -2556,6 +2744,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2569,7 +2763,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index de19e9c..88b9d1f 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1292,6 +1292,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index ea3f59a..62bf8b7 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2996,3 +3001,41 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionTreeNode ptnode, + TupleTableSlot *slot, EState *estate) +{ + int i_leaf_partition; + Oid failed_at; + + i_leaf_partition = get_partition_for_tuple(ptnode, slot, estate, + &failed_at); + + if (i_leaf_partition < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return i_leaf_partition; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..d0a5306 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,31 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + saved_resultRelInfo = resultRelInfo; + + if (mtstate->mt_partition_tree_root) + { + int i_leaf_partition; + ExprContext *econtext = GetPerTupleExprContext(estate); + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_tree_root, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < mtstate->mt_num_partitions); + + resultRelInfo = mtstate->mt_partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = mtstate->mt_partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +537,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1597,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1688,98 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + int i, + j, + num_leaf_parts; + List *leaf_part_oids; + ListCell *cell; + ResultRelInfo *leaf_rel_rri; + + mtstate->mt_partition_tree_root = RelationGetPartitionTreeNode(rel); + leaf_part_oids = get_leaf_partition_oids_v2(mtstate->mt_partition_tree_root); + num_leaf_parts = list_length(leaf_part_oids); + + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_rel_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_part_oids) + { + Oid ftoid = lfirst_oid(cell); + Relation leaf_rel; + + leaf_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(leaf_rel, CMD_INSERT); + + InitResultRelInfo(leaf_rel_rri, + leaf_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_rel_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_rel_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_rel_rri, false); + + if (leaf_rel_rri->ri_FdwRoutine) + { + ListCell *lc; + List *fdw_private; + int k; + + /* + * There are as many fdw_private's in fdwPrivLists as there + * are FDW partitions, but we must find the intended for the + * this foreign table. + */ + k = 0; + foreach(lc, node->fdwPartitionOids) + { + if (lfirst_oid(lc) == ftoid) + break; + k++; + } + + Assert(k < num_leaf_parts); + fdw_private = (List *) list_nth(node->fdwPrivLists, k); + Assert(fdw_private != NIL); + + leaf_rel_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_rel_rri, + fdw_private, + 0, + eflags); + j++; + } + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(leaf_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(leaf_rel), + gettext_noop("could not convert row type")); + + leaf_rel_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2097,23 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(node->ps.state, + resultRelInfo); + + if (node->mt_partition_tupconv_maps[i]) + pfree(node->mt_partition_tupconv_maps[i]); + } + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index c0d48d0..ca48547 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -188,6 +188,7 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(withCheckOptionLists); COPY_NODE_FIELD(returningLists); COPY_NODE_FIELD(fdwPrivLists); + COPY_NODE_FIELD(fdwPartitionOids); COPY_BITMAPSET_FIELD(fdwDirectModifyPlans); COPY_NODE_FIELD(rowMarks); COPY_SCALAR_FIELD(epqParam); diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index d5eb0cc..7a219e5 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -340,6 +340,7 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(withCheckOptionLists); WRITE_NODE_FIELD(returningLists); WRITE_NODE_FIELD(fdwPrivLists); + WRITE_NODE_FIELD(fdwPartitionOids); WRITE_BITMAPSET_FIELD(fdwDirectModifyPlans); WRITE_NODE_FIELD(rowMarks); WRITE_INT_FIELD(epqParam); diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 655aa1c..d047160 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -1495,6 +1495,7 @@ _readModifyTable(void) READ_NODE_FIELD(withCheckOptionLists); READ_NODE_FIELD(returningLists); READ_NODE_FIELD(fdwPrivLists); + READ_NODE_FIELD(fdwPartitionOids); READ_BITMAPSET_FIELD(fdwDirectModifyPlans); READ_NODE_FIELD(rowMarks); READ_INT_FIELD(epqParam); diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index ad49674..29f40fe 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -22,6 +22,7 @@ #include "access/stratnum.h" #include "access/sysattr.h" #include "catalog/pg_class.h" +#include "catalog/pg_inherits_fn.h" #include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/extensible.h" @@ -6159,6 +6160,82 @@ make_modifytable(PlannerInfo *root, node->fdwPrivLists = fdw_private_list; node->fdwDirectModifyPlans = direct_modify_plans; + /* Collect insert plans for all FDW-managed partitions */ + if (node->operation == CMD_INSERT) + { + RangeTblEntry *rte, + **saved_simple_rte_array; + List *partition_oids, + *fdw_partition_oids; + + Assert(list_length(resultRelations) == 1); + rte = rt_fetch(linitial_int(resultRelations), root->parse->rtable); + Assert(rte->rtekind == RTE_RELATION); + + if (rte->relkind != RELKIND_PARTITIONED_TABLE) + return node; + + partition_oids = find_all_inheritors(rte->relid, NoLock, NULL); + + /* Discard any previous content which is useless anyway */ + fdw_private_list = NIL; + fdw_partition_oids = NIL; + + /* + * To force the FDW driver fetch the intended RTE, we need to temporarily + * switch root->simple_rte_array to the one holding only that RTE at a + * designated index, for every foreign table. + */ + saved_simple_rte_array = root->simple_rte_array; + root->simple_rte_array = (RangeTblEntry **) + palloc0(2 * sizeof(RangeTblEntry *)); + foreach(lc, partition_oids) + { + Oid myoid = lfirst_oid(lc); + FdwRoutine *fdwroutine; + List *fdw_private; + + /* + * We are only interested in foreign tables. Note that this will + * also eliminate any partitioned tables since foreign tables can + * only ever be leaf partitions. + */ + if (!oid_is_foreign_table(myoid)) + continue; + + fdw_partition_oids = lappend_oid(fdw_partition_oids, myoid); + + fdwroutine = GetFdwRoutineByRelId(myoid); + if (fdwroutine && fdwroutine->PlanForeignModify) + { + RangeTblEntry *fdw_rte; + + fdw_rte = copyObject(rte); + fdw_rte->relid = myoid; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + + /* + * Assumes PlanForeignModify() uses planner_rt_fetch(), also, + * see the above comment. + */ + root->simple_rte_array[1] = fdw_rte; + + fdw_private = fdwroutine->PlanForeignModify(root, node, 1, 0); + pfree(fdw_rte); + } + else + fdw_private = NIL; + + fdw_private_list = lappend(fdw_private_list, fdw_private); + } + + pfree(root->simple_rte_array); + root->simple_rte_array = saved_simple_rte_array; + + node->fdwPrivLists = fdw_private_list; + node->fdwPartitionOids = fdw_partition_oids; + } + return node; } diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index a2cbf14..e85ca0a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1715,3 +1715,16 @@ has_row_triggers(PlannerInfo *root, Index rti, CmdType event) heap_close(relation, NoLock); return result; } + +bool +oid_is_foreign_table(Oid relid) +{ + Relation rel; + char relkind; + + rel = heap_open(relid, NoLock); + relkind = rel->rd_rel->relkind; + heap_close(rel, NoLock); + + return relkind == RELKIND_FOREIGN_TABLE; +} diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index ab6d3a8..09727cf 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -56,4 +58,9 @@ extern List *RelationGetPartitionQual(Relation rel, bool recurse); /* For tuple routing */ extern PartitionTreeNode RelationGetPartitionTreeNode(Relation rel); extern List *get_leaf_partition_oids_v2(PartitionTreeNode ptnode); + +extern int get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..c62946f 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..ce01008 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,15 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionTreeNodeData *mt_partition_tree_root; + /* Partition descriptor node tree */ + ResultRelInfo *mt_partitions; /* Per leaf partition target + * relations */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per leaf partition + * tuple conversion map */ + int mt_num_partitions; /* Number of leaf partition target + * relations in the above array */ } ModifyTableState; /* ---------------- diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index e2fbc7d..d82222c 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -189,6 +189,7 @@ typedef struct ModifyTable List *returningLists; /* per-target-table RETURNING tlists */ List *fdwPrivLists; /* per-target-table FDW private data lists */ Bitmapset *fdwDirectModifyPlans; /* indices of FDW DM plans */ + List *fdwPartitionOids; /* OIDs of FDW-managed partition */ List *rowMarks; /* PlanRowMarks (non-locking only) */ int epqParam; /* ID of Param for EvalPlanQual re-eval */ OnConflictAction onConflictAction; /* ON CONFLICT action */ diff --git a/src/include/optimizer/plancat.h b/src/include/optimizer/plancat.h index 125274e..fac606c 100644 --- a/src/include/optimizer/plancat.h +++ b/src/include/optimizer/plancat.h @@ -56,5 +56,6 @@ extern Selectivity join_selectivity(PlannerInfo *root, SpecialJoinInfo *sjinfo); extern bool has_row_triggers(PlannerInfo *root, Index rti, CmdType event); +extern bool oid_is_foreign_table(Oid relid); #endif /* PLANCAT_H */ diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index ebc0ba5..3d97742 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -222,6 +222,62 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- fail (no list partition defined which accepts nulls) +insert into list_parted (b) values (1); +ERROR: no partition of relation "list_parted" found for row +DETAIL: Failing row contains (null, 1). +create table part_nulls partition of list_parted for values in (null); +-- ok +insert into list_parted (b) values (1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_nulls | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(7 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects @@ -230,9 +286,10 @@ drop cascades to table part_a_10_a_20 drop cascades to table part_b_1_b_10 drop cascades to table part_b_10_b_20 drop table list_parted cascade; -NOTICE: drop cascades to 5 other objects +NOTICE: drop cascades to 6 other objects DETAIL: drop cascades to table part_aa_bb drop cascades to table part_cc_dd drop cascades to table part_ee_ff drop cascades to table part_ee_ff_1_10 drop cascades to table part_ee_ff_10_20 +drop cascades to table part_nulls diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index 4bf042e..d1b5a09 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -137,6 +137,34 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- fail (no list partition defined which accepts nulls) +insert into list_parted (b) values (1); +create table part_nulls partition of list_parted for values in (null); +-- ok +insert into list_parted (b) values (1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------294A99728041A5608B577083 Content-Type: text/x-diff; name="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-11.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-11"; filename*1=".patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH 7/8] Tuple routing for partitioned tables. @ 2016-07-27 07:59 amit <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: amit @ 2016-07-27 07:59 UTC (permalink / raw) Both COPY FROM and INSERT. --- src/backend/catalog/partition.c | 325 ++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 203 +++++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 47 +++++- src/backend/executor/nodeModifyTable.c | 142 ++++++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 1 + src/backend/optimizer/plan/createplan.c | 77 ++++++++ src/backend/optimizer/util/plancat.c | 13 ++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 7 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 10 + src/include/nodes/plannodes.h | 1 + src/include/optimizer/plancat.h | 1 + src/test/regress/expected/insert.out | 52 +++++ src/test/regress/sql/insert.sql | 25 +++ 18 files changed, 914 insertions(+), 7 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index 6301d91..e3a0848 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -181,6 +181,18 @@ static List *generate_partition_qual(Relation rel, bool recurse); static PartitionTreeNode GetPartitionTreeNodeRecurse(Relation rel, int offset); static int get_leaf_partition_count(PartitionTreeNode ptnode); +/* Support get_partition_for_tuple() */ +static PartitionKeyExecInfo *BuildPartitionKeyExecInfo(Relation rel); +static void FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); +static int list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull); +static int range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum *tuple); + /* List partition related support functions */ static bool equal_list_info(PartitionKey key, PartitionListInfo *l1, PartitionListInfo *l2); @@ -194,6 +206,8 @@ static PartitionRangeBound *copy_range_bound(PartitionKey key, PartitionRangeBou static bool equal_range_info(PartitionKey key, PartitionRangeInfo *r1, PartitionRangeInfo *r2); static int32 partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg); +static int32 partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg); static bool partition_rbound_eq(PartitionKey key, PartitionRangeBound *b1, PartitionRangeBound *b2); typedef int32 (*partition_rbound_bsearch_cmp_fn) (PartitionKey, @@ -1437,7 +1451,7 @@ GetPartitionTreeNodeRecurse(Relation rel, int offset) /* First build our own node */ parent = (PartitionTreeNode) palloc0(sizeof(PartitionTreeNodeData)); - parent->pkinfo = NULL; + parent->pkinfo = BuildPartitionKeyExecInfo(rel); parent->pdesc = RelationGetPartitionDesc(rel); parent->relid = RelationGetRelid(rel); parent->offset = offset; @@ -1521,6 +1535,281 @@ get_leaf_partition_count(PartitionTreeNode ptnode) return result; } +/* + * BuildPartitionKeyExecInfo + * Construct a list of PartitionKeyExecInfo records for an open + * relation + * + * PartitionKeyExecInfo stores the information about the partition key + * that's needed when inserting tuples into a partitioned table; especially, + * partition key expression state if there are any expression columns in + * the partition key. Normally we build a PartitionKeyExecInfo for a + * partitioned table just once per command, and then use it for (potentially) + * many tuples. + * + */ +static PartitionKeyExecInfo * +BuildPartitionKeyExecInfo(Relation rel) +{ + PartitionKeyExecInfo *pkinfo; + + pkinfo = (PartitionKeyExecInfo *) palloc0(sizeof(PartitionKeyExecInfo)); + pkinfo->pi_Key = RelationGetPartitionKey(rel); + pkinfo->pi_ExpressionState = NIL; + + return pkinfo; +} + +/* + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for partition key columns + */ +static void +FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pkinfo->pi_Key->partexprs != NIL && pkinfo->pi_ExpressionState == NIL) + { + /* First time through, set up expression evaluation state */ + pkinfo->pi_ExpressionState = (List *) + ExecPrepareExpr((Expr *) pkinfo->pi_Key->partexprs, + estate); + /* Check caller has set up context correctly */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + } + + partexpr_item = list_head(pkinfo->pi_ExpressionState); + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + { + AttrNumber keycol = pkinfo->pi_Key->partattrs[i]; + Datum pkDatum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + pkDatum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + pkDatum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = pkDatum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Recursively finds the "leaf" partition for tuple + * + * Returns -1 if no partition is found and sets *failed_at to the OID of + * the partitioned table whose partition was not found. + */ +int +get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + Relation partRel; + PartitionKeyExecInfo *pkinfo = ptnode->pkinfo; + PartitionTreeNode node; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int i; + int index; + + /* Guard against stack overflow due to overly deep partition tree */ + check_stack_depth(); + + if (ptnode->pdesc->nparts == 0) + { + *failed_at = ptnode->relid; + return -1; + } + + /* Extract partition key from tuple */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + FormPartitionKeyDatum(pkinfo, slot, estate, values, isnull); + + /* Disallow nulls, if range partition key */ + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + if (isnull[i] && pkinfo->pi_Key->strategy == PARTITION_STRATEGY_RANGE) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key contains null"))); + + switch (pkinfo->pi_Key->strategy) + { + case PARTITION_STRATEGY_LIST: + index = list_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values[0], isnull[0]); + break; + + case PARTITION_STRATEGY_RANGE: + index = range_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values); + break; + } + + /* No partition found at this level */ + if (index < 0) + { + *failed_at = ptnode->relid; + return index; + } + + partRel = heap_open(ptnode->pdesc->oids[index], NoLock); + + /* Don't recurse if the index'th partition is a leaf partition. */ + if (partRel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) + { + PartitionTreeNode prev; + + /* + * Index returned above considers only this parent (of which partRel + * is a partition). We however want to return the index across the + * the whole partition tree. If partRel is the leftmost partition + * of parent (index = 0) or if none of the left siblings are + * partitioned tables, we can simply return parent's offset plus + * partRel's index as the result. Otherwise, we find the node + * corresponding to the rightmost partitioned table among partRel's + * left siblings and use the information therein. + */ + prev = node = ptnode->downlink; + if (node && node->index < index) + { + /* + * Find the partition tree node such that its index value is the + * greatest value less than the above returned index. + */ + while (node) + { + if (node->index > index) + { + node = prev; + break; + } + + prev = node; + node = node->next; + } + + if (!node) + node = prev; + Assert (node != NULL); + + index = node->offset + node->num_leaf_parts + + (index - node->index - 1); + } + else + /* + * The easy case where all of partRel's left siblings (if any) are + * leaf partitions. + */ + index = ptnode->offset + index; + + heap_close(partRel, NoLock); + return index; + } + + heap_close(partRel, NoLock); + + /* + * Need to recurse as the selected partition is a partitioned table + * itself. Locate the corresponding PartitionTreeNode to pass it down. + */ + node = ptnode->downlink; + while (node->next != NULL && node->index != index) + node = node->next; + Assert (node != NULL); + + return get_partition_for_tuple(node, slot, estate, failed_at); +} + +/* + * list_partition_for_tuple + * Find the list partition for a tuple + * + * Returns -1 if none found. + */ +static int +list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull) +{ + PartitionListInfo listinfo; + int found; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_LIST); + listinfo = pdesc->boundinfo->bounds.lists; + + if (isnull && listinfo.has_null) + return listinfo.null_index; + else if (!isnull) + { + found = partition_list_values_bsearch(key, + listinfo.values, + listinfo.nvalues, + value); + if (found >= 0) + return listinfo.indexes[found]; + } + + /* Control reaches here if isnull and !listinfo->has_null */ + return -1; +} + +/* + * range_partition_for_tuple + * Search the range partition for a range key ('values') + * + * Returns -1 if none found. + */ +static int +range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, Datum *tuple) +{ + int offset; + PartitionRangeInfo rangeinfo; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_RANGE); + rangeinfo = pdesc->boundinfo->bounds.ranges; + + offset = partition_rbound_bsearch(key, + rangeinfo.bounds, rangeinfo.nbounds, + tuple, partition_rbound_datum_cmp, + true, NULL); + + /* + * Offset returned is such that the bound at offset is found to be + * less or equal with the tuple. That is, the tuple belongs to the + * partition with the rangeinfo.bounds[offset] as the lower bound and + * rangeinfo.bounds[offset+1] as the upper bound, provided the latter + * is indeed marked !lower (that is, it's an upper bound). If it turns + * out that it is a lower bound then the corresponding index will be -1, + * which means no valid partition exists. + */ + return rangeinfo.indexes[offset+1]; +} + /* List partition related support functions */ /* @@ -1767,6 +2056,40 @@ partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg) } /* + * Return whether bound <=, =, >= partition key of tuple + * + * The 3rd argument is void * so that it can be used with + * partition_rbound_bsearch() + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg) +{ + Datum *datums1 = bound->datums, + *datums2 = (Datum *) arg; + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (bound->infinite[i]) + return bound->lower ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + datums1[i], datums2[i])); + if (cmpval != 0) + break; + } + + /* If datums are equal and this is an upper bound, tuple > bound */ + if (cmpval == 0 && !bound->lower) + return -1; + + return cmpval; +} + +/* * Return whether two range bounds are equal simply by comparing datums */ static bool diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..6fb376f 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -30,6 +30,7 @@ #include "commands/defrem.h" #include "commands/trigger.h" #include "executor/executor.h" +#include "foreign/fdwapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -161,6 +162,11 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionTreeNode ptnode; /* partition descriptor node tree */ + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; + List *partition_fdw_priv_lists; + int num_partitions; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1403,94 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + List *leaf_parts; + ListCell *cell; + int i; + int num_leaf_parts; + ResultRelInfo *leaf_part_rri; + PlannerInfo *root = makeNode(PlannerInfo); /* mostly dummy */ + Query *parse = makeNode(Query); /* ditto */ + ModifyTable *plan = makeNode(ModifyTable); /* ditto */ + RangeTblEntry *fdw_rte = makeNode(RangeTblEntry); /* ditto */ + List *fdw_private_lists = NIL; + + cstate->ptnode = RelationGetPartitionTreeNode(rel); + leaf_parts = get_leaf_partition_oids(cstate->ptnode); + num_leaf_parts = list_length(leaf_parts); + + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + /* For use below, iff a partition found to be a foreign table */ + plan->operation = CMD_INSERT; + plan->plans = list_make1(makeNode(Result)); + fdw_rte->rtekind = RTE_RELATION; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + parse->rtable = list_make1(fdw_rte); + root->parse = parse; + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation part_rel; + + part_rel = heap_open(lfirst_oid(cell), RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + /* Special dance for foreign tables */ + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + fdw_rte->relid = RelationGetRelid(part_rel); + fdw_private = leaf_part_rri->ri_FdwRoutine->PlanForeignModify(root, + plan, + 1, + 0); + fdw_private_lists = lappend(fdw_private_lists, fdw_private); + } + + if (!equalTupleDescs(tupDesc, RelationGetDescr(part_rel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + + cstate->partition_fdw_priv_lists = fdw_private_lists; + pfree(fdw_rte); + pfree(plan); + pfree(parse); + pfree(root); + } } else { @@ -1692,6 +1786,8 @@ ClosePipeToProgram(CopyState cstate) static void EndCopy(CopyState cstate) { + int i; + if (cstate->is_program) { ClosePipeToProgram(cstate); @@ -1705,6 +1801,23 @@ EndCopy(CopyState cstate) cstate->filename))); } + /* Close all partitions and indices thereof */ + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + /* XXX - EState not handy here to pass to EndForeignModify() */ + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(NULL, resultRelInfo); + + if (cstate->partition_tupconv_maps[i]) + pfree(cstate->partition_tupconv_maps[i]); + } + MemoryContextDelete(cstate->copycontext); pfree(cstate); } @@ -2255,6 +2368,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2395,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2506,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2534,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->ptnode != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2439,10 +2556,46 @@ CopyFrom(CopyState cstate) */ ExecBSInsertTriggers(estate, resultRelInfo); + /* Initialize FDW partition insert plans */ + if (cstate->ptnode) + { + int i, + j; + List *fdw_private_lists = cstate->partition_fdw_priv_lists; + ModifyTableState *mtstate = makeNode(ModifyTableState); + ResultRelInfo *leaf_part_rri; + + /* Mostly dummy containing enough state for BeginForeignModify */ + mtstate->ps.state = estate; + mtstate->operation = CMD_INSERT; + + j = 0; + leaf_part_rri = cstate->partitions; + for (i = 0; i < cstate->num_partitions; i++) + { + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + Assert(fdw_private_lists); + fdw_private = list_nth(fdw_private_lists, j++); + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, 0); + } + leaf_part_rri++; + } + } + values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2647,31 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition */ + saved_resultRelInfo = resultRelInfo; + if (cstate->ptnode) + { + int i_leaf_partition; + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + cstate->ptnode, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < cstate->num_partitions); + + resultRelInfo = cstate->partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = cstate->partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2523,7 +2701,16 @@ CopyFrom(CopyState cstate) resultRelInfo->ri_PartitionCheck) ExecConstraints(resultRelInfo, slot, estate); - if (useHeapMultiInsert) + if (resultRelInfo->ri_FdwRoutine) + { + resultRelInfo->ri_FdwRoutine->ExecForeignInsert(estate, + resultRelInfo, + slot, + NULL); + /* AFTER ROW INSERT Triggers */ + ExecARInsertTriggers(estate, resultRelInfo, tuple, NIL); + } + else if (useHeapMultiInsert) { /* Add this tuple to the tuple buffer */ if (nBufferedTuples == 0) @@ -2553,7 +2740,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2765,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2784,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8cd3614..b51d59a 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1293,6 +1293,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index ea3f59a..62bf8b7 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2996,3 +3001,41 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionTreeNode ptnode, + TupleTableSlot *slot, EState *estate) +{ + int i_leaf_partition; + Oid failed_at; + + i_leaf_partition = get_partition_for_tuple(ptnode, slot, estate, + &failed_at); + + if (i_leaf_partition < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return i_leaf_partition; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..5c2bf6c 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,31 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + saved_resultRelInfo = resultRelInfo; + + if (mtstate->mt_partition_tree_root) + { + int i_leaf_partition; + ExprContext *econtext = GetPerTupleExprContext(estate); + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_tree_root, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < mtstate->mt_num_partitions); + + resultRelInfo = mtstate->mt_partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = mtstate->mt_partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +537,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1597,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1688,98 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + mtstate->mt_partition_tree_root = RelationGetPartitionTreeNode(rel); + leaf_parts = get_leaf_partition_oids(mtstate->mt_partition_tree_root); + num_leaf_parts = list_length(leaf_parts); + + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (leaf_part_rri->ri_FdwRoutine) + { + ListCell *lc; + List *fdw_private; + int k; + + /* + * There are as many fdw_private's in fdwPrivLists as there + * are FDW partitions, but we must find the intended for the + * this foreign table. + */ + k = 0; + foreach(lc, node->fdwPartitionOids) + { + if (lfirst_oid(lc) == ftoid) + break; + k++; + } + + Assert(k < num_leaf_parts); + fdw_private = (List *) list_nth(node->fdwPrivLists, k); + Assert(fdw_private != NIL); + + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, + eflags); + j++; + } + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2097,23 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(node->ps.state, + resultRelInfo); + + if (node->mt_partition_tupconv_maps[i]) + pfree(node->mt_partition_tupconv_maps[i]); + } + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 28d0036..470dee7 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -188,6 +188,7 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(withCheckOptionLists); COPY_NODE_FIELD(returningLists); COPY_NODE_FIELD(fdwPrivLists); + COPY_NODE_FIELD(fdwPartitionOids); COPY_BITMAPSET_FIELD(fdwDirectModifyPlans); COPY_NODE_FIELD(rowMarks); COPY_SCALAR_FIELD(epqParam); diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 0d858f5..5c8eced 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -340,6 +340,7 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(withCheckOptionLists); WRITE_NODE_FIELD(returningLists); WRITE_NODE_FIELD(fdwPrivLists); + WRITE_NODE_FIELD(fdwPartitionOids); WRITE_BITMAPSET_FIELD(fdwDirectModifyPlans); WRITE_NODE_FIELD(rowMarks); WRITE_INT_FIELD(epqParam); diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c587d4e..ddd78c7 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -1495,6 +1495,7 @@ _readModifyTable(void) READ_NODE_FIELD(withCheckOptionLists); READ_NODE_FIELD(returningLists); READ_NODE_FIELD(fdwPrivLists); + READ_NODE_FIELD(fdwPartitionOids); READ_BITMAPSET_FIELD(fdwDirectModifyPlans); READ_NODE_FIELD(rowMarks); READ_INT_FIELD(epqParam); diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index ad49674..29f40fe 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -22,6 +22,7 @@ #include "access/stratnum.h" #include "access/sysattr.h" #include "catalog/pg_class.h" +#include "catalog/pg_inherits_fn.h" #include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/extensible.h" @@ -6159,6 +6160,82 @@ make_modifytable(PlannerInfo *root, node->fdwPrivLists = fdw_private_list; node->fdwDirectModifyPlans = direct_modify_plans; + /* Collect insert plans for all FDW-managed partitions */ + if (node->operation == CMD_INSERT) + { + RangeTblEntry *rte, + **saved_simple_rte_array; + List *partition_oids, + *fdw_partition_oids; + + Assert(list_length(resultRelations) == 1); + rte = rt_fetch(linitial_int(resultRelations), root->parse->rtable); + Assert(rte->rtekind == RTE_RELATION); + + if (rte->relkind != RELKIND_PARTITIONED_TABLE) + return node; + + partition_oids = find_all_inheritors(rte->relid, NoLock, NULL); + + /* Discard any previous content which is useless anyway */ + fdw_private_list = NIL; + fdw_partition_oids = NIL; + + /* + * To force the FDW driver fetch the intended RTE, we need to temporarily + * switch root->simple_rte_array to the one holding only that RTE at a + * designated index, for every foreign table. + */ + saved_simple_rte_array = root->simple_rte_array; + root->simple_rte_array = (RangeTblEntry **) + palloc0(2 * sizeof(RangeTblEntry *)); + foreach(lc, partition_oids) + { + Oid myoid = lfirst_oid(lc); + FdwRoutine *fdwroutine; + List *fdw_private; + + /* + * We are only interested in foreign tables. Note that this will + * also eliminate any partitioned tables since foreign tables can + * only ever be leaf partitions. + */ + if (!oid_is_foreign_table(myoid)) + continue; + + fdw_partition_oids = lappend_oid(fdw_partition_oids, myoid); + + fdwroutine = GetFdwRoutineByRelId(myoid); + if (fdwroutine && fdwroutine->PlanForeignModify) + { + RangeTblEntry *fdw_rte; + + fdw_rte = copyObject(rte); + fdw_rte->relid = myoid; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + + /* + * Assumes PlanForeignModify() uses planner_rt_fetch(), also, + * see the above comment. + */ + root->simple_rte_array[1] = fdw_rte; + + fdw_private = fdwroutine->PlanForeignModify(root, node, 1, 0); + pfree(fdw_rte); + } + else + fdw_private = NIL; + + fdw_private_list = lappend(fdw_private_list, fdw_private); + } + + pfree(root->simple_rte_array); + root->simple_rte_array = saved_simple_rte_array; + + node->fdwPrivLists = fdw_private_list; + node->fdwPartitionOids = fdw_partition_oids; + } + return node; } diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index a2cbf14..e85ca0a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1715,3 +1715,16 @@ has_row_triggers(PlannerInfo *root, Index rti, CmdType event) heap_close(relation, NoLock); return result; } + +bool +oid_is_foreign_table(Oid relid) +{ + Relation rel; + char relkind; + + rel = heap_open(relid, NoLock); + relkind = rel->rd_rel->relkind; + heap_close(rel, NoLock); + + return relkind == RELKIND_FOREIGN_TABLE; +} diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 057b1e3..86cc598 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -50,4 +52,9 @@ extern List *RelationGetPartitionQual(Relation rel, bool recurse); /* For tuple routing */ extern PartitionTreeNode RelationGetPartitionTreeNode(Relation rel); extern List *get_leaf_partition_oids(PartitionTreeNode ptnode); + +extern int get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..c62946f 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..ce01008 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,15 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionTreeNodeData *mt_partition_tree_root; + /* Partition descriptor node tree */ + ResultRelInfo *mt_partitions; /* Per leaf partition target + * relations */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per leaf partition + * tuple conversion map */ + int mt_num_partitions; /* Number of leaf partition target + * relations in the above array */ } ModifyTableState; /* ---------------- diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index e2fbc7d..d82222c 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -189,6 +189,7 @@ typedef struct ModifyTable List *returningLists; /* per-target-table RETURNING tlists */ List *fdwPrivLists; /* per-target-table FDW private data lists */ Bitmapset *fdwDirectModifyPlans; /* indices of FDW DM plans */ + List *fdwPartitionOids; /* OIDs of FDW-managed partition */ List *rowMarks; /* PlanRowMarks (non-locking only) */ int epqParam; /* ID of Param for EvalPlanQual re-eval */ OnConflictAction onConflictAction; /* ON CONFLICT action */ diff --git a/src/include/optimizer/plancat.h b/src/include/optimizer/plancat.h index 125274e..fac606c 100644 --- a/src/include/optimizer/plancat.h +++ b/src/include/optimizer/plancat.h @@ -56,5 +56,6 @@ extern Selectivity join_selectivity(PlannerInfo *root, SpecialJoinInfo *sjinfo); extern bool has_row_triggers(PlannerInfo *root, Index rti, CmdType event); +extern bool oid_is_foreign_table(Oid relid); #endif /* PLANCAT_H */ diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 9ae6b09..d5dcb59 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -227,6 +227,58 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index b6e821e..fbd30d9 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,31 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------20C4AAD7DBE3F2B5281ABBA6 Content-Type: text/x-diff; name="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-13.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-13"; filename*1=".patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. @ 2021-03-04 09:10 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Heikki Linnakangas @ 2021-03-04 09:10 UTC (permalink / raw) Now that we don't support the old-style COPY protocol anymore, we can force four bytes of lookahead like was suggested in an existing comment, and simplify the loop in CopyReadLineText(). Discussion: https://www.postgresql.org/message-id/9ec25819-0a8a-d51a-17dc-4150bb3cca3b%40iki.fi --- src/backend/commands/copyfromparse.c | 119 ++++++++--------------- src/include/commands/copyfrom_internal.h | 3 +- 2 files changed, 40 insertions(+), 82 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index ce24a1528bd..c2efe7e0782 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -46,21 +46,6 @@ * empty statements. See http://www.cit.gu.edu.au/~anthony/info/C/C.macros. */ -/* - * This keeps the character read at the top of the loop in the buffer - * even if there is more than one read-ahead. - */ -#define IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(extralen) \ -if (1) \ -{ \ - if (raw_buf_ptr + (extralen) >= copy_buf_len && !hit_eof) \ - { \ - raw_buf_ptr = prev_raw_ptr; /* undo fetch */ \ - need_data = true; \ - continue; \ - } \ -} else ((void) 0) - /* This consumes the remainder of the buffer and breaks */ #define IF_NEED_REFILL_AND_EOF_BREAK(extralen) \ if (1) \ @@ -118,7 +103,7 @@ static int CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread); static inline bool CopyGetInt32(CopyFromState cstate, int32 *val); static inline bool CopyGetInt16(CopyFromState cstate, int16 *val); -static bool CopyLoadRawBuf(CopyFromState cstate); +static bool CopyLoadRawBuf(CopyFromState cstate, int minread); static int CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes); void @@ -209,7 +194,7 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) ereport(ERROR, (errcode_for_file_access(), errmsg("could not read from COPY file: %m"))); - if (bytesread == 0) + if (bytesread < maxread) cstate->reached_eof = true; break; case COPY_FRONTEND: @@ -278,6 +263,8 @@ CopyGetData(CopyFromState cstate, void *databuf, int minread, int maxread) break; case COPY_CALLBACK: bytesread = cstate->data_source_cb(databuf, minread, maxread); + if (bytesread < minread) + cstate->reached_eof = true; break; } @@ -329,14 +316,13 @@ CopyGetInt16(CopyFromState cstate, int16 *val) /* * CopyLoadRawBuf loads some more data into raw_buf * - * Returns true if able to obtain at least one more byte, else false. + * Returns true if able to obtain at least 'minread' bytes, else false. * * If RAW_BUF_BYTES(cstate) > 0, the unprocessed bytes are moved to the start - * of the buffer and then we load more data after that. This case occurs only - * when a multibyte character crosses a bufferload boundary. + * of the buffer and then we load more data after that. */ static bool -CopyLoadRawBuf(CopyFromState cstate) +CopyLoadRawBuf(CopyFromState cstate, int minread) { int nbytes = RAW_BUF_BYTES(cstate); int inbytes; @@ -347,14 +333,15 @@ CopyLoadRawBuf(CopyFromState cstate) nbytes); inbytes = CopyGetData(cstate, cstate->raw_buf + nbytes, - 1, RAW_BUF_SIZE - nbytes); + minread, RAW_BUF_SIZE - nbytes); nbytes += inbytes; cstate->raw_buf[nbytes] = '\0'; cstate->raw_buf_index = 0; cstate->raw_buf_len = nbytes; cstate->bytes_processed += inbytes; pgstat_progress_update_param(PROGRESS_COPY_BYTES_PROCESSED, cstate->bytes_processed); - return (inbytes > 0); + + return (inbytes >= minread); } /* @@ -389,7 +376,7 @@ CopyReadBinaryData(CopyFromState cstate, char *dest, int nbytes) /* Load more data if buffer is empty. */ if (RAW_BUF_BYTES(cstate) == 0) { - if (!CopyLoadRawBuf(cstate)) + if (!CopyLoadRawBuf(cstate, 1)) break; /* EOF */ } @@ -678,7 +665,7 @@ CopyReadLine(CopyFromState cstate) do { cstate->raw_buf_index = cstate->raw_buf_len; - } while (CopyLoadRawBuf(cstate)); + } while (CopyLoadRawBuf(cstate, 1)); } } else @@ -747,7 +734,6 @@ CopyReadLineText(CopyFromState cstate) char *copy_raw_buf; int raw_buf_ptr; int copy_buf_len; - bool need_data = false; bool hit_eof = false; bool result = false; char mblen_str[2]; @@ -794,6 +780,7 @@ CopyReadLineText(CopyFromState cstate) copy_raw_buf = cstate->raw_buf; raw_buf_ptr = cstate->raw_buf_index; copy_buf_len = cstate->raw_buf_len; + hit_eof = cstate->reached_eof; for (;;) { @@ -801,38 +788,40 @@ CopyReadLineText(CopyFromState cstate) char c; /* - * Load more data if needed. Ideally we would just force four bytes - * of read-ahead and avoid the many calls to - * IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(), but the COPY_OLD_FE protocol - * does not allow us to read too far ahead or we might read into the - * next data, so we read-ahead only as far we know we can. One - * optimization would be to read-ahead four byte here if - * cstate->copy_src != COPY_OLD_FE, but it hardly seems worth it, - * considering the size of the buffer. + * Load more data if needed. + * + * We need to look ahead max three bytes in one iteration of the loop + * (for the sequence \.<CR><NL>), so make sure we have at least four + * bytes in the buffer. Note that we always guarantee that there is + * one \0 in the buffer, after last valid byte; the lookaheads below + * rely on that. */ - if (raw_buf_ptr >= copy_buf_len || need_data) +#define COPY_READ_LINE_LOOKAHEAD 4 + if (raw_buf_ptr + COPY_READ_LINE_LOOKAHEAD >= copy_buf_len) { - REFILL_LINEBUF; + if (!hit_eof) + { + REFILL_LINEBUF; - /* - * Try to read some more data. This will certainly reset - * raw_buf_index to zero, and raw_buf_ptr must go with it. - */ - if (!CopyLoadRawBuf(cstate)) - hit_eof = true; - raw_buf_ptr = 0; - copy_buf_len = cstate->raw_buf_len; + /* + * Try to read some more data. This will certainly reset + * raw_buf_index to zero, and raw_buf_ptr must go with it. + */ + if (!CopyLoadRawBuf(cstate, COPY_READ_LINE_LOOKAHEAD)) + hit_eof = true; + raw_buf_ptr = 0; + copy_buf_len = cstate->raw_buf_len; + } /* * If we are completely out of data, break out of the loop, * reporting EOF. */ - if (copy_buf_len <= 0) + if (copy_buf_len - raw_buf_ptr <= 0) { result = true; break; } - need_data = false; } /* OK to fetch a character */ @@ -841,20 +830,6 @@ CopyReadLineText(CopyFromState cstate) if (cstate->opts.csv_mode) { - /* - * If character is '\\' or '\r', we may need to look ahead below. - * Force fetch of the next character if we don't already have it. - * We need to do this before changing CSV state, in case one of - * these characters is also the quote or escape character. - * - * Note: old-protocol does not like forced prefetch, but it's OK - * here since we cannot validly be at EOF. - */ - if (c == '\\' || c == '\r') - { - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - } - /* * Dealing with quotes and escapes here is mildly tricky. If the * quote char is also the escape char, there's no problem - we @@ -888,14 +863,9 @@ CopyReadLineText(CopyFromState cstate) cstate->eol_type == EOL_CRNL) { /* - * If need more data, go back to loop top to load it. - * - * Note that if we are at EOF, c will wind up as '\0' because - * of the guaranteed pad of raw_buf. + * Look at the next character. If we're at EOF, c2 will wind + * up as '\0' because of the guaranteed pad of raw_buf. */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - - /* get next char */ c = copy_raw_buf[raw_buf_ptr]; if (c == '\n') @@ -961,7 +931,6 @@ CopyReadLineText(CopyFromState cstate) { char c2; - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); IF_NEED_REFILL_AND_EOF_BREAK(0); /* ----- @@ -976,16 +945,9 @@ CopyReadLineText(CopyFromState cstate) { raw_buf_ptr++; /* consume the '.' */ - /* - * Note: if we loop back for more data here, it does not - * matter that the CSV state change checks are re-executed; we - * will come back here with no important state changed. - */ if (cstate->eol_type == EOL_CRNL) { - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 == '\n') @@ -1008,9 +970,7 @@ CopyReadLineText(CopyFromState cstate) } } - /* Get the next character */ - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(0); - /* if hit_eof, c2 will become '\0' */ + /* Get next character. If hit_eof, c2 will become '\0' */ c2 = copy_raw_buf[raw_buf_ptr++]; if (c2 != '\r' && c2 != '\n') @@ -1095,7 +1055,6 @@ not_end_of_copy: mblen_str[0] = c; mblen = pg_encoding_mblen(cstate->file_encoding, mblen_str); - IF_NEED_REFILL_AND_NOT_EOF_CONTINUE(mblen - 1); IF_NEED_REFILL_AND_EOF_BREAK(mblen - 1); raw_buf_ptr += mblen - 1; } diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 705f5b615be..c088c4facdb 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -70,8 +70,7 @@ typedef struct CopyFromStateData CopySource copy_src; /* type of copy source */ FILE *copy_file; /* used if copy_src == COPY_FILE */ StringInfo fe_msgbuf; /* used if copy_src == COPY_NEW_FE */ - bool reached_eof; /* true if we read to end of copy data (not - * all copy_src types maintain this) */ + bool reached_eof; /* true if we read to end of copy data */ EolType eol_type; /* EOL type of input */ int file_encoding; /* file or remote side's character encoding */ -- 2.30.1 --------------5F54C02640018A10FACC0C96-- ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Proposal: Support custom authentication methods using hooks @ 2022-02-25 01:47 Tom Lane <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Tom Lane @ 2022-02-25 01:47 UTC (permalink / raw) To: Jeff Davis <[email protected]>; +Cc: samay sharma <[email protected]>; [email protected] Jeff Davis <[email protected]> writes: > On Thu, 2022-02-17 at 11:25 -0800, samay sharma wrote: >> To enable this, I've proposed adding a new authentication method >> "custom" which can be specified in pg_hba.conf and takes a mandatory >> argument "provider" specifying which authentication provider to use. > One caveat is that this only works given information available from > existing authentication methods, because that's all the client > supports. In practice, it seems to only be useful with plaintext > password authentication over an SSL connection. ... and, since we can't readily enforce that the client only sends those cleartext passwords over suitably-encrypted connections, this could easily be a net negative for security. Not sure that I think it's a good idea. regards, tom lane ^ permalink raw reply [nested|flat] 12+ messages in thread
end of thread, other threads:[~2022-02-25 01:47 UTC | newest] Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2016-07-27 06:47 [PATCH 6/7] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 06:47 [PATCH 6/7] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 06:47 [PATCH 6/7] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 06:47 [PATCH 6/7] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 07:59 [PATCH 7/8] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 07:59 [PATCH 7/8] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 07:59 [PATCH 7/8] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 07:59 [PATCH 7/8] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 07:59 [PATCH 7/8] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 07:59 [PATCH 7/8] Tuple routing for partitioned tables. amit <[email protected]> 2021-03-04 09:10 [PATCH v2 1/1] Simplify COPY FROM parsing by forcing lookahead. Heikki Linnakangas <[email protected]> 2022-02-25 01:47 Re: Proposal: Support custom authentication methods using hooks Tom Lane <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox