agora inbox for [email protected]help / color / mirror / Atom feed
[PATCH 6/7] Tuple routing for partitioned tables. 14+ messages / 4 participants [nested] [flat]
* [PATCH 6/7] Tuple routing for partitioned tables. @ 2016-07-27 06:47 amit <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: amit @ 2016-07-27 06:47 UTC (permalink / raw) Both COPY FROM and INSERT are covered by this commit. Routing to foreing partitions is not supported at the moment. To implement tuple-routing, introduce a PartitionDispatch data structure. Each partitioned table in a partition tree gets one and contains info such as a pointer to its partition descriptor, partition key execution state, global sequence numbers of its leaf partitions and a way to link to the PartitionDispatch objects of any of its partitions that are partitioned themselves. Starting with the PartitionDispatch object of the root partitioned table and a tuple to route, one can get the global sequence number of the leaf partition that the tuple gets routed to, if one exists. --- src/backend/catalog/partition.c | 342 +++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 151 ++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 58 ++++++- src/backend/executor/nodeModifyTable.c | 130 ++++++++++++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 11 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 8 + src/test/regress/expected/insert.out | 52 +++++ src/test/regress/sql/insert.sql | 25 +++ 11 files changed, 786 insertions(+), 6 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index 197fcef..710829a 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -113,6 +113,28 @@ typedef struct PartitionRangeBound bool lower; /* this is the lower (vs upper) bound */ } PartitionRangeBound; +/*----------------------- + * PartitionDispatch - information about one partitioned table in a partition + * hiearchy required to route a tuple to one of its partitions + * + * relid OID of the table + * key Partition key information of the table + * keystate Execution state required for expressions in the partition key + * partdesc Partition descriptor of the table + * indexes Array with partdesc->nparts members (for details on what + * individual members represent, see how they are set in + * RelationGetPartitionDispatchInfo()) + *----------------------- + */ +typedef struct PartitionDispatchData +{ + Oid relid; + PartitionKey key; + List *keystate; /* list of ExprState */ + PartitionDesc partdesc; + int *indexes; +} PartitionDispatchData; + static int32 qsort_partition_list_value_cmp(const void *a, const void *b, void *arg); static int32 qsort_partition_rbound_cmp(const void *a, const void *b, void *arg); @@ -126,12 +148,22 @@ static PartitionRangeBound *make_one_range_bound(PartitionKey key, int index, Li static int32 partition_rbound_cmp(PartitionKey key, Datum *datums1, RangeDatumContent *content1, bool lower1, PartitionRangeBound *b2); +static int32 partition_rbound_datum_cmp(PartitionKey key, + Datum *rb_datums, RangeDatumContent *rb_content, + Datum *tuple_datums); static int32 partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, int offset, void *probe, bool probe_is_bound); static int partition_bound_bsearch(PartitionKey key, PartitionBoundInfo boundinfo, void *probe, bool probe_is_bound, bool *is_equal); +/* Support get_partition_for_tuple() */ +static void FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); + /* * RelationBuildPartitionDesc * Form rel's partition descriptor @@ -895,6 +927,115 @@ RelationGetPartitionQual(Relation rel, bool recurse) return generate_partition_qual(rel, recurse); } +/* Turn an array of OIDs with N elements into a list */ +#define OID_ARRAY_TO_LIST(arr, N, list) \ + do\ + {\ + int i;\ + for (i = 0; i < (N); i++)\ + (list) = lappend_oid((list), (arr)[i]);\ + } while(0) + +/* + * RelationGetPartitionDispatchInfo + * Returns information necessary to route tuples down a partition tree + * + * All the partitions will be locked with lockmode, unless it is NoLock. + * A list of the OIDs of all the leaf partition of rel is returned in + * *leaf_part_oids. + */ +PartitionDispatch * +RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + List **leaf_part_oids) +{ + PartitionDesc rootpartdesc = RelationGetPartitionDesc(rel); + PartitionDispatchData **pd; + List *all_parts = NIL, + *parted_rels = NIL; + ListCell *lc; + int i, + k, + num_parted; + + /* + * Lock partitions and collect OIDs of the partitioned ones to prepare + * their PartitionDispatch objects. + * + * Cannot use find_all_inheritors() here, because then the order of OIDs + * in parted_rels list would be unknown, which does not help because down + * below, we assign indexes within individual PartitionDispatch in an + * order that's predetermined (determined by the order of OIDs in + * individual partition descriptors). + */ + parted_rels = lappend_oid(parted_rels, RelationGetRelid(rel)); + num_parted = 1; + OID_ARRAY_TO_LIST(rootpartdesc->oids, rootpartdesc->nparts, all_parts); + foreach(lc, all_parts) + { + Relation partrel = heap_open(lfirst_oid(lc), lockmode); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + + /* + * If this partition is a partitined table, add its children to to the + * end of the list, so that they are processed as well. + */ + if (partdesc) + { + num_parted++; + parted_rels = lappend_oid(parted_rels, lfirst_oid(lc)); + OID_ARRAY_TO_LIST(partdesc->oids, partdesc->nparts, all_parts); + } + + heap_close(partrel, NoLock); + } + + /* Generate PartitionDispatch objects for all partitioned tables */ + pd = (PartitionDispatchData **) palloc(num_parted * + sizeof(PartitionDispatchData *)); + *leaf_part_oids = NIL; + i = k = 0; + foreach(lc, parted_rels) + { + /* We locked all partitions above */ + Relation partrel = heap_open(lfirst_oid(lc), NoLock); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + int j, + m; + + pd[i] = (PartitionDispatch) palloc(sizeof(PartitionDispatchData)); + pd[i]->relid = RelationGetRelid(partrel); + pd[i]->key = RelationGetPartitionKey(partrel); + pd[i]->keystate = NIL; + pd[i]->partdesc = partdesc; + pd[i]->indexes = (int *) palloc(partdesc->nparts * sizeof(int)); + heap_close(partrel, NoLock); + + m = 0; + for (j = 0; j < partdesc->nparts; j++) + { + Oid partrelid = partdesc->oids[j]; + + if (get_rel_relkind(partrelid) != RELKIND_PARTITIONED_TABLE) + { + *leaf_part_oids = lappend_oid(*leaf_part_oids, partrelid); + pd[i]->indexes[j] = k++; + } + else + { + /* + * We can assign indexes this way because of the way + * parted_rels has been generated. + */ + pd[i]->indexes[j] = -(i + 1 + m); + m++; + } + } + i++; + } + + return pd; +} + /* Module-local functions */ /* @@ -1329,6 +1470,172 @@ generate_partition_qual(Relation rel, bool recurse) return result; } +/* ---------------- + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for the partition key + * of a tuple. + * + * pkinfo partition key execution info + * slot Heap tuple from which to extract partition key + * estate executor state for evaluating any partition key + * expressions (must be non-NULL) + * values Array of partition key Datums (output area) + * isnull Array of is-null indicators (output area) + * + * the ecxt_scantuple slot of estate's per-tuple expr context must point to + * the heap tuple passed in. + * ---------------- + */ +static void +FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pd->key->partexprs != NIL && pd->keystate == NIL) + { + /* Check caller has set up context correctly */ + Assert(estate != NULL && + GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + + /* First time through, set up expression evaluation state */ + pd->keystate = (List *) ExecPrepareExpr((Expr *) pd->key->partexprs, + estate); + } + + partexpr_item = list_head(pd->keystate); + for (i = 0; i < pd->key->partnatts; i++) + { + AttrNumber keycol = pd->key->partattrs[i]; + Datum datum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + datum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + datum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = datum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Finds a leaf partition for tuple contained in *slot + * + * Returned value is the sequence number of the leaf partition thus found, + * or -1 if no leaf partition is found for the tuple. *failed_at is set + * to the OID of the partitioned table whose partition was not found in + * the latter case. + */ +int +get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + PartitionDispatch parent; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int cur_offset, + cur_index; + int i; + + /* start with the root partitioned table */ + parent = pd[0]; + while(true) + { + PartitionKey key = parent->key; + PartitionDesc partdesc = parent->partdesc; + + /* Quick exit */ + if (partdesc->nparts == 0) + { + *failed_at = parent->relid; + return -1; + } + + /* Extract partition key from tuple */ + FormPartitionKeyDatum(parent, slot, estate, values, isnull); + + if (key->strategy == PARTITION_STRATEGY_RANGE) + { + /* Disallow nulls in the range partition key of the tuple */ + for (i = 0; i < key->partnatts; i++) + if (isnull[i]) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key of row contains null"))); + } + + if (partdesc->boundinfo->has_null && isnull[0]) + /* Tuple maps to the null-accepting list partition */ + cur_index = partdesc->boundinfo->null_index; + else + { + /* Else bsearch in partdesc->boundinfo */ + bool equal = false; + + cur_offset = partition_bound_bsearch(key, partdesc->boundinfo, + values, false, &equal); + switch (key->strategy) + { + case PARTITION_STRATEGY_LIST: + if (cur_offset >= 0 && equal) + cur_index = partdesc->boundinfo->indexes[cur_offset]; + else + cur_index = -1; + break; + + case PARTITION_STRATEGY_RANGE: + /* + * Offset returned is such that the bound at offset is + * found to be less or equal with the tuple. So, the + * bound at offset+1 would be the upper bound. + */ + cur_index = partdesc->boundinfo->indexes[cur_offset+1]; + break; + } + } + + /* + * cur_index < 0 means we failed to find a partition of this parent. + * cur_index >= 0 means we either found the leaf partition, or the + * next parent to find a partition of. + */ + if (cur_index < 0) + { + *failed_at = parent->relid; + return -1; + } + else if (parent->indexes[cur_index] < 0) + parent = pd[-parent->indexes[cur_index]]; + else + break; + } + + return parent->indexes[cur_index]; +} + /* * qsort_partition_list_value_cmp * @@ -1461,6 +1768,36 @@ partition_rbound_cmp(PartitionKey key, } /* + * partition_rbound_datum_cmp + * + * Return whether range bound (specified in rb_datums, rb_content, and + * rb_lower) <=, =, >= partition key of tuple (tuple_datums) + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, + Datum *rb_datums, RangeDatumContent *rb_content, + Datum *tuple_datums) +{ + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (rb_content[i] != RANGE_DATUM_FINITE) + return rb_content[i] == RANGE_DATUM_NEG_INF ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + rb_datums[i], + tuple_datums[i])); + if (cmpval != 0) + break; + } + + return cmpval; +} + +/* * partition_bound_cmp * * Return whether the bound at offset in boundinfo is <=, =, >= the argument @@ -1499,7 +1836,10 @@ partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, bound_datums, content, lower, (PartitionRangeBound *) probe); } - + else + cmpval = partition_rbound_datum_cmp(key, + bound_datums, content, + (Datum *) probe); break; } } diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..7d76ead 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -161,6 +161,10 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionDispatch *partition_dispatch_info; + int num_partitions; + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1401,67 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + List *leaf_parts; + ListCell *cell; + int i, + num_leaf_parts; + ResultRelInfo *leaf_part_rri; + + /* Get the tuple-routing information and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + num_leaf_parts = list_length(leaf_parts); + cstate->partition_dispatch_info = pd; + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) palloc(num_leaf_parts * + sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation partrel; + + /* + * All partitions locked above; will be closed after CopyFrom is + * finished. + */ + partrel = heap_open(lfirst_oid(cell), NoLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(partrel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + partrel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(tupDesc, RelationGetDescr(partrel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(partrel), + gettext_noop("could not convert row type")); + leaf_part_rri++; + i++; + } + } } else { @@ -2255,6 +2320,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2347,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2458,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2486,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->partition_dispatch_info != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2442,7 +2511,11 @@ CopyFrom(CopyState cstate) values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2567,56 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition to heap_insert the tuple into */ + if (cstate->partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + cstate->partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < cstate->num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding + * to the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = cstate->partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* + * For ExecInsertIndexTuples() to work on the partition's indexes + */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the + * partition rowtype. + */ + map = cstate->partition_tupconv_maps[leaf_part_index]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2553,7 +2676,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2701,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2720,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); @@ -2614,6 +2745,20 @@ CopyFrom(CopyState cstate) ExecCloseIndices(resultRelInfo); + /* Close all partitions and indices thereof */ + if (cstate->partition_dispatch_info) + { + int i; + + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + } + FreeExecutorState(estate); /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 99c9d10..8a8eb01 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1322,6 +1322,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index c7a6347..54fb771 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2990,3 +2995,52 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +/* + * ExecFindPartition -- Find a leaf partition in the partition tree rooted + * at parent, for the heap tuple contained in *slot + * + * estate must be non-NULL; we'll need it to compute any expressions in the + * partition key(s) + * + * If no leaf partition is found, this routine errors out with the appropriate + * error message, else it returns the leaf partition sequence number returned + * by get_partition_for_tuple() unchanged. + */ +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, + TupleTableSlot *slot, EState *estate) +{ + int result; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + result = get_partition_for_tuple(pd, slot, estate, &failed_at); + if (result < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return result; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 6eccfb7..87a04aa 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,56 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + /* Determine the partition to heap_insert the tuple into */ + if (mtstate->mt_partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, false); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +562,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1622,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1713,69 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + /* Form the partition node tree and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + mtstate->mt_partition_dispatch_info = pd; + num_leaf_parts = list_length(leaf_parts); + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2093,15 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + /* * Free the exprcontext */ diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 36f8c54..88380ba 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -806,8 +806,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 70d8325..f76c5d9 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -36,6 +38,7 @@ typedef struct PartitionDescData } PartitionDescData; typedef struct PartitionDescData *PartitionDesc; +typedef struct PartitionDispatchData *PartitionDispatch; extern void RelationBuildPartitionDesc(Relation relation); extern bool partition_bounds_equal(PartitionKey key, @@ -45,4 +48,12 @@ extern void check_new_partition_bound(char *relname, Relation parent, Node *boun extern Oid get_partition_parent(Oid relid); extern List *get_qual_from_partbound(Relation rel, Relation parent, Node *bound); extern List *RelationGetPartitionQual(Relation rel, bool recurse); + +/* For tuple routing */ +extern PartitionDispatch *RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + List **leaf_part_oids); +extern int get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..b4d09f9 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..606cb21 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,13 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionDispatchData **mt_partition_dispatch_info; + /* Tuple-routing support info */ + int mt_num_partitions; /* Number of members in the + * following arrays */ + ResultRelInfo *mt_partitions; /* Per partition result relation */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per partition tuple conversion map */ } ModifyTableState; /* ---------------- diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index cc9f957..9706034 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -223,6 +223,58 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index 3a9430e..afecb74 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -137,6 +137,31 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------96825311778D254EC2B71375 Content-Type: text/x-diff; name="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-18.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-18"; filename*1=".patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH 6/7] Tuple routing for partitioned tables. @ 2016-07-27 06:47 amit <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: amit @ 2016-07-27 06:47 UTC (permalink / raw) Both COPY FROM and INSERT are covered by this commit. Routing to foreing partitions is not supported at the moment. To implement tuple-routing, introduce a PartitionDispatch data structure. Each partitioned table in a partition tree gets one and contains info such as a pointer to its partition descriptor, partition key execution state, global sequence numbers of its leaf partitions and a way to link to the PartitionDispatch objects of any of its partitions that are partitioned themselves. Starting with the PartitionDispatch object of the root partitioned table and a tuple to route, one can get the global sequence number of the leaf partition that the tuple gets routed to, if one exists. --- src/backend/catalog/partition.c | 342 ++++++++++++++++++++++++++++++++ src/backend/commands/copy.c | 151 ++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 58 ++++++- src/backend/executor/nodeModifyTable.c | 130 ++++++++++++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 11 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 8 + src/test/regress/expected/insert.out | 52 +++++ src/test/regress/sql/insert.sql | 25 +++ 11 files changed, 787 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index ba22453..643a258 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -109,6 +109,28 @@ typedef struct PartitionRangeBound bool lower; /* this is the lower (vs upper) bound */ } PartitionRangeBound; +/*----------------------- + * PartitionDispatch - information about one partitioned table in a partition + * hiearchy required to route a tuple to one of its partitions + * + * relid OID of the table + * key Partition key information of the table + * keystate Execution state required for expressions in the partition key + * partdesc Partition descriptor of the table + * indexes Array with partdesc->nparts members (for details on what + * individual members represent, see how they are set in + * RelationGetPartitionDispatchInfo()) + *----------------------- + */ +typedef struct PartitionDispatchData +{ + Oid relid; + PartitionKey key; + List *keystate; /* list of ExprState */ + PartitionDesc partdesc; + int *indexes; +} PartitionDispatchData; + static int32 qsort_partition_list_value_cmp(const void *a, const void *b, void *arg); static int32 qsort_partition_rbound_cmp(const void *a, const void *b, void *arg); @@ -122,12 +144,21 @@ static PartitionRangeBound *make_one_range_bound(PartitionKey key, int index, Li static int32 partition_rbound_cmp(PartitionKey key, Datum *datums1, bool *inf1, bool lower1, PartitionRangeBound *b2); +static int32 partition_rbound_datum_cmp(PartitionKey key, Datum *rb_datums, bool *rb_inf, + Datum *tuple_datums); static int32 partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, int offset, void *probe, bool probe_is_bound); static int partition_bound_bsearch(PartitionKey key, PartitionBoundInfo boundinfo, void *probe, bool probe_is_bound, bool *is_equal); +/* Support get_partition_for_tuple() */ +static void FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); + /* * RelationBuildPartitionDesc * Form rel's partition descriptor @@ -888,6 +919,115 @@ RelationGetPartitionQual(Relation rel, bool recurse) return generate_partition_qual(rel, recurse); } +/* Turn an array of OIDs with N elements into a list */ +#define OID_ARRAY_TO_LIST(arr, N, list) \ + do\ + {\ + int i;\ + for (i = 0; i < (N); i++)\ + (list) = lappend_oid((list), (arr)[i]);\ + } while(0) + +/* + * RelationGetPartitionDispatchInfo + * Returns information necessary to route tuples down a partition tree + * + * All the partitions will be locked with lockmode, unless it is NoLock. + * A list of the OIDs of all the leaf partition of rel is returned in + * *leaf_part_oids. + */ +PartitionDispatch * +RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + List **leaf_part_oids) +{ + PartitionDesc rootpartdesc = RelationGetPartitionDesc(rel); + PartitionDispatchData **pd; + List *all_parts = NIL, + *parted_rels = NIL; + ListCell *lc; + int i, + k, + num_parted; + + /* + * Lock partitions and collect OIDs of the partitioned ones to prepare + * their PartitionDispatch objects. + * + * Cannot use find_all_inheritors() here, because then the order of OIDs + * in parted_rels list would be unknown, which does not help because down + * below, we assign indexes within individual PartitionDispatch in an + * order that's predetermined (determined by the order of OIDs in + * individual partition descriptors). + */ + parted_rels = lappend_oid(parted_rels, RelationGetRelid(rel)); + num_parted = 1; + OID_ARRAY_TO_LIST(rootpartdesc->oids, rootpartdesc->nparts, all_parts); + foreach(lc, all_parts) + { + Relation partrel = heap_open(lfirst_oid(lc), lockmode); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + + /* + * If this partition is a partitined table, add its children to to the + * end of the list, so that they are processed as well. + */ + if (partdesc) + { + num_parted++; + parted_rels = lappend_oid(parted_rels, lfirst_oid(lc)); + OID_ARRAY_TO_LIST(partdesc->oids, partdesc->nparts, all_parts); + } + + heap_close(partrel, NoLock); + } + + /* Generate PartitionDispatch objects for all partitioned tables */ + pd = (PartitionDispatchData **) palloc(num_parted * + sizeof(PartitionDispatchData *)); + *leaf_part_oids = NIL; + i = k = 0; + foreach(lc, parted_rels) + { + /* We locked all partitions above */ + Relation partrel = heap_open(lfirst_oid(lc), NoLock); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + int j, + m; + + pd[i] = (PartitionDispatch) palloc(sizeof(PartitionDispatchData)); + pd[i]->relid = RelationGetRelid(partrel); + pd[i]->key = RelationGetPartitionKey(partrel); + pd[i]->keystate = NIL; + pd[i]->partdesc = partdesc; + pd[i]->indexes = (int *) palloc(partdesc->nparts * sizeof(int)); + heap_close(partrel, NoLock); + + m = 0; + for (j = 0; j < partdesc->nparts; j++) + { + Oid partrelid = partdesc->oids[j]; + + if (get_rel_relkind(partrelid) != RELKIND_PARTITIONED_TABLE) + { + *leaf_part_oids = lappend_oid(*leaf_part_oids, partrelid); + pd[i]->indexes[j] = k++; + } + else + { + /* + * We can assign indexes this way because of the way + * parted_rels has been generated. + */ + pd[i]->indexes[j] = -(i + 1 + m); + m++; + } + } + i++; + } + + return pd; +} + /* Module-local functions */ /* @@ -1328,6 +1468,172 @@ generate_partition_qual(Relation rel, bool recurse) return result; } +/* ---------------- + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for the partition key + * of a tuple. + * + * pkinfo partition key execution info + * slot Heap tuple from which to extract partition key + * estate executor state for evaluating any partition key + * expressions (must be non-NULL) + * values Array of partition key Datums (output area) + * isnull Array of is-null indicators (output area) + * + * the ecxt_scantuple slot of estate's per-tuple expr context must point to + * the heap tuple passed in. + * ---------------- + */ +static void +FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pd->key->partexprs != NIL && pd->keystate == NIL) + { + /* Check caller has set up context correctly */ + Assert(estate != NULL && + GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + + /* First time through, set up expression evaluation state */ + pd->keystate = (List *) ExecPrepareExpr((Expr *) pd->key->partexprs, + estate); + } + + partexpr_item = list_head(pd->keystate); + for (i = 0; i < pd->key->partnatts; i++) + { + AttrNumber keycol = pd->key->partattrs[i]; + Datum datum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + datum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + datum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = datum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Finds a leaf partition for tuple contained in *slot + * + * Returned value is the sequence number of the leaf partition thus found, + * or -1 if no leaf partition is found for the tuple. *failed_at is set + * to the OID of the partitioned table whose partition was not found in + * the latter case. + */ +int +get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + PartitionDispatch parent; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int cur_offset, + cur_index; + int i; + + /* start with the root partitioned table */ + parent = pd[0]; + while(true) + { + PartitionKey key = parent->key; + PartitionDesc partdesc = parent->partdesc; + + /* Quick exit */ + if (partdesc->nparts == 0) + { + *failed_at = parent->relid; + return -1; + } + + /* Extract partition key from tuple */ + FormPartitionKeyDatum(parent, slot, estate, values, isnull); + + if (key->strategy == PARTITION_STRATEGY_RANGE) + { + /* Disallow nulls in the range partition key of the tuple */ + for (i = 0; i < key->partnatts; i++) + if (isnull[i]) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key of row contains null"))); + } + + if (partdesc->boundinfo->has_null && isnull[0]) + /* Tuple maps to the null-accepting list partition */ + cur_index = partdesc->boundinfo->null_index; + else + { + /* Else bsearch in partdesc->boundinfo */ + bool equal = false; + + cur_offset = partition_bound_bsearch(key, partdesc->boundinfo, + values, false, &equal); + switch (key->strategy) + { + case PARTITION_STRATEGY_LIST: + if (cur_offset >= 0 && equal) + cur_index = partdesc->boundinfo->indexes[cur_offset]; + else + cur_index = -1; + break; + + case PARTITION_STRATEGY_RANGE: + /* + * Offset returned is such that the bound at offset is + * found to be less or equal with the tuple. So, the + * bound at offset+1 would be the upper bound. + */ + cur_index = partdesc->boundinfo->indexes[cur_offset+1]; + break; + } + } + + /* + * cur_index < 0 means we failed to find a partition of this parent. + * cur_index >= 0 means we either found the leaf partition, or the + * next parent to find a partition of. + */ + if (cur_index < 0) + { + *failed_at = parent->relid; + return -1; + } + else if (parent->indexes[cur_index] < 0) + parent = pd[-parent->indexes[cur_index]]; + else + break; + } + + return parent->indexes[cur_index]; +} + /* List partition related support functions */ /* @@ -1461,6 +1767,35 @@ partition_rbound_cmp(PartitionKey key, } /* + * partition_rbound_datum_cmp + * + * Return whether range bound (specified in datums, finite, and lower) is + * <=, =, >= partition key of tuple (tuple_datums) + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, Datum *rb_datums, bool *rb_inf, + Datum *tuple_datums) +{ + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (rb_inf[i]) + return rb_inf[i] == RANGE_DATUM_NEG_INF ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + rb_datums[i], + tuple_datums[i])); + if (cmpval != 0) + break; + } + + return cmpval; +} + +/* * partition_bound_cmp * * Return whether the bound at offset in boundinfo is <=, =, >= the argument @@ -1502,6 +1837,13 @@ partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, cmpval = partition_rbound_cmp(key, datums1, inf1, lower1, b2); } + else + { + Datum *datums2 = (Datum *) probe; + + cmpval = partition_rbound_datum_cmp(key, datums1, inf1, + datums2); + } } break; } diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..7d76ead 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -161,6 +161,10 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionDispatch *partition_dispatch_info; + int num_partitions; + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1401,67 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + List *leaf_parts; + ListCell *cell; + int i, + num_leaf_parts; + ResultRelInfo *leaf_part_rri; + + /* Get the tuple-routing information and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + num_leaf_parts = list_length(leaf_parts); + cstate->partition_dispatch_info = pd; + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) palloc(num_leaf_parts * + sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation partrel; + + /* + * All partitions locked above; will be closed after CopyFrom is + * finished. + */ + partrel = heap_open(lfirst_oid(cell), NoLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(partrel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + partrel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(tupDesc, RelationGetDescr(partrel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(partrel), + gettext_noop("could not convert row type")); + leaf_part_rri++; + i++; + } + } } else { @@ -2255,6 +2320,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2347,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2458,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2486,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->partition_dispatch_info != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2442,7 +2511,11 @@ CopyFrom(CopyState cstate) values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2567,56 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition to heap_insert the tuple into */ + if (cstate->partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + cstate->partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < cstate->num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding + * to the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = cstate->partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* + * For ExecInsertIndexTuples() to work on the partition's indexes + */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the + * partition rowtype. + */ + map = cstate->partition_tupconv_maps[leaf_part_index]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2553,7 +2676,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2701,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2720,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); @@ -2614,6 +2745,20 @@ CopyFrom(CopyState cstate) ExecCloseIndices(resultRelInfo); + /* Close all partitions and indices thereof */ + if (cstate->partition_dispatch_info) + { + int i; + + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + } + FreeExecutorState(estate); /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index b27da1d..be08eff 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1293,6 +1293,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index c7a6347..54fb771 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2990,3 +2995,52 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +/* + * ExecFindPartition -- Find a leaf partition in the partition tree rooted + * at parent, for the heap tuple contained in *slot + * + * estate must be non-NULL; we'll need it to compute any expressions in the + * partition key(s) + * + * If no leaf partition is found, this routine errors out with the appropriate + * error message, else it returns the leaf partition sequence number returned + * by get_partition_for_tuple() unchanged. + */ +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, + TupleTableSlot *slot, EState *estate) +{ + int result; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + result = get_partition_for_tuple(pd, slot, estate, &failed_at); + if (result < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return result; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..aa1d8b9 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,56 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + /* Determine the partition to heap_insert the tuple into */ + if (mtstate->mt_partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, false); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +562,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1622,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1713,69 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + /* Form the partition node tree and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + mtstate->mt_partition_dispatch_info = pd; + num_leaf_parts = list_length(leaf_parts); + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2093,15 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + /* * Free the exprcontext */ diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 70d8325..f76c5d9 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -36,6 +38,7 @@ typedef struct PartitionDescData } PartitionDescData; typedef struct PartitionDescData *PartitionDesc; +typedef struct PartitionDispatchData *PartitionDispatch; extern void RelationBuildPartitionDesc(Relation relation); extern bool partition_bounds_equal(PartitionKey key, @@ -45,4 +48,12 @@ extern void check_new_partition_bound(char *relname, Relation parent, Node *boun extern Oid get_partition_parent(Oid relid); extern List *get_qual_from_partbound(Relation rel, Relation parent, Node *bound); extern List *RelationGetPartitionQual(Relation rel, bool recurse); + +/* For tuple routing */ +extern PartitionDispatch *RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + List **leaf_part_oids); +extern int get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..b4d09f9 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..606cb21 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,13 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionDispatchData **mt_partition_dispatch_info; + /* Tuple-routing support info */ + int mt_num_partitions; /* Number of members in the + * following arrays */ + ResultRelInfo *mt_partitions; /* Per partition result relation */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per partition tuple conversion map */ } ModifyTableState; /* ---------------- diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 9ae6b09..d5dcb59 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -227,6 +227,58 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index b6e821e..fbd30d9 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,31 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------F7EE68175C11124DB3D461AC Content-Type: text/x-diff; name="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-17.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-17"; filename*1=".patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH 6/7] Tuple routing for partitioned tables. @ 2016-07-27 06:47 amit <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: amit @ 2016-07-27 06:47 UTC (permalink / raw) Both COPY FROM and INSERT are covered by this commit. Routing to foreing partitions is not supported at the moment. To implement tuple-routing, introduce a PartitionDispatch data structure. Each partitioned table in a partition tree gets one and contains info such as a pointer to its partition descriptor, partition key execution state, global sequence numbers of its leaf partitions and a way to link to the PartitionDispatch objects of any of its partitions that are partitioned themselves. Starting with the PartitionDispatch object of the root partitioned table and a tuple to route, one can get the global sequence number of the leaf partition that the tuple gets routed to, if one exists. --- src/backend/catalog/partition.c | 328 ++++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 171 ++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 58 +++++- src/backend/executor/nodeModifyTable.c | 147 +++++++++++++++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 35 ++++ src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 10 + src/test/regress/expected/insert.out | 55 ++++++ src/test/regress/sql/insert.sql | 27 +++ 11 files changed, 840 insertions(+), 6 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index 0a4f95fc3f..b2cb742264 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -129,6 +129,9 @@ static PartitionRangeBound *make_one_range_bound(PartitionKey key, int index, static int32 partition_rbound_cmp(PartitionKey key, Datum *datums1, RangeDatumContent *content1, bool lower1, PartitionRangeBound *b2); +static int32 partition_rbound_datum_cmp(PartitionKey key, + Datum *rb_datums, RangeDatumContent *rb_content, + Datum *tuple_datums); static int32 partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, @@ -137,6 +140,13 @@ static int partition_bound_bsearch(PartitionKey key, PartitionBoundInfo boundinfo, void *probe, bool probe_is_bound, bool *is_equal); +/* Support get_partition_for_tuple() */ +static void FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); + /* * RelationBuildPartitionDesc * Form rel's partition descriptor @@ -916,6 +926,119 @@ RelationGetPartitionQual(Relation rel, bool recurse) return generate_partition_qual(rel, recurse); } +/* Turn an array of OIDs with N elements into a list */ +#define OID_ARRAY_TO_LIST(arr, N, list) \ + do\ + {\ + int i;\ + for (i = 0; i < (N); i++)\ + (list) = lappend_oid((list), (arr)[i]);\ + } while(0) + +/* + * RelationGetPartitionDispatchInfo + * Returns information necessary to route tuples down a partition tree + * + * All the partitions will be locked with lockmode, unless it is NoLock. + * A list of the OIDs of all the leaf partition of rel is returned in + * *leaf_part_oids. + */ +PartitionDispatch * +RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + int *num_parted, List **leaf_part_oids) +{ + PartitionDesc rootpartdesc = RelationGetPartitionDesc(rel); + PartitionDispatchData **pd; + List *all_parts = NIL, + *parted_rels; + ListCell *lc; + int i, + k; + + /* + * Lock partitions and make a list of the partitioned ones to prepare + * their PartitionDispatch objects below. + * + * Cannot use find_all_inheritors() here, because then the order of OIDs + * in parted_rels list would be unknown, which does not help, because we + * we assign indexes within individual PartitionDispatch in an order that + * is predetermined (determined by the order of OIDs in individual + * partition descriptors). + */ + *num_parted = 1; + parted_rels = list_make1(rel); + OID_ARRAY_TO_LIST(rootpartdesc->oids, rootpartdesc->nparts, all_parts); + foreach(lc, all_parts) + { + Relation partrel = heap_open(lfirst_oid(lc), lockmode); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + + /* + * If this partition is a partitioned table, add its children to the + * end of the list, so that they are processed as well. + */ + if (partdesc) + { + (*num_parted)++; + parted_rels = lappend(parted_rels, partrel); + OID_ARRAY_TO_LIST(partdesc->oids, partdesc->nparts, all_parts); + } + else + heap_close(partrel, NoLock); + + /* + * We keep the partitioned ones open until we're done using the + * information being collected here (for example, see + * ExecEndModifyTable). + */ + } + + /* Generate PartitionDispatch objects for all partitioned tables */ + pd = (PartitionDispatchData **) palloc(*num_parted * + sizeof(PartitionDispatchData *)); + *leaf_part_oids = NIL; + i = k = 0; + foreach(lc, parted_rels) + { + Relation partrel = lfirst(lc); + PartitionKey partkey = RelationGetPartitionKey(partrel); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + int j, + m; + + pd[i] = (PartitionDispatch) palloc(sizeof(PartitionDispatchData)); + pd[i]->reldesc = partrel; + pd[i]->key = partkey; + pd[i]->keystate = NIL; + pd[i]->partdesc = partdesc; + pd[i]->indexes = (int *) palloc(partdesc->nparts * sizeof(int)); + + m = 0; + for (j = 0; j < partdesc->nparts; j++) + { + Oid partrelid = partdesc->oids[j]; + + if (get_rel_relkind(partrelid) != RELKIND_PARTITIONED_TABLE) + { + *leaf_part_oids = lappend_oid(*leaf_part_oids, partrelid); + pd[i]->indexes[j] = k++; + } + else + { + /* + * We can assign indexes this way because of the way + * parted_rels has been generated. + */ + pd[i]->indexes[j] = -(i + 1 + m); + m++; + } + } + i++; + } + + return pd; +} + /* Module-local functions */ /* @@ -1367,6 +1490,176 @@ generate_partition_qual(Relation rel, bool recurse) return result; } +/* ---------------- + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for the partition key + * of a tuple. + * + * pkinfo partition key execution info + * slot Heap tuple from which to extract partition key + * estate executor state for evaluating any partition key + * expressions (must be non-NULL) + * values Array of partition key Datums (output area) + * isnull Array of is-null indicators (output area) + * + * the ecxt_scantuple slot of estate's per-tuple expr context must point to + * the heap tuple passed in. + * ---------------- + */ +static void +FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pd->key->partexprs != NIL && pd->keystate == NIL) + { + /* Check caller has set up context correctly */ + Assert(estate != NULL && + GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + + /* First time through, set up expression evaluation state */ + pd->keystate = (List *) ExecPrepareExpr((Expr *) pd->key->partexprs, + estate); + } + + partexpr_item = list_head(pd->keystate); + for (i = 0; i < pd->key->partnatts; i++) + { + AttrNumber keycol = pd->key->partattrs[i]; + Datum datum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + datum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + datum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = datum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Finds a leaf partition for tuple contained in *slot + * + * Returned value is the sequence number of the leaf partition thus found, + * or -1 if no leaf partition is found for the tuple. *failed_at is set + * to the OID of the partitioned table whose partition was not found in + * the latter case. + */ +int +get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + PartitionDispatch parent; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int cur_offset, + cur_index; + int i; + + /* start with the root partitioned table */ + parent = pd[0]; + while(true) + { + PartitionKey key = parent->key; + PartitionDesc partdesc = parent->partdesc; + + /* Quick exit */ + if (partdesc->nparts == 0) + { + *failed_at = RelationGetRelid(parent->reldesc); + return -1; + } + + /* Extract partition key from tuple */ + FormPartitionKeyDatum(parent, slot, estate, values, isnull); + + if (key->strategy == PARTITION_STRATEGY_RANGE) + { + /* Disallow nulls in the range partition key of the tuple */ + for (i = 0; i < key->partnatts; i++) + if (isnull[i]) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key of row contains null"))); + } + + if (partdesc->boundinfo->has_null && isnull[0]) + /* Tuple maps to the null-accepting list partition */ + cur_index = partdesc->boundinfo->null_index; + else + { + /* Else bsearch in partdesc->boundinfo */ + bool equal = false; + + cur_offset = partition_bound_bsearch(key, partdesc->boundinfo, + values, false, &equal); + switch (key->strategy) + { + case PARTITION_STRATEGY_LIST: + if (cur_offset >= 0 && equal) + cur_index = partdesc->boundinfo->indexes[cur_offset]; + else + cur_index = -1; + break; + + case PARTITION_STRATEGY_RANGE: + /* + * Offset returned is such that the bound at offset is + * found to be less or equal with the tuple. So, the + * bound at offset+1 would be the upper bound. + */ + cur_index = partdesc->boundinfo->indexes[cur_offset+1]; + break; + + default: + elog(ERROR, "unexpected partition strategy: %d", + (int) key->strategy); + } + } + + /* + * cur_index < 0 means we failed to find a partition of this parent. + * cur_index >= 0 means we either found the leaf partition, or the + * next parent to find a partition of. + */ + if (cur_index < 0) + { + *failed_at = RelationGetRelid(parent->reldesc); + return -1; + } + else if (parent->indexes[cur_index] < 0) + parent = pd[-parent->indexes[cur_index]]; + else + break; + } + + return parent->indexes[cur_index]; +} + /* * qsort_partition_list_value_cmp * @@ -1499,6 +1792,36 @@ partition_rbound_cmp(PartitionKey key, } /* + * partition_rbound_datum_cmp + * + * Return whether range bound (specified in rb_datums, rb_content, and + * rb_lower) <=, =, >= partition key of tuple (tuple_datums) + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, + Datum *rb_datums, RangeDatumContent *rb_content, + Datum *tuple_datums) +{ + int i; + int32 cmpval = -1; + + for (i = 0; i < key->partnatts; i++) + { + if (rb_content[i] != RANGE_DATUM_FINITE) + return rb_content[i] == RANGE_DATUM_NEG_INF ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + rb_datums[i], + tuple_datums[i])); + if (cmpval != 0) + break; + } + + return cmpval; +} + +/* * partition_bound_cmp * * Return whether the bound at offset in boundinfo is <=, =, >= the argument @@ -1537,7 +1860,10 @@ partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, bound_datums, content, lower, (PartitionRangeBound *) probe); } - + else + cmpval = partition_rbound_datum_cmp(key, + bound_datums, content, + (Datum *) probe); break; } diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index de9e29c81e..3f8d748a82 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -161,6 +161,11 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionDispatch *partition_dispatch_info; + int num_dispatch; + int num_partitions; + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1402,71 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + List *leaf_parts; + ListCell *cell; + int i, + num_parted, + num_leaf_parts; + ResultRelInfo *leaf_part_rri; + + /* Get the tuple-routing information and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &num_parted, &leaf_parts); + num_leaf_parts = list_length(leaf_parts); + cstate->partition_dispatch_info = pd; + cstate->num_dispatch = num_parted; + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) palloc(num_leaf_parts * + sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation partrel; + + /* + * We locked all the partitions above including the leaf + * partitions. Note that each of the relations in + * cstate->partitions will be closed by CopyFrom() after + * it's finished with its processing. + */ + partrel = heap_open(lfirst_oid(cell), NoLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(partrel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + partrel, + 1, /* dummy */ + false, /* no partition constraint check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(tupDesc, RelationGetDescr(partrel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(partrel), + gettext_noop("could not convert row type")); + leaf_part_rri++; + i++; + } + } } else { @@ -2255,6 +2325,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2352,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2463,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2491,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->partition_dispatch_info != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2442,7 +2516,11 @@ CopyFrom(CopyState cstate) values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2572,59 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition to heap_insert the tuple into */ + if (cstate->partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + cstate->partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < cstate->num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding + * to the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = cstate->partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* + * For ExecInsertIndexTuples() to work on the partition's indexes + */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the + * partition rowtype. + */ + map = cstate->partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, true); + } + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2553,7 +2684,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2709,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2728,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); @@ -2614,6 +2753,32 @@ CopyFrom(CopyState cstate) ExecCloseIndices(resultRelInfo); + /* Close all the partitioned tables, leaf partitions, and their indices */ + if (cstate->partition_dispatch_info) + { + int i; + + /* + * Remember cstate->partition_dispatch_info[0] corresponds to the root + * partitioned table, which we must not try to close, because it is + * the main target table of COPY that will be closed eventually by + * DoCopy(). + */ + for (i = 1; i < cstate->num_dispatch; i++) + { + PartitionDispatch pd = cstate->partition_dispatch_info[i]; + + heap_close(pd->reldesc, NoLock); + } + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + } + FreeExecutorState(estate); /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8a803233ca..c77b216d4f 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1322,6 +1322,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index 21b18b82b9..0f47c7e010 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2991,3 +2996,52 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +/* + * ExecFindPartition -- Find a leaf partition in the partition tree rooted + * at parent, for the heap tuple contained in *slot + * + * estate must be non-NULL; we'll need it to compute any expressions in the + * partition key(s) + * + * If no leaf partition is found, this routine errors out with the appropriate + * error message, else it returns the leaf partition sequence number returned + * by get_partition_for_tuple() unchanged. + */ +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, + TupleTableSlot *slot, EState *estate) +{ + int result; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + result = get_partition_for_tuple(pd, slot, estate, &failed_at); + if (result < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return result; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 6eccfb7cec..c0b58d1841 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,56 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + /* Determine the partition to heap_insert the tuple into */ + if (mtstate->mt_partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, true); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +562,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1622,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1713,75 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + int i, + j, + num_parted, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + /* Form the partition node tree and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &num_parted, &leaf_parts); + mtstate->mt_partition_dispatch_info = pd; + mtstate->mt_num_dispatch = num_parted; + num_leaf_parts = list_length(leaf_parts); + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid partrelid = lfirst_oid(cell); + Relation partrel; + + /* + * We locked all the partitions above including the leaf + * partitions. Note that each of the relations in + * mtstate->mt_partitions will be closed by ExecEndModifyTable(). + */ + partrel = heap_open(partrelid, NoLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(partrel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + partrel, + 1, /* dummy */ + false, /* no partition constraint checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (partrel->rd_rel->relhasindex && operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(partrel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(partrel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2099,26 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all the partitioned tables, leaf partitions, and their indices + * + * Remember node->mt_partition_dispatch_info[0] corresponds to the root + * partitioned table, which we must not try to close, because it is the + * main target table of the query that will be closed by ExecEndPlan(). + */ + for (i = 1; i < node->mt_num_dispatch; i++) + { + PartitionDispatch pd = node->mt_partition_dispatch_info[i]; + + heap_close(pd->reldesc, NoLock); + } + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + /* * Free the exprcontext */ diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 1a541788eb..7364346167 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -806,8 +806,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 70d8325137..21effbf87b 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -37,6 +39,30 @@ typedef struct PartitionDescData typedef struct PartitionDescData *PartitionDesc; +/*----------------------- + * PartitionDispatch - information about one partitioned table in a partition + * hiearchy required to route a tuple to one of its partitions + * + * reldesc Relation descriptor of the table + * key Partition key information of the table + * keystate Execution state required for expressions in the partition key + * partdesc Partition descriptor of the table + * indexes Array with partdesc->nparts members (for details on what + * individual members represent, see how they are set in + * RelationGetPartitionDispatchInfo()) + *----------------------- + */ +typedef struct PartitionDispatchData +{ + Relation reldesc; + PartitionKey key; + List *keystate; /* list of ExprState */ + PartitionDesc partdesc; + int *indexes; +} PartitionDispatchData; + +typedef struct PartitionDispatchData *PartitionDispatch; + extern void RelationBuildPartitionDesc(Relation relation); extern bool partition_bounds_equal(PartitionKey key, PartitionBoundInfo p1, PartitionBoundInfo p2); @@ -45,4 +71,13 @@ extern void check_new_partition_bound(char *relname, Relation parent, Node *boun extern Oid get_partition_parent(Oid relid); extern List *get_qual_from_partbound(Relation rel, Relation parent, Node *bound); extern List *RelationGetPartitionQual(Relation rel, bool recurse); + +/* For tuple routing */ +extern PartitionDispatch *RelationGetPartitionDispatchInfo(Relation rel, + int lockmode, int *num_parted, + List **leaf_part_oids); +extern int get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276be53..b4d09f9564 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index df2dec3a2c..1de5c8196d 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,15 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionDispatchData **mt_partition_dispatch_info; + /* Tuple-routing support info */ + int mt_num_dispatch; /* Number of entries in the above + * array */ + int mt_num_partitions; /* Number of members in the + * following arrays */ + ResultRelInfo *mt_partitions; /* Per partition result relation */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per partition tuple conversion map */ } ModifyTableState; /* ---------------- diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 2e78fd9fc1..561cefa3c4 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -230,6 +230,61 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_ee_ff1 values ('ff', 1); insert into part_ee_ff2 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +-- fail (partition key (b+0) is null) +insert into range_parted values ('a'); +ERROR: range partition key of row contains null +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------+---+---- + part1 | a | 1 + part1 | a | 1 + part2 | a | 10 + part3 | b | 1 + part4 | b | 10 + part4 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_ee_ff not found in both cases) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_ee_ff values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_ee_ff values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +-------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff1 | ff | 1 + part_ee_ff1 | EE | 1 + part_ee_ff2 | ff | 11 + part_ee_ff2 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index eb923646c3..846bb5897a 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,33 @@ insert into part_ee_ff1 values ('cc', 1); insert into part_ee_ff1 values ('ff', 1); insert into part_ee_ff2 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +-- fail (partition key (b+0) is null) +insert into range_parted values ('a'); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_ee_ff not found in both cases) +insert into list_parted values ('EE', 0); +insert into part_ee_ff values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_ee_ff values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 2.11.0 --------------E1933196A000B12EC95A8B6E Content-Type: text/x-diff; name="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-20.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-20"; filename*1=".patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH 6/7] Tuple routing for partitioned tables. @ 2016-07-27 06:47 amit <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: amit @ 2016-07-27 06:47 UTC (permalink / raw) Both COPY FROM and INSERT are covered by this commit. Routing to foreing partitions is not supported at the moment. To implement tuple-routing, introduce a PartitionDispatch data structure. Each partitioned table in a partition tree gets one and contains info such as a pointer to its partition descriptor, partition key execution state, global sequence numbers of its leaf partitions and a way to link to the PartitionDispatch objects of any of its partitions that are partitioned themselves. Starting with the PartitionDispatch object of the root partitioned table and a tuple to route, one can get the global sequence number of the leaf partition that the tuple gets routed to, if one exists. --- src/backend/catalog/partition.c | 342 +++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 154 ++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 58 ++++++- src/backend/executor/nodeModifyTable.c | 130 ++++++++++++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 11 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 8 + src/test/regress/expected/insert.out | 55 +++++ src/test/regress/sql/insert.sql | 27 +++ 11 files changed, 794 insertions(+), 6 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index f439c43..83dc151 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -113,6 +113,28 @@ typedef struct PartitionRangeBound bool lower; /* this is the lower (vs upper) bound */ } PartitionRangeBound; +/*----------------------- + * PartitionDispatch - information about one partitioned table in a partition + * hiearchy required to route a tuple to one of its partitions + * + * relid OID of the table + * key Partition key information of the table + * keystate Execution state required for expressions in the partition key + * partdesc Partition descriptor of the table + * indexes Array with partdesc->nparts members (for details on what + * individual members represent, see how they are set in + * RelationGetPartitionDispatchInfo()) + *----------------------- + */ +typedef struct PartitionDispatchData +{ + Oid relid; + PartitionKey key; + List *keystate; /* list of ExprState */ + PartitionDesc partdesc; + int *indexes; +} PartitionDispatchData; + static int32 qsort_partition_list_value_cmp(const void *a, const void *b, void *arg); static int32 qsort_partition_rbound_cmp(const void *a, const void *b, void *arg); @@ -126,12 +148,22 @@ static PartitionRangeBound *make_one_range_bound(PartitionKey key, int index, Li static int32 partition_rbound_cmp(PartitionKey key, Datum *datums1, RangeDatumContent *content1, bool lower1, PartitionRangeBound *b2); +static int32 partition_rbound_datum_cmp(PartitionKey key, + Datum *rb_datums, RangeDatumContent *rb_content, + Datum *tuple_datums); static int32 partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, int offset, void *probe, bool probe_is_bound); static int partition_bound_bsearch(PartitionKey key, PartitionBoundInfo boundinfo, void *probe, bool probe_is_bound, bool *is_equal); +/* Support get_partition_for_tuple() */ +static void FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); + /* * RelationBuildPartitionDesc * Form rel's partition descriptor @@ -895,6 +927,115 @@ RelationGetPartitionQual(Relation rel, bool recurse) return generate_partition_qual(rel, recurse); } +/* Turn an array of OIDs with N elements into a list */ +#define OID_ARRAY_TO_LIST(arr, N, list) \ + do\ + {\ + int i;\ + for (i = 0; i < (N); i++)\ + (list) = lappend_oid((list), (arr)[i]);\ + } while(0) + +/* + * RelationGetPartitionDispatchInfo + * Returns information necessary to route tuples down a partition tree + * + * All the partitions will be locked with lockmode, unless it is NoLock. + * A list of the OIDs of all the leaf partition of rel is returned in + * *leaf_part_oids. + */ +PartitionDispatch * +RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + List **leaf_part_oids) +{ + PartitionDesc rootpartdesc = RelationGetPartitionDesc(rel); + PartitionDispatchData **pd; + List *all_parts = NIL, + *parted_rels = NIL; + ListCell *lc; + int i, + k, + num_parted; + + /* + * Lock partitions and collect OIDs of the partitioned ones to prepare + * their PartitionDispatch objects. + * + * Cannot use find_all_inheritors() here, because then the order of OIDs + * in parted_rels list would be unknown, which does not help because down + * below, we assign indexes within individual PartitionDispatch in an + * order that's predetermined (determined by the order of OIDs in + * individual partition descriptors). + */ + parted_rels = lappend_oid(parted_rels, RelationGetRelid(rel)); + num_parted = 1; + OID_ARRAY_TO_LIST(rootpartdesc->oids, rootpartdesc->nparts, all_parts); + foreach(lc, all_parts) + { + Relation partrel = heap_open(lfirst_oid(lc), lockmode); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + + /* + * If this partition is a partitined table, add its children to to the + * end of the list, so that they are processed as well. + */ + if (partdesc) + { + num_parted++; + parted_rels = lappend_oid(parted_rels, lfirst_oid(lc)); + OID_ARRAY_TO_LIST(partdesc->oids, partdesc->nparts, all_parts); + } + + heap_close(partrel, NoLock); + } + + /* Generate PartitionDispatch objects for all partitioned tables */ + pd = (PartitionDispatchData **) palloc(num_parted * + sizeof(PartitionDispatchData *)); + *leaf_part_oids = NIL; + i = k = 0; + foreach(lc, parted_rels) + { + /* We locked all partitions above */ + Relation partrel = heap_open(lfirst_oid(lc), NoLock); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + int j, + m; + + pd[i] = (PartitionDispatch) palloc(sizeof(PartitionDispatchData)); + pd[i]->relid = RelationGetRelid(partrel); + pd[i]->key = RelationGetPartitionKey(partrel); + pd[i]->keystate = NIL; + pd[i]->partdesc = partdesc; + pd[i]->indexes = (int *) palloc(partdesc->nparts * sizeof(int)); + heap_close(partrel, NoLock); + + m = 0; + for (j = 0; j < partdesc->nparts; j++) + { + Oid partrelid = partdesc->oids[j]; + + if (get_rel_relkind(partrelid) != RELKIND_PARTITIONED_TABLE) + { + *leaf_part_oids = lappend_oid(*leaf_part_oids, partrelid); + pd[i]->indexes[j] = k++; + } + else + { + /* + * We can assign indexes this way because of the way + * parted_rels has been generated. + */ + pd[i]->indexes[j] = -(i + 1 + m); + m++; + } + } + i++; + } + + return pd; +} + /* Module-local functions */ /* @@ -1346,6 +1487,172 @@ generate_partition_qual(Relation rel, bool recurse) return result; } +/* ---------------- + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for the partition key + * of a tuple. + * + * pkinfo partition key execution info + * slot Heap tuple from which to extract partition key + * estate executor state for evaluating any partition key + * expressions (must be non-NULL) + * values Array of partition key Datums (output area) + * isnull Array of is-null indicators (output area) + * + * the ecxt_scantuple slot of estate's per-tuple expr context must point to + * the heap tuple passed in. + * ---------------- + */ +static void +FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pd->key->partexprs != NIL && pd->keystate == NIL) + { + /* Check caller has set up context correctly */ + Assert(estate != NULL && + GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + + /* First time through, set up expression evaluation state */ + pd->keystate = (List *) ExecPrepareExpr((Expr *) pd->key->partexprs, + estate); + } + + partexpr_item = list_head(pd->keystate); + for (i = 0; i < pd->key->partnatts; i++) + { + AttrNumber keycol = pd->key->partattrs[i]; + Datum datum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + datum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + datum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = datum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Finds a leaf partition for tuple contained in *slot + * + * Returned value is the sequence number of the leaf partition thus found, + * or -1 if no leaf partition is found for the tuple. *failed_at is set + * to the OID of the partitioned table whose partition was not found in + * the latter case. + */ +int +get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + PartitionDispatch parent; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int cur_offset, + cur_index; + int i; + + /* start with the root partitioned table */ + parent = pd[0]; + while(true) + { + PartitionKey key = parent->key; + PartitionDesc partdesc = parent->partdesc; + + /* Quick exit */ + if (partdesc->nparts == 0) + { + *failed_at = parent->relid; + return -1; + } + + /* Extract partition key from tuple */ + FormPartitionKeyDatum(parent, slot, estate, values, isnull); + + if (key->strategy == PARTITION_STRATEGY_RANGE) + { + /* Disallow nulls in the range partition key of the tuple */ + for (i = 0; i < key->partnatts; i++) + if (isnull[i]) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key of row contains null"))); + } + + if (partdesc->boundinfo->has_null && isnull[0]) + /* Tuple maps to the null-accepting list partition */ + cur_index = partdesc->boundinfo->null_index; + else + { + /* Else bsearch in partdesc->boundinfo */ + bool equal = false; + + cur_offset = partition_bound_bsearch(key, partdesc->boundinfo, + values, false, &equal); + switch (key->strategy) + { + case PARTITION_STRATEGY_LIST: + if (cur_offset >= 0 && equal) + cur_index = partdesc->boundinfo->indexes[cur_offset]; + else + cur_index = -1; + break; + + case PARTITION_STRATEGY_RANGE: + /* + * Offset returned is such that the bound at offset is + * found to be less or equal with the tuple. So, the + * bound at offset+1 would be the upper bound. + */ + cur_index = partdesc->boundinfo->indexes[cur_offset+1]; + break; + } + } + + /* + * cur_index < 0 means we failed to find a partition of this parent. + * cur_index >= 0 means we either found the leaf partition, or the + * next parent to find a partition of. + */ + if (cur_index < 0) + { + *failed_at = parent->relid; + return -1; + } + else if (parent->indexes[cur_index] < 0) + parent = pd[-parent->indexes[cur_index]]; + else + break; + } + + return parent->indexes[cur_index]; +} + /* * qsort_partition_list_value_cmp * @@ -1478,6 +1785,36 @@ partition_rbound_cmp(PartitionKey key, } /* + * partition_rbound_datum_cmp + * + * Return whether range bound (specified in rb_datums, rb_content, and + * rb_lower) <=, =, >= partition key of tuple (tuple_datums) + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, + Datum *rb_datums, RangeDatumContent *rb_content, + Datum *tuple_datums) +{ + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (rb_content[i] != RANGE_DATUM_FINITE) + return rb_content[i] == RANGE_DATUM_NEG_INF ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + rb_datums[i], + tuple_datums[i])); + if (cmpval != 0) + break; + } + + return cmpval; +} + +/* * partition_bound_cmp * * Return whether the bound at offset in boundinfo is <=, =, >= the argument @@ -1516,7 +1853,10 @@ partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, bound_datums, content, lower, (PartitionRangeBound *) probe); } - + else + cmpval = partition_rbound_datum_cmp(key, + bound_datums, content, + (Datum *) probe); break; } } diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..3f64f3a 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -161,6 +161,10 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionDispatch *partition_dispatch_info; + int num_partitions; + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1401,67 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + List *leaf_parts; + ListCell *cell; + int i, + num_leaf_parts; + ResultRelInfo *leaf_part_rri; + + /* Get the tuple-routing information and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + num_leaf_parts = list_length(leaf_parts); + cstate->partition_dispatch_info = pd; + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) palloc(num_leaf_parts * + sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation partrel; + + /* + * All partitions locked above; will be closed after CopyFrom is + * finished. + */ + partrel = heap_open(lfirst_oid(cell), NoLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(partrel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + partrel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(tupDesc, RelationGetDescr(partrel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(partrel), + gettext_noop("could not convert row type")); + leaf_part_rri++; + i++; + } + } } else { @@ -2255,6 +2320,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2347,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2458,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2486,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->partition_dispatch_info != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2442,7 +2511,11 @@ CopyFrom(CopyState cstate) values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2567,59 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition to heap_insert the tuple into */ + if (cstate->partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + cstate->partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < cstate->num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding + * to the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = cstate->partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* + * For ExecInsertIndexTuples() to work on the partition's indexes + */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the + * partition rowtype. + */ + map = cstate->partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, true); + } + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2553,7 +2679,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2704,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2723,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); @@ -2614,6 +2748,20 @@ CopyFrom(CopyState cstate) ExecCloseIndices(resultRelInfo); + /* Close all partitions and indices thereof */ + if (cstate->partition_dispatch_info) + { + int i; + + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + } + FreeExecutorState(estate); /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 71ca739..abfb46b 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1322,6 +1322,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index c7a6347..54fb771 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2990,3 +2995,52 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +/* + * ExecFindPartition -- Find a leaf partition in the partition tree rooted + * at parent, for the heap tuple contained in *slot + * + * estate must be non-NULL; we'll need it to compute any expressions in the + * partition key(s) + * + * If no leaf partition is found, this routine errors out with the appropriate + * error message, else it returns the leaf partition sequence number returned + * by get_partition_for_tuple() unchanged. + */ +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, + TupleTableSlot *slot, EState *estate) +{ + int result; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + result = get_partition_for_tuple(pd, slot, estate, &failed_at); + if (result < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return result; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 6eccfb7..ceb5d44 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,56 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + /* Determine the partition to heap_insert the tuple into */ + if (mtstate->mt_partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, true); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +562,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1622,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1713,69 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + /* Form the partition node tree and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + mtstate->mt_partition_dispatch_info = pd; + num_leaf_parts = list_length(leaf_parts); + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2093,15 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + /* * Free the exprcontext */ diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 36f8c54..88380ba 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -806,8 +806,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 70d8325..f76c5d9 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -36,6 +38,7 @@ typedef struct PartitionDescData } PartitionDescData; typedef struct PartitionDescData *PartitionDesc; +typedef struct PartitionDispatchData *PartitionDispatch; extern void RelationBuildPartitionDesc(Relation relation); extern bool partition_bounds_equal(PartitionKey key, @@ -45,4 +48,12 @@ extern void check_new_partition_bound(char *relname, Relation parent, Node *boun extern Oid get_partition_parent(Oid relid); extern List *get_qual_from_partbound(Relation rel, Relation parent, Node *bound); extern List *RelationGetPartitionQual(Relation rel, bool recurse); + +/* For tuple routing */ +extern PartitionDispatch *RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + List **leaf_part_oids); +extern int get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..b4d09f9 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..606cb21 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,13 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionDispatchData **mt_partition_dispatch_info; + /* Tuple-routing support info */ + int mt_num_partitions; /* Number of members in the + * following arrays */ + ResultRelInfo *mt_partitions; /* Per partition result relation */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per partition tuple conversion map */ } ModifyTableState; /* ---------------- diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 2e78fd9..561cefa 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -230,6 +230,61 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_ee_ff1 values ('ff', 1); insert into part_ee_ff2 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +-- fail (partition key (b+0) is null) +insert into range_parted values ('a'); +ERROR: range partition key of row contains null +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------+---+---- + part1 | a | 1 + part1 | a | 1 + part2 | a | 10 + part3 | b | 1 + part4 | b | 10 + part4 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_ee_ff not found in both cases) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_ee_ff values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_ee_ff values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +-------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff1 | ff | 1 + part_ee_ff1 | EE | 1 + part_ee_ff2 | ff | 11 + part_ee_ff2 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index eb92364..846bb58 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,33 @@ insert into part_ee_ff1 values ('cc', 1); insert into part_ee_ff1 values ('ff', 1); insert into part_ee_ff2 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +-- fail (partition key (b+0) is null) +insert into range_parted values ('a'); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_ee_ff not found in both cases) +insert into list_parted values ('EE', 0); +insert into part_ee_ff values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_ee_ff values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------AEEF86BC59A941B2E83AF069 Content-Type: text/x-diff; name="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-19.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-19"; filename*1=".patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH 7/8] Tuple routing for partitioned tables. @ 2016-07-27 07:59 amit <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: amit @ 2016-07-27 07:59 UTC (permalink / raw) Both COPY FROM and INSERT. --- src/backend/catalog/partition.c | 317 ++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 222 +++++++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 59 ++++++- src/backend/executor/nodeModifyTable.c | 160 ++++++++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 1 + src/backend/optimizer/plan/createplan.c | 76 ++++++++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 7 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 10 + src/include/nodes/plannodes.h | 1 + src/test/regress/expected/insert.out | 52 +++++ src/test/regress/sql/insert.sql | 25 +++ 16 files changed, 940 insertions(+), 7 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index 7bc4b15..4e253c1 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -185,6 +185,18 @@ static List *generate_partition_qual(Relation rel, bool recurse); static PartitionTreeNode GetPartitionTreeNodeRecurse(Relation rel, int offset, int lockmode); static int get_leaf_partition_count(PartitionTreeNode parent); +/* Support get_partition_for_tuple() */ +static PartitionKeyExecInfo *BuildPartitionKeyExecInfo(Relation rel); +static void FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); +static int list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull); +static int range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum *tuple); + /* List partition related support functions */ static bool equal_list_info(PartitionKey key, PartitionListInfo *l1, PartitionListInfo *l2); @@ -198,6 +210,8 @@ static PartitionRangeBound *copy_range_bound(PartitionKey key, PartitionRangeBou static bool equal_range_info(PartitionKey key, PartitionRangeInfo *r1, PartitionRangeInfo *r2); static int32 partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg); +static int32 partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg); static bool partition_rbound_eq(PartitionKey key, PartitionRangeBound *b1, PartitionRangeBound *b2); typedef int32 (*partition_rbound_bsearch_cmp_fn) (PartitionKey, @@ -1443,7 +1457,7 @@ GetPartitionTreeNodeRecurse(Relation rel, int offset, int lockmode) /* First build our own node */ parent = (PartitionTreeNode) palloc0(sizeof(PartitionTreeNodeData)); - parent->pkinfo = NULL; + parent->pkinfo = BuildPartitionKeyExecInfo(rel); parent->pdesc = RelationGetPartitionDesc(rel); parent->relid = RelationGetRelid(rel); parent->offset = offset; @@ -1527,6 +1541,273 @@ get_leaf_partition_count(PartitionTreeNode parent) return result; } +/* + * BuildPartitionKeyExecInfo + * Construct a list of PartitionKeyExecInfo records for an open + * relation + * + * PartitionKeyExecInfo stores the information about the partition key + * that's needed when inserting tuples into a partitioned table; especially, + * partition key expression state if there are any expression columns in + * the partition key. Normally we build a PartitionKeyExecInfo for a + * partitioned table just once per command, and then use it for (potentially) + * many tuples. + * + */ +static PartitionKeyExecInfo * +BuildPartitionKeyExecInfo(Relation rel) +{ + PartitionKeyExecInfo *pkinfo; + + pkinfo = (PartitionKeyExecInfo *) palloc0(sizeof(PartitionKeyExecInfo)); + pkinfo->pi_Key = RelationGetPartitionKey(rel); + pkinfo->pi_ExpressionState = NIL; + + return pkinfo; +} + +/* ---------------- + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for the partition key + * of a tuple. + * + * pkinfo partition key execution info + * slot Heap tuple from which to extract partition key + * estate executor state for evaluating any partition key + * expressions (must be non-NULL) + * values Array of partition key Datums (output area) + * isnull Array of is-null indicators (output area) + * + * the ecxt_scantuple slot of estate's per-tuple expr context must point to + * the heap tuple passed in. + * ---------------- + */ +static void +FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pkinfo->pi_Key->partexprs != NIL && pkinfo->pi_ExpressionState == NIL) + { + /* Check caller has set up context correctly */ + Assert(estate != NULL && + GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + + /* First time through, set up expression evaluation state */ + pkinfo->pi_ExpressionState = (List *) + ExecPrepareExpr((Expr *) pkinfo->pi_Key->partexprs, + estate); + } + + partexpr_item = list_head(pkinfo->pi_ExpressionState); + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + { + AttrNumber keycol = pkinfo->pi_Key->partattrs[i]; + Datum pkDatum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + pkDatum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + pkDatum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = pkDatum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Recursively finds the "leaf" partition for tuple + * + * parent is the root of the partition tree through which we are trying + * to route the tuple contained in *slot. + * + * Returned value is the sequence number of the leaf partition thus found, + * or -1 if no leaf partition is found for the tuple. *failed_at is set + * to the OID of the partitioned table whose partition was not found in + * the latter case. + */ +int +get_partition_for_tuple(PartitionTreeNode parent, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + PartitionKeyExecInfo *pkinfo = parent->pkinfo; + PartitionTreeNode node, + prev; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int i; + int cur_idx; + + /* Guard against stack overflow due to overly deep partition tree */ + check_stack_depth(); + + if (parent->pdesc->nparts == 0) + { + *failed_at = parent->relid; + return -1; + } + + /* Extract partition key from tuple */ + FormPartitionKeyDatum(pkinfo, slot, estate, values, isnull); + + switch (pkinfo->pi_Key->strategy) + { + case PARTITION_STRATEGY_LIST: + cur_idx = list_partition_for_tuple(pkinfo->pi_Key, parent->pdesc, + values[0], isnull[0]); + break; + + case PARTITION_STRATEGY_RANGE: + /* Disallow nulls in the partition key of the tuple */ + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + if (isnull[i]) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key of row contains null"))); + + cur_idx = range_partition_for_tuple(pkinfo->pi_Key, parent->pdesc, + values); + break; + } + + /* No partition found at this level */ + if (cur_idx < 0) + { + *failed_at = parent->relid; + return cur_idx; + } + + /* + * The partition we found may either be a leaf partition or partitioned + * itself. In the latter case, we need to recurse by finding the proper + * node (ie, such that node->index == cur_idx) to pass down. + */ + prev = parent; + node = parent->downlink; + while (node != NULL) + { + if (node->index >= cur_idx) + break; + + prev = node; + node = node->next; + } + + /* + * If we couldn't find a node, that means cur_idx is actually a leaf + * partition. In the simpler case where all of the left siblings are leaf + * partitions themselves, we can get the correct index to return by adding + * cur_idx to the parent node's offset. Otherwise, we need to compensate + * for the leaf partitions in the partition subtrees of all the left + * siblings that are partitioned, which, fortunately it suffices to look + * at the rightmost such node. + */ + Assert(prev != NULL); + if (!node || cur_idx < node->index) + return prev == parent ? prev->offset + cur_idx + : prev->offset + prev->num_leaf_parts + + (cur_idx - prev->index - 1); + + /* + * Must recurse because it turns out that the partition at cur_idx is + * partitioned itself + */ + Assert (node != NULL && node->index == cur_idx); + + return get_partition_for_tuple(node, slot, estate, failed_at); +} + +/* + * list_partition_for_tuple + * Find the list partition for a tuple (arg 'value' contains the + * list partition key of the original tuple) + * + * Returns -1 if none found. + */ +static int +list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull) +{ + PartitionListInfo listinfo; + int found; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_LIST); + listinfo = pdesc->boundinfo->bounds.lists; + + if (isnull && listinfo.has_null) + return listinfo.null_index; + else if (!isnull) + { + found = partition_list_values_bsearch(key, + listinfo.values, + listinfo.nvalues, + value); + if (found >= 0) + return listinfo.indexes[found]; + } + + /* Control reaches here if isnull and !listinfo->has_null */ + return -1; +} + +/* + * range_partition_for_tuple + * Get the index of the range partition for a tuple (arg 'tuple' + * actually contains the range partition key of the original + * tuple) + * + * Returns -1 if none found. + */ +static int +range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, Datum *tuple) +{ + int offset; + PartitionRangeInfo rangeinfo; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_RANGE); + rangeinfo = pdesc->boundinfo->bounds.ranges; + + offset = partition_rbound_bsearch(key, + rangeinfo.bounds, rangeinfo.nbounds, + tuple, partition_rbound_datum_cmp, + true, NULL); + + /* + * Offset returned is such that the bound at offset is found to be less + * or equal with the tuple. That is, the tuple belongs to the partition + * with the rangeinfo.bounds[offset] as the lower bound and + * rangeinfo.bounds[offset+1] as the upper bound, provided the latter is + * indeed an upper (!lower) bound. If it turns out otherwise, the + * corresponding index will be -1, which means no valid partition exists. + */ + return rangeinfo.indexes[offset+1]; +} + /* List partition related support functions */ /* @@ -1772,6 +2053,40 @@ partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg) } /* + * Return whether bound <=, =, >= partition key of tuple + * + * The 3rd argument is void * so that it can be used with + * partition_rbound_bsearch() + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg) +{ + Datum *datums1 = bound->datums, + *datums2 = (Datum *) arg; + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (bound->infinite[i]) + return bound->lower ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + datums1[i], datums2[i])); + if (cmpval != 0) + break; + } + + /* If datums are equal and this is an upper bound, tuple > bound */ + if (cmpval == 0 && !bound->lower) + return -1; + + return cmpval; +} + +/* * Return whether two range bounds are equal simply by comparing datums */ static bool diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..cb0ca0e 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -30,6 +30,7 @@ #include "commands/defrem.h" #include "commands/trigger.h" #include "executor/executor.h" +#include "foreign/fdwapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -161,6 +162,11 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionTreeNode partition_tree; /* partition node tree */ + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; + List *partition_fdw_priv_lists; + int num_partitions; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1403,91 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + List *leaf_parts; + ListCell *cell; + int i; + int num_leaf_parts; + ResultRelInfo *leaf_part_rri; + PlannerInfo *root = makeNode(PlannerInfo); /* mostly dummy */ + ModifyTable *plan = makeNode(ModifyTable); /* ditto */ + RangeTblEntry *fdw_rte = makeNode(RangeTblEntry); /* ditto */ + List *fdw_private_lists = NIL; + + /* Form the partition node tree and lock partitions */ + cstate->partition_tree = RelationGetPartitionTreeNode(rel, + RowExclusiveLock); + leaf_parts = get_leaf_partition_oids(cstate->partition_tree); + num_leaf_parts = list_length(leaf_parts); + + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + /* For use below, iff a partition found to be a foreign table */ + plan->operation = CMD_INSERT; + plan->plans = list_make1(makeNode(Result)); + fdw_rte->rtekind = RTE_RELATION; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + root->parse = makeNode(Query); + root->parse->rtable = list_make1(fdw_rte); + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation part_rel; + + part_rel = heap_open(lfirst_oid(cell), RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + /* Special dance for foreign tables */ + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + fdw_rte->relid = RelationGetRelid(part_rel); + fdw_private = leaf_part_rri->ri_FdwRoutine->PlanForeignModify(root, + plan, + 1, + 0); + fdw_private_lists = lappend(fdw_private_lists, fdw_private); + } + + if (!equalTupleDescs(tupDesc, RelationGetDescr(part_rel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + + cstate->partition_fdw_priv_lists = fdw_private_lists; + } } else { @@ -2255,6 +2346,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2373,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2484,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2512,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->partition_tree != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2439,10 +2534,46 @@ CopyFrom(CopyState cstate) */ ExecBSInsertTriggers(estate, resultRelInfo); + /* Initialize FDW partition insert plans */ + if (cstate->partition_tree) + { + int i, + j; + List *fdw_private_lists = cstate->partition_fdw_priv_lists; + ModifyTableState *mtstate = makeNode(ModifyTableState); + ResultRelInfo *leaf_part_rri; + + /* Mostly dummy containing enough state for BeginForeignModify */ + mtstate->ps.state = estate; + mtstate->operation = CMD_INSERT; + + j = 0; + leaf_part_rri = cstate->partitions; + for (i = 0; i < cstate->num_partitions; i++) + { + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + Assert(fdw_private_lists); + fdw_private = list_nth(fdw_private_lists, j++); + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, 0); + } + leaf_part_rri++; + } + } + values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2625,50 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition to heap_insert the tuple into */ + if (cstate->partition_tree) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + cstate->partition_tree, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < cstate->num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding + * to the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = cstate->partitions + leaf_part_index; + + /* + * For ExecInsertIndexTuples() to work on the partition's indexes + */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the + * partition rowtype. + */ + map = cstate->partition_tupconv_maps[leaf_part_index]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2523,7 +2698,16 @@ CopyFrom(CopyState cstate) resultRelInfo->ri_PartitionCheck) ExecConstraints(resultRelInfo, slot, estate); - if (useHeapMultiInsert) + if (resultRelInfo->ri_FdwRoutine) + { + resultRelInfo->ri_FdwRoutine->ExecForeignInsert(estate, + resultRelInfo, + slot, + NULL); + /* AFTER ROW INSERT Triggers */ + ExecARInsertTriggers(estate, resultRelInfo, tuple, NIL); + } + else if (useHeapMultiInsert) { /* Add this tuple to the tuple buffer */ if (nBufferedTuples == 0) @@ -2553,7 +2737,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2762,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2781,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); @@ -2614,6 +2806,28 @@ CopyFrom(CopyState cstate) ExecCloseIndices(resultRelInfo); + /* + * Close all partitions and indices thereof, also clean up any + * state of FDW modify plans. + */ + if (cstate->partition_tree) + { + int i; + + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(estate, + resultRelInfo); + } + } + FreeExecutorState(estate); /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 3b72ae3..6478211 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1293,6 +1293,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index ea3f59a..3b4dd38 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2996,3 +3001,53 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +/* + * ExecFindPartition -- Find a leaf partition in the partition tree rooted + * at parent, for the heap tuple contained in *slot + * + * estate must be non-NULL; we'll need it to compute any expressions in the + * partition key(s) + * + * If no leaf partition is found, this routine errors out with the appropriate + * error message, else it returns the leaf partition sequence number returned + * by get_partition_for_tuple() unchanged. + */ +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionTreeNode parent, + TupleTableSlot *slot, EState *estate) +{ + int result; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + result = get_partition_for_tuple(parent, slot, estate, &failed_at); + + if (result < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return result; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..3523a2d 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,50 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + /* Determine the partition to heap_insert the tuple into */ + if (mtstate->mt_partition_tree) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_tree, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + leaf_part_index; + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, false); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +556,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1616,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1707,100 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + /* Form the partition node tree and lock partitions */ + mtstate->mt_partition_tree = RelationGetPartitionTreeNode(rel, + RowExclusiveLock); + leaf_parts = get_leaf_partition_oids(mtstate->mt_partition_tree); + num_leaf_parts = list_length(leaf_parts); + + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (leaf_part_rri->ri_FdwRoutine) + { + ListCell *lc; + List *fdw_private; + int k; + + /* + * There are as many fdw_private's in fdwPrivLists as there + * are FDW partitions, but we must find the intended for the + * this foreign table. + */ + k = 0; + foreach(lc, node->fdwPartitionOids) + { + if (lfirst_oid(lc) == ftoid) + break; + k++; + } + + Assert(k < num_leaf_parts); + fdw_private = (List *) list_nth(node->fdwPrivLists, k); + Assert(fdw_private != NIL); + + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, + eflags); + j++; + } + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2118,20 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(node->ps.state, + resultRelInfo); + } + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 28d0036..470dee7 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -188,6 +188,7 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(withCheckOptionLists); COPY_NODE_FIELD(returningLists); COPY_NODE_FIELD(fdwPrivLists); + COPY_NODE_FIELD(fdwPartitionOids); COPY_BITMAPSET_FIELD(fdwDirectModifyPlans); COPY_NODE_FIELD(rowMarks); COPY_SCALAR_FIELD(epqParam); diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 0d858f5..5c8eced 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -340,6 +340,7 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(withCheckOptionLists); WRITE_NODE_FIELD(returningLists); WRITE_NODE_FIELD(fdwPrivLists); + WRITE_NODE_FIELD(fdwPartitionOids); WRITE_BITMAPSET_FIELD(fdwDirectModifyPlans); WRITE_NODE_FIELD(rowMarks); WRITE_INT_FIELD(epqParam); diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c587d4e..ddd78c7 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -1495,6 +1495,7 @@ _readModifyTable(void) READ_NODE_FIELD(withCheckOptionLists); READ_NODE_FIELD(returningLists); READ_NODE_FIELD(fdwPrivLists); + READ_NODE_FIELD(fdwPartitionOids); READ_BITMAPSET_FIELD(fdwDirectModifyPlans); READ_NODE_FIELD(rowMarks); READ_INT_FIELD(epqParam); diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index ad49674..caf2f44 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -22,6 +22,7 @@ #include "access/stratnum.h" #include "access/sysattr.h" #include "catalog/pg_class.h" +#include "catalog/pg_inherits_fn.h" #include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/extensible.h" @@ -6159,6 +6160,81 @@ make_modifytable(PlannerInfo *root, node->fdwPrivLists = fdw_private_list; node->fdwDirectModifyPlans = direct_modify_plans; + /* Collect insert plans for all FDW-managed partitions */ + if (node->operation == CMD_INSERT) + { + RangeTblEntry *rte, + **saved_simple_rte_array; + List *partition_oids, + *fdw_partition_oids; + + Assert(list_length(resultRelations) == 1); + rte = rt_fetch(linitial_int(resultRelations), root->parse->rtable); + Assert(rte->rtekind == RTE_RELATION); + + if (rte->relkind != RELKIND_PARTITIONED_TABLE) + return node; + + partition_oids = find_all_inheritors(rte->relid, RowExclusiveLock, + NULL); + + /* Discard any previous content which is useless anyway */ + fdw_private_list = NIL; + fdw_partition_oids = NIL; + + /* + * To force the FDW driver fetch the intended RTE, we need to temporarily + * switch root->simple_rte_array to the one holding only that RTE at a + * designated index, for every foreign table. + */ + saved_simple_rte_array = root->simple_rte_array; + root->simple_rte_array = (RangeTblEntry **) + palloc0(2 * sizeof(RangeTblEntry *)); + foreach(lc, partition_oids) + { + Oid myoid = lfirst_oid(lc); + FdwRoutine *fdwroutine; + List *fdw_private; + + /* + * We are only interested in foreign tables. Note that this will + * also eliminate any partitioned tables since foreign tables can + * only ever be leaf partitions. + */ + if (get_rel_relkind(myoid) != RELKIND_FOREIGN_TABLE) + continue; + + fdw_partition_oids = lappend_oid(fdw_partition_oids, myoid); + + fdwroutine = GetFdwRoutineByRelId(myoid); + if (fdwroutine && fdwroutine->PlanForeignModify) + { + RangeTblEntry *fdw_rte; + + fdw_rte = copyObject(rte); + fdw_rte->relid = myoid; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + + /* + * Assumes PlanForeignModify() uses planner_rt_fetch(), also, + * see the above comment. + */ + root->simple_rte_array[1] = fdw_rte; + + fdw_private = fdwroutine->PlanForeignModify(root, node, 1, 0); + } + else + fdw_private = NIL; + + fdw_private_list = lappend(fdw_private_list, fdw_private); + } + + root->simple_rte_array = saved_simple_rte_array; + + node->fdwPrivLists = fdw_private_list; + node->fdwPartitionOids = fdw_partition_oids; + } + return node; } diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 672183b..e1ccaf8 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -50,4 +52,9 @@ extern List *RelationGetPartitionQual(Relation rel, bool recurse); /* For tuple routing */ extern PartitionTreeNode RelationGetPartitionTreeNode(Relation rel, int lockmode); extern List *get_leaf_partition_oids(PartitionTreeNode parent); + +extern int get_partition_for_tuple(PartitionTreeNode parent, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..fadc402 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionTreeNode parent, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..06675bc 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,15 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionTreeNodeData *mt_partition_tree; + /* Partition descriptor node tree */ + ResultRelInfo *mt_partitions; /* Per leaf partition target + * relations */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per leaf partition + * tuple conversion map */ + int mt_num_partitions; /* Number of leaf partition target + * relations in the above array */ } ModifyTableState; /* ---------------- diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index e2fbc7d..d82222c 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -189,6 +189,7 @@ typedef struct ModifyTable List *returningLists; /* per-target-table RETURNING tlists */ List *fdwPrivLists; /* per-target-table FDW private data lists */ Bitmapset *fdwDirectModifyPlans; /* indices of FDW DM plans */ + List *fdwPartitionOids; /* OIDs of FDW-managed partition */ List *rowMarks; /* PlanRowMarks (non-locking only) */ int epqParam; /* ID of Param for EvalPlanQual re-eval */ OnConflictAction onConflictAction; /* ON CONFLICT action */ diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 9ae6b09..d5dcb59 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -227,6 +227,58 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index b6e821e..fbd30d9 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,31 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------1C84B3F970C18697850BA485 Content-Type: text/x-diff; name="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-15.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-15"; filename*1=".patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH 7/8] Tuple routing for partitioned tables. @ 2016-07-27 07:59 amit <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: amit @ 2016-07-27 07:59 UTC (permalink / raw) Both COPY FROM and INSERT are covered by this commit. Routing to foreing partitions is not supported at the moment. --- src/backend/catalog/partition.c | 289 +++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 151 ++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 58 ++++++- src/backend/executor/nodeModifyTable.c | 130 ++++++++++++++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 6 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 8 + src/test/regress/expected/insert.out | 52 ++++++ src/test/regress/sql/insert.sql | 25 +++ 11 files changed, 728 insertions(+), 6 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index 03c7fa1..d758500 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -161,6 +161,18 @@ static Oid get_partition_operator(PartitionKey key, int col, StrategyNumber stra static List *generate_partition_qual(Relation rel, bool recurse); +/* Support get_partition_for_tuple() */ +static PartitionKeyExecInfo *BuildPartitionKeyExecInfo(Relation rel); +static void FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); +static int list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull); +static int range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum *tuple); + /* List partition related support functions */ static bool equal_list_info(PartitionKey key, PartitionListInfo *l1, PartitionListInfo *l2); @@ -174,6 +186,8 @@ static PartitionRangeBound *copy_range_bound(PartitionKey key, PartitionRangeBou static bool equal_range_info(PartitionKey key, PartitionRangeInfo *r1, PartitionRangeInfo *r2); static int32 partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg); +static int32 partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg); static bool partition_rbound_eq(PartitionKey key, PartitionRangeBound *b1, PartitionRangeBound *b2); typedef int32 (*partition_rbound_bsearch_cmp_fn) (PartitionKey, @@ -988,7 +1002,7 @@ RelationGetPartitionDispatchInfo(Relation rel, int lockmode, pd[i] = (PartitionDispatch) palloc(sizeof(PartitionDispatchData)); pd[i]->relid = RelationGetRelid(partrel); - pd[i]->pkinfo = NULL; + pd[i]->pkinfo = BuildPartitionKeyExecInfo(partrel); pd[i]->partdesc = partdesc; pd[i]->indexes = (int *) palloc(partdesc->nparts * sizeof(int)); heap_close(partrel, NoLock); @@ -1459,6 +1473,245 @@ generate_partition_qual(Relation rel, bool recurse) return result; } +/* + * BuildPartitionKeyExecInfo + * Construct a list of PartitionKeyExecInfo records for an open + * relation + * + * PartitionKeyExecInfo stores the information about the partition key + * that's needed when inserting tuples into a partitioned table; especially, + * partition key expression state if there are any expression columns in + * the partition key. Normally we build a PartitionKeyExecInfo for a + * partitioned table just once per command, and then use it for (potentially) + * many tuples. + * + */ +static PartitionKeyExecInfo * +BuildPartitionKeyExecInfo(Relation rel) +{ + PartitionKeyExecInfo *pkinfo; + + pkinfo = (PartitionKeyExecInfo *) palloc0(sizeof(PartitionKeyExecInfo)); + pkinfo->pi_Key = RelationGetPartitionKey(rel); + pkinfo->pi_ExpressionState = NIL; + + return pkinfo; +} + +/* ---------------- + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for the partition key + * of a tuple. + * + * pkinfo partition key execution info + * slot Heap tuple from which to extract partition key + * estate executor state for evaluating any partition key + * expressions (must be non-NULL) + * values Array of partition key Datums (output area) + * isnull Array of is-null indicators (output area) + * + * the ecxt_scantuple slot of estate's per-tuple expr context must point to + * the heap tuple passed in. + * ---------------- + */ +static void +FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pkinfo->pi_Key->partexprs != NIL && pkinfo->pi_ExpressionState == NIL) + { + /* Check caller has set up context correctly */ + Assert(estate != NULL && + GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + + /* First time through, set up expression evaluation state */ + pkinfo->pi_ExpressionState = (List *) + ExecPrepareExpr((Expr *) pkinfo->pi_Key->partexprs, + estate); + } + + partexpr_item = list_head(pkinfo->pi_ExpressionState); + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + { + AttrNumber keycol = pkinfo->pi_Key->partattrs[i]; + Datum pkDatum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + pkDatum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + pkDatum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = pkDatum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Finds a leaf partition for tuple contained in *slot + * + * Returned value is the sequence number of the leaf partition thus found, + * or -1 if no leaf partition is found for the tuple. *failed_at is set + * to the OID of the partitioned table whose partition was not found in + * the latter case. + */ +int +get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + PartitionDispatch parent; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int cur_idx; + int i; + + /* start with the root partitioned table */ + parent = pd[0]; + while(1) + { + PartitionDesc partdesc = parent->partdesc; + PartitionKeyExecInfo *pkinfo = parent->pkinfo; + + /* Quick exit */ + if (partdesc->nparts == 0) + { + *failed_at = parent->relid; + return -1; + } + + /* Extract partition key from tuple */ + FormPartitionKeyDatum(pkinfo, slot, estate, values, isnull); + + switch (pkinfo->pi_Key->strategy) + { + case PARTITION_STRATEGY_LIST: + cur_idx = list_partition_for_tuple(pkinfo->pi_Key, partdesc, + values[0], isnull[0]); + break; + + case PARTITION_STRATEGY_RANGE: + /* Disallow nulls in the partition key of the tuple */ + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + if (isnull[i]) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key of row contains null"))); + + cur_idx = range_partition_for_tuple(pkinfo->pi_Key, partdesc, + values); + break; + } + + /* + * cur_idx < 0 means we failed to find a partition of this parent. + * cur_idx >= 0 means we either found the leaf partition we have been + * looking for, or the next parent to find a partition of. + */ + if (cur_idx < 0) + { + *failed_at = parent->relid; + return -1; + } + else if (parent->indexes[cur_idx] < 0) + parent = pd[-parent->indexes[cur_idx]]; + else + break; + } + + return parent->indexes[cur_idx]; +} + +/* + * list_partition_for_tuple + * Find the list partition for a tuple (arg 'value' contains the + * list partition key of the original tuple) + * + * Returns -1 if none found. + */ +static int +list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull) +{ + PartitionListInfo listinfo; + int found; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_LIST); + listinfo = pdesc->boundinfo->bounds.lists; + + if (isnull && listinfo.has_null) + return listinfo.null_index; + else if (!isnull) + { + found = partition_list_values_bsearch(key, + listinfo.values, + listinfo.nvalues, + value); + if (found >= 0) + return listinfo.indexes[found]; + } + + /* Control reaches here if isnull and !listinfo->has_null */ + return -1; +} + +/* + * range_partition_for_tuple + * Get the index of the range partition for a tuple (arg 'tuple' + * actually contains the range partition key of the original + * tuple) + * + * Returns -1 if none found. + */ +static int +range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, Datum *tuple) +{ + int offset; + PartitionRangeInfo rangeinfo; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_RANGE); + rangeinfo = pdesc->boundinfo->bounds.ranges; + + offset = partition_rbound_bsearch(key, + rangeinfo.bounds, rangeinfo.nbounds, + tuple, partition_rbound_datum_cmp, + true, NULL); + + /* + * Offset returned is such that the bound at offset is found to be less + * or equal with the tuple. That is, the tuple belongs to the partition + * with the rangeinfo.bounds[offset] as the lower bound and + * rangeinfo.bounds[offset+1] as the upper bound, provided the latter is + * indeed an upper (!lower) bound. If it turns out otherwise, the + * corresponding index will be -1, which means no valid partition exists. + */ + return rangeinfo.indexes[offset+1]; +} + /* List partition related support functions */ /* @@ -1704,6 +1957,40 @@ partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg) } /* + * Return whether bound <=, =, >= partition key of tuple + * + * The 3rd argument is void * so that it can be used with + * partition_rbound_bsearch() + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg) +{ + Datum *datums1 = bound->datums, + *datums2 = (Datum *) arg; + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (bound->infinite[i]) + return bound->lower ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + datums1[i], datums2[i])); + if (cmpval != 0) + break; + } + + /* If datums are equal and this is an upper bound, tuple > bound */ + if (cmpval == 0 && !bound->lower) + return -1; + + return cmpval; +} + +/* * Return whether two range bounds are equal simply by comparing datums */ static bool diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..7d76ead 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -161,6 +161,10 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionDispatch *partition_dispatch_info; + int num_partitions; + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1401,67 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + List *leaf_parts; + ListCell *cell; + int i, + num_leaf_parts; + ResultRelInfo *leaf_part_rri; + + /* Get the tuple-routing information and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + num_leaf_parts = list_length(leaf_parts); + cstate->partition_dispatch_info = pd; + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) palloc(num_leaf_parts * + sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation partrel; + + /* + * All partitions locked above; will be closed after CopyFrom is + * finished. + */ + partrel = heap_open(lfirst_oid(cell), NoLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(partrel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + partrel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(tupDesc, RelationGetDescr(partrel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(partrel), + gettext_noop("could not convert row type")); + leaf_part_rri++; + i++; + } + } } else { @@ -2255,6 +2320,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2347,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2458,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2486,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->partition_dispatch_info != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2442,7 +2511,11 @@ CopyFrom(CopyState cstate) values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2567,56 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition to heap_insert the tuple into */ + if (cstate->partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + cstate->partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < cstate->num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding + * to the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = cstate->partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* + * For ExecInsertIndexTuples() to work on the partition's indexes + */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the + * partition rowtype. + */ + map = cstate->partition_tupconv_maps[leaf_part_index]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2553,7 +2676,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2701,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2720,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); @@ -2614,6 +2745,20 @@ CopyFrom(CopyState cstate) ExecCloseIndices(resultRelInfo); + /* Close all partitions and indices thereof */ + if (cstate->partition_dispatch_info) + { + int i; + + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + } + FreeExecutorState(estate); /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 3b72ae3..6478211 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1293,6 +1293,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index c7a6347..54fb771 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2990,3 +2995,52 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +/* + * ExecFindPartition -- Find a leaf partition in the partition tree rooted + * at parent, for the heap tuple contained in *slot + * + * estate must be non-NULL; we'll need it to compute any expressions in the + * partition key(s) + * + * If no leaf partition is found, this routine errors out with the appropriate + * error message, else it returns the leaf partition sequence number returned + * by get_partition_for_tuple() unchanged. + */ +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, + TupleTableSlot *slot, EState *estate) +{ + int result; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + result = get_partition_for_tuple(pd, slot, estate, &failed_at); + if (result < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return result; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..aa1d8b9 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,56 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + /* Determine the partition to heap_insert the tuple into */ + if (mtstate->mt_partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, false); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +562,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1622,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1713,69 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + /* Form the partition node tree and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + mtstate->mt_partition_dispatch_info = pd; + num_leaf_parts = list_length(leaf_parts); + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2093,15 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + /* * Free the exprcontext */ diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index fb43cc1..1d231a3 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -50,4 +52,8 @@ extern List *RelationGetPartitionQual(Relation rel, bool recurse); /* For tuple routing */ extern PartitionDispatch *RelationGetPartitionDispatchInfo(Relation rel, int lockmode, List **leaf_part_oids); +extern int get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..b4d09f9 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..606cb21 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,13 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionDispatchData **mt_partition_dispatch_info; + /* Tuple-routing support info */ + int mt_num_partitions; /* Number of members in the + * following arrays */ + ResultRelInfo *mt_partitions; /* Per partition result relation */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per partition tuple conversion map */ } ModifyTableState; /* ---------------- diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 9ae6b09..d5dcb59 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -227,6 +227,58 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index b6e821e..fbd30d9 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,31 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------38C0D572AC793CAA7276C062 Content-Type: text/x-diff; name="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-16.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-16"; filename*1=".patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH 7/8] Tuple routing for partitioned tables. @ 2016-07-27 07:59 amit <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: amit @ 2016-07-27 07:59 UTC (permalink / raw) Both COPY FROM and INSERT. --- src/backend/catalog/partition.c | 327 ++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 203 +++++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 49 +++++- src/backend/executor/nodeModifyTable.c | 156 +++++++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 1 + src/backend/optimizer/plan/createplan.c | 77 +++++++ src/backend/optimizer/util/plancat.c | 13 ++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 7 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 10 + src/include/nodes/plannodes.h | 1 + src/include/optimizer/plancat.h | 1 + src/test/regress/expected/insert.out | 52 +++++ src/test/regress/sql/insert.sql | 25 +++ 18 files changed, 932 insertions(+), 7 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index a5800fb..e7e0c7c 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -182,6 +182,18 @@ static List *generate_partition_qual(Relation rel, bool recurse); static PartitionTreeNode GetPartitionTreeNodeRecurse(Relation rel, int offset); static int get_leaf_partition_count(PartitionTreeNode ptnode); +/* Support get_partition_for_tuple() */ +static PartitionKeyExecInfo *BuildPartitionKeyExecInfo(Relation rel); +static void FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); +static int list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull); +static int range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum *tuple); + /* List partition related support functions */ static bool equal_list_info(PartitionKey key, PartitionListInfo *l1, PartitionListInfo *l2); @@ -195,6 +207,8 @@ static PartitionRangeBound *copy_range_bound(PartitionKey key, PartitionRangeBou static bool equal_range_info(PartitionKey key, PartitionRangeInfo *r1, PartitionRangeInfo *r2); static int32 partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg); +static int32 partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg); static bool partition_rbound_eq(PartitionKey key, PartitionRangeBound *b1, PartitionRangeBound *b2); typedef int32 (*partition_rbound_bsearch_cmp_fn) (PartitionKey, @@ -1438,7 +1452,7 @@ GetPartitionTreeNodeRecurse(Relation rel, int offset) /* First build our own node */ parent = (PartitionTreeNode) palloc0(sizeof(PartitionTreeNodeData)); - parent->pkinfo = NULL; + parent->pkinfo = BuildPartitionKeyExecInfo(rel); parent->pdesc = RelationGetPartitionDesc(rel); parent->relid = RelationGetRelid(rel); parent->offset = offset; @@ -1522,6 +1536,283 @@ get_leaf_partition_count(PartitionTreeNode ptnode) return result; } +/* + * BuildPartitionKeyExecInfo + * Construct a list of PartitionKeyExecInfo records for an open + * relation + * + * PartitionKeyExecInfo stores the information about the partition key + * that's needed when inserting tuples into a partitioned table; especially, + * partition key expression state if there are any expression columns in + * the partition key. Normally we build a PartitionKeyExecInfo for a + * partitioned table just once per command, and then use it for (potentially) + * many tuples. + * + */ +static PartitionKeyExecInfo * +BuildPartitionKeyExecInfo(Relation rel) +{ + PartitionKeyExecInfo *pkinfo; + + pkinfo = (PartitionKeyExecInfo *) palloc0(sizeof(PartitionKeyExecInfo)); + pkinfo->pi_Key = RelationGetPartitionKey(rel); + pkinfo->pi_ExpressionState = NIL; + + return pkinfo; +} + +/* + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for partition key columns + */ +static void +FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pkinfo->pi_Key->partexprs != NIL && pkinfo->pi_ExpressionState == NIL) + { + /* First time through, set up expression evaluation state */ + pkinfo->pi_ExpressionState = (List *) + ExecPrepareExpr((Expr *) pkinfo->pi_Key->partexprs, + estate); + /* Check caller has set up context correctly */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + } + + partexpr_item = list_head(pkinfo->pi_ExpressionState); + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + { + AttrNumber keycol = pkinfo->pi_Key->partattrs[i]; + Datum pkDatum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + pkDatum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + pkDatum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = pkDatum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Recursively finds the "leaf" partition for tuple + * + * Returns -1 if no partition is found and sets *failed_at to the OID of + * the partitioned table whose partition was not found. + */ +int +get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + Relation partRel; + PartitionKeyExecInfo *pkinfo = ptnode->pkinfo; + PartitionTreeNode node; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int i; + int index; + + /* Guard against stack overflow due to overly deep partition tree */ + check_stack_depth(); + + if (ptnode->pdesc->nparts == 0) + { + *failed_at = ptnode->relid; + return -1; + } + + /* Extract partition key from tuple */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + FormPartitionKeyDatum(pkinfo, slot, estate, values, isnull); + + /* Disallow nulls, if range partition key */ + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + if (isnull[i] && pkinfo->pi_Key->strategy == PARTITION_STRATEGY_RANGE) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key contains null"))); + + switch (pkinfo->pi_Key->strategy) + { + case PARTITION_STRATEGY_LIST: + index = list_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values[0], isnull[0]); + break; + + case PARTITION_STRATEGY_RANGE: + index = range_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values); + break; + } + + /* No partition found at this level */ + if (index < 0) + { + *failed_at = ptnode->relid; + return index; + } + + partRel = heap_open(ptnode->pdesc->oids[index], NoLock); + + /* Don't recurse if the index'th partition is a leaf partition. */ + if (partRel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) + { + PartitionTreeNode prev; + + /* + * Index returned above considers only this parent (of which partRel + * is a partition). We however want to return the index across the + * the whole partition tree. If partRel is the leftmost partition + * of parent (index = 0) or if none of the left siblings are + * partitioned tables, we can simply return parent's offset plus + * partRel's index as the result. Otherwise, we find the node + * corresponding to the rightmost partitioned table among partRel's + * left siblings and use the information therein. + */ + prev = node = ptnode->downlink; + if (node && node->index < index) + { + /* + * Find the partition tree node such that its index value is the + * greatest value less than the above returned index. + */ + while (node) + { + if (node->index > index) + { + node = prev; + break; + } + + prev = node; + node = node->next; + } + + if (!node) + node = prev; + Assert (node != NULL); + + index = node->offset + node->num_leaf_parts + + (index - node->index - 1); + } + else + /* + * The easy case where all of partRel's left siblings (if any) are + * leaf partitions. + */ + index = ptnode->offset + index; + + heap_close(partRel, NoLock); + return index; + } + + heap_close(partRel, NoLock); + + /* + * Need to recurse as the selected partition is a partitioned table + * itself. Locate the corresponding PartitionTreeNode to pass it down. + */ + node = ptnode->downlink; + while (node->next != NULL && node->index != index) + node = node->next; + Assert (node != NULL); + + return get_partition_for_tuple(node, slot, estate, failed_at); +} + +/* + * list_partition_for_tuple + * Find the list partition for a tuple (arg 'value' contains the + * list partition key of the original tuple) + * + * Returns -1 if none found. + */ +static int +list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull) +{ + PartitionListInfo listinfo; + int found; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_LIST); + listinfo = pdesc->boundinfo->bounds.lists; + + if (isnull && listinfo.has_null) + return listinfo.null_index; + else if (!isnull) + { + found = partition_list_values_bsearch(key, + listinfo.values, + listinfo.nvalues, + value); + if (found >= 0) + return listinfo.indexes[found]; + } + + /* Control reaches here if isnull and !listinfo->has_null */ + return -1; +} + +/* + * range_partition_for_tuple + * Get the index of the range partition for a tuple (arg 'tuple' + * actually contains the range partition key of the original + * tuple) + * + * Returns -1 if none found. + */ +static int +range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, Datum *tuple) +{ + int offset; + PartitionRangeInfo rangeinfo; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_RANGE); + rangeinfo = pdesc->boundinfo->bounds.ranges; + + offset = partition_rbound_bsearch(key, + rangeinfo.bounds, rangeinfo.nbounds, + tuple, partition_rbound_datum_cmp, + true, NULL); + + /* + * Offset returned is such that the bound at offset is found to be less + * or equal with the tuple. That is, the tuple belongs to the partition + * with the rangeinfo.bounds[offset] as the lower bound and + * rangeinfo.bounds[offset+1] as the upper bound, provided the latter is + * indeed an upper (!lower) bound. If it turns out otherwise, the + * corresponding index will be -1, which means no valid partition exists. + */ + return rangeinfo.indexes[offset+1]; +} + /* List partition related support functions */ /* @@ -1767,6 +2058,40 @@ partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg) } /* + * Return whether bound <=, =, >= partition key of tuple + * + * The 3rd argument is void * so that it can be used with + * partition_rbound_bsearch() + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg) +{ + Datum *datums1 = bound->datums, + *datums2 = (Datum *) arg; + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (bound->infinite[i]) + return bound->lower ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + datums1[i], datums2[i])); + if (cmpval != 0) + break; + } + + /* If datums are equal and this is an upper bound, tuple > bound */ + if (cmpval == 0 && !bound->lower) + return -1; + + return cmpval; +} + +/* * Return whether two range bounds are equal simply by comparing datums */ static bool diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..6fb376f 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -30,6 +30,7 @@ #include "commands/defrem.h" #include "commands/trigger.h" #include "executor/executor.h" +#include "foreign/fdwapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -161,6 +162,11 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionTreeNode ptnode; /* partition descriptor node tree */ + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; + List *partition_fdw_priv_lists; + int num_partitions; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1403,94 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + List *leaf_parts; + ListCell *cell; + int i; + int num_leaf_parts; + ResultRelInfo *leaf_part_rri; + PlannerInfo *root = makeNode(PlannerInfo); /* mostly dummy */ + Query *parse = makeNode(Query); /* ditto */ + ModifyTable *plan = makeNode(ModifyTable); /* ditto */ + RangeTblEntry *fdw_rte = makeNode(RangeTblEntry); /* ditto */ + List *fdw_private_lists = NIL; + + cstate->ptnode = RelationGetPartitionTreeNode(rel); + leaf_parts = get_leaf_partition_oids(cstate->ptnode); + num_leaf_parts = list_length(leaf_parts); + + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + /* For use below, iff a partition found to be a foreign table */ + plan->operation = CMD_INSERT; + plan->plans = list_make1(makeNode(Result)); + fdw_rte->rtekind = RTE_RELATION; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + parse->rtable = list_make1(fdw_rte); + root->parse = parse; + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation part_rel; + + part_rel = heap_open(lfirst_oid(cell), RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + /* Special dance for foreign tables */ + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + fdw_rte->relid = RelationGetRelid(part_rel); + fdw_private = leaf_part_rri->ri_FdwRoutine->PlanForeignModify(root, + plan, + 1, + 0); + fdw_private_lists = lappend(fdw_private_lists, fdw_private); + } + + if (!equalTupleDescs(tupDesc, RelationGetDescr(part_rel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + + cstate->partition_fdw_priv_lists = fdw_private_lists; + pfree(fdw_rte); + pfree(plan); + pfree(parse); + pfree(root); + } } else { @@ -1692,6 +1786,8 @@ ClosePipeToProgram(CopyState cstate) static void EndCopy(CopyState cstate) { + int i; + if (cstate->is_program) { ClosePipeToProgram(cstate); @@ -1705,6 +1801,23 @@ EndCopy(CopyState cstate) cstate->filename))); } + /* Close all partitions and indices thereof */ + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + /* XXX - EState not handy here to pass to EndForeignModify() */ + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(NULL, resultRelInfo); + + if (cstate->partition_tupconv_maps[i]) + pfree(cstate->partition_tupconv_maps[i]); + } + MemoryContextDelete(cstate->copycontext); pfree(cstate); } @@ -2255,6 +2368,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2395,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2506,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2534,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->ptnode != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2439,10 +2556,46 @@ CopyFrom(CopyState cstate) */ ExecBSInsertTriggers(estate, resultRelInfo); + /* Initialize FDW partition insert plans */ + if (cstate->ptnode) + { + int i, + j; + List *fdw_private_lists = cstate->partition_fdw_priv_lists; + ModifyTableState *mtstate = makeNode(ModifyTableState); + ResultRelInfo *leaf_part_rri; + + /* Mostly dummy containing enough state for BeginForeignModify */ + mtstate->ps.state = estate; + mtstate->operation = CMD_INSERT; + + j = 0; + leaf_part_rri = cstate->partitions; + for (i = 0; i < cstate->num_partitions; i++) + { + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + Assert(fdw_private_lists); + fdw_private = list_nth(fdw_private_lists, j++); + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, 0); + } + leaf_part_rri++; + } + } + values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2647,31 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition */ + saved_resultRelInfo = resultRelInfo; + if (cstate->ptnode) + { + int i_leaf_partition; + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + cstate->ptnode, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < cstate->num_partitions); + + resultRelInfo = cstate->partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = cstate->partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2523,7 +2701,16 @@ CopyFrom(CopyState cstate) resultRelInfo->ri_PartitionCheck) ExecConstraints(resultRelInfo, slot, estate); - if (useHeapMultiInsert) + if (resultRelInfo->ri_FdwRoutine) + { + resultRelInfo->ri_FdwRoutine->ExecForeignInsert(estate, + resultRelInfo, + slot, + NULL); + /* AFTER ROW INSERT Triggers */ + ExecARInsertTriggers(estate, resultRelInfo, tuple, NIL); + } + else if (useHeapMultiInsert) { /* Add this tuple to the tuple buffer */ if (nBufferedTuples == 0) @@ -2553,7 +2740,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2765,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2784,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 3b72ae3..6478211 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1293,6 +1293,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index ea3f59a..b5125a8 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2996,3 +3001,43 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionTreeNode ptnode, + TupleTableSlot *slot, EState *estate) +{ + int i_leaf_partition; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + i_leaf_partition = get_partition_for_tuple(ptnode, slot, estate, + &failed_at); + + if (i_leaf_partition < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return i_leaf_partition; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..44801b8 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,45 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + if (mtstate->mt_partition_tree_root) + { + int i_leaf_partition; + TupleConversionMap *map; + + /* + * Get the index of leaf partitions that we'll use to get the correct + * ResultRelInfo from mtstate->mt_partitions[]. + */ + i_leaf_partition = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_tree_root, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + i_leaf_partition; + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[i_leaf_partition]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, false); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +551,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1611,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1702,98 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + mtstate->mt_partition_tree_root = RelationGetPartitionTreeNode(rel); + leaf_parts = get_leaf_partition_oids(mtstate->mt_partition_tree_root); + num_leaf_parts = list_length(leaf_parts); + + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (leaf_part_rri->ri_FdwRoutine) + { + ListCell *lc; + List *fdw_private; + int k; + + /* + * There are as many fdw_private's in fdwPrivLists as there + * are FDW partitions, but we must find the intended for the + * this foreign table. + */ + k = 0; + foreach(lc, node->fdwPartitionOids) + { + if (lfirst_oid(lc) == ftoid) + break; + k++; + } + + Assert(k < num_leaf_parts); + fdw_private = (List *) list_nth(node->fdwPrivLists, k); + Assert(fdw_private != NIL); + + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, + eflags); + j++; + } + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2111,23 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(node->ps.state, + resultRelInfo); + + if (node->mt_partition_tupconv_maps[i]) + pfree(node->mt_partition_tupconv_maps[i]); + } + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 28d0036..470dee7 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -188,6 +188,7 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(withCheckOptionLists); COPY_NODE_FIELD(returningLists); COPY_NODE_FIELD(fdwPrivLists); + COPY_NODE_FIELD(fdwPartitionOids); COPY_BITMAPSET_FIELD(fdwDirectModifyPlans); COPY_NODE_FIELD(rowMarks); COPY_SCALAR_FIELD(epqParam); diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 0d858f5..5c8eced 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -340,6 +340,7 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(withCheckOptionLists); WRITE_NODE_FIELD(returningLists); WRITE_NODE_FIELD(fdwPrivLists); + WRITE_NODE_FIELD(fdwPartitionOids); WRITE_BITMAPSET_FIELD(fdwDirectModifyPlans); WRITE_NODE_FIELD(rowMarks); WRITE_INT_FIELD(epqParam); diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c587d4e..ddd78c7 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -1495,6 +1495,7 @@ _readModifyTable(void) READ_NODE_FIELD(withCheckOptionLists); READ_NODE_FIELD(returningLists); READ_NODE_FIELD(fdwPrivLists); + READ_NODE_FIELD(fdwPartitionOids); READ_BITMAPSET_FIELD(fdwDirectModifyPlans); READ_NODE_FIELD(rowMarks); READ_INT_FIELD(epqParam); diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index ad49674..29f40fe 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -22,6 +22,7 @@ #include "access/stratnum.h" #include "access/sysattr.h" #include "catalog/pg_class.h" +#include "catalog/pg_inherits_fn.h" #include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/extensible.h" @@ -6159,6 +6160,82 @@ make_modifytable(PlannerInfo *root, node->fdwPrivLists = fdw_private_list; node->fdwDirectModifyPlans = direct_modify_plans; + /* Collect insert plans for all FDW-managed partitions */ + if (node->operation == CMD_INSERT) + { + RangeTblEntry *rte, + **saved_simple_rte_array; + List *partition_oids, + *fdw_partition_oids; + + Assert(list_length(resultRelations) == 1); + rte = rt_fetch(linitial_int(resultRelations), root->parse->rtable); + Assert(rte->rtekind == RTE_RELATION); + + if (rte->relkind != RELKIND_PARTITIONED_TABLE) + return node; + + partition_oids = find_all_inheritors(rte->relid, NoLock, NULL); + + /* Discard any previous content which is useless anyway */ + fdw_private_list = NIL; + fdw_partition_oids = NIL; + + /* + * To force the FDW driver fetch the intended RTE, we need to temporarily + * switch root->simple_rte_array to the one holding only that RTE at a + * designated index, for every foreign table. + */ + saved_simple_rte_array = root->simple_rte_array; + root->simple_rte_array = (RangeTblEntry **) + palloc0(2 * sizeof(RangeTblEntry *)); + foreach(lc, partition_oids) + { + Oid myoid = lfirst_oid(lc); + FdwRoutine *fdwroutine; + List *fdw_private; + + /* + * We are only interested in foreign tables. Note that this will + * also eliminate any partitioned tables since foreign tables can + * only ever be leaf partitions. + */ + if (!oid_is_foreign_table(myoid)) + continue; + + fdw_partition_oids = lappend_oid(fdw_partition_oids, myoid); + + fdwroutine = GetFdwRoutineByRelId(myoid); + if (fdwroutine && fdwroutine->PlanForeignModify) + { + RangeTblEntry *fdw_rte; + + fdw_rte = copyObject(rte); + fdw_rte->relid = myoid; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + + /* + * Assumes PlanForeignModify() uses planner_rt_fetch(), also, + * see the above comment. + */ + root->simple_rte_array[1] = fdw_rte; + + fdw_private = fdwroutine->PlanForeignModify(root, node, 1, 0); + pfree(fdw_rte); + } + else + fdw_private = NIL; + + fdw_private_list = lappend(fdw_private_list, fdw_private); + } + + pfree(root->simple_rte_array); + root->simple_rte_array = saved_simple_rte_array; + + node->fdwPrivLists = fdw_private_list; + node->fdwPartitionOids = fdw_partition_oids; + } + return node; } diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index a2cbf14..e85ca0a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1715,3 +1715,16 @@ has_row_triggers(PlannerInfo *root, Index rti, CmdType event) heap_close(relation, NoLock); return result; } + +bool +oid_is_foreign_table(Oid relid) +{ + Relation rel; + char relkind; + + rel = heap_open(relid, NoLock); + relkind = rel->rd_rel->relkind; + heap_close(rel, NoLock); + + return relkind == RELKIND_FOREIGN_TABLE; +} diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 057b1e3..86cc598 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -50,4 +52,9 @@ extern List *RelationGetPartitionQual(Relation rel, bool recurse); /* For tuple routing */ extern PartitionTreeNode RelationGetPartitionTreeNode(Relation rel); extern List *get_leaf_partition_oids(PartitionTreeNode ptnode); + +extern int get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..c62946f 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..ce01008 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,15 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionTreeNodeData *mt_partition_tree_root; + /* Partition descriptor node tree */ + ResultRelInfo *mt_partitions; /* Per leaf partition target + * relations */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per leaf partition + * tuple conversion map */ + int mt_num_partitions; /* Number of leaf partition target + * relations in the above array */ } ModifyTableState; /* ---------------- diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index e2fbc7d..d82222c 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -189,6 +189,7 @@ typedef struct ModifyTable List *returningLists; /* per-target-table RETURNING tlists */ List *fdwPrivLists; /* per-target-table FDW private data lists */ Bitmapset *fdwDirectModifyPlans; /* indices of FDW DM plans */ + List *fdwPartitionOids; /* OIDs of FDW-managed partition */ List *rowMarks; /* PlanRowMarks (non-locking only) */ int epqParam; /* ID of Param for EvalPlanQual re-eval */ OnConflictAction onConflictAction; /* ON CONFLICT action */ diff --git a/src/include/optimizer/plancat.h b/src/include/optimizer/plancat.h index 125274e..fac606c 100644 --- a/src/include/optimizer/plancat.h +++ b/src/include/optimizer/plancat.h @@ -56,5 +56,6 @@ extern Selectivity join_selectivity(PlannerInfo *root, SpecialJoinInfo *sjinfo); extern bool has_row_triggers(PlannerInfo *root, Index rti, CmdType event); +extern bool oid_is_foreign_table(Oid relid); #endif /* PLANCAT_H */ diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 9ae6b09..d5dcb59 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -227,6 +227,58 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index b6e821e..fbd30d9 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,31 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------30A9D66B3BEA4523792AC3CB Content-Type: text/x-diff; name="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-14.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-14"; filename*1=".patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH 7/8] Tuple routing for partitioned tables. @ 2016-07-27 07:59 amit <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: amit @ 2016-07-27 07:59 UTC (permalink / raw) Both COPY FROM and INSERT. --- src/backend/catalog/partition.c | 327 ++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 205 +++++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 47 +++++- src/backend/executor/nodeModifyTable.c | 142 +++++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 1 + src/backend/optimizer/plan/createplan.c | 77 +++++++ src/backend/optimizer/util/plancat.c | 13 ++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 7 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 10 + src/include/nodes/plannodes.h | 1 + src/include/optimizer/plancat.h | 1 + src/test/regress/expected/insert.out | 59 ++++++- src/test/regress/sql/insert.sql | 28 +++ 18 files changed, 926 insertions(+), 9 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index ae9e945..373e9e5 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -181,6 +181,18 @@ static List *generate_partition_qual(Relation rel, bool recurse); static PartitionTreeNode GetPartitionTreeNodeRecurse(Relation rel, int offset); static int get_leaf_partition_count(PartitionTreeNode ptnode); +/* Support get_partition_for_tuple() */ +static PartitionKeyExecInfo *BuildPartitionKeyExecInfo(Relation rel); +static void FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); +static int list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull); +static int range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum *tuple); + /* List partition related support functions */ static bool equal_list_info(PartitionKey key, PartitionListInfo *l1, PartitionListInfo *l2); @@ -194,6 +206,8 @@ static PartitionRangeBound *copy_range_bound(PartitionKey key, PartitionRangeBou static bool equal_range_info(PartitionKey key, PartitionRangeInfo *r1, PartitionRangeInfo *r2); static int32 partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg); +static int32 partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg); static bool partition_rbound_eq(PartitionKey key, PartitionRangeBound *b1, PartitionRangeBound *b2); typedef int32 (*partition_rbound_bsearch_cmp_fn) (PartitionKey, @@ -1442,7 +1456,7 @@ GetPartitionTreeNodeRecurse(Relation rel, int offset) /* First build our own node */ parent = (PartitionTreeNode) palloc0(sizeof(PartitionTreeNodeData)); - parent->pkinfo = NULL; + parent->pkinfo = BuildPartitionKeyExecInfo(rel); parent->pdesc = RelationGetPartitionDesc(rel); parent->relid = RelationGetRelid(rel); parent->offset = offset; @@ -1526,6 +1540,284 @@ get_leaf_partition_count(PartitionTreeNode ptnode) return result; } +/* + * BuildPartitionKeyExecInfo + * Construct a list of PartitionKeyExecInfo records for an open + * relation + * + * PartitionKeyExecInfo stores the information about the partition key + * that's needed when inserting tuples into a partitioned table; especially, + * partition key expression state if there are any expression columns in + * the partition key. Normally we build a PartitionKeyExecInfo for a + * partitioned table just once per command, and then use it for (potentially) + * many tuples. + * + */ +static PartitionKeyExecInfo * +BuildPartitionKeyExecInfo(Relation rel) +{ + PartitionKeyExecInfo *pkinfo; + + pkinfo = (PartitionKeyExecInfo *) palloc0(sizeof(PartitionKeyExecInfo)); + pkinfo->pi_Key = RelationGetPartitionKey(rel); + pkinfo->pi_ExpressionState = NIL; + + return pkinfo; +} + +/* + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for partition key columns + */ +static void +FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pkinfo->pi_Key->partexprs != NIL && pkinfo->pi_ExpressionState == NIL) + { + /* First time through, set up expression evaluation state */ + pkinfo->pi_ExpressionState = (List *) + ExecPrepareExpr((Expr *) pkinfo->pi_Key->partexprs, + estate); + /* Check caller has set up context correctly */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + } + + partexpr_item = list_head(pkinfo->pi_ExpressionState); + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + { + AttrNumber keycol = pkinfo->pi_Key->partattrs[i]; + Datum pkDatum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + pkDatum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + pkDatum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = pkDatum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Recursively finds the "leaf" partition for tuple + * + * Returns -1 if no partition is found and sets *failed_at to the OID of + * the partitioned table whose partition was not found. + */ +int +get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + Relation partRel; + PartitionKeyExecInfo *pkinfo = ptnode->pkinfo; + PartitionTreeNode node; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int i; + int index; + + /* Guard against stack overflow due to overly deep partition tree */ + check_stack_depth(); + + if (ptnode->pdesc->nparts == 0) + { + *failed_at = ptnode->relid; + return -1; + } + + /* Extract partition key from tuple */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + FormPartitionKeyDatum(pkinfo, slot, estate, values, isnull); + + /* Disallow nulls, if range partition key */ + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + if (isnull[i] && pkinfo->pi_Key->strategy == PARTITION_STRATEGY_RANGE) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key contains null"))); + + switch (pkinfo->pi_Key->strategy) + { + case PARTITION_STRATEGY_LIST: + index = list_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values[0], isnull[0]); + break; + + case PARTITION_STRATEGY_RANGE: + index = range_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values); + break; + } + + /* No partition found at this level */ + if (index < 0) + { + *failed_at = ptnode->relid; + return index; + } + + partRel = heap_open(ptnode->pdesc->oids[index], NoLock); + + /* Don't recurse if the index'th partition is a leaf partition. */ + if (partRel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) + { + PartitionTreeNode prev; + + /* + * Index returned above is the array index within pdesc->parts[] of + * the parent rel, however, we want to return the leaf partition index + * across the whole partition tree. Note that some partitions within + * pdesc->parts[] may be partitioned themselves and hence stand for + * the leaf partitions in their partition subtrees. We would need to + * skip past the indexes of leaf partitions of all such partition + * subtrees if they are to left of the above returned index. In fact, + * finding the PartitionTreeNode of the rightmost subtree is enough + * since its offset counts the leaf partitions on its left including + * those of partition subtrees to its left. + */ + prev = node = ptnode->downlink; + if (node && node->index < index) + { + /* + * Find the partition tree node such that its index value is the + * greatest value less than the above returned index. + */ + while (node) + { + if (node->index > index) + { + node = prev; + break; + } + + prev = node; + node = node->next; + } + + if (!node) + node = prev; + Assert (node != NULL); + + index = node->offset + node->num_leaf_parts + + (index - node->index - 1); + } + else + /* + * The easy case where we don't have any partition subtree to the + * left of the index. + */ + index = ptnode->offset + index; + + heap_close(partRel, NoLock); + return index; + } + + heap_close(partRel, NoLock); + + /* + * Need to perform recursion as the selected partition is partitioned + * itself. Locate the PartitionTreeNode corresponding to the partition + * passing it down. + */ + node = ptnode->downlink; + while (node->next != NULL && node->index != index) + node = node->next; + Assert (node != NULL); + + return get_partition_for_tuple(node, slot, estate, failed_at); +} + +/* + * list_partition_for_tuple + * Find the list partition for a tuple + * + * Returns -1 if none found. + */ +static int +list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull) +{ + PartitionListInfo listinfo; + int found; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_LIST); + listinfo = pdesc->boundinfo->bounds.lists; + + if (isnull && listinfo.has_null) + return listinfo.null_index; + else if (!isnull) + { + found = partition_list_values_bsearch(key, + listinfo.values, + listinfo.nvalues, + value); + if (found >= 0) + return listinfo.indexes[found]; + } + + /* Control reaches here if isnull and !listinfo->has_null */ + return -1; +} + +/* + * range_partition_for_tuple + * Search the range partition for a range key ('values') + * + * Returns -1 if none found. + */ +static int +range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, Datum *tuple) +{ + int offset; + PartitionRangeInfo rangeinfo; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_RANGE); + rangeinfo = pdesc->boundinfo->bounds.ranges; + + offset = partition_rbound_bsearch(key, + rangeinfo.bounds, rangeinfo.nbounds, + tuple, partition_rbound_datum_cmp, + true, NULL); + + /* + * Offset returned is such that the bound at offset is found to be + * less or equal with the tuple. That is, the tuple belongs to the + * partition with the rangeinfo.bounds[offset] as the lower bound and + * rangeinfo.bounds[offset+1] as the upper bound, provided the latter + * is indeed marked !lower (that is, it's an upper bound). If it turns + * out that it is a lower bound then the corresponding index will be -1, + * which means no valid partition exists. + */ + return rangeinfo.indexes[offset+1]; +} + /* List partition related support functions */ /* @@ -1787,6 +2079,39 @@ partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg) } /* + * Return whether the passed in range bound <=, =, >= tuple specified in arg + * + * The 2nd argument is void * so that it can be used with + * partition_rbound_bsearch() + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg) +{ + Datum *datums1 = bound->datums, + *datums2 = (Datum *) arg; + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (bound->infinite[i]) + return bound->lower ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + datums1[i], datums2[i])); + if (cmpval != 0) + break; + } + + if (cmpval == 0 && !bound->inclusive) + return bound->lower ? 1 : -1; + + return cmpval; +} + +/* * Are two (consecutive) range bounds equal without distinguishing lower * and upper? */ diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 77d4dcb..91e4d12 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -30,6 +30,7 @@ #include "commands/defrem.h" #include "commands/trigger.h" #include "executor/executor.h" +#include "foreign/fdwapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -161,6 +162,11 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionTreeNode ptnode; /* partition descriptor node tree */ + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; + List *partition_fdw_priv_lists; + int num_partitions; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1403,94 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + List *leaf_part_oids; + ListCell *cell; + int i; + int num_leaf_parts; + ResultRelInfo *leaf_rel_rri; + PlannerInfo *root = makeNode(PlannerInfo); /* mostly dummy */ + Query *parse = makeNode(Query); /* ditto */ + ModifyTable *plan = makeNode(ModifyTable); /* ditto */ + RangeTblEntry *fdw_rte = makeNode(RangeTblEntry); /* ditto */ + List *fdw_private_lists = NIL; + + cstate->ptnode = RelationGetPartitionTreeNode(rel); + leaf_part_oids = get_leaf_partition_oids_v2(cstate->ptnode); + num_leaf_parts = list_length(leaf_part_oids); + + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + /* For use below, iff a partition found to be a foreign table */ + plan->operation = CMD_INSERT; + plan->plans = list_make1(makeNode(Result)); + fdw_rte->rtekind = RTE_RELATION; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + parse->rtable = list_make1(fdw_rte); + root->parse = parse; + + leaf_rel_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_part_oids) + { + Relation leaf_rel; + + leaf_rel = heap_open(lfirst_oid(cell), RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(leaf_rel, CMD_INSERT); + + InitResultRelInfo(leaf_rel_rri, + leaf_rel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_rel_rri, false); + + /* Special dance for foreign tables */ + if (leaf_rel_rri->ri_FdwRoutine) + { + List *fdw_private; + + fdw_rte->relid = RelationGetRelid(leaf_rel); + fdw_private = leaf_rel_rri->ri_FdwRoutine->PlanForeignModify(root, + plan, + 1, + 0); + fdw_private_lists = lappend(fdw_private_lists, fdw_private); + } + + if (!equalTupleDescs(tupDesc, RelationGetDescr(leaf_rel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(leaf_rel), + gettext_noop("could not convert row type")); + + leaf_rel_rri++; + i++; + } + + cstate->partition_fdw_priv_lists = fdw_private_lists; + pfree(fdw_rte); + pfree(plan); + pfree(parse); + pfree(root); + } } else { @@ -1692,6 +1786,8 @@ ClosePipeToProgram(CopyState cstate) static void EndCopy(CopyState cstate) { + int i; + if (cstate->is_program) { ClosePipeToProgram(cstate); @@ -1705,6 +1801,23 @@ EndCopy(CopyState cstate) cstate->filename))); } + /* Close all partitions and indices thereof */ + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + /* XXX - EState not handy here to pass to EndForeignModify() */ + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(NULL, resultRelInfo); + + if (cstate->partition_tupconv_maps[i]) + pfree(cstate->partition_tupconv_maps[i]); + } + MemoryContextDelete(cstate->copycontext); pfree(cstate); } @@ -2255,6 +2368,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2275,7 +2389,8 @@ CopyFrom(CopyState cstate) Assert(cstate->rel); - if (cstate->rel->rd_rel->relkind != RELKIND_RELATION) + if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) { if (cstate->rel->rd_rel->relkind == RELKIND_VIEW) ereport(ERROR, @@ -2383,6 +2498,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2410,6 +2526,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->ptnode != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2431,10 +2548,46 @@ CopyFrom(CopyState cstate) */ ExecBSInsertTriggers(estate, resultRelInfo); + /* Initialize FDW partition insert plans */ + if (cstate->ptnode) + { + int i, + j; + List *fdw_private_lists = cstate->partition_fdw_priv_lists; + ModifyTableState *mtstate = makeNode(ModifyTableState); + ResultRelInfo *leaf_part_rri; + + /* Mostly dummy containing enough state for BeginForeignModify */ + mtstate->ps.state = estate; + mtstate->operation = CMD_INSERT; + + j = 0; + leaf_part_rri = cstate->partitions; + for (i = 0; i < cstate->num_partitions; i++) + { + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + Assert(fdw_private_lists); + fdw_private = list_nth(fdw_private_lists, j++); + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, 0); + } + leaf_part_rri++; + } + } + values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2486,6 +2639,31 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition */ + saved_resultRelInfo = resultRelInfo; + if (cstate->ptnode) + { + int i_leaf_partition; + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + cstate->ptnode, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < cstate->num_partitions); + + resultRelInfo = cstate->partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = cstate->partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2506,7 +2684,16 @@ CopyFrom(CopyState cstate) if (cstate->rel->rd_att->constr || resultRelInfo->ri_PartitionCheck) ExecConstraints(resultRelInfo, slot, estate); - if (useHeapMultiInsert) + if (resultRelInfo->ri_FdwRoutine) + { + resultRelInfo->ri_FdwRoutine->ExecForeignInsert(estate, + resultRelInfo, + slot, + NULL); + /* AFTER ROW INSERT Triggers */ + ExecARInsertTriggers(estate, resultRelInfo, tuple, NIL); + } + else if (useHeapMultiInsert) { /* Add this tuple to the tuple buffer */ if (nBufferedTuples == 0) @@ -2536,7 +2723,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, + tuple, mycid, hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self), @@ -2556,6 +2744,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2569,7 +2763,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 395e116..685b184 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1294,6 +1294,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index ea3f59a..62bf8b7 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2996,3 +3001,41 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionTreeNode ptnode, + TupleTableSlot *slot, EState *estate) +{ + int i_leaf_partition; + Oid failed_at; + + i_leaf_partition = get_partition_for_tuple(ptnode, slot, estate, + &failed_at); + + if (i_leaf_partition < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return i_leaf_partition; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..d0a5306 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,31 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + saved_resultRelInfo = resultRelInfo; + + if (mtstate->mt_partition_tree_root) + { + int i_leaf_partition; + ExprContext *econtext = GetPerTupleExprContext(estate); + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_tree_root, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < mtstate->mt_num_partitions); + + resultRelInfo = mtstate->mt_partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = mtstate->mt_partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +537,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1597,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1688,98 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + int i, + j, + num_leaf_parts; + List *leaf_part_oids; + ListCell *cell; + ResultRelInfo *leaf_rel_rri; + + mtstate->mt_partition_tree_root = RelationGetPartitionTreeNode(rel); + leaf_part_oids = get_leaf_partition_oids_v2(mtstate->mt_partition_tree_root); + num_leaf_parts = list_length(leaf_part_oids); + + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_rel_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_part_oids) + { + Oid ftoid = lfirst_oid(cell); + Relation leaf_rel; + + leaf_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(leaf_rel, CMD_INSERT); + + InitResultRelInfo(leaf_rel_rri, + leaf_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_rel_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_rel_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_rel_rri, false); + + if (leaf_rel_rri->ri_FdwRoutine) + { + ListCell *lc; + List *fdw_private; + int k; + + /* + * There are as many fdw_private's in fdwPrivLists as there + * are FDW partitions, but we must find the intended for the + * this foreign table. + */ + k = 0; + foreach(lc, node->fdwPartitionOids) + { + if (lfirst_oid(lc) == ftoid) + break; + k++; + } + + Assert(k < num_leaf_parts); + fdw_private = (List *) list_nth(node->fdwPrivLists, k); + Assert(fdw_private != NIL); + + leaf_rel_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_rel_rri, + fdw_private, + 0, + eflags); + j++; + } + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(leaf_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(leaf_rel), + gettext_noop("could not convert row type")); + + leaf_rel_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2097,23 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(node->ps.state, + resultRelInfo); + + if (node->mt_partition_tupconv_maps[i]) + pfree(node->mt_partition_tupconv_maps[i]); + } + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 28d0036..470dee7 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -188,6 +188,7 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(withCheckOptionLists); COPY_NODE_FIELD(returningLists); COPY_NODE_FIELD(fdwPrivLists); + COPY_NODE_FIELD(fdwPartitionOids); COPY_BITMAPSET_FIELD(fdwDirectModifyPlans); COPY_NODE_FIELD(rowMarks); COPY_SCALAR_FIELD(epqParam); diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 0d858f5..5c8eced 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -340,6 +340,7 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(withCheckOptionLists); WRITE_NODE_FIELD(returningLists); WRITE_NODE_FIELD(fdwPrivLists); + WRITE_NODE_FIELD(fdwPartitionOids); WRITE_BITMAPSET_FIELD(fdwDirectModifyPlans); WRITE_NODE_FIELD(rowMarks); WRITE_INT_FIELD(epqParam); diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c587d4e..ddd78c7 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -1495,6 +1495,7 @@ _readModifyTable(void) READ_NODE_FIELD(withCheckOptionLists); READ_NODE_FIELD(returningLists); READ_NODE_FIELD(fdwPrivLists); + READ_NODE_FIELD(fdwPartitionOids); READ_BITMAPSET_FIELD(fdwDirectModifyPlans); READ_NODE_FIELD(rowMarks); READ_INT_FIELD(epqParam); diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index ad49674..29f40fe 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -22,6 +22,7 @@ #include "access/stratnum.h" #include "access/sysattr.h" #include "catalog/pg_class.h" +#include "catalog/pg_inherits_fn.h" #include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/extensible.h" @@ -6159,6 +6160,82 @@ make_modifytable(PlannerInfo *root, node->fdwPrivLists = fdw_private_list; node->fdwDirectModifyPlans = direct_modify_plans; + /* Collect insert plans for all FDW-managed partitions */ + if (node->operation == CMD_INSERT) + { + RangeTblEntry *rte, + **saved_simple_rte_array; + List *partition_oids, + *fdw_partition_oids; + + Assert(list_length(resultRelations) == 1); + rte = rt_fetch(linitial_int(resultRelations), root->parse->rtable); + Assert(rte->rtekind == RTE_RELATION); + + if (rte->relkind != RELKIND_PARTITIONED_TABLE) + return node; + + partition_oids = find_all_inheritors(rte->relid, NoLock, NULL); + + /* Discard any previous content which is useless anyway */ + fdw_private_list = NIL; + fdw_partition_oids = NIL; + + /* + * To force the FDW driver fetch the intended RTE, we need to temporarily + * switch root->simple_rte_array to the one holding only that RTE at a + * designated index, for every foreign table. + */ + saved_simple_rte_array = root->simple_rte_array; + root->simple_rte_array = (RangeTblEntry **) + palloc0(2 * sizeof(RangeTblEntry *)); + foreach(lc, partition_oids) + { + Oid myoid = lfirst_oid(lc); + FdwRoutine *fdwroutine; + List *fdw_private; + + /* + * We are only interested in foreign tables. Note that this will + * also eliminate any partitioned tables since foreign tables can + * only ever be leaf partitions. + */ + if (!oid_is_foreign_table(myoid)) + continue; + + fdw_partition_oids = lappend_oid(fdw_partition_oids, myoid); + + fdwroutine = GetFdwRoutineByRelId(myoid); + if (fdwroutine && fdwroutine->PlanForeignModify) + { + RangeTblEntry *fdw_rte; + + fdw_rte = copyObject(rte); + fdw_rte->relid = myoid; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + + /* + * Assumes PlanForeignModify() uses planner_rt_fetch(), also, + * see the above comment. + */ + root->simple_rte_array[1] = fdw_rte; + + fdw_private = fdwroutine->PlanForeignModify(root, node, 1, 0); + pfree(fdw_rte); + } + else + fdw_private = NIL; + + fdw_private_list = lappend(fdw_private_list, fdw_private); + } + + pfree(root->simple_rte_array); + root->simple_rte_array = saved_simple_rte_array; + + node->fdwPrivLists = fdw_private_list; + node->fdwPartitionOids = fdw_partition_oids; + } + return node; } diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index a2cbf14..e85ca0a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1715,3 +1715,16 @@ has_row_triggers(PlannerInfo *root, Index rti, CmdType event) heap_close(relation, NoLock); return result; } + +bool +oid_is_foreign_table(Oid relid) +{ + Relation rel; + char relkind; + + rel = heap_open(relid, NoLock); + relkind = rel->rd_rel->relkind; + heap_close(rel, NoLock); + + return relkind == RELKIND_FOREIGN_TABLE; +} diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index ab6d3a8..09727cf 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -56,4 +58,9 @@ extern List *RelationGetPartitionQual(Relation rel, bool recurse); /* For tuple routing */ extern PartitionTreeNode RelationGetPartitionTreeNode(Relation rel); extern List *get_leaf_partition_oids_v2(PartitionTreeNode ptnode); + +extern int get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..c62946f 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..ce01008 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,15 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionTreeNodeData *mt_partition_tree_root; + /* Partition descriptor node tree */ + ResultRelInfo *mt_partitions; /* Per leaf partition target + * relations */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per leaf partition + * tuple conversion map */ + int mt_num_partitions; /* Number of leaf partition target + * relations in the above array */ } ModifyTableState; /* ---------------- diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index e2fbc7d..d82222c 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -189,6 +189,7 @@ typedef struct ModifyTable List *returningLists; /* per-target-table RETURNING tlists */ List *fdwPrivLists; /* per-target-table FDW private data lists */ Bitmapset *fdwDirectModifyPlans; /* indices of FDW DM plans */ + List *fdwPartitionOids; /* OIDs of FDW-managed partition */ List *rowMarks; /* PlanRowMarks (non-locking only) */ int epqParam; /* ID of Param for EvalPlanQual re-eval */ OnConflictAction onConflictAction; /* ON CONFLICT action */ diff --git a/src/include/optimizer/plancat.h b/src/include/optimizer/plancat.h index 125274e..fac606c 100644 --- a/src/include/optimizer/plancat.h +++ b/src/include/optimizer/plancat.h @@ -56,5 +56,6 @@ extern Selectivity join_selectivity(PlannerInfo *root, SpecialJoinInfo *sjinfo); extern bool has_row_triggers(PlannerInfo *root, Index rti, CmdType event); +extern bool oid_is_foreign_table(Oid relid); #endif /* PLANCAT_H */ diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 5d58d23..619336b 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -222,6 +222,62 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- fail (no list partition defined which accepts nulls) +insert into list_parted (b) values (1); +ERROR: no partition of relation "list_parted" found for row +DETAIL: Failing row contains (null, 1). +create table part_nulls partition of list_parted for values in (null); +-- ok +insert into list_parted (b) values (1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_nulls | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(7 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects @@ -230,9 +286,10 @@ drop cascades to table part_a_10_a_20 drop cascades to table part_b_1_b_10 drop cascades to table part_b_10_b_20 drop table list_parted cascade; -NOTICE: drop cascades to 5 other objects +NOTICE: drop cascades to 6 other objects DETAIL: drop cascades to table part_aa_bb drop cascades to table part_cc_dd drop cascades to table part_ee_ff drop cascades to table part_ee_ff_1_10 drop cascades to table part_ee_ff_10_20 +drop cascades to table part_nulls diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index 5aedeef..210fab4 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -137,6 +137,34 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- fail (no list partition defined which accepts nulls) +insert into list_parted (b) values (1); +create table part_nulls partition of list_parted for values in (null); +-- ok +insert into list_parted (b) values (1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------B737FB31608A9E2BB1F071B4 Content-Type: text/x-diff; name="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-12.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-12"; filename*1=".patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH 7/8] Tuple routing for partitioned tables. @ 2016-07-27 07:59 amit <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: amit @ 2016-07-27 07:59 UTC (permalink / raw) Both COPY FROM and INSERT. --- src/backend/catalog/partition.c | 327 ++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 205 +++++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 47 +++++- src/backend/executor/nodeModifyTable.c | 142 +++++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 1 + src/backend/optimizer/plan/createplan.c | 77 +++++++ src/backend/optimizer/util/plancat.c | 13 ++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 7 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 10 + src/include/nodes/plannodes.h | 1 + src/include/optimizer/plancat.h | 1 + src/test/regress/expected/insert.out | 59 ++++++- src/test/regress/sql/insert.sql | 28 +++ 18 files changed, 926 insertions(+), 9 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index 69c3e52..9cba603 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -181,6 +181,18 @@ static List *generate_partition_qual(Relation rel, bool recurse); static PartitionTreeNode GetPartitionTreeNodeRecurse(Relation rel, int offset); static int get_leaf_partition_count(PartitionTreeNode ptnode); +/* Support get_partition_for_tuple() */ +static PartitionKeyExecInfo *BuildPartitionKeyExecInfo(Relation rel); +static void FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); +static int list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull); +static int range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum *tuple); + /* List partition related support functions */ static bool equal_list_info(PartitionKey key, PartitionListInfo *l1, PartitionListInfo *l2); @@ -195,6 +207,8 @@ static PartitionRangeBound *copy_range_bound(PartitionKey key, PartitionRangeBou static bool equal_range_info(PartitionKey key, PartitionRangeInfo *r1, PartitionRangeInfo *r2); static int32 partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg); +static int32 partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg); static bool partition_rbound_eq(PartitionKey key, PartitionRangeBound *b1, PartitionRangeBound *b2); typedef int32 (*partition_rbound_bsearch_cmp_fn) (PartitionKey, @@ -1449,7 +1463,7 @@ GetPartitionTreeNodeRecurse(Relation rel, int offset) /* First build our own node */ parent = (PartitionTreeNode) palloc0(sizeof(PartitionTreeNodeData)); - parent->pkinfo = NULL; + parent->pkinfo = BuildPartitionKeyExecInfo(rel); parent->pdesc = RelationGetPartitionDesc(rel); parent->relid = RelationGetRelid(rel); parent->offset = offset; @@ -1533,6 +1547,284 @@ get_leaf_partition_count(PartitionTreeNode ptnode) return result; } +/* + * BuildPartitionKeyExecInfo + * Construct a list of PartitionKeyExecInfo records for an open + * relation + * + * PartitionKeyExecInfo stores the information about the partition key + * that's needed when inserting tuples into a partitioned table; especially, + * partition key expression state if there are any expression columns in + * the partition key. Normally we build a PartitionKeyExecInfo for a + * partitioned table just once per command, and then use it for (potentially) + * many tuples. + * + */ +static PartitionKeyExecInfo * +BuildPartitionKeyExecInfo(Relation rel) +{ + PartitionKeyExecInfo *pkinfo; + + pkinfo = (PartitionKeyExecInfo *) palloc0(sizeof(PartitionKeyExecInfo)); + pkinfo->pi_Key = RelationGetPartitionKey(rel); + pkinfo->pi_ExpressionState = NIL; + + return pkinfo; +} + +/* + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for partition key columns + */ +static void +FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pkinfo->pi_Key->partexprs != NIL && pkinfo->pi_ExpressionState == NIL) + { + /* First time through, set up expression evaluation state */ + pkinfo->pi_ExpressionState = (List *) + ExecPrepareExpr((Expr *) pkinfo->pi_Key->partexprs, + estate); + /* Check caller has set up context correctly */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + } + + partexpr_item = list_head(pkinfo->pi_ExpressionState); + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + { + AttrNumber keycol = pkinfo->pi_Key->partattrs[i]; + Datum pkDatum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + pkDatum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + pkDatum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = pkDatum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Recursively finds the "leaf" partition for tuple + * + * Returns -1 if no partition is found and sets *failed_at to the OID of + * the partitioned table whose partition was not found. + */ +int +get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + Relation partRel; + PartitionKeyExecInfo *pkinfo = ptnode->pkinfo; + PartitionTreeNode node; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int i; + int index; + + /* Guard against stack overflow due to overly deep partition tree */ + check_stack_depth(); + + if (ptnode->pdesc->nparts == 0) + { + *failed_at = ptnode->relid; + return -1; + } + + /* Extract partition key from tuple */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + FormPartitionKeyDatum(pkinfo, slot, estate, values, isnull); + + /* Disallow nulls, if range partition key */ + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + if (isnull[i] && pkinfo->pi_Key->strategy == PARTITION_STRATEGY_RANGE) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key contains null"))); + + switch (pkinfo->pi_Key->strategy) + { + case PARTITION_STRATEGY_LIST: + index = list_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values[0], isnull[0]); + break; + + case PARTITION_STRATEGY_RANGE: + index = range_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values); + break; + } + + /* No partition found at this level */ + if (index < 0) + { + *failed_at = ptnode->relid; + return index; + } + + partRel = heap_open(ptnode->pdesc->oids[index], NoLock); + + /* Don't recurse if the index'th partition is a leaf partition. */ + if (partRel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) + { + PartitionTreeNode prev; + + /* + * Index returned above is the array index within pdesc->parts[] of + * the parent rel, however, we want to return the leaf partition index + * across the whole partition tree. Note that some partitions within + * pdesc->parts[] may be partitioned themselves and hence stand for + * the leaf partitions in their partition subtrees. We would need to + * skip past the indexes of leaf partitions of all such partition + * subtrees if they are to left of the above returned index. In fact, + * finding the PartitionTreeNode of the rightmost subtree is enough + * since its offset counts the leaf partitions on its left including + * those of partition subtrees to its left. + */ + prev = node = ptnode->downlink; + if (node && node->index < index) + { + /* + * Find the partition tree node such that its index value is the + * greatest value less than the above returned index. + */ + while (node) + { + if (node->index > index) + { + node = prev; + break; + } + + prev = node; + node = node->next; + } + + if (!node) + node = prev; + Assert (node != NULL); + + index = node->offset + node->num_leaf_parts + + (index - node->index - 1); + } + else + /* + * The easy case where we don't have any partition subtree to the + * left of the index. + */ + index = ptnode->offset + index; + + heap_close(partRel, NoLock); + return index; + } + + heap_close(partRel, NoLock); + + /* + * Need to perform recursion as the selected partition is partitioned + * itself. Locate the PartitionTreeNode corresponding to the partition + * passing it down. + */ + node = ptnode->downlink; + while (node->next != NULL && node->index != index) + node = node->next; + Assert (node != NULL); + + return get_partition_for_tuple(node, slot, estate, failed_at); +} + +/* + * list_partition_for_tuple + * Find the list partition for a tuple + * + * Returns -1 if none found. + */ +static int +list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull) +{ + PartitionListInfo listinfo; + int found; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_LIST); + listinfo = pdesc->boundinfo->bounds.lists; + + if (isnull && listinfo.has_null) + return listinfo.null_index; + else if (!isnull) + { + found = partition_list_values_bsearch(key, + listinfo.values, + listinfo.nvalues, + value); + if (found >= 0) + return listinfo.indexes[found]; + } + + /* Control reaches here if isnull and !listinfo->has_null */ + return -1; +} + +/* + * range_partition_for_tuple + * Search the range partition for a range key ('values') + * + * Returns -1 if none found. + */ +static int +range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, Datum *tuple) +{ + int offset; + PartitionRangeInfo rangeinfo; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_RANGE); + rangeinfo = pdesc->boundinfo->bounds.ranges; + + offset = partition_rbound_bsearch(key, + rangeinfo.bounds, rangeinfo.nbounds, + tuple, partition_rbound_datum_cmp, + true, NULL); + + /* + * Offset returned is such that the bound at offset is found to be + * less or equal with the tuple. That is, the tuple belongs to the + * partition with the rangeinfo.bounds[offset] as the lower bound and + * rangeinfo.bounds[offset+1] as the upper bound, provided the latter + * is indeed marked !lower (that is, it's an upper bound). If it turns + * out that it is a lower bound then the corresponding index will be -1, + * which means no valid partition exists. + */ + return rangeinfo.indexes[offset+1]; +} + /* List partition related support functions */ /* @@ -1798,6 +2090,39 @@ partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg) } /* + * Return whether the passed in range bound <=, =, >= tuple specified in arg + * + * The 2nd argument is void * so that it can be used with + * partition_rbound_bsearch() + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg) +{ + Datum *datums1 = bound->datums, + *datums2 = (Datum *) arg; + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (bound->infinite[i]) + return bound->lower ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + datums1[i], datums2[i])); + if (cmpval != 0) + break; + } + + if (cmpval == 0 && !bound->inclusive) + return bound->lower ? 1 : -1; + + return cmpval; +} + +/* * Are two (consecutive) range bounds equal without distinguishing lower * and upper? */ diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 77d4dcb..91e4d12 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -30,6 +30,7 @@ #include "commands/defrem.h" #include "commands/trigger.h" #include "executor/executor.h" +#include "foreign/fdwapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -161,6 +162,11 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionTreeNode ptnode; /* partition descriptor node tree */ + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; + List *partition_fdw_priv_lists; + int num_partitions; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1403,94 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + List *leaf_part_oids; + ListCell *cell; + int i; + int num_leaf_parts; + ResultRelInfo *leaf_rel_rri; + PlannerInfo *root = makeNode(PlannerInfo); /* mostly dummy */ + Query *parse = makeNode(Query); /* ditto */ + ModifyTable *plan = makeNode(ModifyTable); /* ditto */ + RangeTblEntry *fdw_rte = makeNode(RangeTblEntry); /* ditto */ + List *fdw_private_lists = NIL; + + cstate->ptnode = RelationGetPartitionTreeNode(rel); + leaf_part_oids = get_leaf_partition_oids_v2(cstate->ptnode); + num_leaf_parts = list_length(leaf_part_oids); + + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + /* For use below, iff a partition found to be a foreign table */ + plan->operation = CMD_INSERT; + plan->plans = list_make1(makeNode(Result)); + fdw_rte->rtekind = RTE_RELATION; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + parse->rtable = list_make1(fdw_rte); + root->parse = parse; + + leaf_rel_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_part_oids) + { + Relation leaf_rel; + + leaf_rel = heap_open(lfirst_oid(cell), RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(leaf_rel, CMD_INSERT); + + InitResultRelInfo(leaf_rel_rri, + leaf_rel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_rel_rri, false); + + /* Special dance for foreign tables */ + if (leaf_rel_rri->ri_FdwRoutine) + { + List *fdw_private; + + fdw_rte->relid = RelationGetRelid(leaf_rel); + fdw_private = leaf_rel_rri->ri_FdwRoutine->PlanForeignModify(root, + plan, + 1, + 0); + fdw_private_lists = lappend(fdw_private_lists, fdw_private); + } + + if (!equalTupleDescs(tupDesc, RelationGetDescr(leaf_rel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(leaf_rel), + gettext_noop("could not convert row type")); + + leaf_rel_rri++; + i++; + } + + cstate->partition_fdw_priv_lists = fdw_private_lists; + pfree(fdw_rte); + pfree(plan); + pfree(parse); + pfree(root); + } } else { @@ -1692,6 +1786,8 @@ ClosePipeToProgram(CopyState cstate) static void EndCopy(CopyState cstate) { + int i; + if (cstate->is_program) { ClosePipeToProgram(cstate); @@ -1705,6 +1801,23 @@ EndCopy(CopyState cstate) cstate->filename))); } + /* Close all partitions and indices thereof */ + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + /* XXX - EState not handy here to pass to EndForeignModify() */ + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(NULL, resultRelInfo); + + if (cstate->partition_tupconv_maps[i]) + pfree(cstate->partition_tupconv_maps[i]); + } + MemoryContextDelete(cstate->copycontext); pfree(cstate); } @@ -2255,6 +2368,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2275,7 +2389,8 @@ CopyFrom(CopyState cstate) Assert(cstate->rel); - if (cstate->rel->rd_rel->relkind != RELKIND_RELATION) + if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) { if (cstate->rel->rd_rel->relkind == RELKIND_VIEW) ereport(ERROR, @@ -2383,6 +2498,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2410,6 +2526,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->ptnode != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2431,10 +2548,46 @@ CopyFrom(CopyState cstate) */ ExecBSInsertTriggers(estate, resultRelInfo); + /* Initialize FDW partition insert plans */ + if (cstate->ptnode) + { + int i, + j; + List *fdw_private_lists = cstate->partition_fdw_priv_lists; + ModifyTableState *mtstate = makeNode(ModifyTableState); + ResultRelInfo *leaf_part_rri; + + /* Mostly dummy containing enough state for BeginForeignModify */ + mtstate->ps.state = estate; + mtstate->operation = CMD_INSERT; + + j = 0; + leaf_part_rri = cstate->partitions; + for (i = 0; i < cstate->num_partitions; i++) + { + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + Assert(fdw_private_lists); + fdw_private = list_nth(fdw_private_lists, j++); + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, 0); + } + leaf_part_rri++; + } + } + values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2486,6 +2639,31 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition */ + saved_resultRelInfo = resultRelInfo; + if (cstate->ptnode) + { + int i_leaf_partition; + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + cstate->ptnode, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < cstate->num_partitions); + + resultRelInfo = cstate->partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = cstate->partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2506,7 +2684,16 @@ CopyFrom(CopyState cstate) if (cstate->rel->rd_att->constr || resultRelInfo->ri_PartitionCheck) ExecConstraints(resultRelInfo, slot, estate); - if (useHeapMultiInsert) + if (resultRelInfo->ri_FdwRoutine) + { + resultRelInfo->ri_FdwRoutine->ExecForeignInsert(estate, + resultRelInfo, + slot, + NULL); + /* AFTER ROW INSERT Triggers */ + ExecARInsertTriggers(estate, resultRelInfo, tuple, NIL); + } + else if (useHeapMultiInsert) { /* Add this tuple to the tuple buffer */ if (nBufferedTuples == 0) @@ -2536,7 +2723,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, + tuple, mycid, hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self), @@ -2556,6 +2744,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2569,7 +2763,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index de19e9c..88b9d1f 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1292,6 +1292,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index ea3f59a..62bf8b7 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2996,3 +3001,41 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionTreeNode ptnode, + TupleTableSlot *slot, EState *estate) +{ + int i_leaf_partition; + Oid failed_at; + + i_leaf_partition = get_partition_for_tuple(ptnode, slot, estate, + &failed_at); + + if (i_leaf_partition < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return i_leaf_partition; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..d0a5306 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,31 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + saved_resultRelInfo = resultRelInfo; + + if (mtstate->mt_partition_tree_root) + { + int i_leaf_partition; + ExprContext *econtext = GetPerTupleExprContext(estate); + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_tree_root, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < mtstate->mt_num_partitions); + + resultRelInfo = mtstate->mt_partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = mtstate->mt_partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +537,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1597,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1688,98 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + int i, + j, + num_leaf_parts; + List *leaf_part_oids; + ListCell *cell; + ResultRelInfo *leaf_rel_rri; + + mtstate->mt_partition_tree_root = RelationGetPartitionTreeNode(rel); + leaf_part_oids = get_leaf_partition_oids_v2(mtstate->mt_partition_tree_root); + num_leaf_parts = list_length(leaf_part_oids); + + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_rel_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_part_oids) + { + Oid ftoid = lfirst_oid(cell); + Relation leaf_rel; + + leaf_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(leaf_rel, CMD_INSERT); + + InitResultRelInfo(leaf_rel_rri, + leaf_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_rel_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_rel_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_rel_rri, false); + + if (leaf_rel_rri->ri_FdwRoutine) + { + ListCell *lc; + List *fdw_private; + int k; + + /* + * There are as many fdw_private's in fdwPrivLists as there + * are FDW partitions, but we must find the intended for the + * this foreign table. + */ + k = 0; + foreach(lc, node->fdwPartitionOids) + { + if (lfirst_oid(lc) == ftoid) + break; + k++; + } + + Assert(k < num_leaf_parts); + fdw_private = (List *) list_nth(node->fdwPrivLists, k); + Assert(fdw_private != NIL); + + leaf_rel_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_rel_rri, + fdw_private, + 0, + eflags); + j++; + } + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(leaf_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(leaf_rel), + gettext_noop("could not convert row type")); + + leaf_rel_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2097,23 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(node->ps.state, + resultRelInfo); + + if (node->mt_partition_tupconv_maps[i]) + pfree(node->mt_partition_tupconv_maps[i]); + } + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index c0d48d0..ca48547 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -188,6 +188,7 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(withCheckOptionLists); COPY_NODE_FIELD(returningLists); COPY_NODE_FIELD(fdwPrivLists); + COPY_NODE_FIELD(fdwPartitionOids); COPY_BITMAPSET_FIELD(fdwDirectModifyPlans); COPY_NODE_FIELD(rowMarks); COPY_SCALAR_FIELD(epqParam); diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index d5eb0cc..7a219e5 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -340,6 +340,7 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(withCheckOptionLists); WRITE_NODE_FIELD(returningLists); WRITE_NODE_FIELD(fdwPrivLists); + WRITE_NODE_FIELD(fdwPartitionOids); WRITE_BITMAPSET_FIELD(fdwDirectModifyPlans); WRITE_NODE_FIELD(rowMarks); WRITE_INT_FIELD(epqParam); diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 655aa1c..d047160 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -1495,6 +1495,7 @@ _readModifyTable(void) READ_NODE_FIELD(withCheckOptionLists); READ_NODE_FIELD(returningLists); READ_NODE_FIELD(fdwPrivLists); + READ_NODE_FIELD(fdwPartitionOids); READ_BITMAPSET_FIELD(fdwDirectModifyPlans); READ_NODE_FIELD(rowMarks); READ_INT_FIELD(epqParam); diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index ad49674..29f40fe 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -22,6 +22,7 @@ #include "access/stratnum.h" #include "access/sysattr.h" #include "catalog/pg_class.h" +#include "catalog/pg_inherits_fn.h" #include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/extensible.h" @@ -6159,6 +6160,82 @@ make_modifytable(PlannerInfo *root, node->fdwPrivLists = fdw_private_list; node->fdwDirectModifyPlans = direct_modify_plans; + /* Collect insert plans for all FDW-managed partitions */ + if (node->operation == CMD_INSERT) + { + RangeTblEntry *rte, + **saved_simple_rte_array; + List *partition_oids, + *fdw_partition_oids; + + Assert(list_length(resultRelations) == 1); + rte = rt_fetch(linitial_int(resultRelations), root->parse->rtable); + Assert(rte->rtekind == RTE_RELATION); + + if (rte->relkind != RELKIND_PARTITIONED_TABLE) + return node; + + partition_oids = find_all_inheritors(rte->relid, NoLock, NULL); + + /* Discard any previous content which is useless anyway */ + fdw_private_list = NIL; + fdw_partition_oids = NIL; + + /* + * To force the FDW driver fetch the intended RTE, we need to temporarily + * switch root->simple_rte_array to the one holding only that RTE at a + * designated index, for every foreign table. + */ + saved_simple_rte_array = root->simple_rte_array; + root->simple_rte_array = (RangeTblEntry **) + palloc0(2 * sizeof(RangeTblEntry *)); + foreach(lc, partition_oids) + { + Oid myoid = lfirst_oid(lc); + FdwRoutine *fdwroutine; + List *fdw_private; + + /* + * We are only interested in foreign tables. Note that this will + * also eliminate any partitioned tables since foreign tables can + * only ever be leaf partitions. + */ + if (!oid_is_foreign_table(myoid)) + continue; + + fdw_partition_oids = lappend_oid(fdw_partition_oids, myoid); + + fdwroutine = GetFdwRoutineByRelId(myoid); + if (fdwroutine && fdwroutine->PlanForeignModify) + { + RangeTblEntry *fdw_rte; + + fdw_rte = copyObject(rte); + fdw_rte->relid = myoid; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + + /* + * Assumes PlanForeignModify() uses planner_rt_fetch(), also, + * see the above comment. + */ + root->simple_rte_array[1] = fdw_rte; + + fdw_private = fdwroutine->PlanForeignModify(root, node, 1, 0); + pfree(fdw_rte); + } + else + fdw_private = NIL; + + fdw_private_list = lappend(fdw_private_list, fdw_private); + } + + pfree(root->simple_rte_array); + root->simple_rte_array = saved_simple_rte_array; + + node->fdwPrivLists = fdw_private_list; + node->fdwPartitionOids = fdw_partition_oids; + } + return node; } diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index a2cbf14..e85ca0a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1715,3 +1715,16 @@ has_row_triggers(PlannerInfo *root, Index rti, CmdType event) heap_close(relation, NoLock); return result; } + +bool +oid_is_foreign_table(Oid relid) +{ + Relation rel; + char relkind; + + rel = heap_open(relid, NoLock); + relkind = rel->rd_rel->relkind; + heap_close(rel, NoLock); + + return relkind == RELKIND_FOREIGN_TABLE; +} diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index ab6d3a8..09727cf 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -56,4 +58,9 @@ extern List *RelationGetPartitionQual(Relation rel, bool recurse); /* For tuple routing */ extern PartitionTreeNode RelationGetPartitionTreeNode(Relation rel); extern List *get_leaf_partition_oids_v2(PartitionTreeNode ptnode); + +extern int get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..c62946f 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..ce01008 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,15 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionTreeNodeData *mt_partition_tree_root; + /* Partition descriptor node tree */ + ResultRelInfo *mt_partitions; /* Per leaf partition target + * relations */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per leaf partition + * tuple conversion map */ + int mt_num_partitions; /* Number of leaf partition target + * relations in the above array */ } ModifyTableState; /* ---------------- diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index e2fbc7d..d82222c 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -189,6 +189,7 @@ typedef struct ModifyTable List *returningLists; /* per-target-table RETURNING tlists */ List *fdwPrivLists; /* per-target-table FDW private data lists */ Bitmapset *fdwDirectModifyPlans; /* indices of FDW DM plans */ + List *fdwPartitionOids; /* OIDs of FDW-managed partition */ List *rowMarks; /* PlanRowMarks (non-locking only) */ int epqParam; /* ID of Param for EvalPlanQual re-eval */ OnConflictAction onConflictAction; /* ON CONFLICT action */ diff --git a/src/include/optimizer/plancat.h b/src/include/optimizer/plancat.h index 125274e..fac606c 100644 --- a/src/include/optimizer/plancat.h +++ b/src/include/optimizer/plancat.h @@ -56,5 +56,6 @@ extern Selectivity join_selectivity(PlannerInfo *root, SpecialJoinInfo *sjinfo); extern bool has_row_triggers(PlannerInfo *root, Index rti, CmdType event); +extern bool oid_is_foreign_table(Oid relid); #endif /* PLANCAT_H */ diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index ebc0ba5..3d97742 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -222,6 +222,62 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- fail (no list partition defined which accepts nulls) +insert into list_parted (b) values (1); +ERROR: no partition of relation "list_parted" found for row +DETAIL: Failing row contains (null, 1). +create table part_nulls partition of list_parted for values in (null); +-- ok +insert into list_parted (b) values (1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_nulls | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(7 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects @@ -230,9 +286,10 @@ drop cascades to table part_a_10_a_20 drop cascades to table part_b_1_b_10 drop cascades to table part_b_10_b_20 drop table list_parted cascade; -NOTICE: drop cascades to 5 other objects +NOTICE: drop cascades to 6 other objects DETAIL: drop cascades to table part_aa_bb drop cascades to table part_cc_dd drop cascades to table part_ee_ff drop cascades to table part_ee_ff_1_10 drop cascades to table part_ee_ff_10_20 +drop cascades to table part_nulls diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index 4bf042e..d1b5a09 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -137,6 +137,34 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- fail (no list partition defined which accepts nulls) +insert into list_parted (b) values (1); +create table part_nulls partition of list_parted for values in (null); +-- ok +insert into list_parted (b) values (1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------294A99728041A5608B577083 Content-Type: text/x-diff; name="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-11.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-11"; filename*1=".patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH 7/8] Tuple routing for partitioned tables. @ 2016-07-27 07:59 amit <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: amit @ 2016-07-27 07:59 UTC (permalink / raw) Both COPY FROM and INSERT. --- src/backend/catalog/partition.c | 325 ++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 203 +++++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 47 +++++- src/backend/executor/nodeModifyTable.c | 142 ++++++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 1 + src/backend/optimizer/plan/createplan.c | 77 ++++++++ src/backend/optimizer/util/plancat.c | 13 ++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 7 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 10 + src/include/nodes/plannodes.h | 1 + src/include/optimizer/plancat.h | 1 + src/test/regress/expected/insert.out | 52 +++++ src/test/regress/sql/insert.sql | 25 +++ 18 files changed, 914 insertions(+), 7 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index 6301d91..e3a0848 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -181,6 +181,18 @@ static List *generate_partition_qual(Relation rel, bool recurse); static PartitionTreeNode GetPartitionTreeNodeRecurse(Relation rel, int offset); static int get_leaf_partition_count(PartitionTreeNode ptnode); +/* Support get_partition_for_tuple() */ +static PartitionKeyExecInfo *BuildPartitionKeyExecInfo(Relation rel); +static void FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); +static int list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull); +static int range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum *tuple); + /* List partition related support functions */ static bool equal_list_info(PartitionKey key, PartitionListInfo *l1, PartitionListInfo *l2); @@ -194,6 +206,8 @@ static PartitionRangeBound *copy_range_bound(PartitionKey key, PartitionRangeBou static bool equal_range_info(PartitionKey key, PartitionRangeInfo *r1, PartitionRangeInfo *r2); static int32 partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg); +static int32 partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg); static bool partition_rbound_eq(PartitionKey key, PartitionRangeBound *b1, PartitionRangeBound *b2); typedef int32 (*partition_rbound_bsearch_cmp_fn) (PartitionKey, @@ -1437,7 +1451,7 @@ GetPartitionTreeNodeRecurse(Relation rel, int offset) /* First build our own node */ parent = (PartitionTreeNode) palloc0(sizeof(PartitionTreeNodeData)); - parent->pkinfo = NULL; + parent->pkinfo = BuildPartitionKeyExecInfo(rel); parent->pdesc = RelationGetPartitionDesc(rel); parent->relid = RelationGetRelid(rel); parent->offset = offset; @@ -1521,6 +1535,281 @@ get_leaf_partition_count(PartitionTreeNode ptnode) return result; } +/* + * BuildPartitionKeyExecInfo + * Construct a list of PartitionKeyExecInfo records for an open + * relation + * + * PartitionKeyExecInfo stores the information about the partition key + * that's needed when inserting tuples into a partitioned table; especially, + * partition key expression state if there are any expression columns in + * the partition key. Normally we build a PartitionKeyExecInfo for a + * partitioned table just once per command, and then use it for (potentially) + * many tuples. + * + */ +static PartitionKeyExecInfo * +BuildPartitionKeyExecInfo(Relation rel) +{ + PartitionKeyExecInfo *pkinfo; + + pkinfo = (PartitionKeyExecInfo *) palloc0(sizeof(PartitionKeyExecInfo)); + pkinfo->pi_Key = RelationGetPartitionKey(rel); + pkinfo->pi_ExpressionState = NIL; + + return pkinfo; +} + +/* + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for partition key columns + */ +static void +FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pkinfo->pi_Key->partexprs != NIL && pkinfo->pi_ExpressionState == NIL) + { + /* First time through, set up expression evaluation state */ + pkinfo->pi_ExpressionState = (List *) + ExecPrepareExpr((Expr *) pkinfo->pi_Key->partexprs, + estate); + /* Check caller has set up context correctly */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + } + + partexpr_item = list_head(pkinfo->pi_ExpressionState); + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + { + AttrNumber keycol = pkinfo->pi_Key->partattrs[i]; + Datum pkDatum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + pkDatum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + pkDatum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = pkDatum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Recursively finds the "leaf" partition for tuple + * + * Returns -1 if no partition is found and sets *failed_at to the OID of + * the partitioned table whose partition was not found. + */ +int +get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + Relation partRel; + PartitionKeyExecInfo *pkinfo = ptnode->pkinfo; + PartitionTreeNode node; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int i; + int index; + + /* Guard against stack overflow due to overly deep partition tree */ + check_stack_depth(); + + if (ptnode->pdesc->nparts == 0) + { + *failed_at = ptnode->relid; + return -1; + } + + /* Extract partition key from tuple */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + FormPartitionKeyDatum(pkinfo, slot, estate, values, isnull); + + /* Disallow nulls, if range partition key */ + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + if (isnull[i] && pkinfo->pi_Key->strategy == PARTITION_STRATEGY_RANGE) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key contains null"))); + + switch (pkinfo->pi_Key->strategy) + { + case PARTITION_STRATEGY_LIST: + index = list_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values[0], isnull[0]); + break; + + case PARTITION_STRATEGY_RANGE: + index = range_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values); + break; + } + + /* No partition found at this level */ + if (index < 0) + { + *failed_at = ptnode->relid; + return index; + } + + partRel = heap_open(ptnode->pdesc->oids[index], NoLock); + + /* Don't recurse if the index'th partition is a leaf partition. */ + if (partRel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) + { + PartitionTreeNode prev; + + /* + * Index returned above considers only this parent (of which partRel + * is a partition). We however want to return the index across the + * the whole partition tree. If partRel is the leftmost partition + * of parent (index = 0) or if none of the left siblings are + * partitioned tables, we can simply return parent's offset plus + * partRel's index as the result. Otherwise, we find the node + * corresponding to the rightmost partitioned table among partRel's + * left siblings and use the information therein. + */ + prev = node = ptnode->downlink; + if (node && node->index < index) + { + /* + * Find the partition tree node such that its index value is the + * greatest value less than the above returned index. + */ + while (node) + { + if (node->index > index) + { + node = prev; + break; + } + + prev = node; + node = node->next; + } + + if (!node) + node = prev; + Assert (node != NULL); + + index = node->offset + node->num_leaf_parts + + (index - node->index - 1); + } + else + /* + * The easy case where all of partRel's left siblings (if any) are + * leaf partitions. + */ + index = ptnode->offset + index; + + heap_close(partRel, NoLock); + return index; + } + + heap_close(partRel, NoLock); + + /* + * Need to recurse as the selected partition is a partitioned table + * itself. Locate the corresponding PartitionTreeNode to pass it down. + */ + node = ptnode->downlink; + while (node->next != NULL && node->index != index) + node = node->next; + Assert (node != NULL); + + return get_partition_for_tuple(node, slot, estate, failed_at); +} + +/* + * list_partition_for_tuple + * Find the list partition for a tuple + * + * Returns -1 if none found. + */ +static int +list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull) +{ + PartitionListInfo listinfo; + int found; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_LIST); + listinfo = pdesc->boundinfo->bounds.lists; + + if (isnull && listinfo.has_null) + return listinfo.null_index; + else if (!isnull) + { + found = partition_list_values_bsearch(key, + listinfo.values, + listinfo.nvalues, + value); + if (found >= 0) + return listinfo.indexes[found]; + } + + /* Control reaches here if isnull and !listinfo->has_null */ + return -1; +} + +/* + * range_partition_for_tuple + * Search the range partition for a range key ('values') + * + * Returns -1 if none found. + */ +static int +range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, Datum *tuple) +{ + int offset; + PartitionRangeInfo rangeinfo; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_RANGE); + rangeinfo = pdesc->boundinfo->bounds.ranges; + + offset = partition_rbound_bsearch(key, + rangeinfo.bounds, rangeinfo.nbounds, + tuple, partition_rbound_datum_cmp, + true, NULL); + + /* + * Offset returned is such that the bound at offset is found to be + * less or equal with the tuple. That is, the tuple belongs to the + * partition with the rangeinfo.bounds[offset] as the lower bound and + * rangeinfo.bounds[offset+1] as the upper bound, provided the latter + * is indeed marked !lower (that is, it's an upper bound). If it turns + * out that it is a lower bound then the corresponding index will be -1, + * which means no valid partition exists. + */ + return rangeinfo.indexes[offset+1]; +} + /* List partition related support functions */ /* @@ -1767,6 +2056,40 @@ partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg) } /* + * Return whether bound <=, =, >= partition key of tuple + * + * The 3rd argument is void * so that it can be used with + * partition_rbound_bsearch() + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg) +{ + Datum *datums1 = bound->datums, + *datums2 = (Datum *) arg; + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (bound->infinite[i]) + return bound->lower ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + datums1[i], datums2[i])); + if (cmpval != 0) + break; + } + + /* If datums are equal and this is an upper bound, tuple > bound */ + if (cmpval == 0 && !bound->lower) + return -1; + + return cmpval; +} + +/* * Return whether two range bounds are equal simply by comparing datums */ static bool diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..6fb376f 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -30,6 +30,7 @@ #include "commands/defrem.h" #include "commands/trigger.h" #include "executor/executor.h" +#include "foreign/fdwapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -161,6 +162,11 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionTreeNode ptnode; /* partition descriptor node tree */ + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; + List *partition_fdw_priv_lists; + int num_partitions; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1403,94 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + List *leaf_parts; + ListCell *cell; + int i; + int num_leaf_parts; + ResultRelInfo *leaf_part_rri; + PlannerInfo *root = makeNode(PlannerInfo); /* mostly dummy */ + Query *parse = makeNode(Query); /* ditto */ + ModifyTable *plan = makeNode(ModifyTable); /* ditto */ + RangeTblEntry *fdw_rte = makeNode(RangeTblEntry); /* ditto */ + List *fdw_private_lists = NIL; + + cstate->ptnode = RelationGetPartitionTreeNode(rel); + leaf_parts = get_leaf_partition_oids(cstate->ptnode); + num_leaf_parts = list_length(leaf_parts); + + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + /* For use below, iff a partition found to be a foreign table */ + plan->operation = CMD_INSERT; + plan->plans = list_make1(makeNode(Result)); + fdw_rte->rtekind = RTE_RELATION; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + parse->rtable = list_make1(fdw_rte); + root->parse = parse; + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation part_rel; + + part_rel = heap_open(lfirst_oid(cell), RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + /* Special dance for foreign tables */ + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + fdw_rte->relid = RelationGetRelid(part_rel); + fdw_private = leaf_part_rri->ri_FdwRoutine->PlanForeignModify(root, + plan, + 1, + 0); + fdw_private_lists = lappend(fdw_private_lists, fdw_private); + } + + if (!equalTupleDescs(tupDesc, RelationGetDescr(part_rel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + + cstate->partition_fdw_priv_lists = fdw_private_lists; + pfree(fdw_rte); + pfree(plan); + pfree(parse); + pfree(root); + } } else { @@ -1692,6 +1786,8 @@ ClosePipeToProgram(CopyState cstate) static void EndCopy(CopyState cstate) { + int i; + if (cstate->is_program) { ClosePipeToProgram(cstate); @@ -1705,6 +1801,23 @@ EndCopy(CopyState cstate) cstate->filename))); } + /* Close all partitions and indices thereof */ + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + /* XXX - EState not handy here to pass to EndForeignModify() */ + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(NULL, resultRelInfo); + + if (cstate->partition_tupconv_maps[i]) + pfree(cstate->partition_tupconv_maps[i]); + } + MemoryContextDelete(cstate->copycontext); pfree(cstate); } @@ -2255,6 +2368,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2395,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2506,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2534,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->ptnode != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2439,10 +2556,46 @@ CopyFrom(CopyState cstate) */ ExecBSInsertTriggers(estate, resultRelInfo); + /* Initialize FDW partition insert plans */ + if (cstate->ptnode) + { + int i, + j; + List *fdw_private_lists = cstate->partition_fdw_priv_lists; + ModifyTableState *mtstate = makeNode(ModifyTableState); + ResultRelInfo *leaf_part_rri; + + /* Mostly dummy containing enough state for BeginForeignModify */ + mtstate->ps.state = estate; + mtstate->operation = CMD_INSERT; + + j = 0; + leaf_part_rri = cstate->partitions; + for (i = 0; i < cstate->num_partitions; i++) + { + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + Assert(fdw_private_lists); + fdw_private = list_nth(fdw_private_lists, j++); + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, 0); + } + leaf_part_rri++; + } + } + values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2647,31 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition */ + saved_resultRelInfo = resultRelInfo; + if (cstate->ptnode) + { + int i_leaf_partition; + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + cstate->ptnode, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < cstate->num_partitions); + + resultRelInfo = cstate->partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = cstate->partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2523,7 +2701,16 @@ CopyFrom(CopyState cstate) resultRelInfo->ri_PartitionCheck) ExecConstraints(resultRelInfo, slot, estate); - if (useHeapMultiInsert) + if (resultRelInfo->ri_FdwRoutine) + { + resultRelInfo->ri_FdwRoutine->ExecForeignInsert(estate, + resultRelInfo, + slot, + NULL); + /* AFTER ROW INSERT Triggers */ + ExecARInsertTriggers(estate, resultRelInfo, tuple, NIL); + } + else if (useHeapMultiInsert) { /* Add this tuple to the tuple buffer */ if (nBufferedTuples == 0) @@ -2553,7 +2740,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2765,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2784,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8cd3614..b51d59a 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1293,6 +1293,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index ea3f59a..62bf8b7 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2996,3 +3001,41 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionTreeNode ptnode, + TupleTableSlot *slot, EState *estate) +{ + int i_leaf_partition; + Oid failed_at; + + i_leaf_partition = get_partition_for_tuple(ptnode, slot, estate, + &failed_at); + + if (i_leaf_partition < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return i_leaf_partition; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..5c2bf6c 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,31 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + saved_resultRelInfo = resultRelInfo; + + if (mtstate->mt_partition_tree_root) + { + int i_leaf_partition; + ExprContext *econtext = GetPerTupleExprContext(estate); + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_tree_root, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < mtstate->mt_num_partitions); + + resultRelInfo = mtstate->mt_partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = mtstate->mt_partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +537,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1597,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1688,98 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + mtstate->mt_partition_tree_root = RelationGetPartitionTreeNode(rel); + leaf_parts = get_leaf_partition_oids(mtstate->mt_partition_tree_root); + num_leaf_parts = list_length(leaf_parts); + + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (leaf_part_rri->ri_FdwRoutine) + { + ListCell *lc; + List *fdw_private; + int k; + + /* + * There are as many fdw_private's in fdwPrivLists as there + * are FDW partitions, but we must find the intended for the + * this foreign table. + */ + k = 0; + foreach(lc, node->fdwPartitionOids) + { + if (lfirst_oid(lc) == ftoid) + break; + k++; + } + + Assert(k < num_leaf_parts); + fdw_private = (List *) list_nth(node->fdwPrivLists, k); + Assert(fdw_private != NIL); + + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, + eflags); + j++; + } + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2097,23 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(node->ps.state, + resultRelInfo); + + if (node->mt_partition_tupconv_maps[i]) + pfree(node->mt_partition_tupconv_maps[i]); + } + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 28d0036..470dee7 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -188,6 +188,7 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(withCheckOptionLists); COPY_NODE_FIELD(returningLists); COPY_NODE_FIELD(fdwPrivLists); + COPY_NODE_FIELD(fdwPartitionOids); COPY_BITMAPSET_FIELD(fdwDirectModifyPlans); COPY_NODE_FIELD(rowMarks); COPY_SCALAR_FIELD(epqParam); diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 0d858f5..5c8eced 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -340,6 +340,7 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(withCheckOptionLists); WRITE_NODE_FIELD(returningLists); WRITE_NODE_FIELD(fdwPrivLists); + WRITE_NODE_FIELD(fdwPartitionOids); WRITE_BITMAPSET_FIELD(fdwDirectModifyPlans); WRITE_NODE_FIELD(rowMarks); WRITE_INT_FIELD(epqParam); diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c587d4e..ddd78c7 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -1495,6 +1495,7 @@ _readModifyTable(void) READ_NODE_FIELD(withCheckOptionLists); READ_NODE_FIELD(returningLists); READ_NODE_FIELD(fdwPrivLists); + READ_NODE_FIELD(fdwPartitionOids); READ_BITMAPSET_FIELD(fdwDirectModifyPlans); READ_NODE_FIELD(rowMarks); READ_INT_FIELD(epqParam); diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index ad49674..29f40fe 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -22,6 +22,7 @@ #include "access/stratnum.h" #include "access/sysattr.h" #include "catalog/pg_class.h" +#include "catalog/pg_inherits_fn.h" #include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/extensible.h" @@ -6159,6 +6160,82 @@ make_modifytable(PlannerInfo *root, node->fdwPrivLists = fdw_private_list; node->fdwDirectModifyPlans = direct_modify_plans; + /* Collect insert plans for all FDW-managed partitions */ + if (node->operation == CMD_INSERT) + { + RangeTblEntry *rte, + **saved_simple_rte_array; + List *partition_oids, + *fdw_partition_oids; + + Assert(list_length(resultRelations) == 1); + rte = rt_fetch(linitial_int(resultRelations), root->parse->rtable); + Assert(rte->rtekind == RTE_RELATION); + + if (rte->relkind != RELKIND_PARTITIONED_TABLE) + return node; + + partition_oids = find_all_inheritors(rte->relid, NoLock, NULL); + + /* Discard any previous content which is useless anyway */ + fdw_private_list = NIL; + fdw_partition_oids = NIL; + + /* + * To force the FDW driver fetch the intended RTE, we need to temporarily + * switch root->simple_rte_array to the one holding only that RTE at a + * designated index, for every foreign table. + */ + saved_simple_rte_array = root->simple_rte_array; + root->simple_rte_array = (RangeTblEntry **) + palloc0(2 * sizeof(RangeTblEntry *)); + foreach(lc, partition_oids) + { + Oid myoid = lfirst_oid(lc); + FdwRoutine *fdwroutine; + List *fdw_private; + + /* + * We are only interested in foreign tables. Note that this will + * also eliminate any partitioned tables since foreign tables can + * only ever be leaf partitions. + */ + if (!oid_is_foreign_table(myoid)) + continue; + + fdw_partition_oids = lappend_oid(fdw_partition_oids, myoid); + + fdwroutine = GetFdwRoutineByRelId(myoid); + if (fdwroutine && fdwroutine->PlanForeignModify) + { + RangeTblEntry *fdw_rte; + + fdw_rte = copyObject(rte); + fdw_rte->relid = myoid; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + + /* + * Assumes PlanForeignModify() uses planner_rt_fetch(), also, + * see the above comment. + */ + root->simple_rte_array[1] = fdw_rte; + + fdw_private = fdwroutine->PlanForeignModify(root, node, 1, 0); + pfree(fdw_rte); + } + else + fdw_private = NIL; + + fdw_private_list = lappend(fdw_private_list, fdw_private); + } + + pfree(root->simple_rte_array); + root->simple_rte_array = saved_simple_rte_array; + + node->fdwPrivLists = fdw_private_list; + node->fdwPartitionOids = fdw_partition_oids; + } + return node; } diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index a2cbf14..e85ca0a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1715,3 +1715,16 @@ has_row_triggers(PlannerInfo *root, Index rti, CmdType event) heap_close(relation, NoLock); return result; } + +bool +oid_is_foreign_table(Oid relid) +{ + Relation rel; + char relkind; + + rel = heap_open(relid, NoLock); + relkind = rel->rd_rel->relkind; + heap_close(rel, NoLock); + + return relkind == RELKIND_FOREIGN_TABLE; +} diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 057b1e3..86cc598 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -50,4 +52,9 @@ extern List *RelationGetPartitionQual(Relation rel, bool recurse); /* For tuple routing */ extern PartitionTreeNode RelationGetPartitionTreeNode(Relation rel); extern List *get_leaf_partition_oids(PartitionTreeNode ptnode); + +extern int get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..c62946f 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..ce01008 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,15 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionTreeNodeData *mt_partition_tree_root; + /* Partition descriptor node tree */ + ResultRelInfo *mt_partitions; /* Per leaf partition target + * relations */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per leaf partition + * tuple conversion map */ + int mt_num_partitions; /* Number of leaf partition target + * relations in the above array */ } ModifyTableState; /* ---------------- diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index e2fbc7d..d82222c 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -189,6 +189,7 @@ typedef struct ModifyTable List *returningLists; /* per-target-table RETURNING tlists */ List *fdwPrivLists; /* per-target-table FDW private data lists */ Bitmapset *fdwDirectModifyPlans; /* indices of FDW DM plans */ + List *fdwPartitionOids; /* OIDs of FDW-managed partition */ List *rowMarks; /* PlanRowMarks (non-locking only) */ int epqParam; /* ID of Param for EvalPlanQual re-eval */ OnConflictAction onConflictAction; /* ON CONFLICT action */ diff --git a/src/include/optimizer/plancat.h b/src/include/optimizer/plancat.h index 125274e..fac606c 100644 --- a/src/include/optimizer/plancat.h +++ b/src/include/optimizer/plancat.h @@ -56,5 +56,6 @@ extern Selectivity join_selectivity(PlannerInfo *root, SpecialJoinInfo *sjinfo); extern bool has_row_triggers(PlannerInfo *root, Index rti, CmdType event); +extern bool oid_is_foreign_table(Oid relid); #endif /* PLANCAT_H */ diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 9ae6b09..d5dcb59 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -227,6 +227,58 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index b6e821e..fbd30d9 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,31 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------20C4AAD7DBE3F2B5281ABBA6 Content-Type: text/x-diff; name="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-13.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-13"; filename*1=".patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* [PATCH 1/4] bootstrap: convert Typ to a List* @ 2020-11-20 02:48 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: Justin Pryzby @ 2020-11-20 02:48 UTC (permalink / raw) --- src/backend/bootstrap/bootstrap.c | 89 ++++++++++++++----------------- 1 file changed, 41 insertions(+), 48 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..1b940d9d27 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -58,7 +58,7 @@ static void BootstrapModeMain(void); static void bootstrap_signals(void); static void ShutdownAuxiliaryProcess(int code, Datum arg); static Form_pg_attribute AllocateAttribute(void); -static void populate_typ_array(void); +static void populate_typ(void); static Oid gettype(char *type); static void cleanup(void); @@ -159,7 +159,7 @@ struct typmap FormData_pg_type am_typ; }; -static struct typmap **Typ = NULL; +static List *Typ = NIL; /* List of struct typmap* */ static struct typmap *Ap = NULL; static Datum values[MAXATTR]; /* current row's attribute values */ @@ -595,10 +595,10 @@ boot_openrel(char *relname) /* * pg_type must be filled before any OPEN command is executed, hence we - * can now populate the Typ array if we haven't yet. + * can now populate Typ if we haven't yet. */ - if (Typ == NULL) - populate_typ_array(); + if (Typ == NIL) + populate_typ(); if (boot_reldesc != NULL) closerel(NULL); @@ -688,7 +688,7 @@ DefineAttr(char *name, char *type, int attnum, int nullness) typeoid = gettype(type); - if (Typ != NULL) + if (Typ != NIL) { attrtypes[attnum]->atttypid = Ap->am_oid; attrtypes[attnum]->attlen = Ap->am_typ.typlen; @@ -866,47 +866,36 @@ cleanup(void) } /* ---------------- - * populate_typ_array + * populate_typ * - * Load the Typ array by reading pg_type. + * Load Typ by reading pg_type. * ---------------- */ static void -populate_typ_array(void) +populate_typ(void) { Relation rel; TableScanDesc scan; HeapTuple tup; - int nalloc; - int i; - - Assert(Typ == NULL); + MemoryContext old; - nalloc = 512; - Typ = (struct typmap **) - MemoryContextAlloc(TopMemoryContext, nalloc * sizeof(struct typmap *)); + Assert(Typ == NIL); rel = table_open(TypeRelationId, NoLock); scan = table_beginscan_catalog(rel, 0, NULL); - i = 0; + old = MemoryContextSwitchTo(TopMemoryContext); while ((tup = heap_getnext(scan, ForwardScanDirection)) != NULL) { Form_pg_type typForm = (Form_pg_type) GETSTRUCT(tup); + struct typmap *newtyp; - /* make sure there will be room for a trailing NULL pointer */ - if (i >= nalloc - 1) - { - nalloc *= 2; - Typ = (struct typmap **) - repalloc(Typ, nalloc * sizeof(struct typmap *)); - } - Typ[i] = (struct typmap *) - MemoryContextAlloc(TopMemoryContext, sizeof(struct typmap)); - Typ[i]->am_oid = typForm->oid; - memcpy(&(Typ[i]->am_typ), typForm, sizeof(Typ[i]->am_typ)); - i++; + newtyp = (struct typmap *) palloc(sizeof(struct typmap)); + Typ = lappend(Typ, newtyp); + + newtyp->am_oid = typForm->oid; + memcpy(&newtyp->am_typ, typForm, sizeof(newtyp->am_typ)); } - Typ[i] = NULL; /* Fill trailing NULL pointer */ + MemoryContextSwitchTo(old); table_endscan(scan); table_close(rel, NoLock); } @@ -916,25 +905,26 @@ populate_typ_array(void) * * NB: this is really ugly; it will return an integer index into TypInfo[], * and not an OID at all, until the first reference to a type not known in - * TypInfo[]. At that point it will read and cache pg_type in the Typ array, + * TypInfo[]. At that point it will read and cache pg_type in Typ, * and subsequently return a real OID (and set the global pointer Ap to * point at the found row in Typ). So caller must check whether Typ is - * still NULL to determine what the return value is! + * still NIL to determine what the return value is! * ---------------- */ static Oid gettype(char *type) { - if (Typ != NULL) + if (Typ != NIL) { - struct typmap **app; + ListCell *lc; - for (app = Typ; *app != NULL; app++) + foreach (lc, Typ) { - if (strncmp(NameStr((*app)->am_typ.typname), type, NAMEDATALEN) == 0) + struct typmap *app = lfirst(lc); + if (strncmp(NameStr(app->am_typ.typname), type, NAMEDATALEN) == 0) { - Ap = *app; - return (*app)->am_oid; + Ap = app; + return app->am_oid; } } } @@ -949,7 +939,7 @@ gettype(char *type) } /* Not in TypInfo, so we'd better be able to read pg_type now */ elog(DEBUG4, "external type: %s", type); - populate_typ_array(); + populate_typ(); return gettype(type); } elog(ERROR, "unrecognized type \"%s\"", type); @@ -977,17 +967,20 @@ boot_get_type_io_data(Oid typid, Oid *typinput, Oid *typoutput) { - if (Typ != NULL) + if (Typ != NIL) { /* We have the boot-time contents of pg_type, so use it */ - struct typmap **app; - struct typmap *ap; - - app = Typ; - while (*app && (*app)->am_oid != typid) - ++app; - ap = *app; - if (ap == NULL) + struct typmap *ap = NULL; + ListCell *lc; + + foreach (lc, Typ) + { + ap = lfirst(lc); + if (ap->am_oid == typid) + break; + } + + if (!ap || ap->am_oid != typid) elog(ERROR, "type OID %u not found in Typ list", typid); *typlen = ap->am_typ.typlen; -- 2.26.2 --------------BCA1373AE29B32B57608F4EE Content-Type: text/x-patch; charset=UTF-8; name="0002-Allow-composite-types-in-bootstrap-20210307.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="0002-Allow-composite-types-in-bootstrap-20210307.patch" ^ permalink raw reply [nested|flat] 14+ messages in thread
* pg16: invalid page/page verification failed @ 2023-10-05 16:45 Justin Pryzby <[email protected]> 0 siblings, 1 reply; 14+ messages in thread From: Justin Pryzby @ 2023-10-05 16:45 UTC (permalink / raw) To: [email protected] On an instance running pg16.0: log_time | 2023-10-05 10:03:00.014-05 backend_type | autovacuum worker left | page verification failed, calculated checksum 5074 but expected 5050 context | while scanning block 119 of relation "public.postgres_log_2023_10_05_0900" This is the only error I've seen so far, and for all I know there's a issue on the storage behind the VM, or a cosmic ray hit. But I moved the table out of the way and saved a copy of get_raw_page() in case someone wants to ask about it. public | BROKEN_postgres_log_2023_10_05_0900 | table | postgres | permanent | heap | 1664 kB This table is what it sounds like: a partition into which CSV logs are COPY'ed. It would've been created around 8am. There's no special params set for the table nor for autovacuum. Although we have a ZFS tablespace, these tables aren't on it, and full_page_writes=on. There's no crashes, and the instance has been up since it was pg_upgraded from v15.4 on sep25. pg_stat_all_tables indicates that the table was never (successfully) vacuumed. This was compiled to RPM on centos7, and might include a few commits made since v16.0. postgres=# SELECT * FROM heap_page_item_attrs(get_raw_page(801594131::regclass::text, 119), 801594131); lp | lp_off | lp_flags | lp_len | t_xmin | t_xmax | t_field3 | t_ctid | t_infomask2 | t_infomask | t_hoff | t_bits | t_oid | t_attrs 1 | 2304 | 1 | 16 | | | | | | | | | | 2 | 8160 | 1 | 16 | | | | | | | | | | 3 | 8144 | 1 | 16 | | | | | | | | | | ...all the same except for lp_off... 365 | 2352 | 1 | 16 | | | | | | | | | | 366 | 2336 | 1 | 16 | | | | | | | | | | 367 | 2320 | 1 | 16 | | | | | | | | | | postgres=# SELECT FROM (SELECT tuple_data_split(801594131, t_data, t_infomask, t_infomask2, t_bits) a FROM heap_page_items(get_raw_page(801594131::regclass::text, 119))) WHERE a IS NOT NULL; WARNING: page verification failed, calculated checksum 5074 but expected 5050 (0 rows) -- Justin ^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: pg16: invalid page/page verification failed @ 2023-10-05 17:16 Matthias van de Meent <[email protected]> parent: Justin Pryzby <[email protected]> 0 siblings, 1 reply; 14+ messages in thread From: Matthias van de Meent @ 2023-10-05 17:16 UTC (permalink / raw) To: Justin Pryzby <[email protected]>; +Cc: [email protected] On Thu, 5 Oct 2023 at 18:48, Justin Pryzby <[email protected]> wrote: > > On an instance running pg16.0: > > log_time | 2023-10-05 10:03:00.014-05 > backend_type | autovacuum worker > left | page verification failed, calculated checksum 5074 but expected 5050 > context | while scanning block 119 of relation "public.postgres_log_2023_10_05_0900" > > This is the only error I've seen so far, and for all I know there's a > issue on the storage behind the VM, or a cosmic ray hit. But I moved > the table out of the way and saved a copy of get_raw_page() in case > someone wants to ask about it. > > postgres=# SELECT * FROM heap_page_item_attrs(get_raw_page(801594131::regclass::text, 119), 801594131); > lp | lp_off | lp_flags | lp_len | t_xmin | t_xmax | t_field3 | t_ctid | t_infomask2 | t_infomask | t_hoff | t_bits | t_oid | t_attrs > 1 | 2304 | 1 | 16 | | | | | | | | | | > 2 | 8160 | 1 | 16 | | | | | | | | | | > 3 | 8144 | 1 | 16 | | | | | | | | | | > ...all the same except for lp_off... > 365 | 2352 | 1 | 16 | | | | | | | | | | > 366 | 2336 | 1 | 16 | | | | | | | | | | > 367 | 2320 | 1 | 16 | | | | | | | | | | That's not a HEAP page; it looks more like a btree page: lp_len is too short for heap (which starts at lp_len = 24), and there are too many line pointers for an 8KiB heap page. btree often has lp_len of 16: 8 bytes indextuple header, one maxalign of data (e.g. int or bigint). So, assuming it's a block of a different relation kind, then it's also likely it was originally located elsewhere in that other relation, indeed causing the checksum failure. You can further validate this by looking at the page header's pd_special value - if it is 8176, that'd be another indicator for it being a btree. Kind regards, Matthias van de Meent. ^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: pg16: invalid page/page verification failed @ 2023-10-05 18:48 Justin Pryzby <[email protected]> parent: Matthias van de Meent <[email protected]> 0 siblings, 0 replies; 14+ messages in thread From: Justin Pryzby @ 2023-10-05 18:48 UTC (permalink / raw) To: Matthias van de Meent <[email protected]>; +Cc: [email protected] On Thu, Oct 05, 2023 at 07:16:31PM +0200, Matthias van de Meent wrote: > On Thu, 5 Oct 2023 at 18:48, Justin Pryzby <[email protected]> wrote: > > > > On an instance running pg16.0: > > > > log_time | 2023-10-05 10:03:00.014-05 > > backend_type | autovacuum worker > > left | page verification failed, calculated checksum 5074 but expected 5050 > > context | while scanning block 119 of relation "public.postgres_log_2023_10_05_0900" > > > > This is the only error I've seen so far, and for all I know there's a > > issue on the storage behind the VM, or a cosmic ray hit. But I moved > > the table out of the way and saved a copy of get_raw_page() in case > > someone wants to ask about it. > > > > postgres=# SELECT * FROM heap_page_item_attrs(get_raw_page(801594131::regclass::text, 119), 801594131); > > lp | lp_off | lp_flags | lp_len | t_xmin | t_xmax | t_field3 | t_ctid | t_infomask2 | t_infomask | t_hoff | t_bits | t_oid | t_attrs > > 1 | 2304 | 1 | 16 | | | | | | | | | | > > 2 | 8160 | 1 | 16 | | | | | | | | | | > > 3 | 8144 | 1 | 16 | | | | | | | | | | > > ...all the same except for lp_off... > > 365 | 2352 | 1 | 16 | | | | | | | | | | > > 366 | 2336 | 1 | 16 | | | | | | | | | | > > 367 | 2320 | 1 | 16 | | | | | | | | | | > > That's not a HEAP page; it looks more like a btree page: lp_len is too > short for heap (which starts at lp_len = 24), and there are too many > line pointers for an 8KiB heap page. btree often has lp_len of 16: 8 > bytes indextuple header, one maxalign of data (e.g. int or bigint). > > So, assuming it's a block of a different relation kind, then it's also > likely it was originally located elsewhere in that other relation, > indeed causing the checksum failure. You can further validate this by > looking at the page header's pd_special value - if it is 8176, that'd > be another indicator for it being a btree. Nice point. postgres=# SET ignore_checksum_failure=on; SELECT * FROM generate_series(115,119) AS a, page_header(get_raw_page(801594131::regclass::text, a)) AS b; WARNING: page verification failed, calculated checksum 5074 but expected 5050 a | lsn | checksum | flags | lower | upper | special | pagesize | version | prune_xid -----+--------------+----------+-------+-------+-------+---------+----------+---------+----------- 115 | B61/A9436C8 | -23759 | 4 | 92 | 336 | 8192 | 8192 | 4 | 0 116 | B61/A944FA0 | 3907 | 4 | 104 | 224 | 8192 | 8192 | 4 | 0 117 | B61/A946828 | -24448 | 4 | 76 | 264 | 8192 | 8192 | 4 | 0 118 | B61/A94CCE0 | 26915 | 4 | 28 | 6256 | 8192 | 8192 | 4 | 0 119 | B5C/9F30D1C8 | 5050 | 0 | 1492 | 2304 | 8176 | 8192 | 4 | 0 The table itself has a few btree indexes on text columns and a brin index on log_timestamp, but not on the integers. It sounds like it's what's expected at this point, but after I "SET ignore_checksum_failure=on", and read the page in, vacuum kicked off and then crashed (in heap_page_prune() if that half of the stack trace can be trusted). *** stack smashing detected ***: postgres: autovacuum worker postgres terminated < 2023-10-05 12:35:30.764 CDT >LOG: server process (PID 30692) was terminated by signal 11: Segmentation fault < 2023-10-05 12:35:30.764 CDT >DETAIL: Failed process was running: autovacuum: VACUUM ANALYZE public.BROKEN_postgres_log_2023_10_05_0900 I took the opportunity to fsck the FS, which showed no errors. I was curious if the relfilenodes had gotten confused/corrupted/?? But this seems to indicate not; the problem is only one block. postgres=# SELECT oid, relfilenode, oid=relfilenode, relname FROM pg_class WHERE oid BETWEEN 801550000 AND 801594199 ORDER BY 1; oid | relfilenode | ?column? | relname -----------+-------------+----------+------------------------------------------------- 801564542 | 801564542 | t | postgres_log_2023_10_05_0800 801564545 | 801564545 | t | pg_toast_801564542 801564546 | 801564546 | t | pg_toast_801564542_index 801564547 | 801564547 | t | postgres_log_2023_10_05_0800_log_time_idx 801564548 | 801564548 | t | postgres_log_2023_10_05_0800_error_severity_idx 801564549 | 801564549 | t | postgres_log_2023_10_05_0800_error_message_idx 801564550 | 801564550 | t | postgres_log_2023_10_05_0800_duration_idx 801564551 | 801564551 | t | postgres_log_2023_10_05_0800_tempfile_idx 801594131 | 801594131 | t | BROKEN_postgres_log_2023_10_05_0900 801594134 | 801594134 | t | pg_toast_801594131 801594135 | 801594135 | t | pg_toast_801594131_index 801594136 | 801594136 | t | postgres_log_2023_10_05_0900_log_time_idx 801594137 | 801594137 | t | postgres_log_2023_10_05_0900_error_severity_idx 801594138 | 801594138 | t | postgres_log_2023_10_05_0900_error_message_idx 801594139 | 801594139 | t | postgres_log_2023_10_05_0900_duration_idx 801594140 | 801594140 | t | postgres_log_2023_10_05_0900_tempfile_idx Before anybody asks, we didn't retain WAL from this morning. FYI, the storage is ext4/LVM/scsi (it looks like this didn't use vmw_pvscsi but an emulated hardware driver). /dev/mapper/data-postgres on /var/lib/pgsql type ext4 (rw,relatime,seclabel,data=ordered) [ 0.000000] Linux version 3.10.0-1160.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Mon Oct 19 16:18:59 UTC 2020 [ 1.446380] scsi 2:0:1:0: Direct-Access VMware Virtual disk 1.0 PQ: 0 ANSI: 2 [ 1.470764] scsi target2:0:1: Beginning Domain Validation [ 1.471077] scsi target2:0:1: Domain Validation skipping write tests [ 1.471079] scsi target2:0:1: Ending Domain Validation [ 1.471099] scsi target2:0:1: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127) [ 1.484109] sd 2:0:1:0: [sdb] 1048576000 512-byte logical blocks: (536 GB/500 GiB) [ 1.484136] sd 2:0:1:0: [sdb] Write Protect is off [ 1.484139] sd 2:0:1:0: [sdb] Mode Sense: 45 00 00 00 [ 1.484163] sd 2:0:1:0: [sdb] Write cache: disabled, read cache: disabled, doesn't support DPO or FUA [ 1.485808] sd 2:0:1:0: [sdb] Attached SCSI disk [ 4.271339] sd 2:0:1:0: Attached scsi generic sg1 type 0 -- Justin ^ permalink raw reply [nested|flat] 14+ messages in thread
end of thread, other threads:[~2023-10-05 18:48 UTC | newest] Thread overview: 14+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2016-07-27 06:47 [PATCH 6/7] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 06:47 [PATCH 6/7] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 06:47 [PATCH 6/7] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 06:47 [PATCH 6/7] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 07:59 [PATCH 7/8] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 07:59 [PATCH 7/8] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 07:59 [PATCH 7/8] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 07:59 [PATCH 7/8] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 07:59 [PATCH 7/8] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 07:59 [PATCH 7/8] Tuple routing for partitioned tables. amit <[email protected]> 2020-11-20 02:48 [PATCH 1/4] bootstrap: convert Typ to a List* Justin Pryzby <[email protected]> 2023-10-05 16:45 pg16: invalid page/page verification failed Justin Pryzby <[email protected]> 2023-10-05 17:16 ` Re: pg16: invalid page/page verification failed Matthias van de Meent <[email protected]> 2023-10-05 18:48 ` Re: pg16: invalid page/page verification failed Justin Pryzby <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox