agora inbox for [email protected]help / color / mirror / Atom feed
[PATCH 6/7] Tuple routing for partitioned tables. 13+ messages / 4 participants [nested] [flat]
* [PATCH 6/7] Tuple routing for partitioned tables. @ 2016-07-27 06:47 amit <[email protected]> 0 siblings, 0 replies; 13+ messages in thread From: amit @ 2016-07-27 06:47 UTC (permalink / raw) Both COPY FROM and INSERT are covered by this commit. Routing to foreing partitions is not supported at the moment. To implement tuple-routing, introduce a PartitionDispatch data structure. Each partitioned table in a partition tree gets one and contains info such as a pointer to its partition descriptor, partition key execution state, global sequence numbers of its leaf partitions and a way to link to the PartitionDispatch objects of any of its partitions that are partitioned themselves. Starting with the PartitionDispatch object of the root partitioned table and a tuple to route, one can get the global sequence number of the leaf partition that the tuple gets routed to, if one exists. --- src/backend/catalog/partition.c | 342 +++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 151 ++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 58 ++++++- src/backend/executor/nodeModifyTable.c | 130 ++++++++++++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 11 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 8 + src/test/regress/expected/insert.out | 52 +++++ src/test/regress/sql/insert.sql | 25 +++ 11 files changed, 786 insertions(+), 6 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index 197fcef..710829a 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -113,6 +113,28 @@ typedef struct PartitionRangeBound bool lower; /* this is the lower (vs upper) bound */ } PartitionRangeBound; +/*----------------------- + * PartitionDispatch - information about one partitioned table in a partition + * hiearchy required to route a tuple to one of its partitions + * + * relid OID of the table + * key Partition key information of the table + * keystate Execution state required for expressions in the partition key + * partdesc Partition descriptor of the table + * indexes Array with partdesc->nparts members (for details on what + * individual members represent, see how they are set in + * RelationGetPartitionDispatchInfo()) + *----------------------- + */ +typedef struct PartitionDispatchData +{ + Oid relid; + PartitionKey key; + List *keystate; /* list of ExprState */ + PartitionDesc partdesc; + int *indexes; +} PartitionDispatchData; + static int32 qsort_partition_list_value_cmp(const void *a, const void *b, void *arg); static int32 qsort_partition_rbound_cmp(const void *a, const void *b, void *arg); @@ -126,12 +148,22 @@ static PartitionRangeBound *make_one_range_bound(PartitionKey key, int index, Li static int32 partition_rbound_cmp(PartitionKey key, Datum *datums1, RangeDatumContent *content1, bool lower1, PartitionRangeBound *b2); +static int32 partition_rbound_datum_cmp(PartitionKey key, + Datum *rb_datums, RangeDatumContent *rb_content, + Datum *tuple_datums); static int32 partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, int offset, void *probe, bool probe_is_bound); static int partition_bound_bsearch(PartitionKey key, PartitionBoundInfo boundinfo, void *probe, bool probe_is_bound, bool *is_equal); +/* Support get_partition_for_tuple() */ +static void FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); + /* * RelationBuildPartitionDesc * Form rel's partition descriptor @@ -895,6 +927,115 @@ RelationGetPartitionQual(Relation rel, bool recurse) return generate_partition_qual(rel, recurse); } +/* Turn an array of OIDs with N elements into a list */ +#define OID_ARRAY_TO_LIST(arr, N, list) \ + do\ + {\ + int i;\ + for (i = 0; i < (N); i++)\ + (list) = lappend_oid((list), (arr)[i]);\ + } while(0) + +/* + * RelationGetPartitionDispatchInfo + * Returns information necessary to route tuples down a partition tree + * + * All the partitions will be locked with lockmode, unless it is NoLock. + * A list of the OIDs of all the leaf partition of rel is returned in + * *leaf_part_oids. + */ +PartitionDispatch * +RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + List **leaf_part_oids) +{ + PartitionDesc rootpartdesc = RelationGetPartitionDesc(rel); + PartitionDispatchData **pd; + List *all_parts = NIL, + *parted_rels = NIL; + ListCell *lc; + int i, + k, + num_parted; + + /* + * Lock partitions and collect OIDs of the partitioned ones to prepare + * their PartitionDispatch objects. + * + * Cannot use find_all_inheritors() here, because then the order of OIDs + * in parted_rels list would be unknown, which does not help because down + * below, we assign indexes within individual PartitionDispatch in an + * order that's predetermined (determined by the order of OIDs in + * individual partition descriptors). + */ + parted_rels = lappend_oid(parted_rels, RelationGetRelid(rel)); + num_parted = 1; + OID_ARRAY_TO_LIST(rootpartdesc->oids, rootpartdesc->nparts, all_parts); + foreach(lc, all_parts) + { + Relation partrel = heap_open(lfirst_oid(lc), lockmode); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + + /* + * If this partition is a partitined table, add its children to to the + * end of the list, so that they are processed as well. + */ + if (partdesc) + { + num_parted++; + parted_rels = lappend_oid(parted_rels, lfirst_oid(lc)); + OID_ARRAY_TO_LIST(partdesc->oids, partdesc->nparts, all_parts); + } + + heap_close(partrel, NoLock); + } + + /* Generate PartitionDispatch objects for all partitioned tables */ + pd = (PartitionDispatchData **) palloc(num_parted * + sizeof(PartitionDispatchData *)); + *leaf_part_oids = NIL; + i = k = 0; + foreach(lc, parted_rels) + { + /* We locked all partitions above */ + Relation partrel = heap_open(lfirst_oid(lc), NoLock); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + int j, + m; + + pd[i] = (PartitionDispatch) palloc(sizeof(PartitionDispatchData)); + pd[i]->relid = RelationGetRelid(partrel); + pd[i]->key = RelationGetPartitionKey(partrel); + pd[i]->keystate = NIL; + pd[i]->partdesc = partdesc; + pd[i]->indexes = (int *) palloc(partdesc->nparts * sizeof(int)); + heap_close(partrel, NoLock); + + m = 0; + for (j = 0; j < partdesc->nparts; j++) + { + Oid partrelid = partdesc->oids[j]; + + if (get_rel_relkind(partrelid) != RELKIND_PARTITIONED_TABLE) + { + *leaf_part_oids = lappend_oid(*leaf_part_oids, partrelid); + pd[i]->indexes[j] = k++; + } + else + { + /* + * We can assign indexes this way because of the way + * parted_rels has been generated. + */ + pd[i]->indexes[j] = -(i + 1 + m); + m++; + } + } + i++; + } + + return pd; +} + /* Module-local functions */ /* @@ -1329,6 +1470,172 @@ generate_partition_qual(Relation rel, bool recurse) return result; } +/* ---------------- + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for the partition key + * of a tuple. + * + * pkinfo partition key execution info + * slot Heap tuple from which to extract partition key + * estate executor state for evaluating any partition key + * expressions (must be non-NULL) + * values Array of partition key Datums (output area) + * isnull Array of is-null indicators (output area) + * + * the ecxt_scantuple slot of estate's per-tuple expr context must point to + * the heap tuple passed in. + * ---------------- + */ +static void +FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pd->key->partexprs != NIL && pd->keystate == NIL) + { + /* Check caller has set up context correctly */ + Assert(estate != NULL && + GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + + /* First time through, set up expression evaluation state */ + pd->keystate = (List *) ExecPrepareExpr((Expr *) pd->key->partexprs, + estate); + } + + partexpr_item = list_head(pd->keystate); + for (i = 0; i < pd->key->partnatts; i++) + { + AttrNumber keycol = pd->key->partattrs[i]; + Datum datum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + datum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + datum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = datum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Finds a leaf partition for tuple contained in *slot + * + * Returned value is the sequence number of the leaf partition thus found, + * or -1 if no leaf partition is found for the tuple. *failed_at is set + * to the OID of the partitioned table whose partition was not found in + * the latter case. + */ +int +get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + PartitionDispatch parent; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int cur_offset, + cur_index; + int i; + + /* start with the root partitioned table */ + parent = pd[0]; + while(true) + { + PartitionKey key = parent->key; + PartitionDesc partdesc = parent->partdesc; + + /* Quick exit */ + if (partdesc->nparts == 0) + { + *failed_at = parent->relid; + return -1; + } + + /* Extract partition key from tuple */ + FormPartitionKeyDatum(parent, slot, estate, values, isnull); + + if (key->strategy == PARTITION_STRATEGY_RANGE) + { + /* Disallow nulls in the range partition key of the tuple */ + for (i = 0; i < key->partnatts; i++) + if (isnull[i]) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key of row contains null"))); + } + + if (partdesc->boundinfo->has_null && isnull[0]) + /* Tuple maps to the null-accepting list partition */ + cur_index = partdesc->boundinfo->null_index; + else + { + /* Else bsearch in partdesc->boundinfo */ + bool equal = false; + + cur_offset = partition_bound_bsearch(key, partdesc->boundinfo, + values, false, &equal); + switch (key->strategy) + { + case PARTITION_STRATEGY_LIST: + if (cur_offset >= 0 && equal) + cur_index = partdesc->boundinfo->indexes[cur_offset]; + else + cur_index = -1; + break; + + case PARTITION_STRATEGY_RANGE: + /* + * Offset returned is such that the bound at offset is + * found to be less or equal with the tuple. So, the + * bound at offset+1 would be the upper bound. + */ + cur_index = partdesc->boundinfo->indexes[cur_offset+1]; + break; + } + } + + /* + * cur_index < 0 means we failed to find a partition of this parent. + * cur_index >= 0 means we either found the leaf partition, or the + * next parent to find a partition of. + */ + if (cur_index < 0) + { + *failed_at = parent->relid; + return -1; + } + else if (parent->indexes[cur_index] < 0) + parent = pd[-parent->indexes[cur_index]]; + else + break; + } + + return parent->indexes[cur_index]; +} + /* * qsort_partition_list_value_cmp * @@ -1461,6 +1768,36 @@ partition_rbound_cmp(PartitionKey key, } /* + * partition_rbound_datum_cmp + * + * Return whether range bound (specified in rb_datums, rb_content, and + * rb_lower) <=, =, >= partition key of tuple (tuple_datums) + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, + Datum *rb_datums, RangeDatumContent *rb_content, + Datum *tuple_datums) +{ + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (rb_content[i] != RANGE_DATUM_FINITE) + return rb_content[i] == RANGE_DATUM_NEG_INF ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + rb_datums[i], + tuple_datums[i])); + if (cmpval != 0) + break; + } + + return cmpval; +} + +/* * partition_bound_cmp * * Return whether the bound at offset in boundinfo is <=, =, >= the argument @@ -1499,7 +1836,10 @@ partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, bound_datums, content, lower, (PartitionRangeBound *) probe); } - + else + cmpval = partition_rbound_datum_cmp(key, + bound_datums, content, + (Datum *) probe); break; } } diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..7d76ead 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -161,6 +161,10 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionDispatch *partition_dispatch_info; + int num_partitions; + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1401,67 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + List *leaf_parts; + ListCell *cell; + int i, + num_leaf_parts; + ResultRelInfo *leaf_part_rri; + + /* Get the tuple-routing information and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + num_leaf_parts = list_length(leaf_parts); + cstate->partition_dispatch_info = pd; + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) palloc(num_leaf_parts * + sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation partrel; + + /* + * All partitions locked above; will be closed after CopyFrom is + * finished. + */ + partrel = heap_open(lfirst_oid(cell), NoLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(partrel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + partrel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(tupDesc, RelationGetDescr(partrel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(partrel), + gettext_noop("could not convert row type")); + leaf_part_rri++; + i++; + } + } } else { @@ -2255,6 +2320,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2347,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2458,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2486,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->partition_dispatch_info != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2442,7 +2511,11 @@ CopyFrom(CopyState cstate) values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2567,56 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition to heap_insert the tuple into */ + if (cstate->partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + cstate->partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < cstate->num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding + * to the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = cstate->partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* + * For ExecInsertIndexTuples() to work on the partition's indexes + */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the + * partition rowtype. + */ + map = cstate->partition_tupconv_maps[leaf_part_index]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2553,7 +2676,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2701,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2720,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); @@ -2614,6 +2745,20 @@ CopyFrom(CopyState cstate) ExecCloseIndices(resultRelInfo); + /* Close all partitions and indices thereof */ + if (cstate->partition_dispatch_info) + { + int i; + + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + } + FreeExecutorState(estate); /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 99c9d10..8a8eb01 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1322,6 +1322,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index c7a6347..54fb771 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2990,3 +2995,52 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +/* + * ExecFindPartition -- Find a leaf partition in the partition tree rooted + * at parent, for the heap tuple contained in *slot + * + * estate must be non-NULL; we'll need it to compute any expressions in the + * partition key(s) + * + * If no leaf partition is found, this routine errors out with the appropriate + * error message, else it returns the leaf partition sequence number returned + * by get_partition_for_tuple() unchanged. + */ +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, + TupleTableSlot *slot, EState *estate) +{ + int result; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + result = get_partition_for_tuple(pd, slot, estate, &failed_at); + if (result < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return result; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 6eccfb7..87a04aa 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,56 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + /* Determine the partition to heap_insert the tuple into */ + if (mtstate->mt_partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, false); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +562,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1622,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1713,69 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + /* Form the partition node tree and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + mtstate->mt_partition_dispatch_info = pd; + num_leaf_parts = list_length(leaf_parts); + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2093,15 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + /* * Free the exprcontext */ diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 36f8c54..88380ba 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -806,8 +806,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 70d8325..f76c5d9 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -36,6 +38,7 @@ typedef struct PartitionDescData } PartitionDescData; typedef struct PartitionDescData *PartitionDesc; +typedef struct PartitionDispatchData *PartitionDispatch; extern void RelationBuildPartitionDesc(Relation relation); extern bool partition_bounds_equal(PartitionKey key, @@ -45,4 +48,12 @@ extern void check_new_partition_bound(char *relname, Relation parent, Node *boun extern Oid get_partition_parent(Oid relid); extern List *get_qual_from_partbound(Relation rel, Relation parent, Node *bound); extern List *RelationGetPartitionQual(Relation rel, bool recurse); + +/* For tuple routing */ +extern PartitionDispatch *RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + List **leaf_part_oids); +extern int get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..b4d09f9 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..606cb21 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,13 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionDispatchData **mt_partition_dispatch_info; + /* Tuple-routing support info */ + int mt_num_partitions; /* Number of members in the + * following arrays */ + ResultRelInfo *mt_partitions; /* Per partition result relation */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per partition tuple conversion map */ } ModifyTableState; /* ---------------- diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index cc9f957..9706034 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -223,6 +223,58 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index 3a9430e..afecb74 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -137,6 +137,31 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------96825311778D254EC2B71375 Content-Type: text/x-diff; name="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-18.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-18"; filename*1=".patch" ^ permalink raw reply [nested|flat] 13+ messages in thread
* [PATCH 6/7] Tuple routing for partitioned tables. @ 2016-07-27 06:47 amit <[email protected]> 0 siblings, 0 replies; 13+ messages in thread From: amit @ 2016-07-27 06:47 UTC (permalink / raw) Both COPY FROM and INSERT are covered by this commit. Routing to foreing partitions is not supported at the moment. To implement tuple-routing, introduce a PartitionDispatch data structure. Each partitioned table in a partition tree gets one and contains info such as a pointer to its partition descriptor, partition key execution state, global sequence numbers of its leaf partitions and a way to link to the PartitionDispatch objects of any of its partitions that are partitioned themselves. Starting with the PartitionDispatch object of the root partitioned table and a tuple to route, one can get the global sequence number of the leaf partition that the tuple gets routed to, if one exists. --- src/backend/catalog/partition.c | 342 +++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 154 ++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 58 ++++++- src/backend/executor/nodeModifyTable.c | 130 ++++++++++++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 11 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 8 + src/test/regress/expected/insert.out | 55 +++++ src/test/regress/sql/insert.sql | 27 +++ 11 files changed, 794 insertions(+), 6 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index f439c43..83dc151 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -113,6 +113,28 @@ typedef struct PartitionRangeBound bool lower; /* this is the lower (vs upper) bound */ } PartitionRangeBound; +/*----------------------- + * PartitionDispatch - information about one partitioned table in a partition + * hiearchy required to route a tuple to one of its partitions + * + * relid OID of the table + * key Partition key information of the table + * keystate Execution state required for expressions in the partition key + * partdesc Partition descriptor of the table + * indexes Array with partdesc->nparts members (for details on what + * individual members represent, see how they are set in + * RelationGetPartitionDispatchInfo()) + *----------------------- + */ +typedef struct PartitionDispatchData +{ + Oid relid; + PartitionKey key; + List *keystate; /* list of ExprState */ + PartitionDesc partdesc; + int *indexes; +} PartitionDispatchData; + static int32 qsort_partition_list_value_cmp(const void *a, const void *b, void *arg); static int32 qsort_partition_rbound_cmp(const void *a, const void *b, void *arg); @@ -126,12 +148,22 @@ static PartitionRangeBound *make_one_range_bound(PartitionKey key, int index, Li static int32 partition_rbound_cmp(PartitionKey key, Datum *datums1, RangeDatumContent *content1, bool lower1, PartitionRangeBound *b2); +static int32 partition_rbound_datum_cmp(PartitionKey key, + Datum *rb_datums, RangeDatumContent *rb_content, + Datum *tuple_datums); static int32 partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, int offset, void *probe, bool probe_is_bound); static int partition_bound_bsearch(PartitionKey key, PartitionBoundInfo boundinfo, void *probe, bool probe_is_bound, bool *is_equal); +/* Support get_partition_for_tuple() */ +static void FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); + /* * RelationBuildPartitionDesc * Form rel's partition descriptor @@ -895,6 +927,115 @@ RelationGetPartitionQual(Relation rel, bool recurse) return generate_partition_qual(rel, recurse); } +/* Turn an array of OIDs with N elements into a list */ +#define OID_ARRAY_TO_LIST(arr, N, list) \ + do\ + {\ + int i;\ + for (i = 0; i < (N); i++)\ + (list) = lappend_oid((list), (arr)[i]);\ + } while(0) + +/* + * RelationGetPartitionDispatchInfo + * Returns information necessary to route tuples down a partition tree + * + * All the partitions will be locked with lockmode, unless it is NoLock. + * A list of the OIDs of all the leaf partition of rel is returned in + * *leaf_part_oids. + */ +PartitionDispatch * +RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + List **leaf_part_oids) +{ + PartitionDesc rootpartdesc = RelationGetPartitionDesc(rel); + PartitionDispatchData **pd; + List *all_parts = NIL, + *parted_rels = NIL; + ListCell *lc; + int i, + k, + num_parted; + + /* + * Lock partitions and collect OIDs of the partitioned ones to prepare + * their PartitionDispatch objects. + * + * Cannot use find_all_inheritors() here, because then the order of OIDs + * in parted_rels list would be unknown, which does not help because down + * below, we assign indexes within individual PartitionDispatch in an + * order that's predetermined (determined by the order of OIDs in + * individual partition descriptors). + */ + parted_rels = lappend_oid(parted_rels, RelationGetRelid(rel)); + num_parted = 1; + OID_ARRAY_TO_LIST(rootpartdesc->oids, rootpartdesc->nparts, all_parts); + foreach(lc, all_parts) + { + Relation partrel = heap_open(lfirst_oid(lc), lockmode); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + + /* + * If this partition is a partitined table, add its children to to the + * end of the list, so that they are processed as well. + */ + if (partdesc) + { + num_parted++; + parted_rels = lappend_oid(parted_rels, lfirst_oid(lc)); + OID_ARRAY_TO_LIST(partdesc->oids, partdesc->nparts, all_parts); + } + + heap_close(partrel, NoLock); + } + + /* Generate PartitionDispatch objects for all partitioned tables */ + pd = (PartitionDispatchData **) palloc(num_parted * + sizeof(PartitionDispatchData *)); + *leaf_part_oids = NIL; + i = k = 0; + foreach(lc, parted_rels) + { + /* We locked all partitions above */ + Relation partrel = heap_open(lfirst_oid(lc), NoLock); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + int j, + m; + + pd[i] = (PartitionDispatch) palloc(sizeof(PartitionDispatchData)); + pd[i]->relid = RelationGetRelid(partrel); + pd[i]->key = RelationGetPartitionKey(partrel); + pd[i]->keystate = NIL; + pd[i]->partdesc = partdesc; + pd[i]->indexes = (int *) palloc(partdesc->nparts * sizeof(int)); + heap_close(partrel, NoLock); + + m = 0; + for (j = 0; j < partdesc->nparts; j++) + { + Oid partrelid = partdesc->oids[j]; + + if (get_rel_relkind(partrelid) != RELKIND_PARTITIONED_TABLE) + { + *leaf_part_oids = lappend_oid(*leaf_part_oids, partrelid); + pd[i]->indexes[j] = k++; + } + else + { + /* + * We can assign indexes this way because of the way + * parted_rels has been generated. + */ + pd[i]->indexes[j] = -(i + 1 + m); + m++; + } + } + i++; + } + + return pd; +} + /* Module-local functions */ /* @@ -1346,6 +1487,172 @@ generate_partition_qual(Relation rel, bool recurse) return result; } +/* ---------------- + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for the partition key + * of a tuple. + * + * pkinfo partition key execution info + * slot Heap tuple from which to extract partition key + * estate executor state for evaluating any partition key + * expressions (must be non-NULL) + * values Array of partition key Datums (output area) + * isnull Array of is-null indicators (output area) + * + * the ecxt_scantuple slot of estate's per-tuple expr context must point to + * the heap tuple passed in. + * ---------------- + */ +static void +FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pd->key->partexprs != NIL && pd->keystate == NIL) + { + /* Check caller has set up context correctly */ + Assert(estate != NULL && + GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + + /* First time through, set up expression evaluation state */ + pd->keystate = (List *) ExecPrepareExpr((Expr *) pd->key->partexprs, + estate); + } + + partexpr_item = list_head(pd->keystate); + for (i = 0; i < pd->key->partnatts; i++) + { + AttrNumber keycol = pd->key->partattrs[i]; + Datum datum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + datum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + datum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = datum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Finds a leaf partition for tuple contained in *slot + * + * Returned value is the sequence number of the leaf partition thus found, + * or -1 if no leaf partition is found for the tuple. *failed_at is set + * to the OID of the partitioned table whose partition was not found in + * the latter case. + */ +int +get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + PartitionDispatch parent; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int cur_offset, + cur_index; + int i; + + /* start with the root partitioned table */ + parent = pd[0]; + while(true) + { + PartitionKey key = parent->key; + PartitionDesc partdesc = parent->partdesc; + + /* Quick exit */ + if (partdesc->nparts == 0) + { + *failed_at = parent->relid; + return -1; + } + + /* Extract partition key from tuple */ + FormPartitionKeyDatum(parent, slot, estate, values, isnull); + + if (key->strategy == PARTITION_STRATEGY_RANGE) + { + /* Disallow nulls in the range partition key of the tuple */ + for (i = 0; i < key->partnatts; i++) + if (isnull[i]) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key of row contains null"))); + } + + if (partdesc->boundinfo->has_null && isnull[0]) + /* Tuple maps to the null-accepting list partition */ + cur_index = partdesc->boundinfo->null_index; + else + { + /* Else bsearch in partdesc->boundinfo */ + bool equal = false; + + cur_offset = partition_bound_bsearch(key, partdesc->boundinfo, + values, false, &equal); + switch (key->strategy) + { + case PARTITION_STRATEGY_LIST: + if (cur_offset >= 0 && equal) + cur_index = partdesc->boundinfo->indexes[cur_offset]; + else + cur_index = -1; + break; + + case PARTITION_STRATEGY_RANGE: + /* + * Offset returned is such that the bound at offset is + * found to be less or equal with the tuple. So, the + * bound at offset+1 would be the upper bound. + */ + cur_index = partdesc->boundinfo->indexes[cur_offset+1]; + break; + } + } + + /* + * cur_index < 0 means we failed to find a partition of this parent. + * cur_index >= 0 means we either found the leaf partition, or the + * next parent to find a partition of. + */ + if (cur_index < 0) + { + *failed_at = parent->relid; + return -1; + } + else if (parent->indexes[cur_index] < 0) + parent = pd[-parent->indexes[cur_index]]; + else + break; + } + + return parent->indexes[cur_index]; +} + /* * qsort_partition_list_value_cmp * @@ -1478,6 +1785,36 @@ partition_rbound_cmp(PartitionKey key, } /* + * partition_rbound_datum_cmp + * + * Return whether range bound (specified in rb_datums, rb_content, and + * rb_lower) <=, =, >= partition key of tuple (tuple_datums) + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, + Datum *rb_datums, RangeDatumContent *rb_content, + Datum *tuple_datums) +{ + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (rb_content[i] != RANGE_DATUM_FINITE) + return rb_content[i] == RANGE_DATUM_NEG_INF ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + rb_datums[i], + tuple_datums[i])); + if (cmpval != 0) + break; + } + + return cmpval; +} + +/* * partition_bound_cmp * * Return whether the bound at offset in boundinfo is <=, =, >= the argument @@ -1516,7 +1853,10 @@ partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, bound_datums, content, lower, (PartitionRangeBound *) probe); } - + else + cmpval = partition_rbound_datum_cmp(key, + bound_datums, content, + (Datum *) probe); break; } } diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..3f64f3a 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -161,6 +161,10 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionDispatch *partition_dispatch_info; + int num_partitions; + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1401,67 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + List *leaf_parts; + ListCell *cell; + int i, + num_leaf_parts; + ResultRelInfo *leaf_part_rri; + + /* Get the tuple-routing information and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + num_leaf_parts = list_length(leaf_parts); + cstate->partition_dispatch_info = pd; + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) palloc(num_leaf_parts * + sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation partrel; + + /* + * All partitions locked above; will be closed after CopyFrom is + * finished. + */ + partrel = heap_open(lfirst_oid(cell), NoLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(partrel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + partrel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(tupDesc, RelationGetDescr(partrel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(partrel), + gettext_noop("could not convert row type")); + leaf_part_rri++; + i++; + } + } } else { @@ -2255,6 +2320,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2347,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2458,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2486,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->partition_dispatch_info != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2442,7 +2511,11 @@ CopyFrom(CopyState cstate) values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2567,59 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition to heap_insert the tuple into */ + if (cstate->partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + cstate->partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < cstate->num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding + * to the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = cstate->partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* + * For ExecInsertIndexTuples() to work on the partition's indexes + */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the + * partition rowtype. + */ + map = cstate->partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, true); + } + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2553,7 +2679,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2704,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2723,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); @@ -2614,6 +2748,20 @@ CopyFrom(CopyState cstate) ExecCloseIndices(resultRelInfo); + /* Close all partitions and indices thereof */ + if (cstate->partition_dispatch_info) + { + int i; + + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + } + FreeExecutorState(estate); /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 71ca739..abfb46b 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1322,6 +1322,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index c7a6347..54fb771 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2990,3 +2995,52 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +/* + * ExecFindPartition -- Find a leaf partition in the partition tree rooted + * at parent, for the heap tuple contained in *slot + * + * estate must be non-NULL; we'll need it to compute any expressions in the + * partition key(s) + * + * If no leaf partition is found, this routine errors out with the appropriate + * error message, else it returns the leaf partition sequence number returned + * by get_partition_for_tuple() unchanged. + */ +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, + TupleTableSlot *slot, EState *estate) +{ + int result; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + result = get_partition_for_tuple(pd, slot, estate, &failed_at); + if (result < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return result; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 6eccfb7..ceb5d44 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,56 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + /* Determine the partition to heap_insert the tuple into */ + if (mtstate->mt_partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, true); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +562,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1622,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1713,69 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + /* Form the partition node tree and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + mtstate->mt_partition_dispatch_info = pd; + num_leaf_parts = list_length(leaf_parts); + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2093,15 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + /* * Free the exprcontext */ diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 36f8c54..88380ba 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -806,8 +806,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 70d8325..f76c5d9 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -36,6 +38,7 @@ typedef struct PartitionDescData } PartitionDescData; typedef struct PartitionDescData *PartitionDesc; +typedef struct PartitionDispatchData *PartitionDispatch; extern void RelationBuildPartitionDesc(Relation relation); extern bool partition_bounds_equal(PartitionKey key, @@ -45,4 +48,12 @@ extern void check_new_partition_bound(char *relname, Relation parent, Node *boun extern Oid get_partition_parent(Oid relid); extern List *get_qual_from_partbound(Relation rel, Relation parent, Node *bound); extern List *RelationGetPartitionQual(Relation rel, bool recurse); + +/* For tuple routing */ +extern PartitionDispatch *RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + List **leaf_part_oids); +extern int get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..b4d09f9 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..606cb21 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,13 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionDispatchData **mt_partition_dispatch_info; + /* Tuple-routing support info */ + int mt_num_partitions; /* Number of members in the + * following arrays */ + ResultRelInfo *mt_partitions; /* Per partition result relation */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per partition tuple conversion map */ } ModifyTableState; /* ---------------- diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 2e78fd9..561cefa 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -230,6 +230,61 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_ee_ff1 values ('ff', 1); insert into part_ee_ff2 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +-- fail (partition key (b+0) is null) +insert into range_parted values ('a'); +ERROR: range partition key of row contains null +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------+---+---- + part1 | a | 1 + part1 | a | 1 + part2 | a | 10 + part3 | b | 1 + part4 | b | 10 + part4 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_ee_ff not found in both cases) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_ee_ff values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_ee_ff values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +-------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff1 | ff | 1 + part_ee_ff1 | EE | 1 + part_ee_ff2 | ff | 11 + part_ee_ff2 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index eb92364..846bb58 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,33 @@ insert into part_ee_ff1 values ('cc', 1); insert into part_ee_ff1 values ('ff', 1); insert into part_ee_ff2 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +-- fail (partition key (b+0) is null) +insert into range_parted values ('a'); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_ee_ff not found in both cases) +insert into list_parted values ('EE', 0); +insert into part_ee_ff values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_ee_ff values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------AEEF86BC59A941B2E83AF069 Content-Type: text/x-diff; name="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-19.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-19"; filename*1=".patch" ^ permalink raw reply [nested|flat] 13+ messages in thread
* [PATCH 6/7] Tuple routing for partitioned tables. @ 2016-07-27 06:47 amit <[email protected]> 0 siblings, 0 replies; 13+ messages in thread From: amit @ 2016-07-27 06:47 UTC (permalink / raw) Both COPY FROM and INSERT are covered by this commit. Routing to foreing partitions is not supported at the moment. To implement tuple-routing, introduce a PartitionDispatch data structure. Each partitioned table in a partition tree gets one and contains info such as a pointer to its partition descriptor, partition key execution state, global sequence numbers of its leaf partitions and a way to link to the PartitionDispatch objects of any of its partitions that are partitioned themselves. Starting with the PartitionDispatch object of the root partitioned table and a tuple to route, one can get the global sequence number of the leaf partition that the tuple gets routed to, if one exists. --- src/backend/catalog/partition.c | 328 ++++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 171 ++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 58 +++++- src/backend/executor/nodeModifyTable.c | 147 +++++++++++++++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 35 ++++ src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 10 + src/test/regress/expected/insert.out | 55 ++++++ src/test/regress/sql/insert.sql | 27 +++ 11 files changed, 840 insertions(+), 6 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index 0a4f95fc3f..b2cb742264 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -129,6 +129,9 @@ static PartitionRangeBound *make_one_range_bound(PartitionKey key, int index, static int32 partition_rbound_cmp(PartitionKey key, Datum *datums1, RangeDatumContent *content1, bool lower1, PartitionRangeBound *b2); +static int32 partition_rbound_datum_cmp(PartitionKey key, + Datum *rb_datums, RangeDatumContent *rb_content, + Datum *tuple_datums); static int32 partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, @@ -137,6 +140,13 @@ static int partition_bound_bsearch(PartitionKey key, PartitionBoundInfo boundinfo, void *probe, bool probe_is_bound, bool *is_equal); +/* Support get_partition_for_tuple() */ +static void FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); + /* * RelationBuildPartitionDesc * Form rel's partition descriptor @@ -916,6 +926,119 @@ RelationGetPartitionQual(Relation rel, bool recurse) return generate_partition_qual(rel, recurse); } +/* Turn an array of OIDs with N elements into a list */ +#define OID_ARRAY_TO_LIST(arr, N, list) \ + do\ + {\ + int i;\ + for (i = 0; i < (N); i++)\ + (list) = lappend_oid((list), (arr)[i]);\ + } while(0) + +/* + * RelationGetPartitionDispatchInfo + * Returns information necessary to route tuples down a partition tree + * + * All the partitions will be locked with lockmode, unless it is NoLock. + * A list of the OIDs of all the leaf partition of rel is returned in + * *leaf_part_oids. + */ +PartitionDispatch * +RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + int *num_parted, List **leaf_part_oids) +{ + PartitionDesc rootpartdesc = RelationGetPartitionDesc(rel); + PartitionDispatchData **pd; + List *all_parts = NIL, + *parted_rels; + ListCell *lc; + int i, + k; + + /* + * Lock partitions and make a list of the partitioned ones to prepare + * their PartitionDispatch objects below. + * + * Cannot use find_all_inheritors() here, because then the order of OIDs + * in parted_rels list would be unknown, which does not help, because we + * we assign indexes within individual PartitionDispatch in an order that + * is predetermined (determined by the order of OIDs in individual + * partition descriptors). + */ + *num_parted = 1; + parted_rels = list_make1(rel); + OID_ARRAY_TO_LIST(rootpartdesc->oids, rootpartdesc->nparts, all_parts); + foreach(lc, all_parts) + { + Relation partrel = heap_open(lfirst_oid(lc), lockmode); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + + /* + * If this partition is a partitioned table, add its children to the + * end of the list, so that they are processed as well. + */ + if (partdesc) + { + (*num_parted)++; + parted_rels = lappend(parted_rels, partrel); + OID_ARRAY_TO_LIST(partdesc->oids, partdesc->nparts, all_parts); + } + else + heap_close(partrel, NoLock); + + /* + * We keep the partitioned ones open until we're done using the + * information being collected here (for example, see + * ExecEndModifyTable). + */ + } + + /* Generate PartitionDispatch objects for all partitioned tables */ + pd = (PartitionDispatchData **) palloc(*num_parted * + sizeof(PartitionDispatchData *)); + *leaf_part_oids = NIL; + i = k = 0; + foreach(lc, parted_rels) + { + Relation partrel = lfirst(lc); + PartitionKey partkey = RelationGetPartitionKey(partrel); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + int j, + m; + + pd[i] = (PartitionDispatch) palloc(sizeof(PartitionDispatchData)); + pd[i]->reldesc = partrel; + pd[i]->key = partkey; + pd[i]->keystate = NIL; + pd[i]->partdesc = partdesc; + pd[i]->indexes = (int *) palloc(partdesc->nparts * sizeof(int)); + + m = 0; + for (j = 0; j < partdesc->nparts; j++) + { + Oid partrelid = partdesc->oids[j]; + + if (get_rel_relkind(partrelid) != RELKIND_PARTITIONED_TABLE) + { + *leaf_part_oids = lappend_oid(*leaf_part_oids, partrelid); + pd[i]->indexes[j] = k++; + } + else + { + /* + * We can assign indexes this way because of the way + * parted_rels has been generated. + */ + pd[i]->indexes[j] = -(i + 1 + m); + m++; + } + } + i++; + } + + return pd; +} + /* Module-local functions */ /* @@ -1367,6 +1490,176 @@ generate_partition_qual(Relation rel, bool recurse) return result; } +/* ---------------- + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for the partition key + * of a tuple. + * + * pkinfo partition key execution info + * slot Heap tuple from which to extract partition key + * estate executor state for evaluating any partition key + * expressions (must be non-NULL) + * values Array of partition key Datums (output area) + * isnull Array of is-null indicators (output area) + * + * the ecxt_scantuple slot of estate's per-tuple expr context must point to + * the heap tuple passed in. + * ---------------- + */ +static void +FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pd->key->partexprs != NIL && pd->keystate == NIL) + { + /* Check caller has set up context correctly */ + Assert(estate != NULL && + GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + + /* First time through, set up expression evaluation state */ + pd->keystate = (List *) ExecPrepareExpr((Expr *) pd->key->partexprs, + estate); + } + + partexpr_item = list_head(pd->keystate); + for (i = 0; i < pd->key->partnatts; i++) + { + AttrNumber keycol = pd->key->partattrs[i]; + Datum datum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + datum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + datum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = datum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Finds a leaf partition for tuple contained in *slot + * + * Returned value is the sequence number of the leaf partition thus found, + * or -1 if no leaf partition is found for the tuple. *failed_at is set + * to the OID of the partitioned table whose partition was not found in + * the latter case. + */ +int +get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + PartitionDispatch parent; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int cur_offset, + cur_index; + int i; + + /* start with the root partitioned table */ + parent = pd[0]; + while(true) + { + PartitionKey key = parent->key; + PartitionDesc partdesc = parent->partdesc; + + /* Quick exit */ + if (partdesc->nparts == 0) + { + *failed_at = RelationGetRelid(parent->reldesc); + return -1; + } + + /* Extract partition key from tuple */ + FormPartitionKeyDatum(parent, slot, estate, values, isnull); + + if (key->strategy == PARTITION_STRATEGY_RANGE) + { + /* Disallow nulls in the range partition key of the tuple */ + for (i = 0; i < key->partnatts; i++) + if (isnull[i]) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key of row contains null"))); + } + + if (partdesc->boundinfo->has_null && isnull[0]) + /* Tuple maps to the null-accepting list partition */ + cur_index = partdesc->boundinfo->null_index; + else + { + /* Else bsearch in partdesc->boundinfo */ + bool equal = false; + + cur_offset = partition_bound_bsearch(key, partdesc->boundinfo, + values, false, &equal); + switch (key->strategy) + { + case PARTITION_STRATEGY_LIST: + if (cur_offset >= 0 && equal) + cur_index = partdesc->boundinfo->indexes[cur_offset]; + else + cur_index = -1; + break; + + case PARTITION_STRATEGY_RANGE: + /* + * Offset returned is such that the bound at offset is + * found to be less or equal with the tuple. So, the + * bound at offset+1 would be the upper bound. + */ + cur_index = partdesc->boundinfo->indexes[cur_offset+1]; + break; + + default: + elog(ERROR, "unexpected partition strategy: %d", + (int) key->strategy); + } + } + + /* + * cur_index < 0 means we failed to find a partition of this parent. + * cur_index >= 0 means we either found the leaf partition, or the + * next parent to find a partition of. + */ + if (cur_index < 0) + { + *failed_at = RelationGetRelid(parent->reldesc); + return -1; + } + else if (parent->indexes[cur_index] < 0) + parent = pd[-parent->indexes[cur_index]]; + else + break; + } + + return parent->indexes[cur_index]; +} + /* * qsort_partition_list_value_cmp * @@ -1499,6 +1792,36 @@ partition_rbound_cmp(PartitionKey key, } /* + * partition_rbound_datum_cmp + * + * Return whether range bound (specified in rb_datums, rb_content, and + * rb_lower) <=, =, >= partition key of tuple (tuple_datums) + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, + Datum *rb_datums, RangeDatumContent *rb_content, + Datum *tuple_datums) +{ + int i; + int32 cmpval = -1; + + for (i = 0; i < key->partnatts; i++) + { + if (rb_content[i] != RANGE_DATUM_FINITE) + return rb_content[i] == RANGE_DATUM_NEG_INF ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + rb_datums[i], + tuple_datums[i])); + if (cmpval != 0) + break; + } + + return cmpval; +} + +/* * partition_bound_cmp * * Return whether the bound at offset in boundinfo is <=, =, >= the argument @@ -1537,7 +1860,10 @@ partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, bound_datums, content, lower, (PartitionRangeBound *) probe); } - + else + cmpval = partition_rbound_datum_cmp(key, + bound_datums, content, + (Datum *) probe); break; } diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index de9e29c81e..3f8d748a82 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -161,6 +161,11 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionDispatch *partition_dispatch_info; + int num_dispatch; + int num_partitions; + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1402,71 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + List *leaf_parts; + ListCell *cell; + int i, + num_parted, + num_leaf_parts; + ResultRelInfo *leaf_part_rri; + + /* Get the tuple-routing information and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &num_parted, &leaf_parts); + num_leaf_parts = list_length(leaf_parts); + cstate->partition_dispatch_info = pd; + cstate->num_dispatch = num_parted; + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) palloc(num_leaf_parts * + sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation partrel; + + /* + * We locked all the partitions above including the leaf + * partitions. Note that each of the relations in + * cstate->partitions will be closed by CopyFrom() after + * it's finished with its processing. + */ + partrel = heap_open(lfirst_oid(cell), NoLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(partrel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + partrel, + 1, /* dummy */ + false, /* no partition constraint check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(tupDesc, RelationGetDescr(partrel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(partrel), + gettext_noop("could not convert row type")); + leaf_part_rri++; + i++; + } + } } else { @@ -2255,6 +2325,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2352,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2463,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2491,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->partition_dispatch_info != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2442,7 +2516,11 @@ CopyFrom(CopyState cstate) values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2572,59 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition to heap_insert the tuple into */ + if (cstate->partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + cstate->partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < cstate->num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding + * to the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = cstate->partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* + * For ExecInsertIndexTuples() to work on the partition's indexes + */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the + * partition rowtype. + */ + map = cstate->partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, true); + } + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2553,7 +2684,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2709,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2728,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); @@ -2614,6 +2753,32 @@ CopyFrom(CopyState cstate) ExecCloseIndices(resultRelInfo); + /* Close all the partitioned tables, leaf partitions, and their indices */ + if (cstate->partition_dispatch_info) + { + int i; + + /* + * Remember cstate->partition_dispatch_info[0] corresponds to the root + * partitioned table, which we must not try to close, because it is + * the main target table of COPY that will be closed eventually by + * DoCopy(). + */ + for (i = 1; i < cstate->num_dispatch; i++) + { + PartitionDispatch pd = cstate->partition_dispatch_info[i]; + + heap_close(pd->reldesc, NoLock); + } + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + } + FreeExecutorState(estate); /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8a803233ca..c77b216d4f 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1322,6 +1322,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index 21b18b82b9..0f47c7e010 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2991,3 +2996,52 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +/* + * ExecFindPartition -- Find a leaf partition in the partition tree rooted + * at parent, for the heap tuple contained in *slot + * + * estate must be non-NULL; we'll need it to compute any expressions in the + * partition key(s) + * + * If no leaf partition is found, this routine errors out with the appropriate + * error message, else it returns the leaf partition sequence number returned + * by get_partition_for_tuple() unchanged. + */ +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, + TupleTableSlot *slot, EState *estate) +{ + int result; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + result = get_partition_for_tuple(pd, slot, estate, &failed_at); + if (result < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return result; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 6eccfb7cec..c0b58d1841 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,56 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + /* Determine the partition to heap_insert the tuple into */ + if (mtstate->mt_partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, true); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +562,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1622,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1713,75 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + int i, + j, + num_parted, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + /* Form the partition node tree and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &num_parted, &leaf_parts); + mtstate->mt_partition_dispatch_info = pd; + mtstate->mt_num_dispatch = num_parted; + num_leaf_parts = list_length(leaf_parts); + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid partrelid = lfirst_oid(cell); + Relation partrel; + + /* + * We locked all the partitions above including the leaf + * partitions. Note that each of the relations in + * mtstate->mt_partitions will be closed by ExecEndModifyTable(). + */ + partrel = heap_open(partrelid, NoLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(partrel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + partrel, + 1, /* dummy */ + false, /* no partition constraint checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (partrel->rd_rel->relhasindex && operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(partrel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(partrel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2099,26 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all the partitioned tables, leaf partitions, and their indices + * + * Remember node->mt_partition_dispatch_info[0] corresponds to the root + * partitioned table, which we must not try to close, because it is the + * main target table of the query that will be closed by ExecEndPlan(). + */ + for (i = 1; i < node->mt_num_dispatch; i++) + { + PartitionDispatch pd = node->mt_partition_dispatch_info[i]; + + heap_close(pd->reldesc, NoLock); + } + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + /* * Free the exprcontext */ diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 1a541788eb..7364346167 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -806,8 +806,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 70d8325137..21effbf87b 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -37,6 +39,30 @@ typedef struct PartitionDescData typedef struct PartitionDescData *PartitionDesc; +/*----------------------- + * PartitionDispatch - information about one partitioned table in a partition + * hiearchy required to route a tuple to one of its partitions + * + * reldesc Relation descriptor of the table + * key Partition key information of the table + * keystate Execution state required for expressions in the partition key + * partdesc Partition descriptor of the table + * indexes Array with partdesc->nparts members (for details on what + * individual members represent, see how they are set in + * RelationGetPartitionDispatchInfo()) + *----------------------- + */ +typedef struct PartitionDispatchData +{ + Relation reldesc; + PartitionKey key; + List *keystate; /* list of ExprState */ + PartitionDesc partdesc; + int *indexes; +} PartitionDispatchData; + +typedef struct PartitionDispatchData *PartitionDispatch; + extern void RelationBuildPartitionDesc(Relation relation); extern bool partition_bounds_equal(PartitionKey key, PartitionBoundInfo p1, PartitionBoundInfo p2); @@ -45,4 +71,13 @@ extern void check_new_partition_bound(char *relname, Relation parent, Node *boun extern Oid get_partition_parent(Oid relid); extern List *get_qual_from_partbound(Relation rel, Relation parent, Node *bound); extern List *RelationGetPartitionQual(Relation rel, bool recurse); + +/* For tuple routing */ +extern PartitionDispatch *RelationGetPartitionDispatchInfo(Relation rel, + int lockmode, int *num_parted, + List **leaf_part_oids); +extern int get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276be53..b4d09f9564 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index df2dec3a2c..1de5c8196d 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,15 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionDispatchData **mt_partition_dispatch_info; + /* Tuple-routing support info */ + int mt_num_dispatch; /* Number of entries in the above + * array */ + int mt_num_partitions; /* Number of members in the + * following arrays */ + ResultRelInfo *mt_partitions; /* Per partition result relation */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per partition tuple conversion map */ } ModifyTableState; /* ---------------- diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 2e78fd9fc1..561cefa3c4 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -230,6 +230,61 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_ee_ff1 values ('ff', 1); insert into part_ee_ff2 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +-- fail (partition key (b+0) is null) +insert into range_parted values ('a'); +ERROR: range partition key of row contains null +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------+---+---- + part1 | a | 1 + part1 | a | 1 + part2 | a | 10 + part3 | b | 1 + part4 | b | 10 + part4 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_ee_ff not found in both cases) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_ee_ff values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_ee_ff values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +-------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff1 | ff | 1 + part_ee_ff1 | EE | 1 + part_ee_ff2 | ff | 11 + part_ee_ff2 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index eb923646c3..846bb5897a 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,33 @@ insert into part_ee_ff1 values ('cc', 1); insert into part_ee_ff1 values ('ff', 1); insert into part_ee_ff2 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +-- fail (partition key (b+0) is null) +insert into range_parted values ('a'); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_ee_ff not found in both cases) +insert into list_parted values ('EE', 0); +insert into part_ee_ff values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_ee_ff values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 2.11.0 --------------E1933196A000B12EC95A8B6E Content-Type: text/x-diff; name="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-20.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-20"; filename*1=".patch" ^ permalink raw reply [nested|flat] 13+ messages in thread
* [PATCH 6/7] Tuple routing for partitioned tables. @ 2016-07-27 06:47 amit <[email protected]> 0 siblings, 0 replies; 13+ messages in thread From: amit @ 2016-07-27 06:47 UTC (permalink / raw) Both COPY FROM and INSERT are covered by this commit. Routing to foreing partitions is not supported at the moment. To implement tuple-routing, introduce a PartitionDispatch data structure. Each partitioned table in a partition tree gets one and contains info such as a pointer to its partition descriptor, partition key execution state, global sequence numbers of its leaf partitions and a way to link to the PartitionDispatch objects of any of its partitions that are partitioned themselves. Starting with the PartitionDispatch object of the root partitioned table and a tuple to route, one can get the global sequence number of the leaf partition that the tuple gets routed to, if one exists. --- src/backend/catalog/partition.c | 342 ++++++++++++++++++++++++++++++++ src/backend/commands/copy.c | 151 ++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 58 ++++++- src/backend/executor/nodeModifyTable.c | 130 ++++++++++++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 11 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 8 + src/test/regress/expected/insert.out | 52 +++++ src/test/regress/sql/insert.sql | 25 +++ 11 files changed, 787 insertions(+), 5 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index ba22453..643a258 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -109,6 +109,28 @@ typedef struct PartitionRangeBound bool lower; /* this is the lower (vs upper) bound */ } PartitionRangeBound; +/*----------------------- + * PartitionDispatch - information about one partitioned table in a partition + * hiearchy required to route a tuple to one of its partitions + * + * relid OID of the table + * key Partition key information of the table + * keystate Execution state required for expressions in the partition key + * partdesc Partition descriptor of the table + * indexes Array with partdesc->nparts members (for details on what + * individual members represent, see how they are set in + * RelationGetPartitionDispatchInfo()) + *----------------------- + */ +typedef struct PartitionDispatchData +{ + Oid relid; + PartitionKey key; + List *keystate; /* list of ExprState */ + PartitionDesc partdesc; + int *indexes; +} PartitionDispatchData; + static int32 qsort_partition_list_value_cmp(const void *a, const void *b, void *arg); static int32 qsort_partition_rbound_cmp(const void *a, const void *b, void *arg); @@ -122,12 +144,21 @@ static PartitionRangeBound *make_one_range_bound(PartitionKey key, int index, Li static int32 partition_rbound_cmp(PartitionKey key, Datum *datums1, bool *inf1, bool lower1, PartitionRangeBound *b2); +static int32 partition_rbound_datum_cmp(PartitionKey key, Datum *rb_datums, bool *rb_inf, + Datum *tuple_datums); static int32 partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, int offset, void *probe, bool probe_is_bound); static int partition_bound_bsearch(PartitionKey key, PartitionBoundInfo boundinfo, void *probe, bool probe_is_bound, bool *is_equal); +/* Support get_partition_for_tuple() */ +static void FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); + /* * RelationBuildPartitionDesc * Form rel's partition descriptor @@ -888,6 +919,115 @@ RelationGetPartitionQual(Relation rel, bool recurse) return generate_partition_qual(rel, recurse); } +/* Turn an array of OIDs with N elements into a list */ +#define OID_ARRAY_TO_LIST(arr, N, list) \ + do\ + {\ + int i;\ + for (i = 0; i < (N); i++)\ + (list) = lappend_oid((list), (arr)[i]);\ + } while(0) + +/* + * RelationGetPartitionDispatchInfo + * Returns information necessary to route tuples down a partition tree + * + * All the partitions will be locked with lockmode, unless it is NoLock. + * A list of the OIDs of all the leaf partition of rel is returned in + * *leaf_part_oids. + */ +PartitionDispatch * +RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + List **leaf_part_oids) +{ + PartitionDesc rootpartdesc = RelationGetPartitionDesc(rel); + PartitionDispatchData **pd; + List *all_parts = NIL, + *parted_rels = NIL; + ListCell *lc; + int i, + k, + num_parted; + + /* + * Lock partitions and collect OIDs of the partitioned ones to prepare + * their PartitionDispatch objects. + * + * Cannot use find_all_inheritors() here, because then the order of OIDs + * in parted_rels list would be unknown, which does not help because down + * below, we assign indexes within individual PartitionDispatch in an + * order that's predetermined (determined by the order of OIDs in + * individual partition descriptors). + */ + parted_rels = lappend_oid(parted_rels, RelationGetRelid(rel)); + num_parted = 1; + OID_ARRAY_TO_LIST(rootpartdesc->oids, rootpartdesc->nparts, all_parts); + foreach(lc, all_parts) + { + Relation partrel = heap_open(lfirst_oid(lc), lockmode); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + + /* + * If this partition is a partitined table, add its children to to the + * end of the list, so that they are processed as well. + */ + if (partdesc) + { + num_parted++; + parted_rels = lappend_oid(parted_rels, lfirst_oid(lc)); + OID_ARRAY_TO_LIST(partdesc->oids, partdesc->nparts, all_parts); + } + + heap_close(partrel, NoLock); + } + + /* Generate PartitionDispatch objects for all partitioned tables */ + pd = (PartitionDispatchData **) palloc(num_parted * + sizeof(PartitionDispatchData *)); + *leaf_part_oids = NIL; + i = k = 0; + foreach(lc, parted_rels) + { + /* We locked all partitions above */ + Relation partrel = heap_open(lfirst_oid(lc), NoLock); + PartitionDesc partdesc = RelationGetPartitionDesc(partrel); + int j, + m; + + pd[i] = (PartitionDispatch) palloc(sizeof(PartitionDispatchData)); + pd[i]->relid = RelationGetRelid(partrel); + pd[i]->key = RelationGetPartitionKey(partrel); + pd[i]->keystate = NIL; + pd[i]->partdesc = partdesc; + pd[i]->indexes = (int *) palloc(partdesc->nparts * sizeof(int)); + heap_close(partrel, NoLock); + + m = 0; + for (j = 0; j < partdesc->nparts; j++) + { + Oid partrelid = partdesc->oids[j]; + + if (get_rel_relkind(partrelid) != RELKIND_PARTITIONED_TABLE) + { + *leaf_part_oids = lappend_oid(*leaf_part_oids, partrelid); + pd[i]->indexes[j] = k++; + } + else + { + /* + * We can assign indexes this way because of the way + * parted_rels has been generated. + */ + pd[i]->indexes[j] = -(i + 1 + m); + m++; + } + } + i++; + } + + return pd; +} + /* Module-local functions */ /* @@ -1328,6 +1468,172 @@ generate_partition_qual(Relation rel, bool recurse) return result; } +/* ---------------- + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for the partition key + * of a tuple. + * + * pkinfo partition key execution info + * slot Heap tuple from which to extract partition key + * estate executor state for evaluating any partition key + * expressions (must be non-NULL) + * values Array of partition key Datums (output area) + * isnull Array of is-null indicators (output area) + * + * the ecxt_scantuple slot of estate's per-tuple expr context must point to + * the heap tuple passed in. + * ---------------- + */ +static void +FormPartitionKeyDatum(PartitionDispatch pd, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pd->key->partexprs != NIL && pd->keystate == NIL) + { + /* Check caller has set up context correctly */ + Assert(estate != NULL && + GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + + /* First time through, set up expression evaluation state */ + pd->keystate = (List *) ExecPrepareExpr((Expr *) pd->key->partexprs, + estate); + } + + partexpr_item = list_head(pd->keystate); + for (i = 0; i < pd->key->partnatts; i++) + { + AttrNumber keycol = pd->key->partattrs[i]; + Datum datum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + datum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + datum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = datum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Finds a leaf partition for tuple contained in *slot + * + * Returned value is the sequence number of the leaf partition thus found, + * or -1 if no leaf partition is found for the tuple. *failed_at is set + * to the OID of the partitioned table whose partition was not found in + * the latter case. + */ +int +get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + PartitionDispatch parent; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int cur_offset, + cur_index; + int i; + + /* start with the root partitioned table */ + parent = pd[0]; + while(true) + { + PartitionKey key = parent->key; + PartitionDesc partdesc = parent->partdesc; + + /* Quick exit */ + if (partdesc->nparts == 0) + { + *failed_at = parent->relid; + return -1; + } + + /* Extract partition key from tuple */ + FormPartitionKeyDatum(parent, slot, estate, values, isnull); + + if (key->strategy == PARTITION_STRATEGY_RANGE) + { + /* Disallow nulls in the range partition key of the tuple */ + for (i = 0; i < key->partnatts; i++) + if (isnull[i]) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key of row contains null"))); + } + + if (partdesc->boundinfo->has_null && isnull[0]) + /* Tuple maps to the null-accepting list partition */ + cur_index = partdesc->boundinfo->null_index; + else + { + /* Else bsearch in partdesc->boundinfo */ + bool equal = false; + + cur_offset = partition_bound_bsearch(key, partdesc->boundinfo, + values, false, &equal); + switch (key->strategy) + { + case PARTITION_STRATEGY_LIST: + if (cur_offset >= 0 && equal) + cur_index = partdesc->boundinfo->indexes[cur_offset]; + else + cur_index = -1; + break; + + case PARTITION_STRATEGY_RANGE: + /* + * Offset returned is such that the bound at offset is + * found to be less or equal with the tuple. So, the + * bound at offset+1 would be the upper bound. + */ + cur_index = partdesc->boundinfo->indexes[cur_offset+1]; + break; + } + } + + /* + * cur_index < 0 means we failed to find a partition of this parent. + * cur_index >= 0 means we either found the leaf partition, or the + * next parent to find a partition of. + */ + if (cur_index < 0) + { + *failed_at = parent->relid; + return -1; + } + else if (parent->indexes[cur_index] < 0) + parent = pd[-parent->indexes[cur_index]]; + else + break; + } + + return parent->indexes[cur_index]; +} + /* List partition related support functions */ /* @@ -1461,6 +1767,35 @@ partition_rbound_cmp(PartitionKey key, } /* + * partition_rbound_datum_cmp + * + * Return whether range bound (specified in datums, finite, and lower) is + * <=, =, >= partition key of tuple (tuple_datums) + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, Datum *rb_datums, bool *rb_inf, + Datum *tuple_datums) +{ + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (rb_inf[i]) + return rb_inf[i] == RANGE_DATUM_NEG_INF ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + rb_datums[i], + tuple_datums[i])); + if (cmpval != 0) + break; + } + + return cmpval; +} + +/* * partition_bound_cmp * * Return whether the bound at offset in boundinfo is <=, =, >= the argument @@ -1502,6 +1837,13 @@ partition_bound_cmp(PartitionKey key, PartitionBoundInfo boundinfo, cmpval = partition_rbound_cmp(key, datums1, inf1, lower1, b2); } + else + { + Datum *datums2 = (Datum *) probe; + + cmpval = partition_rbound_datum_cmp(key, datums1, inf1, + datums2); + } } break; } diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..7d76ead 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -161,6 +161,10 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionDispatch *partition_dispatch_info; + int num_partitions; + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1401,67 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + List *leaf_parts; + ListCell *cell; + int i, + num_leaf_parts; + ResultRelInfo *leaf_part_rri; + + /* Get the tuple-routing information and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + num_leaf_parts = list_length(leaf_parts); + cstate->partition_dispatch_info = pd; + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) palloc(num_leaf_parts * + sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation partrel; + + /* + * All partitions locked above; will be closed after CopyFrom is + * finished. + */ + partrel = heap_open(lfirst_oid(cell), NoLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(partrel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + partrel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(tupDesc, RelationGetDescr(partrel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(partrel), + gettext_noop("could not convert row type")); + leaf_part_rri++; + i++; + } + } } else { @@ -2255,6 +2320,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2347,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2458,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2486,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->partition_dispatch_info != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2442,7 +2511,11 @@ CopyFrom(CopyState cstate) values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2567,56 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition to heap_insert the tuple into */ + if (cstate->partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + cstate->partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < cstate->num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding + * to the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = cstate->partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* + * For ExecInsertIndexTuples() to work on the partition's indexes + */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the + * partition rowtype. + */ + map = cstate->partition_tupconv_maps[leaf_part_index]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2553,7 +2676,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2701,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2720,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); @@ -2614,6 +2745,20 @@ CopyFrom(CopyState cstate) ExecCloseIndices(resultRelInfo); + /* Close all partitions and indices thereof */ + if (cstate->partition_dispatch_info) + { + int i; + + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + } + FreeExecutorState(estate); /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index b27da1d..be08eff 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1293,6 +1293,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index c7a6347..54fb771 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2990,3 +2995,52 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +/* + * ExecFindPartition -- Find a leaf partition in the partition tree rooted + * at parent, for the heap tuple contained in *slot + * + * estate must be non-NULL; we'll need it to compute any expressions in the + * partition key(s) + * + * If no leaf partition is found, this routine errors out with the appropriate + * error message, else it returns the leaf partition sequence number returned + * by get_partition_for_tuple() unchanged. + */ +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, + TupleTableSlot *slot, EState *estate) +{ + int result; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + result = get_partition_for_tuple(pd, slot, estate, &failed_at); + if (result < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return result; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..aa1d8b9 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,56 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + /* Determine the partition to heap_insert the tuple into */ + if (mtstate->mt_partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, false); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +562,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1622,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1713,69 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + /* Form the partition node tree and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + mtstate->mt_partition_dispatch_info = pd; + num_leaf_parts = list_length(leaf_parts); + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2093,15 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + /* * Free the exprcontext */ diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 70d8325..f76c5d9 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -36,6 +38,7 @@ typedef struct PartitionDescData } PartitionDescData; typedef struct PartitionDescData *PartitionDesc; +typedef struct PartitionDispatchData *PartitionDispatch; extern void RelationBuildPartitionDesc(Relation relation); extern bool partition_bounds_equal(PartitionKey key, @@ -45,4 +48,12 @@ extern void check_new_partition_bound(char *relname, Relation parent, Node *boun extern Oid get_partition_parent(Oid relid); extern List *get_qual_from_partbound(Relation rel, Relation parent, Node *bound); extern List *RelationGetPartitionQual(Relation rel, bool recurse); + +/* For tuple routing */ +extern PartitionDispatch *RelationGetPartitionDispatchInfo(Relation rel, int lockmode, + List **leaf_part_oids); +extern int get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..b4d09f9 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..606cb21 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,13 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionDispatchData **mt_partition_dispatch_info; + /* Tuple-routing support info */ + int mt_num_partitions; /* Number of members in the + * following arrays */ + ResultRelInfo *mt_partitions; /* Per partition result relation */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per partition tuple conversion map */ } ModifyTableState; /* ---------------- diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 9ae6b09..d5dcb59 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -227,6 +227,58 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index b6e821e..fbd30d9 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,31 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------F7EE68175C11124DB3D461AC Content-Type: text/x-diff; name="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-17.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0007-Update-DDL-Partitioning-chapter-to-reflect-new-devel-17"; filename*1=".patch" ^ permalink raw reply [nested|flat] 13+ messages in thread
* [PATCH 7/8] Tuple routing for partitioned tables. @ 2016-07-27 07:59 amit <[email protected]> 0 siblings, 0 replies; 13+ messages in thread From: amit @ 2016-07-27 07:59 UTC (permalink / raw) Both COPY FROM and INSERT. --- src/backend/catalog/partition.c | 317 ++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 222 +++++++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 59 ++++++- src/backend/executor/nodeModifyTable.c | 160 ++++++++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 1 + src/backend/optimizer/plan/createplan.c | 76 ++++++++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 7 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 10 + src/include/nodes/plannodes.h | 1 + src/test/regress/expected/insert.out | 52 +++++ src/test/regress/sql/insert.sql | 25 +++ 16 files changed, 940 insertions(+), 7 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index 7bc4b15..4e253c1 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -185,6 +185,18 @@ static List *generate_partition_qual(Relation rel, bool recurse); static PartitionTreeNode GetPartitionTreeNodeRecurse(Relation rel, int offset, int lockmode); static int get_leaf_partition_count(PartitionTreeNode parent); +/* Support get_partition_for_tuple() */ +static PartitionKeyExecInfo *BuildPartitionKeyExecInfo(Relation rel); +static void FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); +static int list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull); +static int range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum *tuple); + /* List partition related support functions */ static bool equal_list_info(PartitionKey key, PartitionListInfo *l1, PartitionListInfo *l2); @@ -198,6 +210,8 @@ static PartitionRangeBound *copy_range_bound(PartitionKey key, PartitionRangeBou static bool equal_range_info(PartitionKey key, PartitionRangeInfo *r1, PartitionRangeInfo *r2); static int32 partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg); +static int32 partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg); static bool partition_rbound_eq(PartitionKey key, PartitionRangeBound *b1, PartitionRangeBound *b2); typedef int32 (*partition_rbound_bsearch_cmp_fn) (PartitionKey, @@ -1443,7 +1457,7 @@ GetPartitionTreeNodeRecurse(Relation rel, int offset, int lockmode) /* First build our own node */ parent = (PartitionTreeNode) palloc0(sizeof(PartitionTreeNodeData)); - parent->pkinfo = NULL; + parent->pkinfo = BuildPartitionKeyExecInfo(rel); parent->pdesc = RelationGetPartitionDesc(rel); parent->relid = RelationGetRelid(rel); parent->offset = offset; @@ -1527,6 +1541,273 @@ get_leaf_partition_count(PartitionTreeNode parent) return result; } +/* + * BuildPartitionKeyExecInfo + * Construct a list of PartitionKeyExecInfo records for an open + * relation + * + * PartitionKeyExecInfo stores the information about the partition key + * that's needed when inserting tuples into a partitioned table; especially, + * partition key expression state if there are any expression columns in + * the partition key. Normally we build a PartitionKeyExecInfo for a + * partitioned table just once per command, and then use it for (potentially) + * many tuples. + * + */ +static PartitionKeyExecInfo * +BuildPartitionKeyExecInfo(Relation rel) +{ + PartitionKeyExecInfo *pkinfo; + + pkinfo = (PartitionKeyExecInfo *) palloc0(sizeof(PartitionKeyExecInfo)); + pkinfo->pi_Key = RelationGetPartitionKey(rel); + pkinfo->pi_ExpressionState = NIL; + + return pkinfo; +} + +/* ---------------- + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for the partition key + * of a tuple. + * + * pkinfo partition key execution info + * slot Heap tuple from which to extract partition key + * estate executor state for evaluating any partition key + * expressions (must be non-NULL) + * values Array of partition key Datums (output area) + * isnull Array of is-null indicators (output area) + * + * the ecxt_scantuple slot of estate's per-tuple expr context must point to + * the heap tuple passed in. + * ---------------- + */ +static void +FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pkinfo->pi_Key->partexprs != NIL && pkinfo->pi_ExpressionState == NIL) + { + /* Check caller has set up context correctly */ + Assert(estate != NULL && + GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + + /* First time through, set up expression evaluation state */ + pkinfo->pi_ExpressionState = (List *) + ExecPrepareExpr((Expr *) pkinfo->pi_Key->partexprs, + estate); + } + + partexpr_item = list_head(pkinfo->pi_ExpressionState); + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + { + AttrNumber keycol = pkinfo->pi_Key->partattrs[i]; + Datum pkDatum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + pkDatum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + pkDatum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = pkDatum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Recursively finds the "leaf" partition for tuple + * + * parent is the root of the partition tree through which we are trying + * to route the tuple contained in *slot. + * + * Returned value is the sequence number of the leaf partition thus found, + * or -1 if no leaf partition is found for the tuple. *failed_at is set + * to the OID of the partitioned table whose partition was not found in + * the latter case. + */ +int +get_partition_for_tuple(PartitionTreeNode parent, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + PartitionKeyExecInfo *pkinfo = parent->pkinfo; + PartitionTreeNode node, + prev; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int i; + int cur_idx; + + /* Guard against stack overflow due to overly deep partition tree */ + check_stack_depth(); + + if (parent->pdesc->nparts == 0) + { + *failed_at = parent->relid; + return -1; + } + + /* Extract partition key from tuple */ + FormPartitionKeyDatum(pkinfo, slot, estate, values, isnull); + + switch (pkinfo->pi_Key->strategy) + { + case PARTITION_STRATEGY_LIST: + cur_idx = list_partition_for_tuple(pkinfo->pi_Key, parent->pdesc, + values[0], isnull[0]); + break; + + case PARTITION_STRATEGY_RANGE: + /* Disallow nulls in the partition key of the tuple */ + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + if (isnull[i]) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key of row contains null"))); + + cur_idx = range_partition_for_tuple(pkinfo->pi_Key, parent->pdesc, + values); + break; + } + + /* No partition found at this level */ + if (cur_idx < 0) + { + *failed_at = parent->relid; + return cur_idx; + } + + /* + * The partition we found may either be a leaf partition or partitioned + * itself. In the latter case, we need to recurse by finding the proper + * node (ie, such that node->index == cur_idx) to pass down. + */ + prev = parent; + node = parent->downlink; + while (node != NULL) + { + if (node->index >= cur_idx) + break; + + prev = node; + node = node->next; + } + + /* + * If we couldn't find a node, that means cur_idx is actually a leaf + * partition. In the simpler case where all of the left siblings are leaf + * partitions themselves, we can get the correct index to return by adding + * cur_idx to the parent node's offset. Otherwise, we need to compensate + * for the leaf partitions in the partition subtrees of all the left + * siblings that are partitioned, which, fortunately it suffices to look + * at the rightmost such node. + */ + Assert(prev != NULL); + if (!node || cur_idx < node->index) + return prev == parent ? prev->offset + cur_idx + : prev->offset + prev->num_leaf_parts + + (cur_idx - prev->index - 1); + + /* + * Must recurse because it turns out that the partition at cur_idx is + * partitioned itself + */ + Assert (node != NULL && node->index == cur_idx); + + return get_partition_for_tuple(node, slot, estate, failed_at); +} + +/* + * list_partition_for_tuple + * Find the list partition for a tuple (arg 'value' contains the + * list partition key of the original tuple) + * + * Returns -1 if none found. + */ +static int +list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull) +{ + PartitionListInfo listinfo; + int found; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_LIST); + listinfo = pdesc->boundinfo->bounds.lists; + + if (isnull && listinfo.has_null) + return listinfo.null_index; + else if (!isnull) + { + found = partition_list_values_bsearch(key, + listinfo.values, + listinfo.nvalues, + value); + if (found >= 0) + return listinfo.indexes[found]; + } + + /* Control reaches here if isnull and !listinfo->has_null */ + return -1; +} + +/* + * range_partition_for_tuple + * Get the index of the range partition for a tuple (arg 'tuple' + * actually contains the range partition key of the original + * tuple) + * + * Returns -1 if none found. + */ +static int +range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, Datum *tuple) +{ + int offset; + PartitionRangeInfo rangeinfo; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_RANGE); + rangeinfo = pdesc->boundinfo->bounds.ranges; + + offset = partition_rbound_bsearch(key, + rangeinfo.bounds, rangeinfo.nbounds, + tuple, partition_rbound_datum_cmp, + true, NULL); + + /* + * Offset returned is such that the bound at offset is found to be less + * or equal with the tuple. That is, the tuple belongs to the partition + * with the rangeinfo.bounds[offset] as the lower bound and + * rangeinfo.bounds[offset+1] as the upper bound, provided the latter is + * indeed an upper (!lower) bound. If it turns out otherwise, the + * corresponding index will be -1, which means no valid partition exists. + */ + return rangeinfo.indexes[offset+1]; +} + /* List partition related support functions */ /* @@ -1772,6 +2053,40 @@ partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg) } /* + * Return whether bound <=, =, >= partition key of tuple + * + * The 3rd argument is void * so that it can be used with + * partition_rbound_bsearch() + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg) +{ + Datum *datums1 = bound->datums, + *datums2 = (Datum *) arg; + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (bound->infinite[i]) + return bound->lower ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + datums1[i], datums2[i])); + if (cmpval != 0) + break; + } + + /* If datums are equal and this is an upper bound, tuple > bound */ + if (cmpval == 0 && !bound->lower) + return -1; + + return cmpval; +} + +/* * Return whether two range bounds are equal simply by comparing datums */ static bool diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..cb0ca0e 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -30,6 +30,7 @@ #include "commands/defrem.h" #include "commands/trigger.h" #include "executor/executor.h" +#include "foreign/fdwapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -161,6 +162,11 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionTreeNode partition_tree; /* partition node tree */ + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; + List *partition_fdw_priv_lists; + int num_partitions; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1403,91 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + List *leaf_parts; + ListCell *cell; + int i; + int num_leaf_parts; + ResultRelInfo *leaf_part_rri; + PlannerInfo *root = makeNode(PlannerInfo); /* mostly dummy */ + ModifyTable *plan = makeNode(ModifyTable); /* ditto */ + RangeTblEntry *fdw_rte = makeNode(RangeTblEntry); /* ditto */ + List *fdw_private_lists = NIL; + + /* Form the partition node tree and lock partitions */ + cstate->partition_tree = RelationGetPartitionTreeNode(rel, + RowExclusiveLock); + leaf_parts = get_leaf_partition_oids(cstate->partition_tree); + num_leaf_parts = list_length(leaf_parts); + + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + /* For use below, iff a partition found to be a foreign table */ + plan->operation = CMD_INSERT; + plan->plans = list_make1(makeNode(Result)); + fdw_rte->rtekind = RTE_RELATION; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + root->parse = makeNode(Query); + root->parse->rtable = list_make1(fdw_rte); + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation part_rel; + + part_rel = heap_open(lfirst_oid(cell), RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + /* Special dance for foreign tables */ + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + fdw_rte->relid = RelationGetRelid(part_rel); + fdw_private = leaf_part_rri->ri_FdwRoutine->PlanForeignModify(root, + plan, + 1, + 0); + fdw_private_lists = lappend(fdw_private_lists, fdw_private); + } + + if (!equalTupleDescs(tupDesc, RelationGetDescr(part_rel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + + cstate->partition_fdw_priv_lists = fdw_private_lists; + } } else { @@ -2255,6 +2346,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2373,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2484,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2512,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->partition_tree != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2439,10 +2534,46 @@ CopyFrom(CopyState cstate) */ ExecBSInsertTriggers(estate, resultRelInfo); + /* Initialize FDW partition insert plans */ + if (cstate->partition_tree) + { + int i, + j; + List *fdw_private_lists = cstate->partition_fdw_priv_lists; + ModifyTableState *mtstate = makeNode(ModifyTableState); + ResultRelInfo *leaf_part_rri; + + /* Mostly dummy containing enough state for BeginForeignModify */ + mtstate->ps.state = estate; + mtstate->operation = CMD_INSERT; + + j = 0; + leaf_part_rri = cstate->partitions; + for (i = 0; i < cstate->num_partitions; i++) + { + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + Assert(fdw_private_lists); + fdw_private = list_nth(fdw_private_lists, j++); + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, 0); + } + leaf_part_rri++; + } + } + values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2625,50 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition to heap_insert the tuple into */ + if (cstate->partition_tree) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + cstate->partition_tree, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < cstate->num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding + * to the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = cstate->partitions + leaf_part_index; + + /* + * For ExecInsertIndexTuples() to work on the partition's indexes + */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the + * partition rowtype. + */ + map = cstate->partition_tupconv_maps[leaf_part_index]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2523,7 +2698,16 @@ CopyFrom(CopyState cstate) resultRelInfo->ri_PartitionCheck) ExecConstraints(resultRelInfo, slot, estate); - if (useHeapMultiInsert) + if (resultRelInfo->ri_FdwRoutine) + { + resultRelInfo->ri_FdwRoutine->ExecForeignInsert(estate, + resultRelInfo, + slot, + NULL); + /* AFTER ROW INSERT Triggers */ + ExecARInsertTriggers(estate, resultRelInfo, tuple, NIL); + } + else if (useHeapMultiInsert) { /* Add this tuple to the tuple buffer */ if (nBufferedTuples == 0) @@ -2553,7 +2737,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2762,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2781,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); @@ -2614,6 +2806,28 @@ CopyFrom(CopyState cstate) ExecCloseIndices(resultRelInfo); + /* + * Close all partitions and indices thereof, also clean up any + * state of FDW modify plans. + */ + if (cstate->partition_tree) + { + int i; + + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(estate, + resultRelInfo); + } + } + FreeExecutorState(estate); /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 3b72ae3..6478211 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1293,6 +1293,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index ea3f59a..3b4dd38 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2996,3 +3001,53 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +/* + * ExecFindPartition -- Find a leaf partition in the partition tree rooted + * at parent, for the heap tuple contained in *slot + * + * estate must be non-NULL; we'll need it to compute any expressions in the + * partition key(s) + * + * If no leaf partition is found, this routine errors out with the appropriate + * error message, else it returns the leaf partition sequence number returned + * by get_partition_for_tuple() unchanged. + */ +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionTreeNode parent, + TupleTableSlot *slot, EState *estate) +{ + int result; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + result = get_partition_for_tuple(parent, slot, estate, &failed_at); + + if (result < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return result; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..3523a2d 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,50 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + /* Determine the partition to heap_insert the tuple into */ + if (mtstate->mt_partition_tree) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_tree, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + leaf_part_index; + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, false); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +556,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1616,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1707,100 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + /* Form the partition node tree and lock partitions */ + mtstate->mt_partition_tree = RelationGetPartitionTreeNode(rel, + RowExclusiveLock); + leaf_parts = get_leaf_partition_oids(mtstate->mt_partition_tree); + num_leaf_parts = list_length(leaf_parts); + + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (leaf_part_rri->ri_FdwRoutine) + { + ListCell *lc; + List *fdw_private; + int k; + + /* + * There are as many fdw_private's in fdwPrivLists as there + * are FDW partitions, but we must find the intended for the + * this foreign table. + */ + k = 0; + foreach(lc, node->fdwPartitionOids) + { + if (lfirst_oid(lc) == ftoid) + break; + k++; + } + + Assert(k < num_leaf_parts); + fdw_private = (List *) list_nth(node->fdwPrivLists, k); + Assert(fdw_private != NIL); + + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, + eflags); + j++; + } + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2118,20 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(node->ps.state, + resultRelInfo); + } + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 28d0036..470dee7 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -188,6 +188,7 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(withCheckOptionLists); COPY_NODE_FIELD(returningLists); COPY_NODE_FIELD(fdwPrivLists); + COPY_NODE_FIELD(fdwPartitionOids); COPY_BITMAPSET_FIELD(fdwDirectModifyPlans); COPY_NODE_FIELD(rowMarks); COPY_SCALAR_FIELD(epqParam); diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 0d858f5..5c8eced 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -340,6 +340,7 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(withCheckOptionLists); WRITE_NODE_FIELD(returningLists); WRITE_NODE_FIELD(fdwPrivLists); + WRITE_NODE_FIELD(fdwPartitionOids); WRITE_BITMAPSET_FIELD(fdwDirectModifyPlans); WRITE_NODE_FIELD(rowMarks); WRITE_INT_FIELD(epqParam); diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c587d4e..ddd78c7 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -1495,6 +1495,7 @@ _readModifyTable(void) READ_NODE_FIELD(withCheckOptionLists); READ_NODE_FIELD(returningLists); READ_NODE_FIELD(fdwPrivLists); + READ_NODE_FIELD(fdwPartitionOids); READ_BITMAPSET_FIELD(fdwDirectModifyPlans); READ_NODE_FIELD(rowMarks); READ_INT_FIELD(epqParam); diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index ad49674..caf2f44 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -22,6 +22,7 @@ #include "access/stratnum.h" #include "access/sysattr.h" #include "catalog/pg_class.h" +#include "catalog/pg_inherits_fn.h" #include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/extensible.h" @@ -6159,6 +6160,81 @@ make_modifytable(PlannerInfo *root, node->fdwPrivLists = fdw_private_list; node->fdwDirectModifyPlans = direct_modify_plans; + /* Collect insert plans for all FDW-managed partitions */ + if (node->operation == CMD_INSERT) + { + RangeTblEntry *rte, + **saved_simple_rte_array; + List *partition_oids, + *fdw_partition_oids; + + Assert(list_length(resultRelations) == 1); + rte = rt_fetch(linitial_int(resultRelations), root->parse->rtable); + Assert(rte->rtekind == RTE_RELATION); + + if (rte->relkind != RELKIND_PARTITIONED_TABLE) + return node; + + partition_oids = find_all_inheritors(rte->relid, RowExclusiveLock, + NULL); + + /* Discard any previous content which is useless anyway */ + fdw_private_list = NIL; + fdw_partition_oids = NIL; + + /* + * To force the FDW driver fetch the intended RTE, we need to temporarily + * switch root->simple_rte_array to the one holding only that RTE at a + * designated index, for every foreign table. + */ + saved_simple_rte_array = root->simple_rte_array; + root->simple_rte_array = (RangeTblEntry **) + palloc0(2 * sizeof(RangeTblEntry *)); + foreach(lc, partition_oids) + { + Oid myoid = lfirst_oid(lc); + FdwRoutine *fdwroutine; + List *fdw_private; + + /* + * We are only interested in foreign tables. Note that this will + * also eliminate any partitioned tables since foreign tables can + * only ever be leaf partitions. + */ + if (get_rel_relkind(myoid) != RELKIND_FOREIGN_TABLE) + continue; + + fdw_partition_oids = lappend_oid(fdw_partition_oids, myoid); + + fdwroutine = GetFdwRoutineByRelId(myoid); + if (fdwroutine && fdwroutine->PlanForeignModify) + { + RangeTblEntry *fdw_rte; + + fdw_rte = copyObject(rte); + fdw_rte->relid = myoid; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + + /* + * Assumes PlanForeignModify() uses planner_rt_fetch(), also, + * see the above comment. + */ + root->simple_rte_array[1] = fdw_rte; + + fdw_private = fdwroutine->PlanForeignModify(root, node, 1, 0); + } + else + fdw_private = NIL; + + fdw_private_list = lappend(fdw_private_list, fdw_private); + } + + root->simple_rte_array = saved_simple_rte_array; + + node->fdwPrivLists = fdw_private_list; + node->fdwPartitionOids = fdw_partition_oids; + } + return node; } diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 672183b..e1ccaf8 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -50,4 +52,9 @@ extern List *RelationGetPartitionQual(Relation rel, bool recurse); /* For tuple routing */ extern PartitionTreeNode RelationGetPartitionTreeNode(Relation rel, int lockmode); extern List *get_leaf_partition_oids(PartitionTreeNode parent); + +extern int get_partition_for_tuple(PartitionTreeNode parent, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..fadc402 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionTreeNode parent, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..06675bc 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,15 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionTreeNodeData *mt_partition_tree; + /* Partition descriptor node tree */ + ResultRelInfo *mt_partitions; /* Per leaf partition target + * relations */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per leaf partition + * tuple conversion map */ + int mt_num_partitions; /* Number of leaf partition target + * relations in the above array */ } ModifyTableState; /* ---------------- diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index e2fbc7d..d82222c 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -189,6 +189,7 @@ typedef struct ModifyTable List *returningLists; /* per-target-table RETURNING tlists */ List *fdwPrivLists; /* per-target-table FDW private data lists */ Bitmapset *fdwDirectModifyPlans; /* indices of FDW DM plans */ + List *fdwPartitionOids; /* OIDs of FDW-managed partition */ List *rowMarks; /* PlanRowMarks (non-locking only) */ int epqParam; /* ID of Param for EvalPlanQual re-eval */ OnConflictAction onConflictAction; /* ON CONFLICT action */ diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 9ae6b09..d5dcb59 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -227,6 +227,58 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index b6e821e..fbd30d9 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,31 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------1C84B3F970C18697850BA485 Content-Type: text/x-diff; name="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-15.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-15"; filename*1=".patch" ^ permalink raw reply [nested|flat] 13+ messages in thread
* [PATCH 7/8] Tuple routing for partitioned tables. @ 2016-07-27 07:59 amit <[email protected]> 0 siblings, 0 replies; 13+ messages in thread From: amit @ 2016-07-27 07:59 UTC (permalink / raw) Both COPY FROM and INSERT are covered by this commit. Routing to foreing partitions is not supported at the moment. --- src/backend/catalog/partition.c | 289 +++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 151 ++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 58 ++++++- src/backend/executor/nodeModifyTable.c | 130 ++++++++++++++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 6 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 8 + src/test/regress/expected/insert.out | 52 ++++++ src/test/regress/sql/insert.sql | 25 +++ 11 files changed, 728 insertions(+), 6 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index 03c7fa1..d758500 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -161,6 +161,18 @@ static Oid get_partition_operator(PartitionKey key, int col, StrategyNumber stra static List *generate_partition_qual(Relation rel, bool recurse); +/* Support get_partition_for_tuple() */ +static PartitionKeyExecInfo *BuildPartitionKeyExecInfo(Relation rel); +static void FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); +static int list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull); +static int range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum *tuple); + /* List partition related support functions */ static bool equal_list_info(PartitionKey key, PartitionListInfo *l1, PartitionListInfo *l2); @@ -174,6 +186,8 @@ static PartitionRangeBound *copy_range_bound(PartitionKey key, PartitionRangeBou static bool equal_range_info(PartitionKey key, PartitionRangeInfo *r1, PartitionRangeInfo *r2); static int32 partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg); +static int32 partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg); static bool partition_rbound_eq(PartitionKey key, PartitionRangeBound *b1, PartitionRangeBound *b2); typedef int32 (*partition_rbound_bsearch_cmp_fn) (PartitionKey, @@ -988,7 +1002,7 @@ RelationGetPartitionDispatchInfo(Relation rel, int lockmode, pd[i] = (PartitionDispatch) palloc(sizeof(PartitionDispatchData)); pd[i]->relid = RelationGetRelid(partrel); - pd[i]->pkinfo = NULL; + pd[i]->pkinfo = BuildPartitionKeyExecInfo(partrel); pd[i]->partdesc = partdesc; pd[i]->indexes = (int *) palloc(partdesc->nparts * sizeof(int)); heap_close(partrel, NoLock); @@ -1459,6 +1473,245 @@ generate_partition_qual(Relation rel, bool recurse) return result; } +/* + * BuildPartitionKeyExecInfo + * Construct a list of PartitionKeyExecInfo records for an open + * relation + * + * PartitionKeyExecInfo stores the information about the partition key + * that's needed when inserting tuples into a partitioned table; especially, + * partition key expression state if there are any expression columns in + * the partition key. Normally we build a PartitionKeyExecInfo for a + * partitioned table just once per command, and then use it for (potentially) + * many tuples. + * + */ +static PartitionKeyExecInfo * +BuildPartitionKeyExecInfo(Relation rel) +{ + PartitionKeyExecInfo *pkinfo; + + pkinfo = (PartitionKeyExecInfo *) palloc0(sizeof(PartitionKeyExecInfo)); + pkinfo->pi_Key = RelationGetPartitionKey(rel); + pkinfo->pi_ExpressionState = NIL; + + return pkinfo; +} + +/* ---------------- + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for the partition key + * of a tuple. + * + * pkinfo partition key execution info + * slot Heap tuple from which to extract partition key + * estate executor state for evaluating any partition key + * expressions (must be non-NULL) + * values Array of partition key Datums (output area) + * isnull Array of is-null indicators (output area) + * + * the ecxt_scantuple slot of estate's per-tuple expr context must point to + * the heap tuple passed in. + * ---------------- + */ +static void +FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pkinfo->pi_Key->partexprs != NIL && pkinfo->pi_ExpressionState == NIL) + { + /* Check caller has set up context correctly */ + Assert(estate != NULL && + GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + + /* First time through, set up expression evaluation state */ + pkinfo->pi_ExpressionState = (List *) + ExecPrepareExpr((Expr *) pkinfo->pi_Key->partexprs, + estate); + } + + partexpr_item = list_head(pkinfo->pi_ExpressionState); + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + { + AttrNumber keycol = pkinfo->pi_Key->partattrs[i]; + Datum pkDatum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + pkDatum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + pkDatum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = pkDatum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Finds a leaf partition for tuple contained in *slot + * + * Returned value is the sequence number of the leaf partition thus found, + * or -1 if no leaf partition is found for the tuple. *failed_at is set + * to the OID of the partitioned table whose partition was not found in + * the latter case. + */ +int +get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + PartitionDispatch parent; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int cur_idx; + int i; + + /* start with the root partitioned table */ + parent = pd[0]; + while(1) + { + PartitionDesc partdesc = parent->partdesc; + PartitionKeyExecInfo *pkinfo = parent->pkinfo; + + /* Quick exit */ + if (partdesc->nparts == 0) + { + *failed_at = parent->relid; + return -1; + } + + /* Extract partition key from tuple */ + FormPartitionKeyDatum(pkinfo, slot, estate, values, isnull); + + switch (pkinfo->pi_Key->strategy) + { + case PARTITION_STRATEGY_LIST: + cur_idx = list_partition_for_tuple(pkinfo->pi_Key, partdesc, + values[0], isnull[0]); + break; + + case PARTITION_STRATEGY_RANGE: + /* Disallow nulls in the partition key of the tuple */ + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + if (isnull[i]) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key of row contains null"))); + + cur_idx = range_partition_for_tuple(pkinfo->pi_Key, partdesc, + values); + break; + } + + /* + * cur_idx < 0 means we failed to find a partition of this parent. + * cur_idx >= 0 means we either found the leaf partition we have been + * looking for, or the next parent to find a partition of. + */ + if (cur_idx < 0) + { + *failed_at = parent->relid; + return -1; + } + else if (parent->indexes[cur_idx] < 0) + parent = pd[-parent->indexes[cur_idx]]; + else + break; + } + + return parent->indexes[cur_idx]; +} + +/* + * list_partition_for_tuple + * Find the list partition for a tuple (arg 'value' contains the + * list partition key of the original tuple) + * + * Returns -1 if none found. + */ +static int +list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull) +{ + PartitionListInfo listinfo; + int found; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_LIST); + listinfo = pdesc->boundinfo->bounds.lists; + + if (isnull && listinfo.has_null) + return listinfo.null_index; + else if (!isnull) + { + found = partition_list_values_bsearch(key, + listinfo.values, + listinfo.nvalues, + value); + if (found >= 0) + return listinfo.indexes[found]; + } + + /* Control reaches here if isnull and !listinfo->has_null */ + return -1; +} + +/* + * range_partition_for_tuple + * Get the index of the range partition for a tuple (arg 'tuple' + * actually contains the range partition key of the original + * tuple) + * + * Returns -1 if none found. + */ +static int +range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, Datum *tuple) +{ + int offset; + PartitionRangeInfo rangeinfo; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_RANGE); + rangeinfo = pdesc->boundinfo->bounds.ranges; + + offset = partition_rbound_bsearch(key, + rangeinfo.bounds, rangeinfo.nbounds, + tuple, partition_rbound_datum_cmp, + true, NULL); + + /* + * Offset returned is such that the bound at offset is found to be less + * or equal with the tuple. That is, the tuple belongs to the partition + * with the rangeinfo.bounds[offset] as the lower bound and + * rangeinfo.bounds[offset+1] as the upper bound, provided the latter is + * indeed an upper (!lower) bound. If it turns out otherwise, the + * corresponding index will be -1, which means no valid partition exists. + */ + return rangeinfo.indexes[offset+1]; +} + /* List partition related support functions */ /* @@ -1704,6 +1957,40 @@ partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg) } /* + * Return whether bound <=, =, >= partition key of tuple + * + * The 3rd argument is void * so that it can be used with + * partition_rbound_bsearch() + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg) +{ + Datum *datums1 = bound->datums, + *datums2 = (Datum *) arg; + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (bound->infinite[i]) + return bound->lower ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + datums1[i], datums2[i])); + if (cmpval != 0) + break; + } + + /* If datums are equal and this is an upper bound, tuple > bound */ + if (cmpval == 0 && !bound->lower) + return -1; + + return cmpval; +} + +/* * Return whether two range bounds are equal simply by comparing datums */ static bool diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..7d76ead 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -161,6 +161,10 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionDispatch *partition_dispatch_info; + int num_partitions; + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1401,67 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + List *leaf_parts; + ListCell *cell; + int i, + num_leaf_parts; + ResultRelInfo *leaf_part_rri; + + /* Get the tuple-routing information and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + num_leaf_parts = list_length(leaf_parts); + cstate->partition_dispatch_info = pd; + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) palloc(num_leaf_parts * + sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation partrel; + + /* + * All partitions locked above; will be closed after CopyFrom is + * finished. + */ + partrel = heap_open(lfirst_oid(cell), NoLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(partrel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + partrel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(tupDesc, RelationGetDescr(partrel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(partrel), + gettext_noop("could not convert row type")); + leaf_part_rri++; + i++; + } + } } else { @@ -2255,6 +2320,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2347,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2458,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2486,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->partition_dispatch_info != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2442,7 +2511,11 @@ CopyFrom(CopyState cstate) values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2567,56 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition to heap_insert the tuple into */ + if (cstate->partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + cstate->partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < cstate->num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding + * to the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = cstate->partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* + * For ExecInsertIndexTuples() to work on the partition's indexes + */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the + * partition rowtype. + */ + map = cstate->partition_tupconv_maps[leaf_part_index]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2553,7 +2676,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2701,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2720,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); @@ -2614,6 +2745,20 @@ CopyFrom(CopyState cstate) ExecCloseIndices(resultRelInfo); + /* Close all partitions and indices thereof */ + if (cstate->partition_dispatch_info) + { + int i; + + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + } + FreeExecutorState(estate); /* diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 3b72ae3..6478211 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1293,6 +1293,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index c7a6347..54fb771 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2990,3 +2995,52 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +/* + * ExecFindPartition -- Find a leaf partition in the partition tree rooted + * at parent, for the heap tuple contained in *slot + * + * estate must be non-NULL; we'll need it to compute any expressions in the + * partition key(s) + * + * If no leaf partition is found, this routine errors out with the appropriate + * error message, else it returns the leaf partition sequence number returned + * by get_partition_for_tuple() unchanged. + */ +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionDispatch *pd, + TupleTableSlot *slot, EState *estate) +{ + int result; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + result = get_partition_for_tuple(pd, slot, estate, &failed_at); + if (result < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return result; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..aa1d8b9 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,56 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + /* Determine the partition to heap_insert the tuple into */ + if (mtstate->mt_partition_dispatch_info) + { + int leaf_part_index; + TupleConversionMap *map; + + /* + * Away we go ... If we end up not finding a partition after all, + * ExecFindPartition() does not return and errors out instead. + * Otherwise, the returned value is to be used as an index into + * arrays mt_partitions[] and mt_partition_tupconv_maps[] that + * will get us the ResultRelInfo and TupleConversionMap for the + * partition, respectively. + */ + leaf_part_index = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_dispatch_info, + slot, + estate); + Assert(leaf_part_index >= 0 && + leaf_part_index < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + leaf_part_index; + + /* We do not yet have a way to insert into a foreign partition */ + if (resultRelInfo->ri_FdwRoutine) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot route inserted tuples to a foreign table"))); + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[leaf_part_index]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, false); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +562,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1622,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1713,69 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + PartitionDispatch *pd; + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + /* Form the partition node tree and lock partitions */ + pd = RelationGetPartitionDispatchInfo(rel, RowExclusiveLock, + &leaf_parts); + mtstate->mt_partition_dispatch_info = pd; + num_leaf_parts = list_length(leaf_parts); + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2093,15 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + } + /* * Free the exprcontext */ diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index fb43cc1..1d231a3 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -50,4 +52,8 @@ extern List *RelationGetPartitionQual(Relation rel, bool recurse); /* For tuple routing */ extern PartitionDispatch *RelationGetPartitionDispatchInfo(Relation rel, int lockmode, List **leaf_part_oids); +extern int get_partition_for_tuple(PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..b4d09f9 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionDispatch *pd, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..606cb21 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,13 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionDispatchData **mt_partition_dispatch_info; + /* Tuple-routing support info */ + int mt_num_partitions; /* Number of members in the + * following arrays */ + ResultRelInfo *mt_partitions; /* Per partition result relation */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per partition tuple conversion map */ } ModifyTableState; /* ---------------- diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 9ae6b09..d5dcb59 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -227,6 +227,58 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index b6e821e..fbd30d9 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,31 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------38C0D572AC793CAA7276C062 Content-Type: text/x-diff; name="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-16.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-16"; filename*1=".patch" ^ permalink raw reply [nested|flat] 13+ messages in thread
* [PATCH 7/8] Tuple routing for partitioned tables. @ 2016-07-27 07:59 amit <[email protected]> 0 siblings, 0 replies; 13+ messages in thread From: amit @ 2016-07-27 07:59 UTC (permalink / raw) Both COPY FROM and INSERT. --- src/backend/catalog/partition.c | 327 ++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 203 +++++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 49 +++++- src/backend/executor/nodeModifyTable.c | 156 +++++++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 1 + src/backend/optimizer/plan/createplan.c | 77 +++++++ src/backend/optimizer/util/plancat.c | 13 ++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 7 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 10 + src/include/nodes/plannodes.h | 1 + src/include/optimizer/plancat.h | 1 + src/test/regress/expected/insert.out | 52 +++++ src/test/regress/sql/insert.sql | 25 +++ 18 files changed, 932 insertions(+), 7 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index a5800fb..e7e0c7c 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -182,6 +182,18 @@ static List *generate_partition_qual(Relation rel, bool recurse); static PartitionTreeNode GetPartitionTreeNodeRecurse(Relation rel, int offset); static int get_leaf_partition_count(PartitionTreeNode ptnode); +/* Support get_partition_for_tuple() */ +static PartitionKeyExecInfo *BuildPartitionKeyExecInfo(Relation rel); +static void FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); +static int list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull); +static int range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum *tuple); + /* List partition related support functions */ static bool equal_list_info(PartitionKey key, PartitionListInfo *l1, PartitionListInfo *l2); @@ -195,6 +207,8 @@ static PartitionRangeBound *copy_range_bound(PartitionKey key, PartitionRangeBou static bool equal_range_info(PartitionKey key, PartitionRangeInfo *r1, PartitionRangeInfo *r2); static int32 partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg); +static int32 partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg); static bool partition_rbound_eq(PartitionKey key, PartitionRangeBound *b1, PartitionRangeBound *b2); typedef int32 (*partition_rbound_bsearch_cmp_fn) (PartitionKey, @@ -1438,7 +1452,7 @@ GetPartitionTreeNodeRecurse(Relation rel, int offset) /* First build our own node */ parent = (PartitionTreeNode) palloc0(sizeof(PartitionTreeNodeData)); - parent->pkinfo = NULL; + parent->pkinfo = BuildPartitionKeyExecInfo(rel); parent->pdesc = RelationGetPartitionDesc(rel); parent->relid = RelationGetRelid(rel); parent->offset = offset; @@ -1522,6 +1536,283 @@ get_leaf_partition_count(PartitionTreeNode ptnode) return result; } +/* + * BuildPartitionKeyExecInfo + * Construct a list of PartitionKeyExecInfo records for an open + * relation + * + * PartitionKeyExecInfo stores the information about the partition key + * that's needed when inserting tuples into a partitioned table; especially, + * partition key expression state if there are any expression columns in + * the partition key. Normally we build a PartitionKeyExecInfo for a + * partitioned table just once per command, and then use it for (potentially) + * many tuples. + * + */ +static PartitionKeyExecInfo * +BuildPartitionKeyExecInfo(Relation rel) +{ + PartitionKeyExecInfo *pkinfo; + + pkinfo = (PartitionKeyExecInfo *) palloc0(sizeof(PartitionKeyExecInfo)); + pkinfo->pi_Key = RelationGetPartitionKey(rel); + pkinfo->pi_ExpressionState = NIL; + + return pkinfo; +} + +/* + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for partition key columns + */ +static void +FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pkinfo->pi_Key->partexprs != NIL && pkinfo->pi_ExpressionState == NIL) + { + /* First time through, set up expression evaluation state */ + pkinfo->pi_ExpressionState = (List *) + ExecPrepareExpr((Expr *) pkinfo->pi_Key->partexprs, + estate); + /* Check caller has set up context correctly */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + } + + partexpr_item = list_head(pkinfo->pi_ExpressionState); + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + { + AttrNumber keycol = pkinfo->pi_Key->partattrs[i]; + Datum pkDatum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + pkDatum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + pkDatum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = pkDatum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Recursively finds the "leaf" partition for tuple + * + * Returns -1 if no partition is found and sets *failed_at to the OID of + * the partitioned table whose partition was not found. + */ +int +get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + Relation partRel; + PartitionKeyExecInfo *pkinfo = ptnode->pkinfo; + PartitionTreeNode node; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int i; + int index; + + /* Guard against stack overflow due to overly deep partition tree */ + check_stack_depth(); + + if (ptnode->pdesc->nparts == 0) + { + *failed_at = ptnode->relid; + return -1; + } + + /* Extract partition key from tuple */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + FormPartitionKeyDatum(pkinfo, slot, estate, values, isnull); + + /* Disallow nulls, if range partition key */ + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + if (isnull[i] && pkinfo->pi_Key->strategy == PARTITION_STRATEGY_RANGE) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key contains null"))); + + switch (pkinfo->pi_Key->strategy) + { + case PARTITION_STRATEGY_LIST: + index = list_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values[0], isnull[0]); + break; + + case PARTITION_STRATEGY_RANGE: + index = range_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values); + break; + } + + /* No partition found at this level */ + if (index < 0) + { + *failed_at = ptnode->relid; + return index; + } + + partRel = heap_open(ptnode->pdesc->oids[index], NoLock); + + /* Don't recurse if the index'th partition is a leaf partition. */ + if (partRel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) + { + PartitionTreeNode prev; + + /* + * Index returned above considers only this parent (of which partRel + * is a partition). We however want to return the index across the + * the whole partition tree. If partRel is the leftmost partition + * of parent (index = 0) or if none of the left siblings are + * partitioned tables, we can simply return parent's offset plus + * partRel's index as the result. Otherwise, we find the node + * corresponding to the rightmost partitioned table among partRel's + * left siblings and use the information therein. + */ + prev = node = ptnode->downlink; + if (node && node->index < index) + { + /* + * Find the partition tree node such that its index value is the + * greatest value less than the above returned index. + */ + while (node) + { + if (node->index > index) + { + node = prev; + break; + } + + prev = node; + node = node->next; + } + + if (!node) + node = prev; + Assert (node != NULL); + + index = node->offset + node->num_leaf_parts + + (index - node->index - 1); + } + else + /* + * The easy case where all of partRel's left siblings (if any) are + * leaf partitions. + */ + index = ptnode->offset + index; + + heap_close(partRel, NoLock); + return index; + } + + heap_close(partRel, NoLock); + + /* + * Need to recurse as the selected partition is a partitioned table + * itself. Locate the corresponding PartitionTreeNode to pass it down. + */ + node = ptnode->downlink; + while (node->next != NULL && node->index != index) + node = node->next; + Assert (node != NULL); + + return get_partition_for_tuple(node, slot, estate, failed_at); +} + +/* + * list_partition_for_tuple + * Find the list partition for a tuple (arg 'value' contains the + * list partition key of the original tuple) + * + * Returns -1 if none found. + */ +static int +list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull) +{ + PartitionListInfo listinfo; + int found; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_LIST); + listinfo = pdesc->boundinfo->bounds.lists; + + if (isnull && listinfo.has_null) + return listinfo.null_index; + else if (!isnull) + { + found = partition_list_values_bsearch(key, + listinfo.values, + listinfo.nvalues, + value); + if (found >= 0) + return listinfo.indexes[found]; + } + + /* Control reaches here if isnull and !listinfo->has_null */ + return -1; +} + +/* + * range_partition_for_tuple + * Get the index of the range partition for a tuple (arg 'tuple' + * actually contains the range partition key of the original + * tuple) + * + * Returns -1 if none found. + */ +static int +range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, Datum *tuple) +{ + int offset; + PartitionRangeInfo rangeinfo; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_RANGE); + rangeinfo = pdesc->boundinfo->bounds.ranges; + + offset = partition_rbound_bsearch(key, + rangeinfo.bounds, rangeinfo.nbounds, + tuple, partition_rbound_datum_cmp, + true, NULL); + + /* + * Offset returned is such that the bound at offset is found to be less + * or equal with the tuple. That is, the tuple belongs to the partition + * with the rangeinfo.bounds[offset] as the lower bound and + * rangeinfo.bounds[offset+1] as the upper bound, provided the latter is + * indeed an upper (!lower) bound. If it turns out otherwise, the + * corresponding index will be -1, which means no valid partition exists. + */ + return rangeinfo.indexes[offset+1]; +} + /* List partition related support functions */ /* @@ -1767,6 +2058,40 @@ partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg) } /* + * Return whether bound <=, =, >= partition key of tuple + * + * The 3rd argument is void * so that it can be used with + * partition_rbound_bsearch() + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg) +{ + Datum *datums1 = bound->datums, + *datums2 = (Datum *) arg; + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (bound->infinite[i]) + return bound->lower ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + datums1[i], datums2[i])); + if (cmpval != 0) + break; + } + + /* If datums are equal and this is an upper bound, tuple > bound */ + if (cmpval == 0 && !bound->lower) + return -1; + + return cmpval; +} + +/* * Return whether two range bounds are equal simply by comparing datums */ static bool diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..6fb376f 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -30,6 +30,7 @@ #include "commands/defrem.h" #include "commands/trigger.h" #include "executor/executor.h" +#include "foreign/fdwapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -161,6 +162,11 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionTreeNode ptnode; /* partition descriptor node tree */ + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; + List *partition_fdw_priv_lists; + int num_partitions; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1403,94 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + List *leaf_parts; + ListCell *cell; + int i; + int num_leaf_parts; + ResultRelInfo *leaf_part_rri; + PlannerInfo *root = makeNode(PlannerInfo); /* mostly dummy */ + Query *parse = makeNode(Query); /* ditto */ + ModifyTable *plan = makeNode(ModifyTable); /* ditto */ + RangeTblEntry *fdw_rte = makeNode(RangeTblEntry); /* ditto */ + List *fdw_private_lists = NIL; + + cstate->ptnode = RelationGetPartitionTreeNode(rel); + leaf_parts = get_leaf_partition_oids(cstate->ptnode); + num_leaf_parts = list_length(leaf_parts); + + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + /* For use below, iff a partition found to be a foreign table */ + plan->operation = CMD_INSERT; + plan->plans = list_make1(makeNode(Result)); + fdw_rte->rtekind = RTE_RELATION; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + parse->rtable = list_make1(fdw_rte); + root->parse = parse; + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation part_rel; + + part_rel = heap_open(lfirst_oid(cell), RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + /* Special dance for foreign tables */ + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + fdw_rte->relid = RelationGetRelid(part_rel); + fdw_private = leaf_part_rri->ri_FdwRoutine->PlanForeignModify(root, + plan, + 1, + 0); + fdw_private_lists = lappend(fdw_private_lists, fdw_private); + } + + if (!equalTupleDescs(tupDesc, RelationGetDescr(part_rel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + + cstate->partition_fdw_priv_lists = fdw_private_lists; + pfree(fdw_rte); + pfree(plan); + pfree(parse); + pfree(root); + } } else { @@ -1692,6 +1786,8 @@ ClosePipeToProgram(CopyState cstate) static void EndCopy(CopyState cstate) { + int i; + if (cstate->is_program) { ClosePipeToProgram(cstate); @@ -1705,6 +1801,23 @@ EndCopy(CopyState cstate) cstate->filename))); } + /* Close all partitions and indices thereof */ + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + /* XXX - EState not handy here to pass to EndForeignModify() */ + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(NULL, resultRelInfo); + + if (cstate->partition_tupconv_maps[i]) + pfree(cstate->partition_tupconv_maps[i]); + } + MemoryContextDelete(cstate->copycontext); pfree(cstate); } @@ -2255,6 +2368,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2395,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2506,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2534,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->ptnode != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2439,10 +2556,46 @@ CopyFrom(CopyState cstate) */ ExecBSInsertTriggers(estate, resultRelInfo); + /* Initialize FDW partition insert plans */ + if (cstate->ptnode) + { + int i, + j; + List *fdw_private_lists = cstate->partition_fdw_priv_lists; + ModifyTableState *mtstate = makeNode(ModifyTableState); + ResultRelInfo *leaf_part_rri; + + /* Mostly dummy containing enough state for BeginForeignModify */ + mtstate->ps.state = estate; + mtstate->operation = CMD_INSERT; + + j = 0; + leaf_part_rri = cstate->partitions; + for (i = 0; i < cstate->num_partitions; i++) + { + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + Assert(fdw_private_lists); + fdw_private = list_nth(fdw_private_lists, j++); + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, 0); + } + leaf_part_rri++; + } + } + values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2647,31 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition */ + saved_resultRelInfo = resultRelInfo; + if (cstate->ptnode) + { + int i_leaf_partition; + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + cstate->ptnode, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < cstate->num_partitions); + + resultRelInfo = cstate->partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = cstate->partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2523,7 +2701,16 @@ CopyFrom(CopyState cstate) resultRelInfo->ri_PartitionCheck) ExecConstraints(resultRelInfo, slot, estate); - if (useHeapMultiInsert) + if (resultRelInfo->ri_FdwRoutine) + { + resultRelInfo->ri_FdwRoutine->ExecForeignInsert(estate, + resultRelInfo, + slot, + NULL); + /* AFTER ROW INSERT Triggers */ + ExecARInsertTriggers(estate, resultRelInfo, tuple, NIL); + } + else if (useHeapMultiInsert) { /* Add this tuple to the tuple buffer */ if (nBufferedTuples == 0) @@ -2553,7 +2740,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2765,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2784,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 3b72ae3..6478211 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1293,6 +1293,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index ea3f59a..b5125a8 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2996,3 +3001,43 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionTreeNode ptnode, + TupleTableSlot *slot, EState *estate) +{ + int i_leaf_partition; + Oid failed_at; + ExprContext *econtext = GetPerTupleExprContext(estate); + + econtext->ecxt_scantuple = slot; + i_leaf_partition = get_partition_for_tuple(ptnode, slot, estate, + &failed_at); + + if (i_leaf_partition < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return i_leaf_partition; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..44801b8 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,45 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + if (mtstate->mt_partition_tree_root) + { + int i_leaf_partition; + TupleConversionMap *map; + + /* + * Get the index of leaf partitions that we'll use to get the correct + * ResultRelInfo from mtstate->mt_partitions[]. + */ + i_leaf_partition = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_tree_root, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < mtstate->mt_num_partitions); + + /* + * Save the old ResultRelInfo and switch to the one corresponding to + * the selected partition. + */ + saved_resultRelInfo = resultRelInfo; + resultRelInfo = mtstate->mt_partitions + i_leaf_partition; + + /* For ExecInsertIndexTuples() to work on the partition's indexes */ + estate->es_result_relation_info = resultRelInfo; + + /* + * We might need to convert from the parent rowtype to the partition + * rowtype. + */ + map = mtstate->mt_partition_tupconv_maps[i_leaf_partition]; + if (map) + { + tuple = do_convert_tuple(tuple, map); + ExecStoreTuple(tuple, slot, InvalidBuffer, false); + } + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +551,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1611,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1702,98 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + mtstate->mt_partition_tree_root = RelationGetPartitionTreeNode(rel); + leaf_parts = get_leaf_partition_oids(mtstate->mt_partition_tree_root); + num_leaf_parts = list_length(leaf_parts); + + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (leaf_part_rri->ri_FdwRoutine) + { + ListCell *lc; + List *fdw_private; + int k; + + /* + * There are as many fdw_private's in fdwPrivLists as there + * are FDW partitions, but we must find the intended for the + * this foreign table. + */ + k = 0; + foreach(lc, node->fdwPartitionOids) + { + if (lfirst_oid(lc) == ftoid) + break; + k++; + } + + Assert(k < num_leaf_parts); + fdw_private = (List *) list_nth(node->fdwPrivLists, k); + Assert(fdw_private != NIL); + + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, + eflags); + j++; + } + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2111,23 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(node->ps.state, + resultRelInfo); + + if (node->mt_partition_tupconv_maps[i]) + pfree(node->mt_partition_tupconv_maps[i]); + } + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 28d0036..470dee7 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -188,6 +188,7 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(withCheckOptionLists); COPY_NODE_FIELD(returningLists); COPY_NODE_FIELD(fdwPrivLists); + COPY_NODE_FIELD(fdwPartitionOids); COPY_BITMAPSET_FIELD(fdwDirectModifyPlans); COPY_NODE_FIELD(rowMarks); COPY_SCALAR_FIELD(epqParam); diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 0d858f5..5c8eced 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -340,6 +340,7 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(withCheckOptionLists); WRITE_NODE_FIELD(returningLists); WRITE_NODE_FIELD(fdwPrivLists); + WRITE_NODE_FIELD(fdwPartitionOids); WRITE_BITMAPSET_FIELD(fdwDirectModifyPlans); WRITE_NODE_FIELD(rowMarks); WRITE_INT_FIELD(epqParam); diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c587d4e..ddd78c7 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -1495,6 +1495,7 @@ _readModifyTable(void) READ_NODE_FIELD(withCheckOptionLists); READ_NODE_FIELD(returningLists); READ_NODE_FIELD(fdwPrivLists); + READ_NODE_FIELD(fdwPartitionOids); READ_BITMAPSET_FIELD(fdwDirectModifyPlans); READ_NODE_FIELD(rowMarks); READ_INT_FIELD(epqParam); diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index ad49674..29f40fe 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -22,6 +22,7 @@ #include "access/stratnum.h" #include "access/sysattr.h" #include "catalog/pg_class.h" +#include "catalog/pg_inherits_fn.h" #include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/extensible.h" @@ -6159,6 +6160,82 @@ make_modifytable(PlannerInfo *root, node->fdwPrivLists = fdw_private_list; node->fdwDirectModifyPlans = direct_modify_plans; + /* Collect insert plans for all FDW-managed partitions */ + if (node->operation == CMD_INSERT) + { + RangeTblEntry *rte, + **saved_simple_rte_array; + List *partition_oids, + *fdw_partition_oids; + + Assert(list_length(resultRelations) == 1); + rte = rt_fetch(linitial_int(resultRelations), root->parse->rtable); + Assert(rte->rtekind == RTE_RELATION); + + if (rte->relkind != RELKIND_PARTITIONED_TABLE) + return node; + + partition_oids = find_all_inheritors(rte->relid, NoLock, NULL); + + /* Discard any previous content which is useless anyway */ + fdw_private_list = NIL; + fdw_partition_oids = NIL; + + /* + * To force the FDW driver fetch the intended RTE, we need to temporarily + * switch root->simple_rte_array to the one holding only that RTE at a + * designated index, for every foreign table. + */ + saved_simple_rte_array = root->simple_rte_array; + root->simple_rte_array = (RangeTblEntry **) + palloc0(2 * sizeof(RangeTblEntry *)); + foreach(lc, partition_oids) + { + Oid myoid = lfirst_oid(lc); + FdwRoutine *fdwroutine; + List *fdw_private; + + /* + * We are only interested in foreign tables. Note that this will + * also eliminate any partitioned tables since foreign tables can + * only ever be leaf partitions. + */ + if (!oid_is_foreign_table(myoid)) + continue; + + fdw_partition_oids = lappend_oid(fdw_partition_oids, myoid); + + fdwroutine = GetFdwRoutineByRelId(myoid); + if (fdwroutine && fdwroutine->PlanForeignModify) + { + RangeTblEntry *fdw_rte; + + fdw_rte = copyObject(rte); + fdw_rte->relid = myoid; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + + /* + * Assumes PlanForeignModify() uses planner_rt_fetch(), also, + * see the above comment. + */ + root->simple_rte_array[1] = fdw_rte; + + fdw_private = fdwroutine->PlanForeignModify(root, node, 1, 0); + pfree(fdw_rte); + } + else + fdw_private = NIL; + + fdw_private_list = lappend(fdw_private_list, fdw_private); + } + + pfree(root->simple_rte_array); + root->simple_rte_array = saved_simple_rte_array; + + node->fdwPrivLists = fdw_private_list; + node->fdwPartitionOids = fdw_partition_oids; + } + return node; } diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index a2cbf14..e85ca0a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1715,3 +1715,16 @@ has_row_triggers(PlannerInfo *root, Index rti, CmdType event) heap_close(relation, NoLock); return result; } + +bool +oid_is_foreign_table(Oid relid) +{ + Relation rel; + char relkind; + + rel = heap_open(relid, NoLock); + relkind = rel->rd_rel->relkind; + heap_close(rel, NoLock); + + return relkind == RELKIND_FOREIGN_TABLE; +} diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 057b1e3..86cc598 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -50,4 +52,9 @@ extern List *RelationGetPartitionQual(Relation rel, bool recurse); /* For tuple routing */ extern PartitionTreeNode RelationGetPartitionTreeNode(Relation rel); extern List *get_leaf_partition_oids(PartitionTreeNode ptnode); + +extern int get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..c62946f 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..ce01008 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,15 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionTreeNodeData *mt_partition_tree_root; + /* Partition descriptor node tree */ + ResultRelInfo *mt_partitions; /* Per leaf partition target + * relations */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per leaf partition + * tuple conversion map */ + int mt_num_partitions; /* Number of leaf partition target + * relations in the above array */ } ModifyTableState; /* ---------------- diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index e2fbc7d..d82222c 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -189,6 +189,7 @@ typedef struct ModifyTable List *returningLists; /* per-target-table RETURNING tlists */ List *fdwPrivLists; /* per-target-table FDW private data lists */ Bitmapset *fdwDirectModifyPlans; /* indices of FDW DM plans */ + List *fdwPartitionOids; /* OIDs of FDW-managed partition */ List *rowMarks; /* PlanRowMarks (non-locking only) */ int epqParam; /* ID of Param for EvalPlanQual re-eval */ OnConflictAction onConflictAction; /* ON CONFLICT action */ diff --git a/src/include/optimizer/plancat.h b/src/include/optimizer/plancat.h index 125274e..fac606c 100644 --- a/src/include/optimizer/plancat.h +++ b/src/include/optimizer/plancat.h @@ -56,5 +56,6 @@ extern Selectivity join_selectivity(PlannerInfo *root, SpecialJoinInfo *sjinfo); extern bool has_row_triggers(PlannerInfo *root, Index rti, CmdType event); +extern bool oid_is_foreign_table(Oid relid); #endif /* PLANCAT_H */ diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 9ae6b09..d5dcb59 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -227,6 +227,58 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index b6e821e..fbd30d9 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,31 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------30A9D66B3BEA4523792AC3CB Content-Type: text/x-diff; name="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-14.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-14"; filename*1=".patch" ^ permalink raw reply [nested|flat] 13+ messages in thread
* [PATCH 7/8] Tuple routing for partitioned tables. @ 2016-07-27 07:59 amit <[email protected]> 0 siblings, 0 replies; 13+ messages in thread From: amit @ 2016-07-27 07:59 UTC (permalink / raw) Both COPY FROM and INSERT. --- src/backend/catalog/partition.c | 327 ++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 205 +++++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 47 +++++- src/backend/executor/nodeModifyTable.c | 142 +++++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 1 + src/backend/optimizer/plan/createplan.c | 77 +++++++ src/backend/optimizer/util/plancat.c | 13 ++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 7 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 10 + src/include/nodes/plannodes.h | 1 + src/include/optimizer/plancat.h | 1 + src/test/regress/expected/insert.out | 59 ++++++- src/test/regress/sql/insert.sql | 28 +++ 18 files changed, 926 insertions(+), 9 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index ae9e945..373e9e5 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -181,6 +181,18 @@ static List *generate_partition_qual(Relation rel, bool recurse); static PartitionTreeNode GetPartitionTreeNodeRecurse(Relation rel, int offset); static int get_leaf_partition_count(PartitionTreeNode ptnode); +/* Support get_partition_for_tuple() */ +static PartitionKeyExecInfo *BuildPartitionKeyExecInfo(Relation rel); +static void FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); +static int list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull); +static int range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum *tuple); + /* List partition related support functions */ static bool equal_list_info(PartitionKey key, PartitionListInfo *l1, PartitionListInfo *l2); @@ -194,6 +206,8 @@ static PartitionRangeBound *copy_range_bound(PartitionKey key, PartitionRangeBou static bool equal_range_info(PartitionKey key, PartitionRangeInfo *r1, PartitionRangeInfo *r2); static int32 partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg); +static int32 partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg); static bool partition_rbound_eq(PartitionKey key, PartitionRangeBound *b1, PartitionRangeBound *b2); typedef int32 (*partition_rbound_bsearch_cmp_fn) (PartitionKey, @@ -1442,7 +1456,7 @@ GetPartitionTreeNodeRecurse(Relation rel, int offset) /* First build our own node */ parent = (PartitionTreeNode) palloc0(sizeof(PartitionTreeNodeData)); - parent->pkinfo = NULL; + parent->pkinfo = BuildPartitionKeyExecInfo(rel); parent->pdesc = RelationGetPartitionDesc(rel); parent->relid = RelationGetRelid(rel); parent->offset = offset; @@ -1526,6 +1540,284 @@ get_leaf_partition_count(PartitionTreeNode ptnode) return result; } +/* + * BuildPartitionKeyExecInfo + * Construct a list of PartitionKeyExecInfo records for an open + * relation + * + * PartitionKeyExecInfo stores the information about the partition key + * that's needed when inserting tuples into a partitioned table; especially, + * partition key expression state if there are any expression columns in + * the partition key. Normally we build a PartitionKeyExecInfo for a + * partitioned table just once per command, and then use it for (potentially) + * many tuples. + * + */ +static PartitionKeyExecInfo * +BuildPartitionKeyExecInfo(Relation rel) +{ + PartitionKeyExecInfo *pkinfo; + + pkinfo = (PartitionKeyExecInfo *) palloc0(sizeof(PartitionKeyExecInfo)); + pkinfo->pi_Key = RelationGetPartitionKey(rel); + pkinfo->pi_ExpressionState = NIL; + + return pkinfo; +} + +/* + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for partition key columns + */ +static void +FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pkinfo->pi_Key->partexprs != NIL && pkinfo->pi_ExpressionState == NIL) + { + /* First time through, set up expression evaluation state */ + pkinfo->pi_ExpressionState = (List *) + ExecPrepareExpr((Expr *) pkinfo->pi_Key->partexprs, + estate); + /* Check caller has set up context correctly */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + } + + partexpr_item = list_head(pkinfo->pi_ExpressionState); + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + { + AttrNumber keycol = pkinfo->pi_Key->partattrs[i]; + Datum pkDatum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + pkDatum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + pkDatum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = pkDatum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Recursively finds the "leaf" partition for tuple + * + * Returns -1 if no partition is found and sets *failed_at to the OID of + * the partitioned table whose partition was not found. + */ +int +get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + Relation partRel; + PartitionKeyExecInfo *pkinfo = ptnode->pkinfo; + PartitionTreeNode node; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int i; + int index; + + /* Guard against stack overflow due to overly deep partition tree */ + check_stack_depth(); + + if (ptnode->pdesc->nparts == 0) + { + *failed_at = ptnode->relid; + return -1; + } + + /* Extract partition key from tuple */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + FormPartitionKeyDatum(pkinfo, slot, estate, values, isnull); + + /* Disallow nulls, if range partition key */ + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + if (isnull[i] && pkinfo->pi_Key->strategy == PARTITION_STRATEGY_RANGE) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key contains null"))); + + switch (pkinfo->pi_Key->strategy) + { + case PARTITION_STRATEGY_LIST: + index = list_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values[0], isnull[0]); + break; + + case PARTITION_STRATEGY_RANGE: + index = range_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values); + break; + } + + /* No partition found at this level */ + if (index < 0) + { + *failed_at = ptnode->relid; + return index; + } + + partRel = heap_open(ptnode->pdesc->oids[index], NoLock); + + /* Don't recurse if the index'th partition is a leaf partition. */ + if (partRel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) + { + PartitionTreeNode prev; + + /* + * Index returned above is the array index within pdesc->parts[] of + * the parent rel, however, we want to return the leaf partition index + * across the whole partition tree. Note that some partitions within + * pdesc->parts[] may be partitioned themselves and hence stand for + * the leaf partitions in their partition subtrees. We would need to + * skip past the indexes of leaf partitions of all such partition + * subtrees if they are to left of the above returned index. In fact, + * finding the PartitionTreeNode of the rightmost subtree is enough + * since its offset counts the leaf partitions on its left including + * those of partition subtrees to its left. + */ + prev = node = ptnode->downlink; + if (node && node->index < index) + { + /* + * Find the partition tree node such that its index value is the + * greatest value less than the above returned index. + */ + while (node) + { + if (node->index > index) + { + node = prev; + break; + } + + prev = node; + node = node->next; + } + + if (!node) + node = prev; + Assert (node != NULL); + + index = node->offset + node->num_leaf_parts + + (index - node->index - 1); + } + else + /* + * The easy case where we don't have any partition subtree to the + * left of the index. + */ + index = ptnode->offset + index; + + heap_close(partRel, NoLock); + return index; + } + + heap_close(partRel, NoLock); + + /* + * Need to perform recursion as the selected partition is partitioned + * itself. Locate the PartitionTreeNode corresponding to the partition + * passing it down. + */ + node = ptnode->downlink; + while (node->next != NULL && node->index != index) + node = node->next; + Assert (node != NULL); + + return get_partition_for_tuple(node, slot, estate, failed_at); +} + +/* + * list_partition_for_tuple + * Find the list partition for a tuple + * + * Returns -1 if none found. + */ +static int +list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull) +{ + PartitionListInfo listinfo; + int found; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_LIST); + listinfo = pdesc->boundinfo->bounds.lists; + + if (isnull && listinfo.has_null) + return listinfo.null_index; + else if (!isnull) + { + found = partition_list_values_bsearch(key, + listinfo.values, + listinfo.nvalues, + value); + if (found >= 0) + return listinfo.indexes[found]; + } + + /* Control reaches here if isnull and !listinfo->has_null */ + return -1; +} + +/* + * range_partition_for_tuple + * Search the range partition for a range key ('values') + * + * Returns -1 if none found. + */ +static int +range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, Datum *tuple) +{ + int offset; + PartitionRangeInfo rangeinfo; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_RANGE); + rangeinfo = pdesc->boundinfo->bounds.ranges; + + offset = partition_rbound_bsearch(key, + rangeinfo.bounds, rangeinfo.nbounds, + tuple, partition_rbound_datum_cmp, + true, NULL); + + /* + * Offset returned is such that the bound at offset is found to be + * less or equal with the tuple. That is, the tuple belongs to the + * partition with the rangeinfo.bounds[offset] as the lower bound and + * rangeinfo.bounds[offset+1] as the upper bound, provided the latter + * is indeed marked !lower (that is, it's an upper bound). If it turns + * out that it is a lower bound then the corresponding index will be -1, + * which means no valid partition exists. + */ + return rangeinfo.indexes[offset+1]; +} + /* List partition related support functions */ /* @@ -1787,6 +2079,39 @@ partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg) } /* + * Return whether the passed in range bound <=, =, >= tuple specified in arg + * + * The 2nd argument is void * so that it can be used with + * partition_rbound_bsearch() + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg) +{ + Datum *datums1 = bound->datums, + *datums2 = (Datum *) arg; + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (bound->infinite[i]) + return bound->lower ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + datums1[i], datums2[i])); + if (cmpval != 0) + break; + } + + if (cmpval == 0 && !bound->inclusive) + return bound->lower ? 1 : -1; + + return cmpval; +} + +/* * Are two (consecutive) range bounds equal without distinguishing lower * and upper? */ diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 77d4dcb..91e4d12 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -30,6 +30,7 @@ #include "commands/defrem.h" #include "commands/trigger.h" #include "executor/executor.h" +#include "foreign/fdwapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -161,6 +162,11 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionTreeNode ptnode; /* partition descriptor node tree */ + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; + List *partition_fdw_priv_lists; + int num_partitions; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1403,94 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + List *leaf_part_oids; + ListCell *cell; + int i; + int num_leaf_parts; + ResultRelInfo *leaf_rel_rri; + PlannerInfo *root = makeNode(PlannerInfo); /* mostly dummy */ + Query *parse = makeNode(Query); /* ditto */ + ModifyTable *plan = makeNode(ModifyTable); /* ditto */ + RangeTblEntry *fdw_rte = makeNode(RangeTblEntry); /* ditto */ + List *fdw_private_lists = NIL; + + cstate->ptnode = RelationGetPartitionTreeNode(rel); + leaf_part_oids = get_leaf_partition_oids_v2(cstate->ptnode); + num_leaf_parts = list_length(leaf_part_oids); + + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + /* For use below, iff a partition found to be a foreign table */ + plan->operation = CMD_INSERT; + plan->plans = list_make1(makeNode(Result)); + fdw_rte->rtekind = RTE_RELATION; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + parse->rtable = list_make1(fdw_rte); + root->parse = parse; + + leaf_rel_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_part_oids) + { + Relation leaf_rel; + + leaf_rel = heap_open(lfirst_oid(cell), RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(leaf_rel, CMD_INSERT); + + InitResultRelInfo(leaf_rel_rri, + leaf_rel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_rel_rri, false); + + /* Special dance for foreign tables */ + if (leaf_rel_rri->ri_FdwRoutine) + { + List *fdw_private; + + fdw_rte->relid = RelationGetRelid(leaf_rel); + fdw_private = leaf_rel_rri->ri_FdwRoutine->PlanForeignModify(root, + plan, + 1, + 0); + fdw_private_lists = lappend(fdw_private_lists, fdw_private); + } + + if (!equalTupleDescs(tupDesc, RelationGetDescr(leaf_rel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(leaf_rel), + gettext_noop("could not convert row type")); + + leaf_rel_rri++; + i++; + } + + cstate->partition_fdw_priv_lists = fdw_private_lists; + pfree(fdw_rte); + pfree(plan); + pfree(parse); + pfree(root); + } } else { @@ -1692,6 +1786,8 @@ ClosePipeToProgram(CopyState cstate) static void EndCopy(CopyState cstate) { + int i; + if (cstate->is_program) { ClosePipeToProgram(cstate); @@ -1705,6 +1801,23 @@ EndCopy(CopyState cstate) cstate->filename))); } + /* Close all partitions and indices thereof */ + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + /* XXX - EState not handy here to pass to EndForeignModify() */ + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(NULL, resultRelInfo); + + if (cstate->partition_tupconv_maps[i]) + pfree(cstate->partition_tupconv_maps[i]); + } + MemoryContextDelete(cstate->copycontext); pfree(cstate); } @@ -2255,6 +2368,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2275,7 +2389,8 @@ CopyFrom(CopyState cstate) Assert(cstate->rel); - if (cstate->rel->rd_rel->relkind != RELKIND_RELATION) + if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) { if (cstate->rel->rd_rel->relkind == RELKIND_VIEW) ereport(ERROR, @@ -2383,6 +2498,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2410,6 +2526,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->ptnode != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2431,10 +2548,46 @@ CopyFrom(CopyState cstate) */ ExecBSInsertTriggers(estate, resultRelInfo); + /* Initialize FDW partition insert plans */ + if (cstate->ptnode) + { + int i, + j; + List *fdw_private_lists = cstate->partition_fdw_priv_lists; + ModifyTableState *mtstate = makeNode(ModifyTableState); + ResultRelInfo *leaf_part_rri; + + /* Mostly dummy containing enough state for BeginForeignModify */ + mtstate->ps.state = estate; + mtstate->operation = CMD_INSERT; + + j = 0; + leaf_part_rri = cstate->partitions; + for (i = 0; i < cstate->num_partitions; i++) + { + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + Assert(fdw_private_lists); + fdw_private = list_nth(fdw_private_lists, j++); + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, 0); + } + leaf_part_rri++; + } + } + values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2486,6 +2639,31 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition */ + saved_resultRelInfo = resultRelInfo; + if (cstate->ptnode) + { + int i_leaf_partition; + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + cstate->ptnode, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < cstate->num_partitions); + + resultRelInfo = cstate->partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = cstate->partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2506,7 +2684,16 @@ CopyFrom(CopyState cstate) if (cstate->rel->rd_att->constr || resultRelInfo->ri_PartitionCheck) ExecConstraints(resultRelInfo, slot, estate); - if (useHeapMultiInsert) + if (resultRelInfo->ri_FdwRoutine) + { + resultRelInfo->ri_FdwRoutine->ExecForeignInsert(estate, + resultRelInfo, + slot, + NULL); + /* AFTER ROW INSERT Triggers */ + ExecARInsertTriggers(estate, resultRelInfo, tuple, NIL); + } + else if (useHeapMultiInsert) { /* Add this tuple to the tuple buffer */ if (nBufferedTuples == 0) @@ -2536,7 +2723,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, + tuple, mycid, hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self), @@ -2556,6 +2744,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2569,7 +2763,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 395e116..685b184 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1294,6 +1294,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index ea3f59a..62bf8b7 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2996,3 +3001,41 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionTreeNode ptnode, + TupleTableSlot *slot, EState *estate) +{ + int i_leaf_partition; + Oid failed_at; + + i_leaf_partition = get_partition_for_tuple(ptnode, slot, estate, + &failed_at); + + if (i_leaf_partition < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return i_leaf_partition; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..d0a5306 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,31 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + saved_resultRelInfo = resultRelInfo; + + if (mtstate->mt_partition_tree_root) + { + int i_leaf_partition; + ExprContext *econtext = GetPerTupleExprContext(estate); + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_tree_root, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < mtstate->mt_num_partitions); + + resultRelInfo = mtstate->mt_partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = mtstate->mt_partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +537,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1597,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1688,98 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + int i, + j, + num_leaf_parts; + List *leaf_part_oids; + ListCell *cell; + ResultRelInfo *leaf_rel_rri; + + mtstate->mt_partition_tree_root = RelationGetPartitionTreeNode(rel); + leaf_part_oids = get_leaf_partition_oids_v2(mtstate->mt_partition_tree_root); + num_leaf_parts = list_length(leaf_part_oids); + + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_rel_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_part_oids) + { + Oid ftoid = lfirst_oid(cell); + Relation leaf_rel; + + leaf_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(leaf_rel, CMD_INSERT); + + InitResultRelInfo(leaf_rel_rri, + leaf_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_rel_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_rel_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_rel_rri, false); + + if (leaf_rel_rri->ri_FdwRoutine) + { + ListCell *lc; + List *fdw_private; + int k; + + /* + * There are as many fdw_private's in fdwPrivLists as there + * are FDW partitions, but we must find the intended for the + * this foreign table. + */ + k = 0; + foreach(lc, node->fdwPartitionOids) + { + if (lfirst_oid(lc) == ftoid) + break; + k++; + } + + Assert(k < num_leaf_parts); + fdw_private = (List *) list_nth(node->fdwPrivLists, k); + Assert(fdw_private != NIL); + + leaf_rel_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_rel_rri, + fdw_private, + 0, + eflags); + j++; + } + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(leaf_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(leaf_rel), + gettext_noop("could not convert row type")); + + leaf_rel_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2097,23 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(node->ps.state, + resultRelInfo); + + if (node->mt_partition_tupconv_maps[i]) + pfree(node->mt_partition_tupconv_maps[i]); + } + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 28d0036..470dee7 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -188,6 +188,7 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(withCheckOptionLists); COPY_NODE_FIELD(returningLists); COPY_NODE_FIELD(fdwPrivLists); + COPY_NODE_FIELD(fdwPartitionOids); COPY_BITMAPSET_FIELD(fdwDirectModifyPlans); COPY_NODE_FIELD(rowMarks); COPY_SCALAR_FIELD(epqParam); diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 0d858f5..5c8eced 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -340,6 +340,7 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(withCheckOptionLists); WRITE_NODE_FIELD(returningLists); WRITE_NODE_FIELD(fdwPrivLists); + WRITE_NODE_FIELD(fdwPartitionOids); WRITE_BITMAPSET_FIELD(fdwDirectModifyPlans); WRITE_NODE_FIELD(rowMarks); WRITE_INT_FIELD(epqParam); diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c587d4e..ddd78c7 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -1495,6 +1495,7 @@ _readModifyTable(void) READ_NODE_FIELD(withCheckOptionLists); READ_NODE_FIELD(returningLists); READ_NODE_FIELD(fdwPrivLists); + READ_NODE_FIELD(fdwPartitionOids); READ_BITMAPSET_FIELD(fdwDirectModifyPlans); READ_NODE_FIELD(rowMarks); READ_INT_FIELD(epqParam); diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index ad49674..29f40fe 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -22,6 +22,7 @@ #include "access/stratnum.h" #include "access/sysattr.h" #include "catalog/pg_class.h" +#include "catalog/pg_inherits_fn.h" #include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/extensible.h" @@ -6159,6 +6160,82 @@ make_modifytable(PlannerInfo *root, node->fdwPrivLists = fdw_private_list; node->fdwDirectModifyPlans = direct_modify_plans; + /* Collect insert plans for all FDW-managed partitions */ + if (node->operation == CMD_INSERT) + { + RangeTblEntry *rte, + **saved_simple_rte_array; + List *partition_oids, + *fdw_partition_oids; + + Assert(list_length(resultRelations) == 1); + rte = rt_fetch(linitial_int(resultRelations), root->parse->rtable); + Assert(rte->rtekind == RTE_RELATION); + + if (rte->relkind != RELKIND_PARTITIONED_TABLE) + return node; + + partition_oids = find_all_inheritors(rte->relid, NoLock, NULL); + + /* Discard any previous content which is useless anyway */ + fdw_private_list = NIL; + fdw_partition_oids = NIL; + + /* + * To force the FDW driver fetch the intended RTE, we need to temporarily + * switch root->simple_rte_array to the one holding only that RTE at a + * designated index, for every foreign table. + */ + saved_simple_rte_array = root->simple_rte_array; + root->simple_rte_array = (RangeTblEntry **) + palloc0(2 * sizeof(RangeTblEntry *)); + foreach(lc, partition_oids) + { + Oid myoid = lfirst_oid(lc); + FdwRoutine *fdwroutine; + List *fdw_private; + + /* + * We are only interested in foreign tables. Note that this will + * also eliminate any partitioned tables since foreign tables can + * only ever be leaf partitions. + */ + if (!oid_is_foreign_table(myoid)) + continue; + + fdw_partition_oids = lappend_oid(fdw_partition_oids, myoid); + + fdwroutine = GetFdwRoutineByRelId(myoid); + if (fdwroutine && fdwroutine->PlanForeignModify) + { + RangeTblEntry *fdw_rte; + + fdw_rte = copyObject(rte); + fdw_rte->relid = myoid; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + + /* + * Assumes PlanForeignModify() uses planner_rt_fetch(), also, + * see the above comment. + */ + root->simple_rte_array[1] = fdw_rte; + + fdw_private = fdwroutine->PlanForeignModify(root, node, 1, 0); + pfree(fdw_rte); + } + else + fdw_private = NIL; + + fdw_private_list = lappend(fdw_private_list, fdw_private); + } + + pfree(root->simple_rte_array); + root->simple_rte_array = saved_simple_rte_array; + + node->fdwPrivLists = fdw_private_list; + node->fdwPartitionOids = fdw_partition_oids; + } + return node; } diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index a2cbf14..e85ca0a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1715,3 +1715,16 @@ has_row_triggers(PlannerInfo *root, Index rti, CmdType event) heap_close(relation, NoLock); return result; } + +bool +oid_is_foreign_table(Oid relid) +{ + Relation rel; + char relkind; + + rel = heap_open(relid, NoLock); + relkind = rel->rd_rel->relkind; + heap_close(rel, NoLock); + + return relkind == RELKIND_FOREIGN_TABLE; +} diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index ab6d3a8..09727cf 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -56,4 +58,9 @@ extern List *RelationGetPartitionQual(Relation rel, bool recurse); /* For tuple routing */ extern PartitionTreeNode RelationGetPartitionTreeNode(Relation rel); extern List *get_leaf_partition_oids_v2(PartitionTreeNode ptnode); + +extern int get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..c62946f 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..ce01008 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,15 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionTreeNodeData *mt_partition_tree_root; + /* Partition descriptor node tree */ + ResultRelInfo *mt_partitions; /* Per leaf partition target + * relations */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per leaf partition + * tuple conversion map */ + int mt_num_partitions; /* Number of leaf partition target + * relations in the above array */ } ModifyTableState; /* ---------------- diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index e2fbc7d..d82222c 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -189,6 +189,7 @@ typedef struct ModifyTable List *returningLists; /* per-target-table RETURNING tlists */ List *fdwPrivLists; /* per-target-table FDW private data lists */ Bitmapset *fdwDirectModifyPlans; /* indices of FDW DM plans */ + List *fdwPartitionOids; /* OIDs of FDW-managed partition */ List *rowMarks; /* PlanRowMarks (non-locking only) */ int epqParam; /* ID of Param for EvalPlanQual re-eval */ OnConflictAction onConflictAction; /* ON CONFLICT action */ diff --git a/src/include/optimizer/plancat.h b/src/include/optimizer/plancat.h index 125274e..fac606c 100644 --- a/src/include/optimizer/plancat.h +++ b/src/include/optimizer/plancat.h @@ -56,5 +56,6 @@ extern Selectivity join_selectivity(PlannerInfo *root, SpecialJoinInfo *sjinfo); extern bool has_row_triggers(PlannerInfo *root, Index rti, CmdType event); +extern bool oid_is_foreign_table(Oid relid); #endif /* PLANCAT_H */ diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 5d58d23..619336b 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -222,6 +222,62 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- fail (no list partition defined which accepts nulls) +insert into list_parted (b) values (1); +ERROR: no partition of relation "list_parted" found for row +DETAIL: Failing row contains (null, 1). +create table part_nulls partition of list_parted for values in (null); +-- ok +insert into list_parted (b) values (1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_nulls | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(7 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects @@ -230,9 +286,10 @@ drop cascades to table part_a_10_a_20 drop cascades to table part_b_1_b_10 drop cascades to table part_b_10_b_20 drop table list_parted cascade; -NOTICE: drop cascades to 5 other objects +NOTICE: drop cascades to 6 other objects DETAIL: drop cascades to table part_aa_bb drop cascades to table part_cc_dd drop cascades to table part_ee_ff drop cascades to table part_ee_ff_1_10 drop cascades to table part_ee_ff_10_20 +drop cascades to table part_nulls diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index 5aedeef..210fab4 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -137,6 +137,34 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- fail (no list partition defined which accepts nulls) +insert into list_parted (b) values (1); +create table part_nulls partition of list_parted for values in (null); +-- ok +insert into list_parted (b) values (1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------B737FB31608A9E2BB1F071B4 Content-Type: text/x-diff; name="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-12.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-12"; filename*1=".patch" ^ permalink raw reply [nested|flat] 13+ messages in thread
* [PATCH 7/8] Tuple routing for partitioned tables. @ 2016-07-27 07:59 amit <[email protected]> 0 siblings, 0 replies; 13+ messages in thread From: amit @ 2016-07-27 07:59 UTC (permalink / raw) Both COPY FROM and INSERT. --- src/backend/catalog/partition.c | 327 ++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 205 +++++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 47 +++++- src/backend/executor/nodeModifyTable.c | 142 +++++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 1 + src/backend/optimizer/plan/createplan.c | 77 +++++++ src/backend/optimizer/util/plancat.c | 13 ++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 7 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 10 + src/include/nodes/plannodes.h | 1 + src/include/optimizer/plancat.h | 1 + src/test/regress/expected/insert.out | 59 ++++++- src/test/regress/sql/insert.sql | 28 +++ 18 files changed, 926 insertions(+), 9 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index 69c3e52..9cba603 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -181,6 +181,18 @@ static List *generate_partition_qual(Relation rel, bool recurse); static PartitionTreeNode GetPartitionTreeNodeRecurse(Relation rel, int offset); static int get_leaf_partition_count(PartitionTreeNode ptnode); +/* Support get_partition_for_tuple() */ +static PartitionKeyExecInfo *BuildPartitionKeyExecInfo(Relation rel); +static void FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); +static int list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull); +static int range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum *tuple); + /* List partition related support functions */ static bool equal_list_info(PartitionKey key, PartitionListInfo *l1, PartitionListInfo *l2); @@ -195,6 +207,8 @@ static PartitionRangeBound *copy_range_bound(PartitionKey key, PartitionRangeBou static bool equal_range_info(PartitionKey key, PartitionRangeInfo *r1, PartitionRangeInfo *r2); static int32 partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg); +static int32 partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg); static bool partition_rbound_eq(PartitionKey key, PartitionRangeBound *b1, PartitionRangeBound *b2); typedef int32 (*partition_rbound_bsearch_cmp_fn) (PartitionKey, @@ -1449,7 +1463,7 @@ GetPartitionTreeNodeRecurse(Relation rel, int offset) /* First build our own node */ parent = (PartitionTreeNode) palloc0(sizeof(PartitionTreeNodeData)); - parent->pkinfo = NULL; + parent->pkinfo = BuildPartitionKeyExecInfo(rel); parent->pdesc = RelationGetPartitionDesc(rel); parent->relid = RelationGetRelid(rel); parent->offset = offset; @@ -1533,6 +1547,284 @@ get_leaf_partition_count(PartitionTreeNode ptnode) return result; } +/* + * BuildPartitionKeyExecInfo + * Construct a list of PartitionKeyExecInfo records for an open + * relation + * + * PartitionKeyExecInfo stores the information about the partition key + * that's needed when inserting tuples into a partitioned table; especially, + * partition key expression state if there are any expression columns in + * the partition key. Normally we build a PartitionKeyExecInfo for a + * partitioned table just once per command, and then use it for (potentially) + * many tuples. + * + */ +static PartitionKeyExecInfo * +BuildPartitionKeyExecInfo(Relation rel) +{ + PartitionKeyExecInfo *pkinfo; + + pkinfo = (PartitionKeyExecInfo *) palloc0(sizeof(PartitionKeyExecInfo)); + pkinfo->pi_Key = RelationGetPartitionKey(rel); + pkinfo->pi_ExpressionState = NIL; + + return pkinfo; +} + +/* + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for partition key columns + */ +static void +FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pkinfo->pi_Key->partexprs != NIL && pkinfo->pi_ExpressionState == NIL) + { + /* First time through, set up expression evaluation state */ + pkinfo->pi_ExpressionState = (List *) + ExecPrepareExpr((Expr *) pkinfo->pi_Key->partexprs, + estate); + /* Check caller has set up context correctly */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + } + + partexpr_item = list_head(pkinfo->pi_ExpressionState); + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + { + AttrNumber keycol = pkinfo->pi_Key->partattrs[i]; + Datum pkDatum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + pkDatum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + pkDatum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = pkDatum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Recursively finds the "leaf" partition for tuple + * + * Returns -1 if no partition is found and sets *failed_at to the OID of + * the partitioned table whose partition was not found. + */ +int +get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + Relation partRel; + PartitionKeyExecInfo *pkinfo = ptnode->pkinfo; + PartitionTreeNode node; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int i; + int index; + + /* Guard against stack overflow due to overly deep partition tree */ + check_stack_depth(); + + if (ptnode->pdesc->nparts == 0) + { + *failed_at = ptnode->relid; + return -1; + } + + /* Extract partition key from tuple */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + FormPartitionKeyDatum(pkinfo, slot, estate, values, isnull); + + /* Disallow nulls, if range partition key */ + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + if (isnull[i] && pkinfo->pi_Key->strategy == PARTITION_STRATEGY_RANGE) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key contains null"))); + + switch (pkinfo->pi_Key->strategy) + { + case PARTITION_STRATEGY_LIST: + index = list_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values[0], isnull[0]); + break; + + case PARTITION_STRATEGY_RANGE: + index = range_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values); + break; + } + + /* No partition found at this level */ + if (index < 0) + { + *failed_at = ptnode->relid; + return index; + } + + partRel = heap_open(ptnode->pdesc->oids[index], NoLock); + + /* Don't recurse if the index'th partition is a leaf partition. */ + if (partRel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) + { + PartitionTreeNode prev; + + /* + * Index returned above is the array index within pdesc->parts[] of + * the parent rel, however, we want to return the leaf partition index + * across the whole partition tree. Note that some partitions within + * pdesc->parts[] may be partitioned themselves and hence stand for + * the leaf partitions in their partition subtrees. We would need to + * skip past the indexes of leaf partitions of all such partition + * subtrees if they are to left of the above returned index. In fact, + * finding the PartitionTreeNode of the rightmost subtree is enough + * since its offset counts the leaf partitions on its left including + * those of partition subtrees to its left. + */ + prev = node = ptnode->downlink; + if (node && node->index < index) + { + /* + * Find the partition tree node such that its index value is the + * greatest value less than the above returned index. + */ + while (node) + { + if (node->index > index) + { + node = prev; + break; + } + + prev = node; + node = node->next; + } + + if (!node) + node = prev; + Assert (node != NULL); + + index = node->offset + node->num_leaf_parts + + (index - node->index - 1); + } + else + /* + * The easy case where we don't have any partition subtree to the + * left of the index. + */ + index = ptnode->offset + index; + + heap_close(partRel, NoLock); + return index; + } + + heap_close(partRel, NoLock); + + /* + * Need to perform recursion as the selected partition is partitioned + * itself. Locate the PartitionTreeNode corresponding to the partition + * passing it down. + */ + node = ptnode->downlink; + while (node->next != NULL && node->index != index) + node = node->next; + Assert (node != NULL); + + return get_partition_for_tuple(node, slot, estate, failed_at); +} + +/* + * list_partition_for_tuple + * Find the list partition for a tuple + * + * Returns -1 if none found. + */ +static int +list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull) +{ + PartitionListInfo listinfo; + int found; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_LIST); + listinfo = pdesc->boundinfo->bounds.lists; + + if (isnull && listinfo.has_null) + return listinfo.null_index; + else if (!isnull) + { + found = partition_list_values_bsearch(key, + listinfo.values, + listinfo.nvalues, + value); + if (found >= 0) + return listinfo.indexes[found]; + } + + /* Control reaches here if isnull and !listinfo->has_null */ + return -1; +} + +/* + * range_partition_for_tuple + * Search the range partition for a range key ('values') + * + * Returns -1 if none found. + */ +static int +range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, Datum *tuple) +{ + int offset; + PartitionRangeInfo rangeinfo; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_RANGE); + rangeinfo = pdesc->boundinfo->bounds.ranges; + + offset = partition_rbound_bsearch(key, + rangeinfo.bounds, rangeinfo.nbounds, + tuple, partition_rbound_datum_cmp, + true, NULL); + + /* + * Offset returned is such that the bound at offset is found to be + * less or equal with the tuple. That is, the tuple belongs to the + * partition with the rangeinfo.bounds[offset] as the lower bound and + * rangeinfo.bounds[offset+1] as the upper bound, provided the latter + * is indeed marked !lower (that is, it's an upper bound). If it turns + * out that it is a lower bound then the corresponding index will be -1, + * which means no valid partition exists. + */ + return rangeinfo.indexes[offset+1]; +} + /* List partition related support functions */ /* @@ -1798,6 +2090,39 @@ partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg) } /* + * Return whether the passed in range bound <=, =, >= tuple specified in arg + * + * The 2nd argument is void * so that it can be used with + * partition_rbound_bsearch() + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg) +{ + Datum *datums1 = bound->datums, + *datums2 = (Datum *) arg; + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (bound->infinite[i]) + return bound->lower ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + datums1[i], datums2[i])); + if (cmpval != 0) + break; + } + + if (cmpval == 0 && !bound->inclusive) + return bound->lower ? 1 : -1; + + return cmpval; +} + +/* * Are two (consecutive) range bounds equal without distinguishing lower * and upper? */ diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 77d4dcb..91e4d12 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -30,6 +30,7 @@ #include "commands/defrem.h" #include "commands/trigger.h" #include "executor/executor.h" +#include "foreign/fdwapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -161,6 +162,11 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionTreeNode ptnode; /* partition descriptor node tree */ + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; + List *partition_fdw_priv_lists; + int num_partitions; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1403,94 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + List *leaf_part_oids; + ListCell *cell; + int i; + int num_leaf_parts; + ResultRelInfo *leaf_rel_rri; + PlannerInfo *root = makeNode(PlannerInfo); /* mostly dummy */ + Query *parse = makeNode(Query); /* ditto */ + ModifyTable *plan = makeNode(ModifyTable); /* ditto */ + RangeTblEntry *fdw_rte = makeNode(RangeTblEntry); /* ditto */ + List *fdw_private_lists = NIL; + + cstate->ptnode = RelationGetPartitionTreeNode(rel); + leaf_part_oids = get_leaf_partition_oids_v2(cstate->ptnode); + num_leaf_parts = list_length(leaf_part_oids); + + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + /* For use below, iff a partition found to be a foreign table */ + plan->operation = CMD_INSERT; + plan->plans = list_make1(makeNode(Result)); + fdw_rte->rtekind = RTE_RELATION; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + parse->rtable = list_make1(fdw_rte); + root->parse = parse; + + leaf_rel_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_part_oids) + { + Relation leaf_rel; + + leaf_rel = heap_open(lfirst_oid(cell), RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(leaf_rel, CMD_INSERT); + + InitResultRelInfo(leaf_rel_rri, + leaf_rel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_rel_rri, false); + + /* Special dance for foreign tables */ + if (leaf_rel_rri->ri_FdwRoutine) + { + List *fdw_private; + + fdw_rte->relid = RelationGetRelid(leaf_rel); + fdw_private = leaf_rel_rri->ri_FdwRoutine->PlanForeignModify(root, + plan, + 1, + 0); + fdw_private_lists = lappend(fdw_private_lists, fdw_private); + } + + if (!equalTupleDescs(tupDesc, RelationGetDescr(leaf_rel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(leaf_rel), + gettext_noop("could not convert row type")); + + leaf_rel_rri++; + i++; + } + + cstate->partition_fdw_priv_lists = fdw_private_lists; + pfree(fdw_rte); + pfree(plan); + pfree(parse); + pfree(root); + } } else { @@ -1692,6 +1786,8 @@ ClosePipeToProgram(CopyState cstate) static void EndCopy(CopyState cstate) { + int i; + if (cstate->is_program) { ClosePipeToProgram(cstate); @@ -1705,6 +1801,23 @@ EndCopy(CopyState cstate) cstate->filename))); } + /* Close all partitions and indices thereof */ + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + /* XXX - EState not handy here to pass to EndForeignModify() */ + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(NULL, resultRelInfo); + + if (cstate->partition_tupconv_maps[i]) + pfree(cstate->partition_tupconv_maps[i]); + } + MemoryContextDelete(cstate->copycontext); pfree(cstate); } @@ -2255,6 +2368,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2275,7 +2389,8 @@ CopyFrom(CopyState cstate) Assert(cstate->rel); - if (cstate->rel->rd_rel->relkind != RELKIND_RELATION) + if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) { if (cstate->rel->rd_rel->relkind == RELKIND_VIEW) ereport(ERROR, @@ -2383,6 +2498,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2410,6 +2526,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->ptnode != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2431,10 +2548,46 @@ CopyFrom(CopyState cstate) */ ExecBSInsertTriggers(estate, resultRelInfo); + /* Initialize FDW partition insert plans */ + if (cstate->ptnode) + { + int i, + j; + List *fdw_private_lists = cstate->partition_fdw_priv_lists; + ModifyTableState *mtstate = makeNode(ModifyTableState); + ResultRelInfo *leaf_part_rri; + + /* Mostly dummy containing enough state for BeginForeignModify */ + mtstate->ps.state = estate; + mtstate->operation = CMD_INSERT; + + j = 0; + leaf_part_rri = cstate->partitions; + for (i = 0; i < cstate->num_partitions; i++) + { + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + Assert(fdw_private_lists); + fdw_private = list_nth(fdw_private_lists, j++); + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, 0); + } + leaf_part_rri++; + } + } + values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2486,6 +2639,31 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition */ + saved_resultRelInfo = resultRelInfo; + if (cstate->ptnode) + { + int i_leaf_partition; + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + cstate->ptnode, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < cstate->num_partitions); + + resultRelInfo = cstate->partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = cstate->partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2506,7 +2684,16 @@ CopyFrom(CopyState cstate) if (cstate->rel->rd_att->constr || resultRelInfo->ri_PartitionCheck) ExecConstraints(resultRelInfo, slot, estate); - if (useHeapMultiInsert) + if (resultRelInfo->ri_FdwRoutine) + { + resultRelInfo->ri_FdwRoutine->ExecForeignInsert(estate, + resultRelInfo, + slot, + NULL); + /* AFTER ROW INSERT Triggers */ + ExecARInsertTriggers(estate, resultRelInfo, tuple, NIL); + } + else if (useHeapMultiInsert) { /* Add this tuple to the tuple buffer */ if (nBufferedTuples == 0) @@ -2536,7 +2723,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, + tuple, mycid, hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, &(tuple->t_self), @@ -2556,6 +2744,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2569,7 +2763,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index de19e9c..88b9d1f 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1292,6 +1292,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index ea3f59a..62bf8b7 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2996,3 +3001,41 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionTreeNode ptnode, + TupleTableSlot *slot, EState *estate) +{ + int i_leaf_partition; + Oid failed_at; + + i_leaf_partition = get_partition_for_tuple(ptnode, slot, estate, + &failed_at); + + if (i_leaf_partition < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return i_leaf_partition; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..d0a5306 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,31 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + saved_resultRelInfo = resultRelInfo; + + if (mtstate->mt_partition_tree_root) + { + int i_leaf_partition; + ExprContext *econtext = GetPerTupleExprContext(estate); + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_tree_root, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < mtstate->mt_num_partitions); + + resultRelInfo = mtstate->mt_partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = mtstate->mt_partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +537,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1597,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1688,98 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + int i, + j, + num_leaf_parts; + List *leaf_part_oids; + ListCell *cell; + ResultRelInfo *leaf_rel_rri; + + mtstate->mt_partition_tree_root = RelationGetPartitionTreeNode(rel); + leaf_part_oids = get_leaf_partition_oids_v2(mtstate->mt_partition_tree_root); + num_leaf_parts = list_length(leaf_part_oids); + + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_rel_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_part_oids) + { + Oid ftoid = lfirst_oid(cell); + Relation leaf_rel; + + leaf_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(leaf_rel, CMD_INSERT); + + InitResultRelInfo(leaf_rel_rri, + leaf_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_rel_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_rel_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_rel_rri, false); + + if (leaf_rel_rri->ri_FdwRoutine) + { + ListCell *lc; + List *fdw_private; + int k; + + /* + * There are as many fdw_private's in fdwPrivLists as there + * are FDW partitions, but we must find the intended for the + * this foreign table. + */ + k = 0; + foreach(lc, node->fdwPartitionOids) + { + if (lfirst_oid(lc) == ftoid) + break; + k++; + } + + Assert(k < num_leaf_parts); + fdw_private = (List *) list_nth(node->fdwPrivLists, k); + Assert(fdw_private != NIL); + + leaf_rel_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_rel_rri, + fdw_private, + 0, + eflags); + j++; + } + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(leaf_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(leaf_rel), + gettext_noop("could not convert row type")); + + leaf_rel_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2097,23 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(node->ps.state, + resultRelInfo); + + if (node->mt_partition_tupconv_maps[i]) + pfree(node->mt_partition_tupconv_maps[i]); + } + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index c0d48d0..ca48547 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -188,6 +188,7 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(withCheckOptionLists); COPY_NODE_FIELD(returningLists); COPY_NODE_FIELD(fdwPrivLists); + COPY_NODE_FIELD(fdwPartitionOids); COPY_BITMAPSET_FIELD(fdwDirectModifyPlans); COPY_NODE_FIELD(rowMarks); COPY_SCALAR_FIELD(epqParam); diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index d5eb0cc..7a219e5 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -340,6 +340,7 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(withCheckOptionLists); WRITE_NODE_FIELD(returningLists); WRITE_NODE_FIELD(fdwPrivLists); + WRITE_NODE_FIELD(fdwPartitionOids); WRITE_BITMAPSET_FIELD(fdwDirectModifyPlans); WRITE_NODE_FIELD(rowMarks); WRITE_INT_FIELD(epqParam); diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index 655aa1c..d047160 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -1495,6 +1495,7 @@ _readModifyTable(void) READ_NODE_FIELD(withCheckOptionLists); READ_NODE_FIELD(returningLists); READ_NODE_FIELD(fdwPrivLists); + READ_NODE_FIELD(fdwPartitionOids); READ_BITMAPSET_FIELD(fdwDirectModifyPlans); READ_NODE_FIELD(rowMarks); READ_INT_FIELD(epqParam); diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index ad49674..29f40fe 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -22,6 +22,7 @@ #include "access/stratnum.h" #include "access/sysattr.h" #include "catalog/pg_class.h" +#include "catalog/pg_inherits_fn.h" #include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/extensible.h" @@ -6159,6 +6160,82 @@ make_modifytable(PlannerInfo *root, node->fdwPrivLists = fdw_private_list; node->fdwDirectModifyPlans = direct_modify_plans; + /* Collect insert plans for all FDW-managed partitions */ + if (node->operation == CMD_INSERT) + { + RangeTblEntry *rte, + **saved_simple_rte_array; + List *partition_oids, + *fdw_partition_oids; + + Assert(list_length(resultRelations) == 1); + rte = rt_fetch(linitial_int(resultRelations), root->parse->rtable); + Assert(rte->rtekind == RTE_RELATION); + + if (rte->relkind != RELKIND_PARTITIONED_TABLE) + return node; + + partition_oids = find_all_inheritors(rte->relid, NoLock, NULL); + + /* Discard any previous content which is useless anyway */ + fdw_private_list = NIL; + fdw_partition_oids = NIL; + + /* + * To force the FDW driver fetch the intended RTE, we need to temporarily + * switch root->simple_rte_array to the one holding only that RTE at a + * designated index, for every foreign table. + */ + saved_simple_rte_array = root->simple_rte_array; + root->simple_rte_array = (RangeTblEntry **) + palloc0(2 * sizeof(RangeTblEntry *)); + foreach(lc, partition_oids) + { + Oid myoid = lfirst_oid(lc); + FdwRoutine *fdwroutine; + List *fdw_private; + + /* + * We are only interested in foreign tables. Note that this will + * also eliminate any partitioned tables since foreign tables can + * only ever be leaf partitions. + */ + if (!oid_is_foreign_table(myoid)) + continue; + + fdw_partition_oids = lappend_oid(fdw_partition_oids, myoid); + + fdwroutine = GetFdwRoutineByRelId(myoid); + if (fdwroutine && fdwroutine->PlanForeignModify) + { + RangeTblEntry *fdw_rte; + + fdw_rte = copyObject(rte); + fdw_rte->relid = myoid; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + + /* + * Assumes PlanForeignModify() uses planner_rt_fetch(), also, + * see the above comment. + */ + root->simple_rte_array[1] = fdw_rte; + + fdw_private = fdwroutine->PlanForeignModify(root, node, 1, 0); + pfree(fdw_rte); + } + else + fdw_private = NIL; + + fdw_private_list = lappend(fdw_private_list, fdw_private); + } + + pfree(root->simple_rte_array); + root->simple_rte_array = saved_simple_rte_array; + + node->fdwPrivLists = fdw_private_list; + node->fdwPartitionOids = fdw_partition_oids; + } + return node; } diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index a2cbf14..e85ca0a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1715,3 +1715,16 @@ has_row_triggers(PlannerInfo *root, Index rti, CmdType event) heap_close(relation, NoLock); return result; } + +bool +oid_is_foreign_table(Oid relid) +{ + Relation rel; + char relkind; + + rel = heap_open(relid, NoLock); + relkind = rel->rd_rel->relkind; + heap_close(rel, NoLock); + + return relkind == RELKIND_FOREIGN_TABLE; +} diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index ab6d3a8..09727cf 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -56,4 +58,9 @@ extern List *RelationGetPartitionQual(Relation rel, bool recurse); /* For tuple routing */ extern PartitionTreeNode RelationGetPartitionTreeNode(Relation rel); extern List *get_leaf_partition_oids_v2(PartitionTreeNode ptnode); + +extern int get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..c62946f 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..ce01008 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,15 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionTreeNodeData *mt_partition_tree_root; + /* Partition descriptor node tree */ + ResultRelInfo *mt_partitions; /* Per leaf partition target + * relations */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per leaf partition + * tuple conversion map */ + int mt_num_partitions; /* Number of leaf partition target + * relations in the above array */ } ModifyTableState; /* ---------------- diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index e2fbc7d..d82222c 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -189,6 +189,7 @@ typedef struct ModifyTable List *returningLists; /* per-target-table RETURNING tlists */ List *fdwPrivLists; /* per-target-table FDW private data lists */ Bitmapset *fdwDirectModifyPlans; /* indices of FDW DM plans */ + List *fdwPartitionOids; /* OIDs of FDW-managed partition */ List *rowMarks; /* PlanRowMarks (non-locking only) */ int epqParam; /* ID of Param for EvalPlanQual re-eval */ OnConflictAction onConflictAction; /* ON CONFLICT action */ diff --git a/src/include/optimizer/plancat.h b/src/include/optimizer/plancat.h index 125274e..fac606c 100644 --- a/src/include/optimizer/plancat.h +++ b/src/include/optimizer/plancat.h @@ -56,5 +56,6 @@ extern Selectivity join_selectivity(PlannerInfo *root, SpecialJoinInfo *sjinfo); extern bool has_row_triggers(PlannerInfo *root, Index rti, CmdType event); +extern bool oid_is_foreign_table(Oid relid); #endif /* PLANCAT_H */ diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index ebc0ba5..3d97742 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -222,6 +222,62 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- fail (no list partition defined which accepts nulls) +insert into list_parted (b) values (1); +ERROR: no partition of relation "list_parted" found for row +DETAIL: Failing row contains (null, 1). +create table part_nulls partition of list_parted for values in (null); +-- ok +insert into list_parted (b) values (1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_nulls | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(7 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects @@ -230,9 +286,10 @@ drop cascades to table part_a_10_a_20 drop cascades to table part_b_1_b_10 drop cascades to table part_b_10_b_20 drop table list_parted cascade; -NOTICE: drop cascades to 5 other objects +NOTICE: drop cascades to 6 other objects DETAIL: drop cascades to table part_aa_bb drop cascades to table part_cc_dd drop cascades to table part_ee_ff drop cascades to table part_ee_ff_1_10 drop cascades to table part_ee_ff_10_20 +drop cascades to table part_nulls diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index 4bf042e..d1b5a09 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -137,6 +137,34 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- fail (no list partition defined which accepts nulls) +insert into list_parted (b) values (1); +create table part_nulls partition of list_parted for values in (null); +-- ok +insert into list_parted (b) values (1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------294A99728041A5608B577083 Content-Type: text/x-diff; name="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-11.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-11"; filename*1=".patch" ^ permalink raw reply [nested|flat] 13+ messages in thread
* [PATCH 7/8] Tuple routing for partitioned tables. @ 2016-07-27 07:59 amit <[email protected]> 0 siblings, 0 replies; 13+ messages in thread From: amit @ 2016-07-27 07:59 UTC (permalink / raw) Both COPY FROM and INSERT. --- src/backend/catalog/partition.c | 325 ++++++++++++++++++++++++++++++- src/backend/commands/copy.c | 203 +++++++++++++++++++- src/backend/commands/tablecmds.c | 1 + src/backend/executor/execMain.c | 47 +++++- src/backend/executor/nodeModifyTable.c | 142 ++++++++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/outfuncs.c | 1 + src/backend/nodes/readfuncs.c | 1 + src/backend/optimizer/plan/createplan.c | 77 ++++++++ src/backend/optimizer/util/plancat.c | 13 ++ src/backend/parser/analyze.c | 8 + src/include/catalog/partition.h | 7 + src/include/executor/executor.h | 6 + src/include/nodes/execnodes.h | 10 + src/include/nodes/plannodes.h | 1 + src/include/optimizer/plancat.h | 1 + src/test/regress/expected/insert.out | 52 +++++ src/test/regress/sql/insert.sql | 25 +++ 18 files changed, 914 insertions(+), 7 deletions(-) diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index 6301d91..e3a0848 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -181,6 +181,18 @@ static List *generate_partition_qual(Relation rel, bool recurse); static PartitionTreeNode GetPartitionTreeNodeRecurse(Relation rel, int offset); static int get_leaf_partition_count(PartitionTreeNode ptnode); +/* Support get_partition_for_tuple() */ +static PartitionKeyExecInfo *BuildPartitionKeyExecInfo(Relation rel); +static void FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull); +static int list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull); +static int range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum *tuple); + /* List partition related support functions */ static bool equal_list_info(PartitionKey key, PartitionListInfo *l1, PartitionListInfo *l2); @@ -194,6 +206,8 @@ static PartitionRangeBound *copy_range_bound(PartitionKey key, PartitionRangeBou static bool equal_range_info(PartitionKey key, PartitionRangeInfo *r1, PartitionRangeInfo *r2); static int32 partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg); +static int32 partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg); static bool partition_rbound_eq(PartitionKey key, PartitionRangeBound *b1, PartitionRangeBound *b2); typedef int32 (*partition_rbound_bsearch_cmp_fn) (PartitionKey, @@ -1437,7 +1451,7 @@ GetPartitionTreeNodeRecurse(Relation rel, int offset) /* First build our own node */ parent = (PartitionTreeNode) palloc0(sizeof(PartitionTreeNodeData)); - parent->pkinfo = NULL; + parent->pkinfo = BuildPartitionKeyExecInfo(rel); parent->pdesc = RelationGetPartitionDesc(rel); parent->relid = RelationGetRelid(rel); parent->offset = offset; @@ -1521,6 +1535,281 @@ get_leaf_partition_count(PartitionTreeNode ptnode) return result; } +/* + * BuildPartitionKeyExecInfo + * Construct a list of PartitionKeyExecInfo records for an open + * relation + * + * PartitionKeyExecInfo stores the information about the partition key + * that's needed when inserting tuples into a partitioned table; especially, + * partition key expression state if there are any expression columns in + * the partition key. Normally we build a PartitionKeyExecInfo for a + * partitioned table just once per command, and then use it for (potentially) + * many tuples. + * + */ +static PartitionKeyExecInfo * +BuildPartitionKeyExecInfo(Relation rel) +{ + PartitionKeyExecInfo *pkinfo; + + pkinfo = (PartitionKeyExecInfo *) palloc0(sizeof(PartitionKeyExecInfo)); + pkinfo->pi_Key = RelationGetPartitionKey(rel); + pkinfo->pi_ExpressionState = NIL; + + return pkinfo; +} + +/* + * FormPartitionKeyDatum + * Construct values[] and isnull[] arrays for partition key columns + */ +static void +FormPartitionKeyDatum(PartitionKeyExecInfo *pkinfo, + TupleTableSlot *slot, + EState *estate, + Datum *values, + bool *isnull) +{ + ListCell *partexpr_item; + int i; + + if (pkinfo->pi_Key->partexprs != NIL && pkinfo->pi_ExpressionState == NIL) + { + /* First time through, set up expression evaluation state */ + pkinfo->pi_ExpressionState = (List *) + ExecPrepareExpr((Expr *) pkinfo->pi_Key->partexprs, + estate); + /* Check caller has set up context correctly */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + } + + partexpr_item = list_head(pkinfo->pi_ExpressionState); + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + { + AttrNumber keycol = pkinfo->pi_Key->partattrs[i]; + Datum pkDatum; + bool isNull; + + if (keycol != 0) + { + /* Plain column; get the value directly from the heap tuple */ + pkDatum = slot_getattr(slot, keycol, &isNull); + } + else + { + /* Expression; need to evaluate it */ + if (partexpr_item == NULL) + elog(ERROR, "wrong number of partition key expressions"); + pkDatum = ExecEvalExprSwitchContext((ExprState *) lfirst(partexpr_item), + GetPerTupleExprContext(estate), + &isNull, + NULL); + partexpr_item = lnext(partexpr_item); + } + values[i] = pkDatum; + isnull[i] = isNull; + } + + if (partexpr_item != NULL) + elog(ERROR, "wrong number of partition key expressions"); +} + +/* + * get_partition_for_tuple + * Recursively finds the "leaf" partition for tuple + * + * Returns -1 if no partition is found and sets *failed_at to the OID of + * the partitioned table whose partition was not found. + */ +int +get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at) +{ + Relation partRel; + PartitionKeyExecInfo *pkinfo = ptnode->pkinfo; + PartitionTreeNode node; + Datum values[PARTITION_MAX_KEYS]; + bool isnull[PARTITION_MAX_KEYS]; + int i; + int index; + + /* Guard against stack overflow due to overly deep partition tree */ + check_stack_depth(); + + if (ptnode->pdesc->nparts == 0) + { + *failed_at = ptnode->relid; + return -1; + } + + /* Extract partition key from tuple */ + Assert(GetPerTupleExprContext(estate)->ecxt_scantuple == slot); + FormPartitionKeyDatum(pkinfo, slot, estate, values, isnull); + + /* Disallow nulls, if range partition key */ + for (i = 0; i < pkinfo->pi_Key->partnatts; i++) + if (isnull[i] && pkinfo->pi_Key->strategy == PARTITION_STRATEGY_RANGE) + ereport(ERROR, + (errcode(ERRCODE_NULL_VALUE_NOT_ALLOWED), + errmsg("range partition key contains null"))); + + switch (pkinfo->pi_Key->strategy) + { + case PARTITION_STRATEGY_LIST: + index = list_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values[0], isnull[0]); + break; + + case PARTITION_STRATEGY_RANGE: + index = range_partition_for_tuple(pkinfo->pi_Key, ptnode->pdesc, + values); + break; + } + + /* No partition found at this level */ + if (index < 0) + { + *failed_at = ptnode->relid; + return index; + } + + partRel = heap_open(ptnode->pdesc->oids[index], NoLock); + + /* Don't recurse if the index'th partition is a leaf partition. */ + if (partRel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) + { + PartitionTreeNode prev; + + /* + * Index returned above considers only this parent (of which partRel + * is a partition). We however want to return the index across the + * the whole partition tree. If partRel is the leftmost partition + * of parent (index = 0) or if none of the left siblings are + * partitioned tables, we can simply return parent's offset plus + * partRel's index as the result. Otherwise, we find the node + * corresponding to the rightmost partitioned table among partRel's + * left siblings and use the information therein. + */ + prev = node = ptnode->downlink; + if (node && node->index < index) + { + /* + * Find the partition tree node such that its index value is the + * greatest value less than the above returned index. + */ + while (node) + { + if (node->index > index) + { + node = prev; + break; + } + + prev = node; + node = node->next; + } + + if (!node) + node = prev; + Assert (node != NULL); + + index = node->offset + node->num_leaf_parts + + (index - node->index - 1); + } + else + /* + * The easy case where all of partRel's left siblings (if any) are + * leaf partitions. + */ + index = ptnode->offset + index; + + heap_close(partRel, NoLock); + return index; + } + + heap_close(partRel, NoLock); + + /* + * Need to recurse as the selected partition is a partitioned table + * itself. Locate the corresponding PartitionTreeNode to pass it down. + */ + node = ptnode->downlink; + while (node->next != NULL && node->index != index) + node = node->next; + Assert (node != NULL); + + return get_partition_for_tuple(node, slot, estate, failed_at); +} + +/* + * list_partition_for_tuple + * Find the list partition for a tuple + * + * Returns -1 if none found. + */ +static int +list_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, + Datum value, bool isnull) +{ + PartitionListInfo listinfo; + int found; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_LIST); + listinfo = pdesc->boundinfo->bounds.lists; + + if (isnull && listinfo.has_null) + return listinfo.null_index; + else if (!isnull) + { + found = partition_list_values_bsearch(key, + listinfo.values, + listinfo.nvalues, + value); + if (found >= 0) + return listinfo.indexes[found]; + } + + /* Control reaches here if isnull and !listinfo->has_null */ + return -1; +} + +/* + * range_partition_for_tuple + * Search the range partition for a range key ('values') + * + * Returns -1 if none found. + */ +static int +range_partition_for_tuple(PartitionKey key, PartitionDesc pdesc, Datum *tuple) +{ + int offset; + PartitionRangeInfo rangeinfo; + + Assert(pdesc->nparts > 0); + Assert(pdesc->boundinfo->strategy == PARTITION_STRATEGY_RANGE); + rangeinfo = pdesc->boundinfo->bounds.ranges; + + offset = partition_rbound_bsearch(key, + rangeinfo.bounds, rangeinfo.nbounds, + tuple, partition_rbound_datum_cmp, + true, NULL); + + /* + * Offset returned is such that the bound at offset is found to be + * less or equal with the tuple. That is, the tuple belongs to the + * partition with the rangeinfo.bounds[offset] as the lower bound and + * rangeinfo.bounds[offset+1] as the upper bound, provided the latter + * is indeed marked !lower (that is, it's an upper bound). If it turns + * out that it is a lower bound then the corresponding index will be -1, + * which means no valid partition exists. + */ + return rangeinfo.indexes[offset+1]; +} + /* List partition related support functions */ /* @@ -1767,6 +2056,40 @@ partition_rbound_cmp(PartitionKey key, PartitionRangeBound *b1, void *arg) } /* + * Return whether bound <=, =, >= partition key of tuple + * + * The 3rd argument is void * so that it can be used with + * partition_rbound_bsearch() + */ +static int32 +partition_rbound_datum_cmp(PartitionKey key, PartitionRangeBound *bound, + void *arg) +{ + Datum *datums1 = bound->datums, + *datums2 = (Datum *) arg; + int i; + int32 cmpval; + + for (i = 0; i < key->partnatts; i++) + { + if (bound->infinite[i]) + return bound->lower ? -1 : 1; + + cmpval = DatumGetInt32(FunctionCall2Coll(&key->partsupfunc[i], + key->partcollation[i], + datums1[i], datums2[i])); + if (cmpval != 0) + break; + } + + /* If datums are equal and this is an upper bound, tuple > bound */ + if (cmpval == 0 && !bound->lower) + return -1; + + return cmpval; +} + +/* * Return whether two range bounds are equal simply by comparing datums */ static bool diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c index 7a2bf94..6fb376f 100644 --- a/src/backend/commands/copy.c +++ b/src/backend/commands/copy.c @@ -30,6 +30,7 @@ #include "commands/defrem.h" #include "commands/trigger.h" #include "executor/executor.h" +#include "foreign/fdwapi.h" #include "libpq/libpq.h" #include "libpq/pqformat.h" #include "mb/pg_wchar.h" @@ -161,6 +162,11 @@ typedef struct CopyStateData ExprState **defexprs; /* array of default att expressions */ bool volatile_defexprs; /* is any of defexprs volatile? */ List *range_table; + PartitionTreeNode ptnode; /* partition descriptor node tree */ + ResultRelInfo *partitions; + TupleConversionMap **partition_tupconv_maps; + List *partition_fdw_priv_lists; + int num_partitions; /* * These variables are used to reduce overhead in textual COPY FROM. @@ -1397,6 +1403,94 @@ BeginCopy(ParseState *pstate, (errcode(ERRCODE_UNDEFINED_COLUMN), errmsg("table \"%s\" does not have OIDs", RelationGetRelationName(cstate->rel)))); + + /* + * Initialize state for CopyFrom tuple routing. Watch out for + * any foreign partitions. + */ + if (is_from && rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + List *leaf_parts; + ListCell *cell; + int i; + int num_leaf_parts; + ResultRelInfo *leaf_part_rri; + PlannerInfo *root = makeNode(PlannerInfo); /* mostly dummy */ + Query *parse = makeNode(Query); /* ditto */ + ModifyTable *plan = makeNode(ModifyTable); /* ditto */ + RangeTblEntry *fdw_rte = makeNode(RangeTblEntry); /* ditto */ + List *fdw_private_lists = NIL; + + cstate->ptnode = RelationGetPartitionTreeNode(rel); + leaf_parts = get_leaf_partition_oids(cstate->ptnode); + num_leaf_parts = list_length(leaf_parts); + + cstate->num_partitions = num_leaf_parts; + cstate->partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + cstate->partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + /* For use below, iff a partition found to be a foreign table */ + plan->operation = CMD_INSERT; + plan->plans = list_make1(makeNode(Result)); + fdw_rte->rtekind = RTE_RELATION; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + parse->rtable = list_make1(fdw_rte); + root->parse = parse; + + leaf_part_rri = cstate->partitions; + i = 0; + foreach(cell, leaf_parts) + { + Relation part_rel; + + part_rel = heap_open(lfirst_oid(cell), RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation. + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition check */ + 0); + + /* Open partition indices */ + ExecOpenIndices(leaf_part_rri, false); + + /* Special dance for foreign tables */ + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + fdw_rte->relid = RelationGetRelid(part_rel); + fdw_private = leaf_part_rri->ri_FdwRoutine->PlanForeignModify(root, + plan, + 1, + 0); + fdw_private_lists = lappend(fdw_private_lists, fdw_private); + } + + if (!equalTupleDescs(tupDesc, RelationGetDescr(part_rel))) + cstate->partition_tupconv_maps[i] = + convert_tuples_by_name(tupDesc, + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + + cstate->partition_fdw_priv_lists = fdw_private_lists; + pfree(fdw_rte); + pfree(plan); + pfree(parse); + pfree(root); + } } else { @@ -1692,6 +1786,8 @@ ClosePipeToProgram(CopyState cstate) static void EndCopy(CopyState cstate) { + int i; + if (cstate->is_program) { ClosePipeToProgram(cstate); @@ -1705,6 +1801,23 @@ EndCopy(CopyState cstate) cstate->filename))); } + /* Close all partitions and indices thereof */ + for (i = 0; i < cstate->num_partitions; i++) + { + ResultRelInfo *resultRelInfo = cstate->partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + /* XXX - EState not handy here to pass to EndForeignModify() */ + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(NULL, resultRelInfo); + + if (cstate->partition_tupconv_maps[i]) + pfree(cstate->partition_tupconv_maps[i]); + } + MemoryContextDelete(cstate->copycontext); pfree(cstate); } @@ -2255,6 +2368,7 @@ CopyFrom(CopyState cstate) Datum *values; bool *nulls; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; EState *estate = CreateExecutorState(); /* for ExecConstraints() */ ExprContext *econtext; TupleTableSlot *myslot; @@ -2281,6 +2395,7 @@ CopyFrom(CopyState cstate) * only hint about them in the view case.) */ if (cstate->rel->rd_rel->relkind != RELKIND_RELATION && + cstate->rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE && !(cstate->rel->trigdesc && cstate->rel->trigdesc->trig_insert_instead_row)) { @@ -2391,6 +2506,7 @@ CopyFrom(CopyState cstate) InitResultRelInfo(resultRelInfo, cstate->rel, 1, /* dummy rangetable index */ + true, /* do load partition check expression */ 0); ExecOpenIndices(resultRelInfo, false); @@ -2418,6 +2534,7 @@ CopyFrom(CopyState cstate) if ((resultRelInfo->ri_TrigDesc != NULL && (resultRelInfo->ri_TrigDesc->trig_insert_before_row || resultRelInfo->ri_TrigDesc->trig_insert_instead_row)) || + cstate->ptnode != NULL || cstate->volatile_defexprs) { useHeapMultiInsert = false; @@ -2439,10 +2556,46 @@ CopyFrom(CopyState cstate) */ ExecBSInsertTriggers(estate, resultRelInfo); + /* Initialize FDW partition insert plans */ + if (cstate->ptnode) + { + int i, + j; + List *fdw_private_lists = cstate->partition_fdw_priv_lists; + ModifyTableState *mtstate = makeNode(ModifyTableState); + ResultRelInfo *leaf_part_rri; + + /* Mostly dummy containing enough state for BeginForeignModify */ + mtstate->ps.state = estate; + mtstate->operation = CMD_INSERT; + + j = 0; + leaf_part_rri = cstate->partitions; + for (i = 0; i < cstate->num_partitions; i++) + { + if (leaf_part_rri->ri_FdwRoutine) + { + List *fdw_private; + + Assert(fdw_private_lists); + fdw_private = list_nth(fdw_private_lists, j++); + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, 0); + } + leaf_part_rri++; + } + } + values = (Datum *) palloc(tupDesc->natts * sizeof(Datum)); nulls = (bool *) palloc(tupDesc->natts * sizeof(bool)); - bistate = GetBulkInsertState(); + if (useHeapMultiInsert) + bistate = GetBulkInsertState(); + else + bistate = NULL; + econtext = GetPerTupleExprContext(estate); /* Set up callback to identify error line number */ @@ -2494,6 +2647,31 @@ CopyFrom(CopyState cstate) slot = myslot; ExecStoreTuple(tuple, slot, InvalidBuffer, false); + /* Determine the partition */ + saved_resultRelInfo = resultRelInfo; + if (cstate->ptnode) + { + int i_leaf_partition; + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + cstate->ptnode, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < cstate->num_partitions); + + resultRelInfo = cstate->partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = cstate->partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + + tuple->t_tableOid = RelationGetRelid(resultRelInfo->ri_RelationDesc); + } + skip_tuple = false; /* BEFORE ROW INSERT Triggers */ @@ -2523,7 +2701,16 @@ CopyFrom(CopyState cstate) resultRelInfo->ri_PartitionCheck) ExecConstraints(resultRelInfo, slot, estate); - if (useHeapMultiInsert) + if (resultRelInfo->ri_FdwRoutine) + { + resultRelInfo->ri_FdwRoutine->ExecForeignInsert(estate, + resultRelInfo, + slot, + NULL); + /* AFTER ROW INSERT Triggers */ + ExecARInsertTriggers(estate, resultRelInfo, tuple, NIL); + } + else if (useHeapMultiInsert) { /* Add this tuple to the tuple buffer */ if (nBufferedTuples == 0) @@ -2553,7 +2740,8 @@ CopyFrom(CopyState cstate) List *recheckIndexes = NIL; /* OK, store the tuple and create index entries for it */ - heap_insert(cstate->rel, tuple, mycid, hi_options, bistate); + heap_insert(resultRelInfo->ri_RelationDesc, tuple, mycid, + hi_options, bistate); if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(slot, @@ -2577,6 +2765,12 @@ CopyFrom(CopyState cstate) * tuples inserted by an INSERT command. */ processed++; + + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } } } @@ -2590,7 +2784,8 @@ CopyFrom(CopyState cstate) /* Done, clean up */ error_context_stack = errcallback.previous; - FreeBulkInsertState(bistate); + if (bistate) + FreeBulkInsertState(bistate); MemoryContextSwitchTo(oldcontext); diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8cd3614..b51d59a 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1293,6 +1293,7 @@ ExecuteTruncate(TruncateStmt *stmt) InitResultRelInfo(resultRelInfo, rel, 0, /* dummy rangetable index */ + false, 0); resultRelInfo++; } diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index ea3f59a..62bf8b7 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -826,6 +826,7 @@ InitPlan(QueryDesc *queryDesc, int eflags) InitResultRelInfo(resultRelInfo, resultRelation, resultRelationIndex, + true, estate->es_instrument); resultRelInfo++; } @@ -1215,6 +1216,7 @@ void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options) { MemSet(resultRelInfo, 0, sizeof(ResultRelInfo)); @@ -1252,8 +1254,10 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo, resultRelInfo->ri_ConstraintExprs = NULL; resultRelInfo->ri_junkFilter = NULL; resultRelInfo->ri_projectReturning = NULL; - resultRelInfo->ri_PartitionCheck = - RelationGetPartitionQual(resultRelationDesc, true); + if (load_partition_check) + resultRelInfo->ri_PartitionCheck = + RelationGetPartitionQual(resultRelationDesc, + true); } /* @@ -1316,6 +1320,7 @@ ExecGetTriggerResultRel(EState *estate, Oid relid) InitResultRelInfo(rInfo, rel, 0, /* dummy rangetable index */ + true, estate->es_instrument); estate->es_trig_target_relations = lappend(estate->es_trig_target_relations, rInfo); @@ -2996,3 +3001,41 @@ EvalPlanQualEnd(EPQState *epqstate) epqstate->planstate = NULL; epqstate->origslot = NULL; } + +int +ExecFindPartition(ResultRelInfo *resultRelInfo, PartitionTreeNode ptnode, + TupleTableSlot *slot, EState *estate) +{ + int i_leaf_partition; + Oid failed_at; + + i_leaf_partition = get_partition_for_tuple(ptnode, slot, estate, + &failed_at); + + if (i_leaf_partition < 0) + { + Relation rel = resultRelInfo->ri_RelationDesc; + char *val_desc; + Bitmapset *insertedCols, + *updatedCols, + *modifiedCols; + TupleDesc tupDesc = RelationGetDescr(rel); + + insertedCols = GetInsertedColumns(resultRelInfo, estate); + updatedCols = GetUpdatedColumns(resultRelInfo, estate); + modifiedCols = bms_union(insertedCols, updatedCols); + val_desc = ExecBuildSlotValueDescription(RelationGetRelid(rel), + slot, + tupDesc, + modifiedCols, + 64); + Assert(OidIsValid(failed_at)); + ereport(ERROR, + (errcode(ERRCODE_CHECK_VIOLATION), + errmsg("no partition of relation \"%s\" found for row", + get_rel_name(failed_at)), + val_desc ? errdetail("Failing row contains %s.", val_desc) : 0)); + } + + return i_leaf_partition; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a612b08..5c2bf6c 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -258,6 +258,7 @@ ExecInsert(ModifyTableState *mtstate, { HeapTuple tuple; ResultRelInfo *resultRelInfo; + ResultRelInfo *saved_resultRelInfo = NULL; Relation resultRelationDesc; Oid newId; List *recheckIndexes = NIL; @@ -272,6 +273,31 @@ ExecInsert(ModifyTableState *mtstate, * get information on the (current) result relation */ resultRelInfo = estate->es_result_relation_info; + + saved_resultRelInfo = resultRelInfo; + + if (mtstate->mt_partition_tree_root) + { + int i_leaf_partition; + ExprContext *econtext = GetPerTupleExprContext(estate); + TupleConversionMap *map; + + econtext->ecxt_scantuple = slot; + i_leaf_partition = ExecFindPartition(resultRelInfo, + mtstate->mt_partition_tree_root, + slot, + estate); + Assert(i_leaf_partition >= 0 && + i_leaf_partition < mtstate->mt_num_partitions); + + resultRelInfo = mtstate->mt_partitions + i_leaf_partition; + estate->es_result_relation_info = resultRelInfo; + + map = mtstate->mt_partition_tupconv_maps[i_leaf_partition]; + if (map) + tuple = do_convert_tuple(tuple, map); + } + resultRelationDesc = resultRelInfo->ri_RelationDesc; /* @@ -511,6 +537,12 @@ ExecInsert(ModifyTableState *mtstate, list_free(recheckIndexes); + if (saved_resultRelInfo) + { + resultRelInfo = saved_resultRelInfo; + estate->es_result_relation_info = resultRelInfo; + } + /* * Check any WITH CHECK OPTION constraints from parent views. We are * required to do this after testing all constraints and uniqueness @@ -1565,6 +1597,7 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) Plan *subplan; ListCell *l; int i; + Relation rel; /* check for unsupported flags */ Assert(!(eflags & (EXEC_FLAG_BACKWARD | EXEC_FLAG_MARK))); @@ -1655,6 +1688,98 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) estate->es_result_relation_info = saved_resultRelInfo; + /* Build state for INSERT tuple routing */ + rel = mtstate->resultRelInfo->ri_RelationDesc; + if (operation == CMD_INSERT && + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + int i, + j, + num_leaf_parts; + List *leaf_parts; + ListCell *cell; + ResultRelInfo *leaf_part_rri; + + mtstate->mt_partition_tree_root = RelationGetPartitionTreeNode(rel); + leaf_parts = get_leaf_partition_oids(mtstate->mt_partition_tree_root); + num_leaf_parts = list_length(leaf_parts); + + mtstate->mt_num_partitions = num_leaf_parts; + mtstate->mt_partitions = (ResultRelInfo *) + palloc0(num_leaf_parts * sizeof(ResultRelInfo)); + mtstate->mt_partition_tupconv_maps = (TupleConversionMap **) + palloc0(num_leaf_parts * sizeof(TupleConversionMap *)); + + leaf_part_rri = mtstate->mt_partitions; + i = j = 0; + foreach(cell, leaf_parts) + { + Oid ftoid = lfirst_oid(cell); + Relation part_rel; + + part_rel = heap_open(ftoid, RowExclusiveLock); + + /* + * Verify result relation is a valid target for the current + * operation + */ + CheckValidResultRel(part_rel, CMD_INSERT); + + InitResultRelInfo(leaf_part_rri, + part_rel, + 1, /* dummy */ + false, /* no need for partition checks */ + eflags); + + /* Open partition indices (note: ON CONFLICT unsupported)*/ + if (leaf_part_rri->ri_RelationDesc->rd_rel->relhasindex && + operation != CMD_DELETE && + leaf_part_rri->ri_IndexRelationDescs == NULL) + ExecOpenIndices(leaf_part_rri, false); + + if (leaf_part_rri->ri_FdwRoutine) + { + ListCell *lc; + List *fdw_private; + int k; + + /* + * There are as many fdw_private's in fdwPrivLists as there + * are FDW partitions, but we must find the intended for the + * this foreign table. + */ + k = 0; + foreach(lc, node->fdwPartitionOids) + { + if (lfirst_oid(lc) == ftoid) + break; + k++; + } + + Assert(k < num_leaf_parts); + fdw_private = (List *) list_nth(node->fdwPrivLists, k); + Assert(fdw_private != NIL); + + leaf_part_rri->ri_FdwRoutine->BeginForeignModify(mtstate, + leaf_part_rri, + fdw_private, + 0, + eflags); + j++; + } + + if (!equalTupleDescs(RelationGetDescr(rel), + RelationGetDescr(part_rel))) + mtstate->mt_partition_tupconv_maps[i] = + convert_tuples_by_name(RelationGetDescr(rel), + RelationGetDescr(part_rel), + gettext_noop("could not convert row type")); + + leaf_part_rri++; + i++; + } + } + /* * Initialize any WITH CHECK OPTION constraints if needed. */ @@ -1972,6 +2097,23 @@ ExecEndModifyTable(ModifyTableState *node) resultRelInfo); } + /* Close all partitions and indices thereof */ + for (i = 0; i < node->mt_num_partitions; i++) + { + ResultRelInfo *resultRelInfo = node->mt_partitions + i; + + ExecCloseIndices(resultRelInfo); + heap_close(resultRelInfo->ri_RelationDesc, NoLock); + + if (resultRelInfo->ri_FdwRoutine && + resultRelInfo->ri_FdwRoutine->EndForeignModify != NULL) + resultRelInfo->ri_FdwRoutine->EndForeignModify(node->ps.state, + resultRelInfo); + + if (node->mt_partition_tupconv_maps[i]) + pfree(node->mt_partition_tupconv_maps[i]); + } + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index 28d0036..470dee7 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -188,6 +188,7 @@ _copyModifyTable(const ModifyTable *from) COPY_NODE_FIELD(withCheckOptionLists); COPY_NODE_FIELD(returningLists); COPY_NODE_FIELD(fdwPrivLists); + COPY_NODE_FIELD(fdwPartitionOids); COPY_BITMAPSET_FIELD(fdwDirectModifyPlans); COPY_NODE_FIELD(rowMarks); COPY_SCALAR_FIELD(epqParam); diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 0d858f5..5c8eced 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -340,6 +340,7 @@ _outModifyTable(StringInfo str, const ModifyTable *node) WRITE_NODE_FIELD(withCheckOptionLists); WRITE_NODE_FIELD(returningLists); WRITE_NODE_FIELD(fdwPrivLists); + WRITE_NODE_FIELD(fdwPartitionOids); WRITE_BITMAPSET_FIELD(fdwDirectModifyPlans); WRITE_NODE_FIELD(rowMarks); WRITE_INT_FIELD(epqParam); diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c index c587d4e..ddd78c7 100644 --- a/src/backend/nodes/readfuncs.c +++ b/src/backend/nodes/readfuncs.c @@ -1495,6 +1495,7 @@ _readModifyTable(void) READ_NODE_FIELD(withCheckOptionLists); READ_NODE_FIELD(returningLists); READ_NODE_FIELD(fdwPrivLists); + READ_NODE_FIELD(fdwPartitionOids); READ_BITMAPSET_FIELD(fdwDirectModifyPlans); READ_NODE_FIELD(rowMarks); READ_INT_FIELD(epqParam); diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index ad49674..29f40fe 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -22,6 +22,7 @@ #include "access/stratnum.h" #include "access/sysattr.h" #include "catalog/pg_class.h" +#include "catalog/pg_inherits_fn.h" #include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/extensible.h" @@ -6159,6 +6160,82 @@ make_modifytable(PlannerInfo *root, node->fdwPrivLists = fdw_private_list; node->fdwDirectModifyPlans = direct_modify_plans; + /* Collect insert plans for all FDW-managed partitions */ + if (node->operation == CMD_INSERT) + { + RangeTblEntry *rte, + **saved_simple_rte_array; + List *partition_oids, + *fdw_partition_oids; + + Assert(list_length(resultRelations) == 1); + rte = rt_fetch(linitial_int(resultRelations), root->parse->rtable); + Assert(rte->rtekind == RTE_RELATION); + + if (rte->relkind != RELKIND_PARTITIONED_TABLE) + return node; + + partition_oids = find_all_inheritors(rte->relid, NoLock, NULL); + + /* Discard any previous content which is useless anyway */ + fdw_private_list = NIL; + fdw_partition_oids = NIL; + + /* + * To force the FDW driver fetch the intended RTE, we need to temporarily + * switch root->simple_rte_array to the one holding only that RTE at a + * designated index, for every foreign table. + */ + saved_simple_rte_array = root->simple_rte_array; + root->simple_rte_array = (RangeTblEntry **) + palloc0(2 * sizeof(RangeTblEntry *)); + foreach(lc, partition_oids) + { + Oid myoid = lfirst_oid(lc); + FdwRoutine *fdwroutine; + List *fdw_private; + + /* + * We are only interested in foreign tables. Note that this will + * also eliminate any partitioned tables since foreign tables can + * only ever be leaf partitions. + */ + if (!oid_is_foreign_table(myoid)) + continue; + + fdw_partition_oids = lappend_oid(fdw_partition_oids, myoid); + + fdwroutine = GetFdwRoutineByRelId(myoid); + if (fdwroutine && fdwroutine->PlanForeignModify) + { + RangeTblEntry *fdw_rte; + + fdw_rte = copyObject(rte); + fdw_rte->relid = myoid; + fdw_rte->relkind = RELKIND_FOREIGN_TABLE; + + /* + * Assumes PlanForeignModify() uses planner_rt_fetch(), also, + * see the above comment. + */ + root->simple_rte_array[1] = fdw_rte; + + fdw_private = fdwroutine->PlanForeignModify(root, node, 1, 0); + pfree(fdw_rte); + } + else + fdw_private = NIL; + + fdw_private_list = lappend(fdw_private_list, fdw_private); + } + + pfree(root->simple_rte_array); + root->simple_rte_array = saved_simple_rte_array; + + node->fdwPrivLists = fdw_private_list; + node->fdwPartitionOids = fdw_partition_oids; + } + return node; } diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index a2cbf14..e85ca0a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1715,3 +1715,16 @@ has_row_triggers(PlannerInfo *root, Index rti, CmdType event) heap_close(relation, NoLock); return result; } + +bool +oid_is_foreign_table(Oid relid) +{ + Relation rel; + char relkind; + + rel = heap_open(relid, NoLock); + relkind = rel->rd_rel->relkind; + heap_close(rel, NoLock); + + return relkind == RELKIND_FOREIGN_TABLE; +} diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c index 6901e08..c10b6c3 100644 --- a/src/backend/parser/analyze.c +++ b/src/backend/parser/analyze.c @@ -798,8 +798,16 @@ transformInsertStmt(ParseState *pstate, InsertStmt *stmt) /* Process ON CONFLICT, if any. */ if (stmt->onConflictClause) + { + /* Bail out if target relation is partitioned table */ + if (pstate->p_target_rangetblentry->relkind == RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("ON CONFLICT clause is not supported with partitioned tables"))); + qry->onConflict = transformOnConflictClause(pstate, stmt->onConflictClause); + } /* * If we have a RETURNING clause, we need to add the target relation to diff --git a/src/include/catalog/partition.h b/src/include/catalog/partition.h index 057b1e3..86cc598 100644 --- a/src/include/catalog/partition.h +++ b/src/include/catalog/partition.h @@ -14,6 +14,8 @@ #define PARTITION_H #include "fmgr.h" +#include "executor/tuptable.h" +#include "nodes/execnodes.h" #include "parser/parse_node.h" #include "utils/rel.h" @@ -50,4 +52,9 @@ extern List *RelationGetPartitionQual(Relation rel, bool recurse); /* For tuple routing */ extern PartitionTreeNode RelationGetPartitionTreeNode(Relation rel); extern List *get_leaf_partition_oids(PartitionTreeNode ptnode); + +extern int get_partition_for_tuple(PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate, + Oid *failed_at); #endif /* PARTITION_H */ diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 136276b..c62946f 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -14,6 +14,7 @@ #ifndef EXECUTOR_H #define EXECUTOR_H +#include "catalog/partition.h" #include "executor/execdesc.h" #include "nodes/parsenodes.h" @@ -188,6 +189,7 @@ extern void CheckValidResultRel(Relation resultRel, CmdType operation); extern void InitResultRelInfo(ResultRelInfo *resultRelInfo, Relation resultRelationDesc, Index resultRelationIndex, + bool load_partition_check, int instrument_options); extern ResultRelInfo *ExecGetTriggerResultRel(EState *estate, Oid relid); extern bool ExecContextForcesOids(PlanState *planstate, bool *hasoids); @@ -211,6 +213,10 @@ extern void EvalPlanQualSetPlan(EPQState *epqstate, extern void EvalPlanQualSetTuple(EPQState *epqstate, Index rti, HeapTuple tuple); extern HeapTuple EvalPlanQualGetTuple(EPQState *epqstate, Index rti); +extern int ExecFindPartition(ResultRelInfo *resultRelInfo, + PartitionTreeNode ptnode, + TupleTableSlot *slot, + EState *estate); #define EvalPlanQualSetSlot(epqstate, slot) ((epqstate)->origslot = (slot)) extern void EvalPlanQualFetchRowMarks(EPQState *epqstate); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index ff8b66b..ce01008 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -16,6 +16,7 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/tupconvert.h" #include "executor/instrument.h" #include "lib/pairingheap.h" #include "nodes/params.h" @@ -1147,6 +1148,15 @@ typedef struct ModifyTableState * tlist */ TupleTableSlot *mt_conflproj; /* CONFLICT ... SET ... projection * target */ + struct PartitionTreeNodeData *mt_partition_tree_root; + /* Partition descriptor node tree */ + ResultRelInfo *mt_partitions; /* Per leaf partition target + * relations */ + TupleConversionMap **mt_partition_tupconv_maps; + /* Per leaf partition + * tuple conversion map */ + int mt_num_partitions; /* Number of leaf partition target + * relations in the above array */ } ModifyTableState; /* ---------------- diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index e2fbc7d..d82222c 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -189,6 +189,7 @@ typedef struct ModifyTable List *returningLists; /* per-target-table RETURNING tlists */ List *fdwPrivLists; /* per-target-table FDW private data lists */ Bitmapset *fdwDirectModifyPlans; /* indices of FDW DM plans */ + List *fdwPartitionOids; /* OIDs of FDW-managed partition */ List *rowMarks; /* PlanRowMarks (non-locking only) */ int epqParam; /* ID of Param for EvalPlanQual re-eval */ OnConflictAction onConflictAction; /* ON CONFLICT action */ diff --git a/src/include/optimizer/plancat.h b/src/include/optimizer/plancat.h index 125274e..fac606c 100644 --- a/src/include/optimizer/plancat.h +++ b/src/include/optimizer/plancat.h @@ -56,5 +56,6 @@ extern Selectivity join_selectivity(PlannerInfo *root, SpecialJoinInfo *sjinfo); extern bool has_row_triggers(PlannerInfo *root, Index rti, CmdType event); +extern bool oid_is_foreign_table(Oid relid); #endif /* PLANCAT_H */ diff --git a/src/test/regress/expected/insert.out b/src/test/regress/expected/insert.out index 9ae6b09..d5dcb59 100644 --- a/src/test/regress/expected/insert.out +++ b/src/test/regress/expected/insert.out @@ -227,6 +227,58 @@ DETAIL: Failing row contains (cc, 1). -- ok insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables +-- fail +insert into range_parted values ('a', 0); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 0). +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +ERROR: no partition of relation "range_parted" found for row +DETAIL: Failing row contains (a, 20). +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + tableoid | a | b +----------------+---+---- + part_a_1_a_10 | a | 1 + part_a_1_a_10 | a | 1 + part_a_10_a_20 | a | 10 + part_b_1_b_10 | b | 1 + part_b_10_b_20 | b | 10 + part_b_10_b_20 | b | 10 +(6 rows) + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +insert into part_EE_FF values ('EE', 0); +ERROR: no partition of relation "part_ee_ff" found for row +DETAIL: Failing row contains (EE, 0). +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + tableoid | a | b +------------------+----+---- + part_aa_bb | aA | + part_cc_dd | cC | 1 + part_null | | 0 + part_null | | 1 + part_ee_ff_1_10 | ff | 1 + part_ee_ff_1_10 | EE | 1 + part_ee_ff_10_20 | ff | 11 + part_ee_ff_10_20 | EE | 10 +(8 rows) + -- cleanup drop table range_parted cascade; NOTICE: drop cascades to 4 other objects diff --git a/src/test/regress/sql/insert.sql b/src/test/regress/sql/insert.sql index b6e821e..fbd30d9 100644 --- a/src/test/regress/sql/insert.sql +++ b/src/test/regress/sql/insert.sql @@ -140,6 +140,31 @@ insert into part_EE_FF_1_10 values ('cc', 1); insert into part_EE_FF_1_10 values ('ff', 1); insert into part_EE_FF_10_20 values ('ff', 11); +-- Check tuple routing for partitioned tables + +-- fail +insert into range_parted values ('a', 0); +-- ok +insert into range_parted values ('a', 1); +insert into range_parted values ('a', 10); +-- fail +insert into range_parted values ('a', 20); +-- ok +insert into range_parted values ('b', 1); +insert into range_parted values ('b', 10); +select tableoid::regclass, * from range_parted; + +-- ok +insert into list_parted values (null, 1); +insert into list_parted (a) values ('aA'); +-- fail (partition of part_EE_FF not found) +insert into list_parted values ('EE', 0); +insert into part_EE_FF values ('EE', 0); +-- ok +insert into list_parted values ('EE', 1); +insert into part_EE_FF values ('EE', 10); +select tableoid::regclass, * from list_parted; + -- cleanup drop table range_parted cascade; drop table list_parted cascade; -- 1.7.1 --------------20C4AAD7DBE3F2B5281ABBA6 Content-Type: text/x-diff; name="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-13.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename*0="0008-Update-DDL-Partitioning-chapter-to-reflect-new-devel-13"; filename*1=".patch" ^ permalink raw reply [nested|flat] 13+ messages in thread
* [PATCH 6/6] Ignore correlation for new BRIN opclasses @ 2020-09-12 13:07 Tomas Vondra <[email protected]> 0 siblings, 0 replies; 13+ messages in thread From: Tomas Vondra @ 2020-09-12 13:07 UTC (permalink / raw) The new BRIN opclasses (bloom and minmax-multi) are less sensitive to poorly correlated data, so just assume the data is perfectly correlated during costing. Author: Tomas Vondra <[email protected]> Discussion: https://postgr.es/m/[email protected] --- src/backend/access/brin/brin_bloom.c | 1 + src/backend/access/brin/brin_minmax_multi.c | 1 + src/backend/utils/adt/selfuncs.c | 19 ++++++++++++++++++- src/include/access/brin_internal.h | 3 +++ 4 files changed, 23 insertions(+), 1 deletion(-) diff --git a/src/backend/access/brin/brin_bloom.c b/src/backend/access/brin/brin_bloom.c index 000c2387ee..0137095660 100644 --- a/src/backend/access/brin/brin_bloom.c +++ b/src/backend/access/brin/brin_bloom.c @@ -385,6 +385,7 @@ brin_bloom_opcinfo(PG_FUNCTION_ARGS) result = palloc0(MAXALIGN(SizeofBrinOpcInfo(1)) + sizeof(BloomOpaque)); + result->oi_ignore_correlation = true; result->oi_nstored = 1; result->oi_regular_nulls = true; result->oi_opaque = (BloomOpaque *) diff --git a/src/backend/access/brin/brin_minmax_multi.c b/src/backend/access/brin/brin_minmax_multi.c index 0a76557cc1..d1f29934fc 100644 --- a/src/backend/access/brin/brin_minmax_multi.c +++ b/src/backend/access/brin/brin_minmax_multi.c @@ -1306,6 +1306,7 @@ brin_minmax_multi_opcinfo(PG_FUNCTION_ARGS) result = palloc0(MAXALIGN(SizeofBrinOpcInfo(1)) + sizeof(MinmaxMultiOpaque)); + result->oi_ignore_correlation = true; result->oi_nstored = 1; result->oi_regular_nulls = true; result->oi_opaque = (MinmaxMultiOpaque *) diff --git a/src/backend/utils/adt/selfuncs.c b/src/backend/utils/adt/selfuncs.c index d5e61664bc..d0cc145938 100644 --- a/src/backend/utils/adt/selfuncs.c +++ b/src/backend/utils/adt/selfuncs.c @@ -98,6 +98,7 @@ #include <math.h> #include "access/brin.h" +#include "access/brin_internal.h" #include "access/brin_page.h" #include "access/gin.h" #include "access/table.h" @@ -7352,7 +7353,8 @@ brincostestimate(PlannerInfo *root, IndexPath *path, double loop_count, double minimalRanges; double estimatedRanges; double selec; - Relation indexRel; + Relation indexRel = NULL; + TupleDesc tupdesc = NULL; ListCell *l; VariableStatData vardata; @@ -7374,6 +7376,7 @@ brincostestimate(PlannerInfo *root, IndexPath *path, double loop_count, */ indexRel = index_open(index->indexoid, NoLock); brinGetStats(indexRel, &statsData); + tupdesc = RelationGetDescr(indexRel); index_close(indexRel, NoLock); /* work out the actual number of ranges in the index */ @@ -7407,6 +7410,17 @@ brincostestimate(PlannerInfo *root, IndexPath *path, double loop_count, { IndexClause *iclause = lfirst_node(IndexClause, l); AttrNumber attnum = index->indexkeys[iclause->indexcol]; + FmgrInfo *opcInfoFn; + BrinOpcInfo *opcInfo; + Form_pg_attribute attr = TupleDescAttr(tupdesc, iclause->indexcol); + bool ignore_correlation; + + opcInfoFn = index_getprocinfo(indexRel, iclause->indexcol + 1, BRIN_PROCNUM_OPCINFO); + + opcInfo = (BrinOpcInfo *) + DatumGetPointer(FunctionCall1(opcInfoFn, attr->atttypid)); + + ignore_correlation = opcInfo->oi_ignore_correlation; /* attempt to lookup stats in relation for this index column */ if (attnum != 0) @@ -7477,6 +7491,9 @@ brincostestimate(PlannerInfo *root, IndexPath *path, double loop_count, if (sslot.nnumbers > 0) varCorrelation = Abs(sslot.numbers[0]); + if (ignore_correlation) + varCorrelation = 1.0; + if (varCorrelation > *indexCorrelation) *indexCorrelation = varCorrelation; diff --git a/src/include/access/brin_internal.h b/src/include/access/brin_internal.h index e7ce4d0209..901febe375 100644 --- a/src/include/access/brin_internal.h +++ b/src/include/access/brin_internal.h @@ -30,6 +30,9 @@ typedef struct BrinOpcInfo /* Regular processing of NULLs in BrinValues? */ bool oi_regular_nulls; + /* Ignore correlation during cost estimation */ + bool oi_ignore_correlation; + /* Opaque pointer for the opclass' private use */ void *oi_opaque; -- 2.26.2 --------------D21F32E0F9E7408E9BE1AF2E-- ^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: Support NOT VALID / VALIDATE constraint options for named NOT NULL constraints @ 2025-04-06 18:02 Alvaro Herrera <[email protected]> 0 siblings, 1 reply; 13+ messages in thread From: Alvaro Herrera @ 2025-04-06 18:02 UTC (permalink / raw) To: jian he <[email protected]>; +Cc: Rushabh Lathia <[email protected]>; pgsql-hackers On 2025-Apr-05, jian he wrote: > hi. > + /* FIXME use CompactAttribute */ > Form_pg_attribute att = TupleDescAttr(relation->rd_att, i - 1); > if (att->attnotnull && att->attnotnullvalid && > !att->attisdropped) > { > NullTest *ntest = makeNode(NullTest); > ntest->arg = (Expr *) makeVar(varno, > i, > att->atttypid, > att->atttypmod, > att->attcollation, > 0); > ntest->nulltesttype = IS_NOT_NULL; > > CompactAttribute doesn't have {atttypmod, attcollation} information, > now it is impossible to use CompactAttribute here, > so I removed this FIXME in get_relation_constraints. Ah, good point. In this new patch, I now consult both TupleDescCompactAttr (to consult attnullability) and then TupleDescAttr (to get collation etc), because now the information we need is split between those two places. It feels a bit nasty TBH. > i am uncomfortable with the change in > 'CREATE TABLE dump_test.test_table_generated' > so I only added 'CONSTRAINT NOT NULL / INVALID' tests in > 002_pg_dump.pl. > so I only added a test case 'CONSTRAINT NOT NULL / INVALID' > to 002_pg_dump.pl. Yeah, good call. > v7-0001 commit message explains what kind of problem > MergeWithExistingConstraint is trying to fix. Umm. I have done away with this again (and the parts in 0002 where you handle the equivalent issue for not-null constraints), because it is still not clear to me exactly what fails if you don't have them. I suggest that we should deal with this separately, and that a patch to deal with it should include a test case that fails if the code fix is not present. > v7-0002 bullet points summary about NOT NULL NOT VALID added to the > commit message. Thanks. > add a test for CREATE TABLE LIKE. > CREATE TABLE LIKE will copy the invalid not-null constraint and will become > valid, i think this is what we want. Yep, thanks. I removed the postgres_fdw changes. What I wrote was untested, and failed as soon as I added a trivial test. Needs more thought. I also did some more polish, and as I said in another email, it seems to me that this approach is also wrong, and that a better approach is to determine the value of CompactAttribute->attnullability using the pg_constraint scan at the time when the tupdesc is built. I coded this, and it almost works, but there are some spots that fail because some tuple descriptors are not built correctly. I'm not sure what to do with this at this stage; I would love to see this patch across the finish line, but it seems a little late now. Note: naturally, patch 0002 in this series would be squashed with 0001 for commit. Patch 0003 is not for commit, just to show what the problems are. If you run the regression tests without 0003, there are surprisingly very few failures, and it's very clear that they are because of tuple descriptors that don't have the attnullability flag set correctly. -- Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/ "La grandeza es una experiencia transitoria. Nunca es consistente. Depende en gran parte de la imaginación humana creadora de mitos" (Irulan) Attachments: [text/x-diff] v8-0001-Allow-NOT-NULL-constraints-to-be-added-as-NOT-VAL.patch (86.5K, ../../[email protected]/2-v8-0001-Allow-NOT-NULL-constraints-to-be-added-as-NOT-VAL.patch) download | inline diff: From adfc93bb7ed8d716f3017fdc6263c21f1abd69ca Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=81lvaro=20Herrera?= <[email protected]> Date: Sat, 5 Apr 2025 19:25:10 +0200 Subject: [PATCH v8 1/3] Allow NOT NULL constraints to be added as NOT VALID MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This required adding a new system column, pg_attribute.attnotnullvalid, in order to allow constructing a TupleDesc without having to scan pg_constraint; at the same time, we keep pg_attribute.attnotnull unchanged, to avoid breaking the countless applications that rely on that. Also add support for ALTER TABLE .. VALIDATE CONSTRAINT for them. Author: Rushabh Lathia <[email protected]> Author: Jian He <[email protected]> Reviewed-by: Álvaro Herrera <[email protected]> Discussion: https://postgr.es/m/CAGPqQf0KitkNack4F5CFkFi-9Dqvp29Ro=EpcWt=4_hs-Rt+bQ@mail.gmail.com --- doc/src/sgml/catalogs.sgml | 11 +- doc/src/sgml/ref/alter_table.sgml | 8 +- src/backend/access/common/tupdesc.c | 11 +- src/backend/bootstrap/bootstrap.c | 3 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 17 +- src/backend/catalog/pg_constraint.c | 46 ++-- src/backend/commands/tablecmds.c | 246 +++++++++++++++++--- src/backend/executor/execMain.c | 1 + src/backend/jit/llvm/llvmjit_deform.c | 10 +- src/backend/optimizer/util/plancat.c | 5 +- src/backend/parser/gram.y | 5 +- src/backend/utils/cache/catcache.c | 1 + src/backend/utils/cache/relcache.c | 3 +- src/bin/pg_dump/pg_dump.c | 186 +++++++++++++-- src/bin/pg_dump/pg_dump.h | 2 + src/bin/pg_dump/t/002_pg_dump.pl | 21 +- src/bin/psql/describe.c | 9 +- src/include/access/tupdesc.h | 9 +- src/include/catalog/catversion.h | 2 +- src/include/catalog/pg_attribute.h | 5 +- src/include/catalog/pg_constraint.h | 2 +- src/test/regress/expected/alter_table.out | 69 ++++++ src/test/regress/expected/constraints.out | 250 +++++++++++++++++++++ src/test/regress/expected/sanity_check.out | 20 ++ src/test/regress/sql/alter_table.sql | 14 ++ src/test/regress/sql/constraints.sql | 166 ++++++++++++++ src/test/regress/sql/sanity_check.sql | 14 ++ 28 files changed, 1047 insertions(+), 92 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 45ba9c5118f..3ab7d7b68aa 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1260,7 +1260,16 @@ <structfield>attnotnull</structfield> <type>bool</type> </para> <para> - This column has a not-null constraint. + This column has a (possibly unvalidated) not-null constraint. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attnotnullvalid</structfield> <type>bool</type> + </para> + <para> + Whether the not-null constraint, if one exists, has been validated. </para></entry> </row> diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml index ece438f0075..a75e75d800d 100644 --- a/doc/src/sgml/ref/alter_table.sgml +++ b/doc/src/sgml/ref/alter_table.sgml @@ -243,6 +243,8 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM entire table; however, if a valid <literal>CHECK</literal> constraint is found which proves no <literal>NULL</literal> can exist, then the table scan is skipped. + If a column has an invalid not-null constraint, + <literal>SET NOT NULL</literal> validates it. </para> <para> @@ -458,8 +460,8 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM <para> This form adds a new constraint to a table using the same constraint syntax as <link linkend="sql-createtable"><command>CREATE TABLE</command></link>, plus the option <literal>NOT - VALID</literal>, which is currently only allowed for foreign key - and CHECK constraints. + VALID</literal>, which is currently only allowed for foreign key, + <literal>CHECK</literal> constraints and not-null constraints. </para> <para> @@ -586,7 +588,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM <term><literal>VALIDATE CONSTRAINT</literal></term> <listitem> <para> - This form validates a foreign key or check constraint that was + This form validates a foreign key, check, or not-null constraint that was previously created as <literal>NOT VALID</literal>, by scanning the table to ensure there are no rows for which the constraint is not satisfied. If the constraint is not enforced, an error is thrown. diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index ed2195f14b2..320de2cbda0 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -74,7 +74,8 @@ populate_compact_attribute_internal(Form_pg_attribute src, dst->atthasmissing = src->atthasmissing; dst->attisdropped = src->attisdropped; dst->attgenerated = (src->attgenerated != '\0'); - dst->attnotnull = src->attnotnull; + dst->attnullability = !src->attnotnull ? ATTNULLABLE_NONE : + src->attnotnullvalid ? ATTNULLABLE_VALID : ATTNULLABLE_INVALID; switch (src->attalign) { @@ -252,6 +253,7 @@ CreateTupleDescCopy(TupleDesc tupdesc) Form_pg_attribute att = TupleDescAttr(desc, i); att->attnotnull = false; + att->attnotnullvalid = false; att->atthasdef = false; att->atthasmissing = false; att->attidentity = '\0'; @@ -298,6 +300,7 @@ CreateTupleDescTruncatedCopy(TupleDesc tupdesc, int natts) Form_pg_attribute att = TupleDescAttr(desc, i); att->attnotnull = false; + att->attnotnullvalid = false; att->atthasdef = false; att->atthasmissing = false; att->attidentity = '\0'; @@ -418,6 +421,7 @@ TupleDescCopy(TupleDesc dst, TupleDesc src) Form_pg_attribute att = TupleDescAttr(dst, i); att->attnotnull = false; + att->attnotnullvalid = false; att->atthasdef = false; att->atthasmissing = false; att->attidentity = '\0'; @@ -464,6 +468,7 @@ TupleDescCopyEntry(TupleDesc dst, AttrNumber dstAttno, /* since we're not copying constraints or defaults, clear these */ dstAtt->attnotnull = false; + dstAtt->attnotnullvalid = false; dstAtt->atthasdef = false; dstAtt->atthasmissing = false; dstAtt->attidentity = '\0'; @@ -613,6 +618,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attnotnull != attr2->attnotnull) return false; + if (attr1->attnotnullvalid != attr2->attnotnullvalid) + return false; if (attr1->atthasdef != attr2->atthasdef) return false; if (attr1->attidentity != attr2->attidentity) @@ -841,6 +848,7 @@ TupleDescInitEntry(TupleDesc desc, att->attndims = attdim; att->attnotnull = false; + att->attnotnullvalid = false; att->atthasdef = false; att->atthasmissing = false; att->attidentity = '\0'; @@ -904,6 +912,7 @@ TupleDescInitBuiltinEntry(TupleDesc desc, att->attndims = attdim; att->attnotnull = false; + att->attnotnullvalid = false; att->atthasdef = false; att->atthasmissing = false; att->attidentity = '\0'; diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6db864892d0..44ec9e58520 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -615,6 +615,9 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attnotnull = true; } } + + /* Not-null constraints on system catalogs are always valid. */ + attrtypes[attnum]->attnotnullvalid = attrtypes[attnum]->attnotnull; } diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index df3231fcd41..bdf55d7dc8d 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -986,6 +986,9 @@ sub morph_row_for_pgattr $row->{attnotnull} = 'f'; } + # Not-null constraints on system catalogs are always valid. + $row->{attnotnullvalid} = $row->{attnotnull}; + Catalog::AddDefaultValues($row, $pgattr_schema, 'pg_attribute'); return; } diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index b807ab66668..503e7cb3f58 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -151,6 +151,7 @@ static const FormData_pg_attribute a1 = { .attalign = TYPALIGN_SHORT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, + .attnotnullvalid = true, .attislocal = true, }; @@ -164,6 +165,7 @@ static const FormData_pg_attribute a2 = { .attalign = TYPALIGN_INT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, + .attnotnullvalid = true, .attislocal = true, }; @@ -177,6 +179,7 @@ static const FormData_pg_attribute a3 = { .attalign = TYPALIGN_INT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, + .attnotnullvalid = true, .attislocal = true, }; @@ -190,6 +193,7 @@ static const FormData_pg_attribute a4 = { .attalign = TYPALIGN_INT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, + .attnotnullvalid = true, .attislocal = true, }; @@ -203,6 +207,7 @@ static const FormData_pg_attribute a5 = { .attalign = TYPALIGN_INT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, + .attnotnullvalid = true, .attislocal = true, }; @@ -222,6 +227,7 @@ static const FormData_pg_attribute a6 = { .attalign = TYPALIGN_INT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, + .attnotnullvalid = true, .attislocal = true, }; @@ -753,6 +759,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attstorage - 1] = CharGetDatum(attrs->attstorage); slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); slot[slotCount]->tts_values[Anum_pg_attribute_attnotnull - 1] = BoolGetDatum(attrs->attnotnull); + slot[slotCount]->tts_values[Anum_pg_attribute_attnotnullvalid - 1] = BoolGetDatum(attrs->attnotnullvalid); slot[slotCount]->tts_values[Anum_pg_attribute_atthasdef - 1] = BoolGetDatum(attrs->atthasdef); slot[slotCount]->tts_values[Anum_pg_attribute_atthasmissing - 1] = BoolGetDatum(attrs->atthasmissing); slot[slotCount]->tts_values[Anum_pg_attribute_attidentity - 1] = CharGetDatum(attrs->attidentity); @@ -1714,6 +1721,7 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Remove any not-null constraint the column may have */ attStruct->attnotnull = false; + attStruct->attnotnullvalid = false; /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; @@ -2616,12 +2624,17 @@ AddRelationNewConstraints(Relation rel, errmsg("cannot add not-null constraint on system column \"%s\"", strVal(linitial(cdef->keys)))); + Assert(cdef->initially_valid != cdef->skip_validation); + /* * If the column already has a not-null constraint, we don't want - * to add another one; just adjust inheritance status as needed. + * to add another one; adjust inheritance status as needed. This + * also checks whether the existing constraint matches the + * requested validity. */ if (AdjustNotNullInheritance(RelationGetRelid(rel), colnum, - is_local, cdef->is_no_inherit)) + is_local, cdef->is_no_inherit, + cdef->skip_validation)) continue; /* diff --git a/src/backend/catalog/pg_constraint.c b/src/backend/catalog/pg_constraint.c index b97960d2766..2f73085961b 100644 --- a/src/backend/catalog/pg_constraint.c +++ b/src/backend/catalog/pg_constraint.c @@ -576,8 +576,8 @@ ChooseConstraintName(const char *name1, const char *name2, /* * Find and return a copy of the pg_constraint tuple that implements a - * validated not-null constraint for the given column of the given relation. - * If no such constraint exists, return NULL. + * (possibly not valid) not-null constraint for the given column of the + * given relation. If no such constraint exists, return NULL. * * XXX This would be easier if we had pg_attribute.notnullconstr with the OID * of the constraint that implements the not-null constraint for that column. @@ -606,13 +606,11 @@ findNotNullConstraintAttnum(Oid relid, AttrNumber attnum) AttrNumber conkey; /* - * We're looking for a NOTNULL constraint that's marked validated, - * with the column we're looking for as the sole element in conkey. + * We're looking for a NOTNULL constraint with the column we're + * looking for as the sole element in conkey. */ if (con->contype != CONSTRAINT_NOTNULL) continue; - if (!con->convalidated) - continue; conkey = extractNotNullColumn(conTup); if (conkey != attnum) @@ -630,9 +628,10 @@ findNotNullConstraintAttnum(Oid relid, AttrNumber attnum) } /* - * Find and return the pg_constraint tuple that implements a validated - * not-null constraint for the given column of the given relation. If - * no such column or no such constraint exists, return NULL. + * Find and return a copy of the pg_constraint tuple that implements a + * (possibly not valid) not-null constraint for the given column of the + * given relation. + * If no such column or no such constraint exists, return NULL. */ HeapTuple findNotNullConstraint(Oid relid, const char *colname) @@ -723,15 +722,19 @@ extractNotNullColumn(HeapTuple constrTup) * * If no not-null constraint is found for the column, return false. * Caller can create one. + * * If a constraint exists but the connoinherit flag is not what the caller - * wants, throw an error about the incompatibility. Otherwise, we adjust - * conislocal/coninhcount and return true. - * In the latter case, if is_local is true we flip conislocal true, or do - * nothing if it's already true; otherwise we increment coninhcount by 1. + * wants, throw an error about the incompatibility. If the desired + * constraint is valid but the existing constraint is not valid, also + * throw an error about that (the opposite case is acceptable). + * + * If everything checks out, we adjust conislocal/coninhcount and return + * true. If is_local is true we flip conislocal true, or do nothing if + * it's already true; otherwise we increment coninhcount by 1. */ bool AdjustNotNullInheritance(Oid relid, AttrNumber attnum, - bool is_local, bool is_no_inherit) + bool is_local, bool is_no_inherit, bool is_notvalid) { HeapTuple tup; @@ -755,6 +758,17 @@ AdjustNotNullInheritance(Oid relid, AttrNumber attnum, errmsg("cannot change NO INHERIT status of NOT NULL constraint \"%s\" on relation \"%s\"", NameStr(conform->conname), get_rel_name(relid))); + /* + * Throw an error if the existing constraint is NOT VALID and caller + * wants a valid one. + */ + if (!is_notvalid && !conform->convalidated) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("incompatible NOT VALID constraint \"%s\" on relation \"%s\"", + NameStr(conform->conname), get_rel_name(relid)), + errhint("You will need to use ALTER TABLE ... VALIDATE CONSTRAINT to validate it.")); + if (!is_local) { if (pg_add_s16_overflow(conform->coninhcount, 1, @@ -832,7 +846,7 @@ RelationGetNotNullConstraints(Oid relid, bool cooked, bool include_noinh) cooked->attnum = colnum; cooked->expr = NULL; cooked->is_enforced = true; - cooked->skip_validation = false; + cooked->skip_validation = !conForm->convalidated; cooked->is_local = true; cooked->inhcount = 0; cooked->is_no_inherit = conForm->connoinherit; @@ -852,7 +866,7 @@ RelationGetNotNullConstraints(Oid relid, bool cooked, bool include_noinh) constr->keys = list_make1(makeString(get_attname(relid, colnum, false))); constr->is_enforced = true; - constr->skip_validation = false; + constr->skip_validation = !conForm->convalidated; constr->initially_valid = true; constr->is_no_inherit = conForm->connoinherit; notnulls = lappend(notnulls, constr); diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 4397123398e..0c7b1231c2e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -435,6 +435,9 @@ static void QueueFKConstraintValidation(List **wqueue, Relation conrel, Relation static void QueueCheckConstraintValidation(List **wqueue, Relation conrel, Relation rel, char *constrName, HeapTuple contuple, bool recurse, bool recursing, LOCKMODE lockmode); +static void QueueNNConstraintValidation(List **wqueue, Relation conrel, Relation rel, + HeapTuple contuple, bool recurse, bool recursing, + LOCKMODE lockmode); static int transformColumnNameList(Oid relId, List *colList, int16 *attnums, Oid *atttypids, Oid *attcollids); static int transformFkeyGetPrimaryKey(Relation pkrel, Oid *indexOid, @@ -498,7 +501,7 @@ static void add_column_collation_dependency(Oid relid, int32 attnum, Oid collid) static ObjectAddress ATExecDropNotNull(Relation rel, const char *colName, bool recurse, LOCKMODE lockmode); static void set_attnotnull(List **wqueue, Relation rel, AttrNumber attnum, - LOCKMODE lockmode); + bool is_valid, bool queue_validation); static ObjectAddress ATExecSetNotNull(List **wqueue, Relation rel, char *constrname, char *colName, bool recurse, bool recursing, @@ -1340,7 +1343,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, nncols = AddRelationNotNullConstraints(rel, stmt->nnconstraints, old_notnulls); foreach_int(attrnum, nncols) - set_attnotnull(NULL, rel, attrnum, NoLock); + set_attnotnull(NULL, rel, attrnum, true, false); ObjectAddressSet(address, RelationRelationId, relationId); @@ -1424,7 +1427,7 @@ BuildDescForRelation(const List *columns) TupleDescInitEntryCollation(desc, attnum, attcollation); /* Fill in additional stuff not handled by TupleDescInitEntry */ - att->attnotnull = entry->is_not_null; + att->attnotnull = att->attnotnullvalid = entry->is_not_null; att->attislocal = entry->is_local; att->attinhcount = entry->inhcount; att->attidentity = entry->identity; @@ -2738,7 +2741,7 @@ MergeAttributes(List *columns, const List *supers, char relpersistence, /* * Request attnotnull on columns that have a not-null constraint - * that's not marked NO INHERIT. + * that's not marked NO INHERIT (even if not valid). */ nnconstrs = RelationGetNotNullConstraints(RelationGetRelid(relation), true, false); @@ -6207,18 +6210,22 @@ ATRewriteTable(AlteredTableInfo *tab, Oid OIDNewHeap) { /* * If we are rebuilding the tuples OR if we added any new but not - * verified not-null constraints, check all not-null constraints. This - * is a bit of overkill but it minimizes risk of bugs. + * verified not-null constraints, check all valid not-null constraints. + * This is a bit of overkill but it minimizes risk of bugs. * * notnull_attrs does *not* collect attribute numbers for not-null * constraints over virtual generated columns; instead, they are * collected in notnull_virtual_attrs. + * + * But we don't need check invalid not-null constraint! this is aligned + * with check constraint behavior. */ for (i = 0; i < newTupDesc->natts; i++) { Form_pg_attribute attr = TupleDescAttr(newTupDesc, i); - if (attr->attnotnull && !attr->attisdropped) + if (attr->attnotnull && attr->attnotnullvalid && + !attr->attisdropped) { if (attr->attgenerated != ATTRIBUTE_GENERATED_VIRTUAL) notnull_attrs = lappend_int(notnull_attrs, attr->attnum); @@ -7788,7 +7795,7 @@ ATExecDropNotNull(Relation rel, const char *colName, bool recurse, /* * Find the constraint that makes this column NOT NULL, and drop it. - * dropconstraint_internal() resets attnotnull. + * dropconstraint_internal() resets attnotnull/attnotnullvalid. */ conTup = findNotNullConstraintAttnum(RelationGetRelid(rel), attnum); if (conTup == NULL) @@ -7809,19 +7816,23 @@ ATExecDropNotNull(Relation rel, const char *colName, bool recurse, } /* - * Helper to set pg_attribute.attnotnull if it isn't set, and to tell phase 3 - * to verify it. + * set_attnotnull + * Helper to update/validate the pg_attribute status of a not-null + * constraint * - * When called to alter an existing table, 'wqueue' must be given so that we - * can queue a check that existing tuples pass the constraint. When called - * from table creation, 'wqueue' should be passed as NULL. + * pg_attribute.attnotnull is set true, if it isn't already. If is_valid + * is true, also set pg_attribute.attnotnullvalid. If queue_validation is + * true, also set up wqueue to validate the constraint. wqueue may be given + * as NULL when validation is not needed (e.g., on table creation). */ static void set_attnotnull(List **wqueue, Relation rel, AttrNumber attnum, - LOCKMODE lockmode) + bool is_valid, bool queue_validation) { Form_pg_attribute attr; + Assert(!queue_validation || wqueue); + CheckAlterTableIsSafe(rel); /* @@ -7832,7 +7843,7 @@ set_attnotnull(List **wqueue, Relation rel, AttrNumber attnum, if (attr->attisdropped) return; - if (!attr->attnotnull) + if (!attr->attnotnull || (is_valid && !attr->attnotnullvalid)) { Relation attr_rel; HeapTuple tuple; @@ -7845,15 +7856,17 @@ set_attnotnull(List **wqueue, Relation rel, AttrNumber attnum, attnum, RelationGetRelid(rel)); attr = (Form_pg_attribute) GETSTRUCT(tuple); - Assert(!attr->attnotnull); + attr->attnotnull = true; + attr->attnotnullvalid = is_valid; CatalogTupleUpdate(attr_rel, &tuple->t_self, tuple); /* * If the nullness isn't already proven by validated constraints, have * ALTER TABLE phase 3 test for it. */ - if (wqueue && !NotNullImpliedByRelConstraints(rel, attr)) + if (queue_validation && wqueue && + !NotNullImpliedByRelConstraints(rel, attr)) { AlteredTableInfo *tab; @@ -7951,6 +7964,15 @@ ATExecSetNotNull(List **wqueue, Relation rel, char *conName, char *colName, conForm->conislocal = true; changed = true; } + else if (!conForm->convalidated) + { + /* + * Flip attnotnull and convalidated, and also validate the + * constraint. + */ + return ATExecValidateConstraint(wqueue, rel, NameStr(conForm->conname), + recurse, recursing, lockmode); + } if (changed) { @@ -8013,8 +8035,8 @@ ATExecSetNotNull(List **wqueue, Relation rel, char *conName, char *colName, InvokeObjectPostAlterHook(RelationRelationId, RelationGetRelid(rel), attnum); - /* Mark pg_attribute.attnotnull for the column */ - set_attnotnull(wqueue, rel, attnum, lockmode); + /* Mark pg_attribute.attnotnull for the column and queue validation */ + set_attnotnull(wqueue, rel, attnum, true, true); /* * Recurse to propagate the constraint to children that don't have one. @@ -9417,7 +9439,6 @@ ATPrepAddPrimaryKey(List **wqueue, Relation rel, AlterTableCmd *cmd, bool recurse, LOCKMODE lockmode, AlterTableUtilityContext *context) { - ListCell *lc; Constraint *pkconstr; pkconstr = castNode(Constraint, cmd->def); @@ -9436,33 +9457,73 @@ ATPrepAddPrimaryKey(List **wqueue, Relation rel, AlterTableCmd *cmd, lockmode); foreach_oid(childrelid, children) { - foreach(lc, pkconstr->keys) + foreach_node(String, attname, pkconstr->keys) { HeapTuple tup; Form_pg_attribute attrForm; - char *attname = strVal(lfirst(lc)); - tup = SearchSysCacheAttName(childrelid, attname); + tup = SearchSysCacheAttName(childrelid, strVal(attname)); if (!tup) elog(ERROR, "cache lookup failed for attribute %s of relation %u", - attname, childrelid); + strVal(attname), childrelid); attrForm = (Form_pg_attribute) GETSTRUCT(tup); if (!attrForm->attnotnull) ereport(ERROR, errmsg("column \"%s\" of table \"%s\" is not marked NOT NULL", - attname, get_rel_name(childrelid))); + strVal(attname), get_rel_name(childrelid))); ReleaseSysCache(tup); } } } - /* Insert not-null constraints in the queue for the PK columns */ - foreach(lc, pkconstr->keys) + /* Verify that columns are not-null, or request that they be made so */ + foreach_node(String, column, pkconstr->keys) { AlterTableCmd *newcmd; Constraint *nnconstr; + HeapTuple tuple; - nnconstr = makeNotNullConstraint(lfirst(lc)); + /* + * First check if a suitable constraint exists. If it does, we don't + * need to request another one. We do need to bail out if it's not + * valid, though. + */ + tuple = findNotNullConstraint(RelationGetRelid(rel), strVal(column)); + if (tuple != NULL) + { + Form_pg_constraint conForm = (Form_pg_constraint) GETSTRUCT(tuple); + + /* a NO INHERIT constraint is no good */ + if (conForm->connoinherit) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot create primary key on column \"%s\"", + strVal(column)), + /*- translator: third %s is a constraint characteristic such as NOT VALID */ + errdetail("The constraint \"%s\" on column \"%s\", marked %s, is incompatible with a primary key.", + NameStr(conForm->conname), strVal(column), "NO INHERIT"), + errhint("You will need to make it inheritable using %s.", + "ALTER TABLE ... ALTER CONSTRAINT ... INHERIT")); + + /* an unvalidated constraint is no good */ + if (!conForm->convalidated) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot create primary key on column \"%s\"", + strVal(column)), + /*- translator: third %s is a constraint characteristic such as NOT VALID */ + errdetail("The constraint \"%s\" on column \"%s\", marked %s, is incompatible with a primary key.", + NameStr(conForm->conname), strVal(column), "NOT VALID"), + errhint("You will need to validate it using %s.", + "ALTER TABLE ... VALIDATE CONSTRAINT")); + + /* All good with this one; don't request another */ + heap_freetuple(tuple); + continue; + } + + /* This column is not already not-null, so add it to the queue */ + nnconstr = makeNotNullConstraint(column); newcmd = makeNode(AlterTableCmd); newcmd->subtype = AT_AddConstraint; @@ -9836,11 +9897,15 @@ ATAddCheckNNConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel, constr->conname = ccon->name; /* - * If adding a not-null constraint, set the pg_attribute flag and tell - * phase 3 to verify existing rows, if needed. + * If adding a valid not-null constraint, set the pg_attribute flag + * and tell phase 3 to verify existing rows, if needed. For an + * invalid constraint, just set attnotnull and attnotnullvalid, + * without queueing verification. */ if (constr->contype == CONSTR_NOTNULL) - set_attnotnull(wqueue, rel, ccon->attnum, lockmode); + set_attnotnull(wqueue, rel, ccon->attnum, + !constr->skip_validation, + !constr->skip_validation); ObjectAddressSet(address, ConstraintRelationId, ccon->conoid); } @@ -12811,10 +12876,11 @@ ATExecValidateConstraint(List **wqueue, Relation rel, char *constrName, con = (Form_pg_constraint) GETSTRUCT(tuple); if (con->contype != CONSTRAINT_FOREIGN && - con->contype != CONSTRAINT_CHECK) + con->contype != CONSTRAINT_CHECK && + con->contype != CONSTRAINT_NOTNULL) ereport(ERROR, (errcode(ERRCODE_WRONG_OBJECT_TYPE), - errmsg("constraint \"%s\" of relation \"%s\" is not a foreign key or check constraint", + errmsg("constraint \"%s\" of relation \"%s\" is not a foreign key, check, or not-null constraint", constrName, RelationGetRelationName(rel)))); if (!con->conenforced) @@ -12833,6 +12899,11 @@ ATExecValidateConstraint(List **wqueue, Relation rel, char *constrName, QueueCheckConstraintValidation(wqueue, conrel, rel, constrName, tuple, recurse, recursing, lockmode); } + else if (con->contype == CONSTRAINT_NOTNULL) + { + QueueNNConstraintValidation(wqueue, conrel, rel, + tuple, recurse, recursing, lockmode); + } ObjectAddressSet(address, ConstraintRelationId, con->oid); } @@ -13049,6 +13120,109 @@ QueueCheckConstraintValidation(List **wqueue, Relation conrel, Relation rel, heap_freetuple(copyTuple); } +/* + * QueueNNConstraintValidation + * + * Add an entry to the wqueue to validate the given not-null constraint in + * Phase 3 and update the convalidated field in the pg_constraint catalog for + * the specified relation and all its inheriting children. + */ +static void +QueueNNConstraintValidation(List **wqueue, Relation conrel, Relation rel, + HeapTuple contuple, bool recurse, bool recursing, + LOCKMODE lockmode) +{ + Form_pg_constraint con; + AlteredTableInfo *tab; + HeapTuple copyTuple; + Form_pg_constraint copy_con; + List *children = NIL; + AttrNumber attnum; + char *colname; + + con = (Form_pg_constraint) GETSTRUCT(contuple); + Assert(con->contype == CONSTRAINT_NOTNULL); + + attnum = extractNotNullColumn(contuple); + + /* + * If we're recursing, we've already done this for parent, so skip it. + * Also, if the constraint is a NO INHERIT constraint, we shouldn't try to + * look for it in the children. + * + * We recurse before validating on the parent, to reduce risk of + * deadlocks. + */ + if (!recursing && !con->connoinherit) + children = find_all_inheritors(RelationGetRelid(rel), lockmode, NULL); + + colname = get_attname(RelationGetRelid(rel), attnum, false); + foreach_oid(childoid, children) + { + Relation childrel; + HeapTuple contup; + Form_pg_constraint childcon; + char *conname; + + if (childoid == RelationGetRelid(rel)) + continue; + + /* + * If we are told not to recurse, there had better not be any child + * tables, because we can't mark the constraint on the parent valid + * unless it is valid for all child tables. + */ + if (!recurse) + ereport(ERROR, + errcode(ERRCODE_INVALID_TABLE_DEFINITION), + errmsg("constraint must be validated on child tables too")); + + /* + * The column on child might have a different attnum, so search by + * column name. + */ + contup = findNotNullConstraint(childoid, colname); + if (!contup) + elog(ERROR, "cache lookup failed for not-null constraint on column \"%s\" of relation \"%s\"", + colname, get_rel_name(childoid)); + childcon = (Form_pg_constraint) GETSTRUCT(contup); + if (childcon->convalidated) + continue; + + /* find_all_inheritors already got lock */ + childrel = table_open(childoid, NoLock); + conname = pstrdup(NameStr(childcon->conname)); + + /* XXX improve ATExecValidateConstraint API to avoid double search */ + ATExecValidateConstraint(wqueue, childrel, conname, + false, true, lockmode); + table_close(childrel, NoLock); + } + + /* Set the flags appropriately without queueing another validation */ + set_attnotnull(NULL, rel, attnum, true, false); + + tab = ATGetQueueEntry(wqueue, rel); + tab->verify_new_notnull = true; + + /* + * Invalidate relcache so that others see the new validated constraint. + */ + CacheInvalidateRelcache(rel); + + /* + * Now update the catalogs, while we have the door open. + */ + copyTuple = heap_copytuple(contuple); + copy_con = (Form_pg_constraint) GETSTRUCT(copyTuple); + copy_con->convalidated = true; + CatalogTupleUpdate(conrel, ©Tuple->t_self, copyTuple); + + InvokeObjectPostAlterHook(ConstraintRelationId, con->oid, 0); + + heap_freetuple(copyTuple); +} + /* * transformColumnNameList - transform list of column names * @@ -13917,10 +14091,11 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha false), RelationGetRelationName(rel))); - /* All good -- reset attnotnull if needed */ + /* All good -- reset attnotnull and attnotnullvalid if needed */ if (attForm->attnotnull) { attForm->attnotnull = false; + attForm->attnotnullvalid = false; CatalogTupleUpdate(attrel, &atttup->t_self, atttup); } @@ -19772,7 +19947,8 @@ PartConstraintImpliedByRelConstraint(Relation scanrel, { Form_pg_attribute att = TupleDescAttr(scanrel->rd_att, i - 1); - if (att->attnotnull && !att->attisdropped) + /* invalid not-null constraint must be ignored */ + if (att->attnotnull && att->attnotnullvalid && !att->attisdropped) { NullTest *ntest = makeNode(NullTest); diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index 2da848970be..c8d44e3086b 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -2074,6 +2074,7 @@ ExecConstraints(ResultRelInfo *resultRelInfo, { for (AttrNumber attnum = 1; attnum <= tupdesc->natts; attnum++) { + /* FIXME use CompactAttribute */ Form_pg_attribute att = TupleDescAttr(tupdesc, attnum - 1); if (att->attnotnull && att->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL) diff --git a/src/backend/jit/llvm/llvmjit_deform.c b/src/backend/jit/llvm/llvmjit_deform.c index 5d169c7a40b..c562edd094b 100644 --- a/src/backend/jit/llvm/llvmjit_deform.c +++ b/src/backend/jit/llvm/llvmjit_deform.c @@ -123,7 +123,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc, * combination of attisdropped && attnotnull combination shouldn't * exist. */ - if (att->attnotnull && + if (att->attnullability == ATTNULLABLE_VALID && !att->atthasmissing && !att->attisdropped) guaranteed_column_number = attnum; @@ -438,7 +438,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc, * into account, because if they're present the heaptuple's natts * would have indicated that a slot_getmissingattrs() is needed. */ - if (!att->attnotnull) + if (att->attnullability != ATTNULLABLE_VALID) { LLVMBasicBlockRef b_ifnotnull; LLVMBasicBlockRef b_ifnull; @@ -604,7 +604,8 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc, known_alignment = -1; attguaranteedalign = false; } - else if (att->attnotnull && attguaranteedalign && known_alignment >= 0) + else if (att->attnullability == ATTNULLABLE_VALID && + attguaranteedalign && known_alignment >= 0) { /* * If the offset to the column was previously known, a NOT NULL & @@ -614,7 +615,8 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc, Assert(att->attlen > 0); known_alignment += att->attlen; } - else if (att->attnotnull && (att->attlen % alignto) == 0) + else if (att->attnullability == ATTNULLABLE_VALID && + (att->attlen % alignto) == 0) { /* * After a NOT NULL fixed-width column with a length that is a diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index 67d879be8b8..191a14b876e 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -177,7 +177,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, { CompactAttribute *attr = TupleDescCompactAttr(relation->rd_att, i); - if (attr->attnotnull) + if (attr->attnullability == ATTNULLABLE_VALID) { rel->notnullattnums = bms_add_member(rel->notnullattnums, i + 1); @@ -1251,6 +1251,7 @@ get_relation_data_width(Oid relid, int32 *attr_widths) * get_relation_constraints * * Retrieve the applicable constraint expressions of the given relation. + * Only constraints that have been validated are considered. * * Returns a List (possibly empty) of constraint expressions. Each one * has been canonicalized, and its Vars are changed to have the varno @@ -1353,7 +1354,7 @@ get_relation_constraints(PlannerInfo *root, { Form_pg_attribute att = TupleDescAttr(relation->rd_att, i - 1); - if (att->attnotnull && !att->attisdropped) + if (att->attnotnull && att->attnotnullvalid && !att->attisdropped) { NullTest *ntest = makeNode(NullTest); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index f1156e2fca3..3c4268b271a 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -4217,11 +4217,10 @@ ConstraintElem: n->contype = CONSTR_NOTNULL; n->location = @1; n->keys = list_make1(makeString($3)); - /* no NOT VALID support yet */ processCASbits($4, @4, "NOT NULL", - NULL, NULL, NULL, NULL, + NULL, NULL, NULL, &n->skip_validation, &n->is_no_inherit, yyscanner); - n->initially_valid = true; + n->initially_valid = !n->skip_validation; $$ = (Node *) n; } | UNIQUE opt_unique_null_treatment '(' columnList opt_without_overlaps ')' opt_c_include opt_definition OptConsTableSpace diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 9ad7681f155..70c11529b90 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1142,6 +1142,7 @@ CatalogCacheInitializeCache(CatCache *cache) keytype = attr->atttypid; /* cache key columns should always be NOT NULL */ Assert(attr->attnotnull); + Assert(attr->attnotnullvalid); } else { diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index 18a14ae186e..c80e40cd47a 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -592,7 +592,7 @@ RelationBuildTupleDesc(Relation relation) /* Update constraint/default info */ if (attp->attnotnull) - constr->has_not_null = true; + constr->has_not_null = true; /* invalid ones included */ if (attp->attgenerated == ATTRIBUTE_GENERATED_STORED) constr->has_generated_stored = true; if (attp->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL) @@ -3573,6 +3573,7 @@ RelationBuildLocalRelation(const char *relname, datt->attidentity = satt->attidentity; datt->attgenerated = satt->attgenerated; datt->attnotnull = satt->attnotnull; + datt->attnotnullvalid = satt->attnotnullvalid; has_not_null |= satt->attnotnull; populate_compact_attribute(rel->rd_att, i); } diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 8e6364d32d7..17c2996faa7 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -350,8 +350,10 @@ static void buildMatViewRefreshDependencies(Archive *fout); static void getTableDataFKConstraints(void); static void determineNotNullFlags(Archive *fout, PGresult *res, int r, TableInfo *tbinfo, int j, - int i_notnull_name, int i_notnull_noinherit, - int i_notnull_islocal); + int i_notnull_name, int i_notnull_invalidoid, + int i_notnull_noinherit, + int i_notnull_islocal, + PQExpBuffer *invalidnotnulloids); static char *format_function_arguments(const FuncInfo *finfo, const char *funcargs, bool is_agg); static char *format_function_signature(Archive *fout, @@ -8984,6 +8986,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) PQExpBuffer q = createPQExpBuffer(); PQExpBuffer tbloids = createPQExpBuffer(); PQExpBuffer checkoids = createPQExpBuffer(); + PQExpBuffer invalidnotnulloids = NULL; PGresult *res; int ntups; int curtblindx; @@ -9003,6 +9006,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) int i_notnull_name; int i_notnull_noinherit; int i_notnull_islocal; + int i_notnull_invalidoid; int i_attoptions; int i_attcollation; int i_attcompression; @@ -9089,6 +9093,10 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) * attnotnull (this cues dumpTableSchema to print the NOT NULL clause * without a name); also, such cases are never NO INHERIT. * + * For invalid constraints, we need to store their OIDs for processing + * elsewhere, so we bring the pg_constraint.oid value when the constraint + * is invalid, and NULL otherwise. + * * We track in notnull_islocal whether the constraint was defined directly * in this table or via an ancestor, for binary upgrade. flagInhAttrs * might modify this later for servers older than 18; it's also in charge @@ -9097,11 +9105,14 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) if (fout->remoteVersion >= 180000) appendPQExpBufferStr(q, "co.conname AS notnull_name,\n" + "CASE WHEN NOT co.convalidated THEN co.oid " + "ELSE NULL END AS notnull_invalidoid,\n" "co.connoinherit AS notnull_noinherit,\n" "co.conislocal AS notnull_islocal,\n"); else appendPQExpBufferStr(q, "CASE WHEN a.attnotnull THEN '' ELSE NULL END AS notnull_name,\n" + "NULL AS notnull_invalidoid,\n" "false AS notnull_noinherit,\n" "a.attislocal AS notnull_islocal,\n"); @@ -9176,6 +9187,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) i_attalign = PQfnumber(res, "attalign"); i_attislocal = PQfnumber(res, "attislocal"); i_notnull_name = PQfnumber(res, "notnull_name"); + i_notnull_invalidoid = PQfnumber(res, "notnull_invalidoid"); i_notnull_noinherit = PQfnumber(res, "notnull_noinherit"); i_notnull_islocal = PQfnumber(res, "notnull_islocal"); i_attoptions = PQfnumber(res, "attoptions"); @@ -9272,8 +9284,11 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* Handle not-null constraint name and flags */ determineNotNullFlags(fout, res, r, tbinfo, j, - i_notnull_name, i_notnull_noinherit, - i_notnull_islocal); + i_notnull_name, + i_notnull_invalidoid, + i_notnull_noinherit, + i_notnull_islocal, + &invalidnotnulloids); tbinfo->attoptions[j] = pg_strdup(PQgetvalue(res, r, i_attoptions)); tbinfo->attcollation[j] = atooid(PQgetvalue(res, r, i_attcollation)); @@ -9294,6 +9309,10 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) } } + /* If invalidnotnulloids has any data, finalize it */ + if (invalidnotnulloids != NULL) + appendPQExpBufferChar(invalidnotnulloids, '}'); + PQclear(res); /* @@ -9426,6 +9445,103 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) PQclear(res); } + /* + * Get info about NOT NULL NOT VALID constraints. This is skipped for a + * data-only dump, as it is only needed for table schemas. + */ + if (dopt->dumpSchema && invalidnotnulloids) + { + ConstraintInfo *constrs; + int numConstrs; + int i_tableoid; + int i_oid; + int i_conrelid; + int i_conname; + int i_consrc; + int i_conislocal; + + pg_log_info("finding invalid not null constraints"); + + resetPQExpBuffer(q); + appendPQExpBuffer(q, + "SELECT c.tableoid, c.oid, conrelid, conname, " + "pg_catalog.pg_get_constraintdef(c.oid) AS consrc, " + "conislocal, convalidated " + "FROM unnest('%s'::pg_catalog.oid[]) AS src(conoid)\n" + "JOIN pg_catalog.pg_constraint c ON (src.conoid = c.oid)\n" + "ORDER BY c.conrelid, c.conname", + invalidnotnulloids->data); + + res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK); + + numConstrs = PQntuples(res); + constrs = (ConstraintInfo *) pg_malloc(numConstrs * sizeof(ConstraintInfo)); + + i_tableoid = PQfnumber(res, "tableoid"); + i_oid = PQfnumber(res, "oid"); + i_conrelid = PQfnumber(res, "conrelid"); + i_conname = PQfnumber(res, "conname"); + i_consrc = PQfnumber(res, "consrc"); + i_conislocal = PQfnumber(res, "conislocal"); + + /* As above, this loop iterates once per table, not once per row */ + curtblindx = -1; + for (int j = 0; j < numConstrs;) + { + Oid conrelid = atooid(PQgetvalue(res, j, i_conrelid)); + TableInfo *tbinfo = NULL; + int numcons; + + /* Count rows for this table */ + for (numcons = 1; numcons < numConstrs - j; numcons++) + if (atooid(PQgetvalue(res, j + numcons, i_conrelid)) != conrelid) + break; + + /* + * Locate the associated TableInfo; we rely on tblinfo[] being in + * OID order. + */ + while (++curtblindx < numTables) + { + tbinfo = &tblinfo[curtblindx]; + if (tbinfo->dobj.catId.oid == conrelid) + break; + } + if (curtblindx >= numTables) + pg_fatal("unrecognized table OID %u", conrelid); + + for (int c = 0; c < numcons; c++, j++) + { + constrs[j].dobj.objType = DO_CONSTRAINT; + constrs[j].dobj.catId.tableoid = atooid(PQgetvalue(res, j, i_tableoid)); + constrs[j].dobj.catId.oid = atooid(PQgetvalue(res, j, i_oid)); + AssignDumpId(&constrs[j].dobj); + constrs[j].dobj.name = pg_strdup(PQgetvalue(res, j, i_conname)); + constrs[j].dobj.namespace = tbinfo->dobj.namespace; + constrs[j].contable = tbinfo; + constrs[j].condomain = NULL; + constrs[j].contype = 'n'; + constrs[j].condef = pg_strdup(PQgetvalue(res, j, i_consrc)); + constrs[j].confrelid = InvalidOid; + constrs[j].conindex = 0; + constrs[j].condeferrable = false; + constrs[j].condeferred = false; + constrs[j].conislocal = (PQgetvalue(res, j, i_conislocal)[0] == 't'); + + /* + * All invalid not-null constraints must be dumped separately, + * because CREATE TABLE would not create them as invalid, and + * also because they must be created after potentially + * violating data has been loaded. + */ + constrs[j].separate = true; + + constrs[j].dobj.dump = tbinfo->dobj.dump; + } + } + PQclear(res); + } + /* * Get info about table CHECK constraints. This is skipped for a * data-only dump, as it is only needed for table schemas. @@ -9570,18 +9686,23 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) * * Result row 'r' is for tbinfo's attribute 'j'. * - * There are three possibilities: + * There are four possibilities: * 1) the column has no not-null constraints. In that case, ->notnull_constrs * (the constraint name) remains NULL. * 2) The column has a constraint with no name (this is the case when * constraints come from pre-18 servers). In this case, ->notnull_constrs * is set to the empty string; dumpTableSchema will print just "NOT NULL". - * 3) The column has a constraint with a known name; in that case + * 3) The column has an invalid not-null constraint. This must be treated + * as a separate object (because it must be created after the table data + * is loaded). So we add its OID to invalidnotnulloids for processing + * elsewhere and do nothing further with it here. We distinguish this + * case because the "notnull_invalidoid" column has been set to a non-NULL + * value, which is the constraint OID. Valid constraints have a null OID. + * 4) The column has a constraint with a known name; in that case * notnull_constrs carries that name and dumpTableSchema will print * "CONSTRAINT the_name NOT NULL". However, if the name is the default * (table_column_not_null), there's no need to print that name in the dump, - * so notnull_constrs is set to the empty string and it behaves as the case - * above. + * so notnull_constrs is set to the empty string and it behaves as case 2. * * In a child table that inherits from a parent already containing NOT NULL * constraints and the columns in the child don't have their own NOT NULL @@ -9593,7 +9714,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) * Any of these constraints might have the NO INHERIT bit. If so we set * ->notnull_noinh and NO INHERIT will be printed by dumpTableSchema. * - * In case 3 above, the name comparison is a bit of a hack; it actually fails + * In case 4 above, the name comparison is a bit of a hack; it actually fails * to do the right thing in all but the trivial case. However, the downside * of getting it wrong is simply that the name is printed rather than * suppressed, so it's not a big deal. @@ -9601,11 +9722,41 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) static void determineNotNullFlags(Archive *fout, PGresult *res, int r, TableInfo *tbinfo, int j, - int i_notnull_name, int i_notnull_noinherit, - int i_notnull_islocal) + int i_notnull_name, + int i_notnull_invalidoid, + int i_notnull_noinherit, + int i_notnull_islocal, + PQExpBuffer *invalidnotnulloids) { DumpOptions *dopt = fout->dopt; + /* + * If this not-null constraint is not valid, list its OID in + * invalidnotnulloids and do nothing further. It'll be processed + * elsewhere later. + * + * Because invalid not-null constraints are rare, we don't want to malloc + * invalidnotnulloids until we're sure we're going it need it, which + * happens here. + */ + if (!PQgetisnull(res, r, i_notnull_invalidoid)) + { + char *constroid = PQgetvalue(res, r, i_notnull_invalidoid); + + if (*invalidnotnulloids == NULL) + { + *invalidnotnulloids = createPQExpBuffer(); + appendPQExpBufferChar(*invalidnotnulloids, '{'); + appendPQExpBuffer(*invalidnotnulloids, "%s", constroid); + } + else + appendPQExpBuffer(*invalidnotnulloids, ",%s", constroid); + + /* nothing else to do */ + tbinfo->notnull_constrs[j] = NULL; + return; + } + /* * notnull_noinh is straight from the query result. notnull_islocal also, * though flagInhAttrs may change that one later in versions < 18. @@ -18152,13 +18303,20 @@ dumpConstraint(Archive *fout, const ConstraintInfo *coninfo) .createStmt = q->data, .dropStmt = delq->data)); } - else if (coninfo->contype == 'c' && tbinfo) + else if ((coninfo->contype == 'c' || coninfo->contype == 'n') && tbinfo) { - /* CHECK constraint on a table */ + /* CHECK or invalid not-null constraint on a table */ /* Ignore if not to be dumped separately, or if it was inherited */ if (coninfo->separate && coninfo->conislocal) { + const char *keyword; + + if (coninfo->contype == 'c') + keyword = "CHECK CONSTRAINT"; + else + keyword = "CONSTRAINT"; + /* not ONLY since we want it to propagate to children */ appendPQExpBuffer(q, "ALTER %sTABLE %s\n", foreign, fmtQualifiedDumpable(tbinfo)); @@ -18178,7 +18336,7 @@ dumpConstraint(Archive *fout, const ConstraintInfo *coninfo) ARCHIVE_OPTS(.tag = tag, .namespace = tbinfo->dobj.namespace->dobj.name, .owner = tbinfo->rolname, - .description = "CHECK CONSTRAINT", + .description = keyword, .section = SECTION_POST_DATA, .createStmt = q->data, .dropStmt = delq->data)); diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index e6f0f86a459..b426b5e4736 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -498,6 +498,8 @@ typedef struct _evttriggerInfo * use a different objType for foreign key constraints, to make it easier * to sort them the way we want. * + * Not-null constraints don't need this, unless they are NOT VALID. + * * Note: condeferrable and condeferred are currently only valid for * unique/primary-key constraints. Otherwise that info is in condef. */ diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 576326daec7..6c03eca8e50 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -744,8 +744,8 @@ my %pgdump_runs = ( schema_only_with_statistics => { dump_cmd => [ 'pg_dump', '--no-sync', - "--file=$tempdir/schema_only_with_statistics.sql", '--schema-only', - '--with-statistics', 'postgres', + "--file=$tempdir/schema_only_with_statistics.sql", + '--schema-only', '--with-statistics', 'postgres', ], }, no_schema => { @@ -1118,6 +1118,23 @@ my %tests = ( }, }, + 'CONSTRAINT NOT NULL / INVALID' => { + create_sql => 'CREATE TABLE dump_test.test_table_nn ( + col1 int); + ALTER TABLE dump_test.test_table_nn ADD CONSTRAINT nn NOT NULL col1 NOT VALID;', + regexp => qr/^ + \QALTER TABLE dump_test.test_table_nn\E \n^\s+ + \QADD CONSTRAINT nn NOT NULL col1 NOT VALID;\E + /xm, + like => { + %full_runs, %dump_test_schema_runs, section_post_data => 1, + }, + unlike => { + exclude_dump_test_schema => 1, + only_dump_measurement => 1, + }, + }, + 'CONSTRAINT PRIMARY KEY / WITHOUT OVERLAPS' => { create_sql => 'CREATE TABLE dump_test.test_table_tpk ( col1 int4range, diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 8970677ac64..1d08268393e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -3106,7 +3106,8 @@ describeOneTableDetails(const char *schemaname, { printfPQExpBuffer(&buf, "SELECT c.conname, a.attname, c.connoinherit,\n" - " c.conislocal, c.coninhcount <> 0\n" + " c.conislocal, c.coninhcount <> 0,\n" + " c.convalidated\n" "FROM pg_catalog.pg_constraint c JOIN\n" " pg_catalog.pg_attribute a ON\n" " (a.attrelid = c.conrelid AND a.attnum = c.conkey[1])\n" @@ -3129,14 +3130,16 @@ describeOneTableDetails(const char *schemaname, { bool islocal = PQgetvalue(result, i, 3)[0] == 't'; bool inherited = PQgetvalue(result, i, 4)[0] == 't'; + bool validated = PQgetvalue(result, i, 5)[0] == 't'; - printfPQExpBuffer(&buf, " \"%s\" NOT NULL \"%s\"%s", + printfPQExpBuffer(&buf, " \"%s\" NOT NULL \"%s\"%s%s", PQgetvalue(result, i, 0), PQgetvalue(result, i, 1), PQgetvalue(result, i, 2)[0] == 't' ? " NO INHERIT" : islocal && inherited ? _(" (local, inherited)") : - inherited ? _(" (inherited)") : ""); + inherited ? _(" (inherited)") : "", + !validated ? " NOT VALID" : ""); printTableAddFooter(&cont, buf.data); } diff --git a/src/include/access/tupdesc.h b/src/include/access/tupdesc.h index 396eeb7a0bb..5523cfcf5aa 100644 --- a/src/include/access/tupdesc.h +++ b/src/include/access/tupdesc.h @@ -42,7 +42,7 @@ typedef struct TupleConstr struct AttrMissing *missing; /* missing attributes values, NULL if none */ uint16 num_defval; uint16 num_check; - bool has_not_null; + bool has_not_null; /* any not-null, including not valid ones */ bool has_generated_stored; bool has_generated_virtual; } TupleConstr; @@ -76,10 +76,15 @@ typedef struct CompactAttribute bool atthasmissing; /* as FormData_pg_attribute.atthasmissing */ bool attisdropped; /* as FormData_pg_attribute.attisdropped */ bool attgenerated; /* FormData_pg_attribute.attgenerated != '\0' */ - bool attnotnull; /* as FormData_pg_attribute.attnotnull */ + char attnullability; /* attnotnull + attnotnullvalid */ uint8 attalignby; /* alignment requirement in bytes */ } CompactAttribute; +/* Valid values for CompactAttribute->attnullability */ +#define ATTNULLABLE_VALID 'v' /* valid constraint exists */ +#define ATTNULLABLE_INVALID 'i' /* constraint exists, marked invalid */ +#define ATTNULLABLE_NONE 'f' /* no constraint exists */ + /* * This struct is passed around within the backend to describe the structure * of tuples. For tuples coming from on-disk relations, the information is diff --git a/src/include/catalog/catversion.h b/src/include/catalog/catversion.h index 208936962ef..b573a0a2828 100644 --- a/src/include/catalog/catversion.h +++ b/src/include/catalog/catversion.h @@ -57,6 +57,6 @@ */ /* yyyymmddN */ -#define CATALOG_VERSION_NO 202504041 +#define CATALOG_VERSION_NO 202504051 #endif diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index deaa515fe53..9b1236e7a90 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -117,9 +117,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, */ char attcompression BKI_DEFAULT('\0'); - /* This flag represents the "NOT NULL" constraint */ + /* Whether a not-null constraint exists for the column */ bool attnotnull; + /* Whether the not-null constraint, if it exists, is valid */ + bool attnotnullvalid; + /* Has DEFAULT value or not */ bool atthasdef BKI_DEFAULT(f); diff --git a/src/include/catalog/pg_constraint.h b/src/include/catalog/pg_constraint.h index 6da164e7e4d..4afceb5c692 100644 --- a/src/include/catalog/pg_constraint.h +++ b/src/include/catalog/pg_constraint.h @@ -264,7 +264,7 @@ extern HeapTuple findNotNullConstraint(Oid relid, const char *colname); extern HeapTuple findDomainNotNullConstraint(Oid typid); extern AttrNumber extractNotNullColumn(HeapTuple constrTup); extern bool AdjustNotNullInheritance(Oid relid, AttrNumber attnum, - bool is_local, bool is_no_inherit); + bool is_local, bool is_no_inherit, bool is_notvalid); extern List *RelationGetNotNullConstraints(Oid relid, bool cooked, bool include_noinh); diff --git a/src/test/regress/expected/alter_table.out b/src/test/regress/expected/alter_table.out index 362f38856d2..8a44321034b 100644 --- a/src/test/regress/expected/alter_table.out +++ b/src/test/regress/expected/alter_table.out @@ -1202,6 +1202,75 @@ alter table atacc1 alter test_a drop not null, alter test_b drop not null; alter table atacc1 add constraint atacc1_constr_b_valid check(test_b is not null); alter table atacc1 alter test_b set not null, alter test_a set not null; drop table atacc1; +-- not null not valid with partitions +CREATE TABLE atnnparted (id int, col1 int) PARTITION BY LIST (id); +ALTER TABLE atnnparted ADD CONSTRAINT dummy_constr NOT NULL id NOT VALID; +CREATE TABLE atnnpart1 (col1 int, id int); +ALTER TABLE atnnpart1 ADD CONSTRAINT another_constr NOT NULL id; +ALTER TABLE atnnpart1 ADD PRIMARY KEY (id); +ALTER TABLE atnnparted ATTACH PARTITION atnnpart1 FOR VALUES IN ('1'); +\d+ atnnpart* + Table "public.atnnpart1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+---------+-----------+----------+---------+---------+--------------+------------- + col1 | integer | | | | plain | | + id | integer | | not null | | plain | | +Partition of: atnnparted FOR VALUES IN (1) +Partition constraint: ((id IS NOT NULL) AND (id = 1)) +Indexes: + "atnnpart1_pkey" PRIMARY KEY, btree (id) +Not-null constraints: + "another_constr" NOT NULL "id" (inherited) + + Index "public.atnnpart1_pkey" + Column | Type | Key? | Definition | Storage | Stats target +--------+---------+------+------------+---------+-------------- + id | integer | yes | id | plain | +primary key, btree, for table "public.atnnpart1" + + Partitioned table "public.atnnparted" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+---------+-----------+----------+---------+---------+--------------+------------- + id | integer | | not null | | plain | | + col1 | integer | | | | plain | | +Partition key: LIST (id) +Not-null constraints: + "dummy_constr" NOT NULL "id" NOT VALID +Partitions: atnnpart1 FOR VALUES IN (1) + +BEGIN; +ALTER TABLE atnnparted VALIDATE CONSTRAINT dummy_constr; +\d+ atnnpart* + Table "public.atnnpart1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+---------+-----------+----------+---------+---------+--------------+------------- + col1 | integer | | | | plain | | + id | integer | | not null | | plain | | +Partition of: atnnparted FOR VALUES IN (1) +Partition constraint: ((id IS NOT NULL) AND (id = 1)) +Indexes: + "atnnpart1_pkey" PRIMARY KEY, btree (id) +Not-null constraints: + "another_constr" NOT NULL "id" (inherited) + + Index "public.atnnpart1_pkey" + Column | Type | Key? | Definition | Storage | Stats target +--------+---------+------+------------+---------+-------------- + id | integer | yes | id | plain | +primary key, btree, for table "public.atnnpart1" + + Partitioned table "public.atnnparted" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+---------+-----------+----------+---------+---------+--------------+------------- + id | integer | | not null | | plain | | + col1 | integer | | | | plain | | +Partition key: LIST (id) +Not-null constraints: + "dummy_constr" NOT NULL "id" +Partitions: atnnpart1 FOR VALUES IN (1) + +ROLLBACK; +-- leave a table in this state for the pg_upgrade test -- test inheritance create table parent (a int); create table child (b varchar(255)) inherits (parent); diff --git a/src/test/regress/expected/constraints.out b/src/test/regress/expected/constraints.out index a719d2f74e9..d8547a9bc81 100644 --- a/src/test/regress/expected/constraints.out +++ b/src/test/regress/expected/constraints.out @@ -1229,6 +1229,13 @@ alter table cnn_uq add unique using index cnn_uq_idx; Indexes: "cnn_uq_idx" UNIQUE CONSTRAINT, btree (a) +-- can't create a primary key on a noinherit not-null +create table cnn_pk (a int not null no inherit); +alter table cnn_pk add primary key (a); +ERROR: cannot create primary key on column "a" +DETAIL: The constraint "cnn_pk_a_not_null" on column "a", marked NO INHERIT, is incompatible with a primary key. +HINT: You will need to make it inheritable using ALTER TABLE ... ALTER CONSTRAINT ... INHERIT. +drop table cnn_pk; -- Ensure partitions are scanned for null values when adding a PK create table cnn2_parted(a int) partition by list (a); create table cnn_part1 partition of cnn2_parted for values in (1, null); @@ -1355,6 +1362,249 @@ Not-null constraints: "ann" NOT NULL "a" "bnn" NOT NULL "b" +-- NOT NULL NOT VALID +PREPARE get_nnconstraint_info(regclass[]) AS +SELECT conrelid::regclass as tabname, conname, convalidated, conislocal, coninhcount +FROM pg_constraint +WHERE conrelid = ANY($1) +ORDER BY conrelid::regclass::text, conname; +CREATE TABLE notnull_tbl1 (a int, b int); +INSERT INTO notnull_tbl1 VALUES (NULL, 1), (300, 3); +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nn NOT NULL a; -- error +ERROR: column "a" of relation "notnull_tbl1" contains null values +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nn NOT NULL a NOT VALID; -- ok +-- even an invalid not-null forbids new nulls +INSERT INTO notnull_tbl1 VALUES (NULL, 4); +ERROR: null value in column "a" of relation "notnull_tbl1" violates not-null constraint +DETAIL: Failing row contains (null, 4). +\d+ notnull_tbl1 + Table "public.notnull_tbl1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+---------+-----------+----------+---------+---------+--------------+------------- + a | integer | | not null | | plain | | + b | integer | | | | plain | | +Not-null constraints: + "nn" NOT NULL "a" NOT VALID + +-- If we have an invalid constraint, we can't have another +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nn1 NOT NULL a NOT VALID NO INHERIT; +ERROR: cannot change NO INHERIT status of NOT NULL constraint "nn" on relation "notnull_tbl1" +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nn NOT NULL a; +ERROR: incompatible NOT VALID constraint "nn" on relation "notnull_tbl1" +HINT: You will need to use ALTER TABLE ... VALIDATE CONSTRAINT to validate it. +-- cannot add primary key on a column with an invalid not-null +ALTER TABLE notnull_tbl1 ADD PRIMARY KEY (a); +ERROR: cannot create primary key on column "a" +DETAIL: The constraint "nn" on column "a", marked NOT VALID, is incompatible with a primary key. +HINT: You will need to validate it using ALTER TABLE ... VALIDATE CONSTRAINT. +-- ALTER column SET NOT NULL validates an invalid constraint (but this fails +-- because of rows with null values) +ALTER TABLE notnull_tbl1 ALTER a SET NOT NULL; +ERROR: column "a" of relation "notnull_tbl1" contains null values +\d+ notnull_tbl1 + Table "public.notnull_tbl1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+---------+-----------+----------+---------+---------+--------------+------------- + a | integer | | not null | | plain | | + b | integer | | | | plain | | +Not-null constraints: + "nn" NOT NULL "a" NOT VALID + +-- Creating a derived table using LIKE gets the constraint, but it's valid +CREATE TABLE notnull_tbl1_copy (LIKE notnull_tbl1); +EXECUTE get_nnconstraint_info('{notnull_tbl1_copy}'); + tabname | conname | convalidated | conislocal | coninhcount +-------------------+---------+--------------+------------+------------- + notnull_tbl1_copy | nn | t | t | 0 +(1 row) + +-- An inheritance child table gets the constraint, but it's valid +CREATE TABLE notnull_tbl1_child (a int, b int) INHERITS (notnull_tbl1); +NOTICE: merging column "a" with inherited definition +NOTICE: merging column "b" with inherited definition +EXECUTE get_nnconstraint_info('{notnull_tbl1_child, notnull_tbl1}'); + tabname | conname | convalidated | conislocal | coninhcount +--------------------+---------+--------------+------------+------------- + notnull_tbl1 | nn | f | t | 0 + notnull_tbl1_child | nn | t | f | 1 +(2 rows) + +-- Also try inheritance added after table creation +CREATE TABLE notnull_tbl1_child2 (c int, b int, a int); +ALTER TABLE notnull_tbl1_child2 INHERIT notnull_tbl1; -- nope +ERROR: column "a" in child table "notnull_tbl1_child2" must be marked NOT NULL +ALTER TABLE notnull_tbl1_child2 ADD NOT NULL a NOT VALID; +ALTER TABLE notnull_tbl1_child2 INHERIT notnull_tbl1; +EXECUTE get_nnconstraint_info('{notnull_tbl1_child2}'); + tabname | conname | convalidated | conislocal | coninhcount +---------------------+--------------------------------+--------------+------------+------------- + notnull_tbl1_child2 | notnull_tbl1_child2_a_not_null | f | t | 1 +(1 row) + +-- VALIDATE CONSTRAINT scans the table +ALTER TABLE notnull_tbl1 VALIDATE CONSTRAINT nn; -- error, nulls exist +ERROR: column "a" of relation "notnull_tbl1" contains null values +UPDATE notnull_tbl1 SET a = 100 WHERE b = 1; +ALTER TABLE notnull_tbl1 VALIDATE CONSTRAINT nn; -- now ok +EXECUTE get_nnconstraint_info('{notnull_tbl1}'); + tabname | conname | convalidated | conislocal | coninhcount +--------------+---------+--------------+------------+------------- + notnull_tbl1 | nn | t | t | 0 +(1 row) + +--- now we can add primary key +ALTER TABLE notnull_tbl1 ADD PRIMARY KEY (a); +DROP TABLE notnull_tbl1, notnull_tbl1_child, notnull_tbl1_child2; +-- dropping an invalid constraint is possible +CREATE TABLE notnull_tbl1 (a int, b int); +ALTER TABLE notnull_tbl1 ADD NOT NULL a NOT VALID, + ADD NOT NULL b NOT VALID; +ALTER TABLE notnull_tbl1 ALTER a DROP NOT NULL; +ALTER TABLE notnull_tbl1 DROP CONSTRAINT notnull_tbl1_b_not_null; +DROP TABLE notnull_tbl1; +-- ALTER .. NO INHERIT works for invalid constraints +CREATE TABLE notnull_tbl1 (a int); +CREATE TABLE notnull_tbl1_chld () INHERITS (notnull_tbl1); +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nntbl1_a NOT NULL a NOT VALID; +ALTER TABLE notnull_tbl1 ALTER CONSTRAINT nntbl1_a NO INHERIT; +-- DROP CONSTRAINT recurses correctly on invalid constraints +ALTER TABLE notnull_tbl1 ALTER CONSTRAINT nntbl1_a INHERIT; +ALTER TABLE notnull_tbl1 DROP CONSTRAINT nntbl1_a; +DROP TABLE notnull_tbl1, notnull_tbl1_chld; +-- if a parent has a valid not null constraint then a child table cannot +-- have an invalid one +CREATE TABLE notnull_tbl1 (a int); +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nn_parent NOT NULL a not valid; +CREATE TABLE notnull_chld0 (a int, CONSTRAINT nn_chld0 NOT NULL a); +ALTER TABLE notnull_tbl1 INHERIT notnull_chld0; --error +ERROR: constraint "nn_parent" conflicts with NOT VALID constraint on child table "notnull_tbl1" +ALTER TABLE notnull_chld0 DROP CONSTRAINT nn_chld0; +ALTER TABLE notnull_chld0 ADD CONSTRAINT nn_chld0 NOT NULL a not valid; +ALTER TABLE notnull_tbl1 INHERIT notnull_chld0; --now ok +-- parents and child not-null will all be validated. +ALTER TABLE notnull_tbl1 VALIDATE CONSTRAINT nn_parent; +EXECUTE get_nnconstraint_info('{notnull_tbl1, notnull_chld0}'); + tabname | conname | convalidated | conislocal | coninhcount +---------------+-----------+--------------+------------+------------- + notnull_chld0 | nn_chld0 | f | t | 0 + notnull_tbl1 | nn_parent | t | t | 1 +(2 rows) + +DROP TABLE notnull_tbl1, notnull_chld0; +-- Test invalid not null on inheritance table. +CREATE TABLE notnull_inhparent (i int); +CREATE TABLE notnull_inhchild (i int) INHERITS (notnull_inhparent); +NOTICE: merging column "i" with inherited definition +CREATE TABLE notnull_inhgrand () INHERITS (notnull_inhparent, notnull_inhchild); +NOTICE: merging multiple inherited definitions of column "i" +ALTER TABLE notnull_inhparent ADD CONSTRAINT nn NOT NULL i NOT VALID; +ALTER TABLE notnull_inhchild ADD CONSTRAINT nn1 NOT NULL i; -- error +ERROR: incompatible NOT VALID constraint "nn" on relation "notnull_inhchild" +HINT: You will need to use ALTER TABLE ... VALIDATE CONSTRAINT to validate it. +EXECUTE get_nnconstraint_info('{notnull_inhparent, notnull_inhchild, notnull_inhgrand}'); + tabname | conname | convalidated | conislocal | coninhcount +-------------------+---------+--------------+------------+------------- + notnull_inhchild | nn | f | f | 1 + notnull_inhgrand | nn | f | f | 2 + notnull_inhparent | nn | f | t | 0 +(3 rows) + +ALTER TABLE notnull_inhparent ALTER i SET NOT NULL; -- ok +EXECUTE get_nnconstraint_info('{notnull_inhparent, notnull_inhchild, notnull_inhgrand}'); + tabname | conname | convalidated | conislocal | coninhcount +-------------------+---------+--------------+------------+------------- + notnull_inhchild | nn | t | f | 1 + notnull_inhgrand | nn | t | f | 2 + notnull_inhparent | nn | t | t | 0 +(3 rows) + +DROP TABLE notnull_inhparent, notnull_inhchild, notnull_inhgrand; +-- Verify NOT NULL VALID/NOT VALID with partition table. +DROP TABLE notnull_tbl1; +ERROR: table "notnull_tbl1" does not exist +CREATE TABLE notnull_tbl1 (a int, b int) PARTITION BY LIST (a); +ALTER TABLE notnull_tbl1 ADD CONSTRAINT notnull_con NOT NULL a NOT VALID; --ok +CREATE TABLE notnull_tbl1_1 PARTITION OF notnull_tbl1 FOR VALUES IN (1,2); +CREATE TABLE notnull_tbl1_2(a int, CONSTRAINT nn2 NOT NULL a, b int); +ALTER TABLE notnull_tbl1 ATTACH PARTITION notnull_tbl1_2 FOR VALUES IN (3,4); +CREATE TABLE notnull_tbl1_3(a int, b int); +INSERT INTO notnull_tbl1_3 values(NULL,1); +ALTER TABLE notnull_tbl1_3 add CONSTRAINT nn3 NOT NULL a NOT VALID; +ALTER TABLE notnull_tbl1 ATTACH PARTITION notnull_tbl1_3 FOR VALUES IN (NULL,5); +EXECUTE get_nnconstraint_info('{notnull_tbl1, notnull_tbl1_1, notnull_tbl1_2, notnull_tbl1_3}'); + tabname | conname | convalidated | conislocal | coninhcount +----------------+-------------+--------------+------------+------------- + notnull_tbl1 | notnull_con | f | t | 0 + notnull_tbl1_1 | notnull_con | t | f | 1 + notnull_tbl1_2 | nn2 | t | f | 1 + notnull_tbl1_3 | nn3 | f | f | 1 +(4 rows) + +ALTER TABLE notnull_tbl1 ALTER COLUMN a SET NOT NULL; --error, notnull_tbl1_3 have null values +ERROR: column "a" of relation "notnull_tbl1_3" contains null values +ALTER TABLE notnull_tbl1_3 VALIDATE CONSTRAINT nn3; --error +ERROR: column "a" of relation "notnull_tbl1_3" contains null values +TRUNCATE notnull_tbl1; +ALTER TABLE notnull_tbl1 ALTER COLUMN a SET NOT NULL; --OK +EXECUTE get_nnconstraint_info('{notnull_tbl1, notnull_tbl1_1, notnull_tbl1_2, notnull_tbl1_3}'); + tabname | conname | convalidated | conislocal | coninhcount +----------------+-------------+--------------+------------+------------- + notnull_tbl1 | notnull_con | t | t | 0 + notnull_tbl1_1 | notnull_con | t | f | 1 + notnull_tbl1_2 | nn2 | t | f | 1 + notnull_tbl1_3 | nn3 | t | f | 1 +(4 rows) + +DROP TABLE notnull_tbl1; +-- partitioned table have not-null, then the partitions can not be NOT NULL NOT VALID. +CREATE TABLE pp_nn (a int, b int, NOT NULL a) PARTITION BY LIST (a); +CREATE TABLE pp_nn_1(a int, b int); +ALTER TABLE pp_nn_1 ADD CONSTRAINT nn1 NOT NULL a NOT VALID; +ALTER TABLE pp_nn ATTACH PARTITION pp_nn_1 FOR VALUES IN (NULL,5); --error +ERROR: constraint "nn1" conflicts with NOT VALID constraint on child table "pp_nn_1" +ALTER TABLE pp_nn_1 VALIDATE CONSTRAINT nn1; +ALTER TABLE pp_nn ATTACH PARTITION pp_nn_1 FOR VALUES IN (NULL,5); --ok +DROP TABLE pp_nn; +-- Create table with NOT NULL INVALID constraint, for pg_upgrade. +CREATE TABLE notnull_tbl1_upg (a int, b int); +INSERT INTO notnull_tbl1_upg VALUES (NULL, 1), (NULL, 2), (300, 3); +ALTER TABLE notnull_tbl1_upg ADD CONSTRAINT nn NOT NULL a NOT VALID; +-- Inherit test for pg_upgrade +CREATE TABLE notnull_parent_upg (a int); +CREATE TABLE notnull_child_upg () INHERITS (notnull_parent_upg); +ALTER TABLE notnull_child_upg ADD CONSTRAINT nn NOT NULL a; +ALTER TABLE notnull_parent_upg ADD CONSTRAINT nn NOT NULL a NOT VALID; +SELECT conrelid::regclass, contype, convalidated, conislocal +FROM pg_catalog.pg_constraint +WHERE conrelid in ('notnull_parent_upg'::regclass, 'notnull_child_upg'::regclass) +ORDER BY 1; + conrelid | contype | convalidated | conislocal +--------------------+---------+--------------+------------ + notnull_parent_upg | n | f | t + notnull_child_upg | n | t | t +(2 rows) + +-- Partition table test, for pg_upgrade +CREATE TABLE notnull_part1_upg (a int, b int) PARTITION BY LIST (a); +ALTER TABLE notnull_part1_upg ADD CONSTRAINT notnull_con NOT NULL a NOT VALID; --ok +CREATE TABLE notnull_part1_1_upg PARTITION OF notnull_part1_upg FOR VALUES IN (1,2); +CREATE TABLE notnull_part1_2_upg (a int, CONSTRAINT nn2 NOT NULL a, b int); +ALTER TABLE notnull_part1_upg ATTACH PARTITION notnull_part1_2_upg FOR VALUES IN (3,4); +CREATE TABLE notnull_part1_3_upg (a int, b int); +INSERT INTO notnull_part1_3_upg values(NULL,1); +ALTER TABLE notnull_part1_3_upg add CONSTRAINT nn3 NOT NULL a NOT VALID; +ALTER TABLE notnull_part1_upg ATTACH PARTITION notnull_part1_3_upg FOR VALUES IN (NULL,5); +EXECUTE get_nnconstraint_info('{notnull_part1_upg, notnull_part1_1_upg, notnull_part1_2_upg, notnull_part1_3_upg}'); + tabname | conname | convalidated | conislocal | coninhcount +---------------------+-------------+--------------+------------+------------- + notnull_part1_1_upg | notnull_con | t | f | 1 + notnull_part1_2_upg | nn2 | t | f | 1 + notnull_part1_3_upg | nn3 | f | f | 1 + notnull_part1_upg | notnull_con | f | t | 0 +(4 rows) + +DEALLOCATE get_nnconstraint_info; +-- end NOT NULL NOT VALID -- Comments -- Setup a low-level role to enforce non-superuser checks. CREATE ROLE regress_constraint_comments; diff --git a/src/test/regress/expected/sanity_check.out b/src/test/regress/expected/sanity_check.out index 8370c1561cc..2e39037e3fb 100644 --- a/src/test/regress/expected/sanity_check.out +++ b/src/test/regress/expected/sanity_check.out @@ -25,3 +25,23 @@ SELECT relname, relkind ---------+--------- (0 rows) +-- pg_attribute sanity check: attnotnullvalid can only be true when +-- attnotnull is valid. +select pa.attrelid::regclass, pa.attname, pa.attnum, + pa.attnotnull, pa.attnotnullvalid +from pg_attribute pa +where not pa.attnotnull and pa.attnotnullvalid; + attrelid | attname | attnum | attnotnull | attnotnullvalid +----------+---------+--------+------------+----------------- +(0 rows) + +-- pg_attribute's attnotnullvalid should match pg_constraint.convalidated +select att.attrelid::regclass, att.attname, att.attnotnull, + att.attnotnullvalid, con.convalidated + from pg_attribute att join pg_constraint con + on (att.attnum = con.conkey[1] and att.attrelid = con.conrelid) + where contype = 'n' and con.convalidated <> att.attnotnullvalid; + attrelid | attname | attnotnull | attnotnullvalid | convalidated +----------+---------+------------+-----------------+-------------- +(0 rows) + diff --git a/src/test/regress/sql/alter_table.sql b/src/test/regress/sql/alter_table.sql index 84e93ef575e..8432e8e3d54 100644 --- a/src/test/regress/sql/alter_table.sql +++ b/src/test/regress/sql/alter_table.sql @@ -910,6 +910,20 @@ alter table atacc1 add constraint atacc1_constr_b_valid check(test_b is not null alter table atacc1 alter test_b set not null, alter test_a set not null; drop table atacc1; +-- not null not valid with partitions +CREATE TABLE atnnparted (id int, col1 int) PARTITION BY LIST (id); +ALTER TABLE atnnparted ADD CONSTRAINT dummy_constr NOT NULL id NOT VALID; +CREATE TABLE atnnpart1 (col1 int, id int); +ALTER TABLE atnnpart1 ADD CONSTRAINT another_constr NOT NULL id; +ALTER TABLE atnnpart1 ADD PRIMARY KEY (id); +ALTER TABLE atnnparted ATTACH PARTITION atnnpart1 FOR VALUES IN ('1'); +\d+ atnnpart* +BEGIN; +ALTER TABLE atnnparted VALIDATE CONSTRAINT dummy_constr; +\d+ atnnpart* +ROLLBACK; +-- leave a table in this state for the pg_upgrade test + -- test inheritance create table parent (a int); create table child (b varchar(255)) inherits (parent); diff --git a/src/test/regress/sql/constraints.sql b/src/test/regress/sql/constraints.sql index 21ce4177de4..8f199c7ba34 100644 --- a/src/test/regress/sql/constraints.sql +++ b/src/test/regress/sql/constraints.sql @@ -761,6 +761,11 @@ create unique index cnn_uq_idx on cnn_uq (a); alter table cnn_uq add unique using index cnn_uq_idx; \d+ cnn_uq +-- can't create a primary key on a noinherit not-null +create table cnn_pk (a int not null no inherit); +alter table cnn_pk add primary key (a); +drop table cnn_pk; + -- Ensure partitions are scanned for null values when adding a PK create table cnn2_parted(a int) partition by list (a); create table cnn_part1 partition of cnn2_parted for values in (1, null); @@ -801,6 +806,167 @@ ALTER TABLE ONLY notnull_tbl6 DROP CONSTRAINT ann; ALTER TABLE ONLY notnull_tbl6 ALTER b DROP NOT NULL; \d+ notnull_tbl6_1 + +-- NOT NULL NOT VALID +PREPARE get_nnconstraint_info(regclass[]) AS +SELECT conrelid::regclass as tabname, conname, convalidated, conislocal, coninhcount +FROM pg_constraint +WHERE conrelid = ANY($1) +ORDER BY conrelid::regclass::text, conname; + +CREATE TABLE notnull_tbl1 (a int, b int); +INSERT INTO notnull_tbl1 VALUES (NULL, 1), (300, 3); +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nn NOT NULL a; -- error +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nn NOT NULL a NOT VALID; -- ok +-- even an invalid not-null forbids new nulls +INSERT INTO notnull_tbl1 VALUES (NULL, 4); +\d+ notnull_tbl1 + +-- If we have an invalid constraint, we can't have another +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nn1 NOT NULL a NOT VALID NO INHERIT; +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nn NOT NULL a; + +-- cannot add primary key on a column with an invalid not-null +ALTER TABLE notnull_tbl1 ADD PRIMARY KEY (a); + +-- ALTER column SET NOT NULL validates an invalid constraint (but this fails +-- because of rows with null values) +ALTER TABLE notnull_tbl1 ALTER a SET NOT NULL; +\d+ notnull_tbl1 + +-- Creating a derived table using LIKE gets the constraint, but it's valid +CREATE TABLE notnull_tbl1_copy (LIKE notnull_tbl1); +EXECUTE get_nnconstraint_info('{notnull_tbl1_copy}'); + +-- An inheritance child table gets the constraint, but it's valid +CREATE TABLE notnull_tbl1_child (a int, b int) INHERITS (notnull_tbl1); +EXECUTE get_nnconstraint_info('{notnull_tbl1_child, notnull_tbl1}'); + +-- Also try inheritance added after table creation +CREATE TABLE notnull_tbl1_child2 (c int, b int, a int); +ALTER TABLE notnull_tbl1_child2 INHERIT notnull_tbl1; -- nope +ALTER TABLE notnull_tbl1_child2 ADD NOT NULL a NOT VALID; +ALTER TABLE notnull_tbl1_child2 INHERIT notnull_tbl1; +EXECUTE get_nnconstraint_info('{notnull_tbl1_child2}'); + +-- VALIDATE CONSTRAINT scans the table +ALTER TABLE notnull_tbl1 VALIDATE CONSTRAINT nn; -- error, nulls exist +UPDATE notnull_tbl1 SET a = 100 WHERE b = 1; +ALTER TABLE notnull_tbl1 VALIDATE CONSTRAINT nn; -- now ok +EXECUTE get_nnconstraint_info('{notnull_tbl1}'); + +--- now we can add primary key +ALTER TABLE notnull_tbl1 ADD PRIMARY KEY (a); +DROP TABLE notnull_tbl1, notnull_tbl1_child, notnull_tbl1_child2; + +-- dropping an invalid constraint is possible +CREATE TABLE notnull_tbl1 (a int, b int); +ALTER TABLE notnull_tbl1 ADD NOT NULL a NOT VALID, + ADD NOT NULL b NOT VALID; +ALTER TABLE notnull_tbl1 ALTER a DROP NOT NULL; +ALTER TABLE notnull_tbl1 DROP CONSTRAINT notnull_tbl1_b_not_null; +DROP TABLE notnull_tbl1; + +-- ALTER .. NO INHERIT works for invalid constraints +CREATE TABLE notnull_tbl1 (a int); +CREATE TABLE notnull_tbl1_chld () INHERITS (notnull_tbl1); +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nntbl1_a NOT NULL a NOT VALID; +ALTER TABLE notnull_tbl1 ALTER CONSTRAINT nntbl1_a NO INHERIT; + +-- DROP CONSTRAINT recurses correctly on invalid constraints +ALTER TABLE notnull_tbl1 ALTER CONSTRAINT nntbl1_a INHERIT; +ALTER TABLE notnull_tbl1 DROP CONSTRAINT nntbl1_a; +DROP TABLE notnull_tbl1, notnull_tbl1_chld; + +-- if a parent has a valid not null constraint then a child table cannot +-- have an invalid one +CREATE TABLE notnull_tbl1 (a int); +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nn_parent NOT NULL a not valid; +CREATE TABLE notnull_chld0 (a int, CONSTRAINT nn_chld0 NOT NULL a); +ALTER TABLE notnull_tbl1 INHERIT notnull_chld0; --error + +ALTER TABLE notnull_chld0 DROP CONSTRAINT nn_chld0; +ALTER TABLE notnull_chld0 ADD CONSTRAINT nn_chld0 NOT NULL a not valid; +ALTER TABLE notnull_tbl1 INHERIT notnull_chld0; --now ok + +-- parents and child not-null will all be validated. +ALTER TABLE notnull_tbl1 VALIDATE CONSTRAINT nn_parent; +EXECUTE get_nnconstraint_info('{notnull_tbl1, notnull_chld0}'); +DROP TABLE notnull_tbl1, notnull_chld0; + +-- Test invalid not null on inheritance table. +CREATE TABLE notnull_inhparent (i int); +CREATE TABLE notnull_inhchild (i int) INHERITS (notnull_inhparent); +CREATE TABLE notnull_inhgrand () INHERITS (notnull_inhparent, notnull_inhchild); +ALTER TABLE notnull_inhparent ADD CONSTRAINT nn NOT NULL i NOT VALID; +ALTER TABLE notnull_inhchild ADD CONSTRAINT nn1 NOT NULL i; -- error +EXECUTE get_nnconstraint_info('{notnull_inhparent, notnull_inhchild, notnull_inhgrand}'); +ALTER TABLE notnull_inhparent ALTER i SET NOT NULL; -- ok +EXECUTE get_nnconstraint_info('{notnull_inhparent, notnull_inhchild, notnull_inhgrand}'); +DROP TABLE notnull_inhparent, notnull_inhchild, notnull_inhgrand; + +-- Verify NOT NULL VALID/NOT VALID with partition table. +DROP TABLE notnull_tbl1; +CREATE TABLE notnull_tbl1 (a int, b int) PARTITION BY LIST (a); +ALTER TABLE notnull_tbl1 ADD CONSTRAINT notnull_con NOT NULL a NOT VALID; --ok +CREATE TABLE notnull_tbl1_1 PARTITION OF notnull_tbl1 FOR VALUES IN (1,2); +CREATE TABLE notnull_tbl1_2(a int, CONSTRAINT nn2 NOT NULL a, b int); +ALTER TABLE notnull_tbl1 ATTACH PARTITION notnull_tbl1_2 FOR VALUES IN (3,4); + +CREATE TABLE notnull_tbl1_3(a int, b int); +INSERT INTO notnull_tbl1_3 values(NULL,1); +ALTER TABLE notnull_tbl1_3 add CONSTRAINT nn3 NOT NULL a NOT VALID; +ALTER TABLE notnull_tbl1 ATTACH PARTITION notnull_tbl1_3 FOR VALUES IN (NULL,5); + +EXECUTE get_nnconstraint_info('{notnull_tbl1, notnull_tbl1_1, notnull_tbl1_2, notnull_tbl1_3}'); +ALTER TABLE notnull_tbl1 ALTER COLUMN a SET NOT NULL; --error, notnull_tbl1_3 have null values +ALTER TABLE notnull_tbl1_3 VALIDATE CONSTRAINT nn3; --error + +TRUNCATE notnull_tbl1; +ALTER TABLE notnull_tbl1 ALTER COLUMN a SET NOT NULL; --OK + +EXECUTE get_nnconstraint_info('{notnull_tbl1, notnull_tbl1_1, notnull_tbl1_2, notnull_tbl1_3}'); +DROP TABLE notnull_tbl1; + +-- partitioned table have not-null, then the partitions can not be NOT NULL NOT VALID. +CREATE TABLE pp_nn (a int, b int, NOT NULL a) PARTITION BY LIST (a); +CREATE TABLE pp_nn_1(a int, b int); +ALTER TABLE pp_nn_1 ADD CONSTRAINT nn1 NOT NULL a NOT VALID; +ALTER TABLE pp_nn ATTACH PARTITION pp_nn_1 FOR VALUES IN (NULL,5); --error +ALTER TABLE pp_nn_1 VALIDATE CONSTRAINT nn1; +ALTER TABLE pp_nn ATTACH PARTITION pp_nn_1 FOR VALUES IN (NULL,5); --ok +DROP TABLE pp_nn; + +-- Create table with NOT NULL INVALID constraint, for pg_upgrade. +CREATE TABLE notnull_tbl1_upg (a int, b int); +INSERT INTO notnull_tbl1_upg VALUES (NULL, 1), (NULL, 2), (300, 3); +ALTER TABLE notnull_tbl1_upg ADD CONSTRAINT nn NOT NULL a NOT VALID; +-- Inherit test for pg_upgrade +CREATE TABLE notnull_parent_upg (a int); +CREATE TABLE notnull_child_upg () INHERITS (notnull_parent_upg); +ALTER TABLE notnull_child_upg ADD CONSTRAINT nn NOT NULL a; +ALTER TABLE notnull_parent_upg ADD CONSTRAINT nn NOT NULL a NOT VALID; +SELECT conrelid::regclass, contype, convalidated, conislocal +FROM pg_catalog.pg_constraint +WHERE conrelid in ('notnull_parent_upg'::regclass, 'notnull_child_upg'::regclass) +ORDER BY 1; + +-- Partition table test, for pg_upgrade +CREATE TABLE notnull_part1_upg (a int, b int) PARTITION BY LIST (a); +ALTER TABLE notnull_part1_upg ADD CONSTRAINT notnull_con NOT NULL a NOT VALID; --ok +CREATE TABLE notnull_part1_1_upg PARTITION OF notnull_part1_upg FOR VALUES IN (1,2); +CREATE TABLE notnull_part1_2_upg (a int, CONSTRAINT nn2 NOT NULL a, b int); +ALTER TABLE notnull_part1_upg ATTACH PARTITION notnull_part1_2_upg FOR VALUES IN (3,4); +CREATE TABLE notnull_part1_3_upg (a int, b int); +INSERT INTO notnull_part1_3_upg values(NULL,1); +ALTER TABLE notnull_part1_3_upg add CONSTRAINT nn3 NOT NULL a NOT VALID; +ALTER TABLE notnull_part1_upg ATTACH PARTITION notnull_part1_3_upg FOR VALUES IN (NULL,5); +EXECUTE get_nnconstraint_info('{notnull_part1_upg, notnull_part1_1_upg, notnull_part1_2_upg, notnull_part1_3_upg}'); +DEALLOCATE get_nnconstraint_info; + +-- end NOT NULL NOT VALID + + -- Comments -- Setup a low-level role to enforce non-superuser checks. CREATE ROLE regress_constraint_comments; diff --git a/src/test/regress/sql/sanity_check.sql b/src/test/regress/sql/sanity_check.sql index 162e5324b5d..6ef36ab84b7 100644 --- a/src/test/regress/sql/sanity_check.sql +++ b/src/test/regress/sql/sanity_check.sql @@ -19,3 +19,17 @@ SELECT relname, relkind FROM pg_class WHERE relkind IN ('v', 'c', 'f', 'p', 'I') AND relfilenode <> 0; + +-- pg_attribute sanity check: attnotnullvalid can only be true when +-- attnotnull is valid. +select pa.attrelid::regclass, pa.attname, pa.attnum, + pa.attnotnull, pa.attnotnullvalid +from pg_attribute pa +where not pa.attnotnull and pa.attnotnullvalid; + +-- pg_attribute's attnotnullvalid should match pg_constraint.convalidated +select att.attrelid::regclass, att.attname, att.attnotnull, + att.attnotnullvalid, con.convalidated + from pg_attribute att join pg_constraint con + on (att.attnum = con.conkey[1] and att.attrelid = con.conrelid) + where contype = 'n' and con.convalidated <> att.attnotnullvalid; -- 2.39.5 [text/x-diff] v8-0002-Remove-attnotnullvalid-again.patch (23.0K, ../../[email protected]/3-v8-0002-Remove-attnotnullvalid-again.patch) download | inline diff: From 1842214bfbaff31a26f2641f9f81872835e041d3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=81lvaro=20Herrera?= <[email protected]> Date: Sun, 6 Apr 2025 19:45:52 +0200 Subject: [PATCH v8 2/3] Remove attnotnullvalid again TupleDesc construction already has a pg_constraint scan. This introduces a bit of ugliness, but the benefit is that we don't need another pg_attribute column. However, some regression tests aren't passing, and it seems to be because we don't set attnullability correctly everywhere. --- doc/src/sgml/catalogs.sgml | 11 +--- src/backend/access/common/tupdesc.c | 17 ++---- src/backend/bootstrap/bootstrap.c | 3 - src/backend/catalog/genbki.pl | 3 - src/backend/catalog/heap.c | 8 --- src/backend/commands/tablecmds.c | 47 +++++++-------- src/backend/optimizer/util/plancat.c | 11 ++-- src/backend/utils/cache/catcache.c | 1 - src/backend/utils/cache/relcache.c | 66 +++++++++++++++++++--- src/include/access/tupdesc.h | 9 +-- src/include/catalog/pg_attribute.h | 3 - src/test/regress/expected/sanity_check.out | 20 ------- src/test/regress/sql/sanity_check.sql | 14 ----- 13 files changed, 100 insertions(+), 113 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 3ab7d7b68aa..cbd4e40a320 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1260,16 +1260,7 @@ <structfield>attnotnull</structfield> <type>bool</type> </para> <para> - This column has a (possibly unvalidated) not-null constraint. - </para></entry> - </row> - - <row> - <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>attnotnullvalid</structfield> <type>bool</type> - </para> - <para> - Whether the not-null constraint, if one exists, has been validated. + This column has a (possibly invalid) not-null constraint. </para></entry> </row> diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 320de2cbda0..5831788cbff 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -74,8 +74,8 @@ populate_compact_attribute_internal(Form_pg_attribute src, dst->atthasmissing = src->atthasmissing; dst->attisdropped = src->attisdropped; dst->attgenerated = (src->attgenerated != '\0'); - dst->attnullability = !src->attnotnull ? ATTNULLABLE_NONE : - src->attnotnullvalid ? ATTNULLABLE_VALID : ATTNULLABLE_INVALID; + dst->attnullability = !src->attnotnull ? ATTNULLABLE_UNRESTRICTED : + ATTNULLABLE_UNKNOWN; switch (src->attalign) { @@ -145,9 +145,10 @@ verify_compact_attribute(TupleDesc tupdesc, int attnum) /* * Make the attcacheoff match since it's been reset to -1 by - * populate_compact_attribute_internal. + * populate_compact_attribute_internal. Same with attnullability. */ tmp.attcacheoff = cattr->attcacheoff; + tmp.attnullability = cattr->attnullability; /* Check the freshly populated CompactAttribute matches the TupleDesc's */ Assert(memcmp(&tmp, cattr, sizeof(CompactAttribute)) == 0); @@ -253,7 +254,6 @@ CreateTupleDescCopy(TupleDesc tupdesc) Form_pg_attribute att = TupleDescAttr(desc, i); att->attnotnull = false; - att->attnotnullvalid = false; att->atthasdef = false; att->atthasmissing = false; att->attidentity = '\0'; @@ -300,7 +300,6 @@ CreateTupleDescTruncatedCopy(TupleDesc tupdesc, int natts) Form_pg_attribute att = TupleDescAttr(desc, i); att->attnotnull = false; - att->attnotnullvalid = false; att->atthasdef = false; att->atthasmissing = false; att->attidentity = '\0'; @@ -421,7 +420,6 @@ TupleDescCopy(TupleDesc dst, TupleDesc src) Form_pg_attribute att = TupleDescAttr(dst, i); att->attnotnull = false; - att->attnotnullvalid = false; att->atthasdef = false; att->atthasmissing = false; att->attidentity = '\0'; @@ -468,7 +466,6 @@ TupleDescCopyEntry(TupleDesc dst, AttrNumber dstAttno, /* since we're not copying constraints or defaults, clear these */ dstAtt->attnotnull = false; - dstAtt->attnotnullvalid = false; dstAtt->atthasdef = false; dstAtt->atthasmissing = false; dstAtt->attidentity = '\0'; @@ -567,6 +564,8 @@ DecrTupleDescRefCount(TupleDesc tupdesc) /* * Compare two TupleDesc structures for logical equality + * + * XXX should we compare CompactAttribute->attnullability here? */ bool equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) @@ -618,8 +617,6 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attnotnull != attr2->attnotnull) return false; - if (attr1->attnotnullvalid != attr2->attnotnullvalid) - return false; if (attr1->atthasdef != attr2->atthasdef) return false; if (attr1->attidentity != attr2->attidentity) @@ -848,7 +845,6 @@ TupleDescInitEntry(TupleDesc desc, att->attndims = attdim; att->attnotnull = false; - att->attnotnullvalid = false; att->atthasdef = false; att->atthasmissing = false; att->attidentity = '\0'; @@ -912,7 +908,6 @@ TupleDescInitBuiltinEntry(TupleDesc desc, att->attndims = attdim; att->attnotnull = false; - att->attnotnullvalid = false; att->atthasdef = false; att->atthasmissing = false; att->attidentity = '\0'; diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 44ec9e58520..6db864892d0 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -615,9 +615,6 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attnotnull = true; } } - - /* Not-null constraints on system catalogs are always valid. */ - attrtypes[attnum]->attnotnullvalid = attrtypes[attnum]->attnotnull; } diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index bdf55d7dc8d..df3231fcd41 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -986,9 +986,6 @@ sub morph_row_for_pgattr $row->{attnotnull} = 'f'; } - # Not-null constraints on system catalogs are always valid. - $row->{attnotnullvalid} = $row->{attnotnull}; - Catalog::AddDefaultValues($row, $pgattr_schema, 'pg_attribute'); return; } diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 503e7cb3f58..fbaed5359ad 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -151,7 +151,6 @@ static const FormData_pg_attribute a1 = { .attalign = TYPALIGN_SHORT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, - .attnotnullvalid = true, .attislocal = true, }; @@ -165,7 +164,6 @@ static const FormData_pg_attribute a2 = { .attalign = TYPALIGN_INT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, - .attnotnullvalid = true, .attislocal = true, }; @@ -179,7 +177,6 @@ static const FormData_pg_attribute a3 = { .attalign = TYPALIGN_INT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, - .attnotnullvalid = true, .attislocal = true, }; @@ -193,7 +190,6 @@ static const FormData_pg_attribute a4 = { .attalign = TYPALIGN_INT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, - .attnotnullvalid = true, .attislocal = true, }; @@ -207,7 +203,6 @@ static const FormData_pg_attribute a5 = { .attalign = TYPALIGN_INT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, - .attnotnullvalid = true, .attislocal = true, }; @@ -227,7 +222,6 @@ static const FormData_pg_attribute a6 = { .attalign = TYPALIGN_INT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, - .attnotnullvalid = true, .attislocal = true, }; @@ -759,7 +753,6 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attstorage - 1] = CharGetDatum(attrs->attstorage); slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); slot[slotCount]->tts_values[Anum_pg_attribute_attnotnull - 1] = BoolGetDatum(attrs->attnotnull); - slot[slotCount]->tts_values[Anum_pg_attribute_attnotnullvalid - 1] = BoolGetDatum(attrs->attnotnullvalid); slot[slotCount]->tts_values[Anum_pg_attribute_atthasdef - 1] = BoolGetDatum(attrs->atthasdef); slot[slotCount]->tts_values[Anum_pg_attribute_atthasmissing - 1] = BoolGetDatum(attrs->atthasmissing); slot[slotCount]->tts_values[Anum_pg_attribute_attidentity - 1] = CharGetDatum(attrs->attidentity); @@ -1721,7 +1714,6 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Remove any not-null constraint the column may have */ attStruct->attnotnull = false; - attStruct->attnotnullvalid = false; /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 0c7b1231c2e..b86c55c7430 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1427,7 +1427,7 @@ BuildDescForRelation(const List *columns) TupleDescInitEntryCollation(desc, attnum, attcollation); /* Fill in additional stuff not handled by TupleDescInitEntry */ - att->attnotnull = att->attnotnullvalid = entry->is_not_null; + att->attnotnull = entry->is_not_null; att->attislocal = entry->is_local; att->attinhcount = entry->inhcount; att->attidentity = entry->identity; @@ -6222,16 +6222,18 @@ ATRewriteTable(AlteredTableInfo *tab, Oid OIDNewHeap) */ for (i = 0; i < newTupDesc->natts; i++) { - Form_pg_attribute attr = TupleDescAttr(newTupDesc, i); + CompactAttribute *attr = TupleDescCompactAttr(newTupDesc, i); - if (attr->attnotnull && attr->attnotnullvalid && + if (attr->attnullability != ATTNULLABLE_UNRESTRICTED && !attr->attisdropped) { - if (attr->attgenerated != ATTRIBUTE_GENERATED_VIRTUAL) - notnull_attrs = lappend_int(notnull_attrs, attr->attnum); + Form_pg_attribute wholeatt = TupleDescAttr(newTupDesc, i); + + if (wholeatt->attgenerated != ATTRIBUTE_GENERATED_VIRTUAL) + notnull_attrs = lappend_int(notnull_attrs, wholeatt->attnum); else notnull_virtual_attrs = lappend_int(notnull_virtual_attrs, - attr->attnum); + wholeatt->attnum); } } if (notnull_attrs || notnull_virtual_attrs) @@ -7795,7 +7797,7 @@ ATExecDropNotNull(Relation rel, const char *colName, bool recurse, /* * Find the constraint that makes this column NOT NULL, and drop it. - * dropconstraint_internal() resets attnotnull/attnotnullvalid. + * dropconstraint_internal() resets attnotnull. */ conTup = findNotNullConstraintAttnum(RelationGetRelid(rel), attnum); if (conTup == NULL) @@ -7820,10 +7822,10 @@ ATExecDropNotNull(Relation rel, const char *colName, bool recurse, * Helper to update/validate the pg_attribute status of a not-null * constraint * - * pg_attribute.attnotnull is set true, if it isn't already. If is_valid - * is true, also set pg_attribute.attnotnullvalid. If queue_validation is - * true, also set up wqueue to validate the constraint. wqueue may be given - * as NULL when validation is not needed (e.g., on table creation). + * pg_attribute.attnotnull is set true, if it isn't already. + * If queue_validation is true, also set up wqueue to validate the constraint. + * wqueue may be given as NULL when validation is not needed (e.g., on table + * creation). */ static void set_attnotnull(List **wqueue, Relation rel, AttrNumber attnum, @@ -7843,7 +7845,7 @@ set_attnotnull(List **wqueue, Relation rel, AttrNumber attnum, if (attr->attisdropped) return; - if (!attr->attnotnull || (is_valid && !attr->attnotnullvalid)) + if (!attr->attnotnull) { Relation attr_rel; HeapTuple tuple; @@ -7858,7 +7860,6 @@ set_attnotnull(List **wqueue, Relation rel, AttrNumber attnum, attr = (Form_pg_attribute) GETSTRUCT(tuple); attr->attnotnull = true; - attr->attnotnullvalid = is_valid; CatalogTupleUpdate(attr_rel, &tuple->t_self, tuple); /* @@ -9899,8 +9900,8 @@ ATAddCheckNNConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel, /* * If adding a valid not-null constraint, set the pg_attribute flag * and tell phase 3 to verify existing rows, if needed. For an - * invalid constraint, just set attnotnull and attnotnullvalid, - * without queueing verification. + * invalid constraint, just set attnotnull, without queueing + * verification. */ if (constr->contype == CONSTR_NOTNULL) set_attnotnull(wqueue, rel, ccon->attnum, @@ -14091,11 +14092,10 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha false), RelationGetRelationName(rel))); - /* All good -- reset attnotnull and attnotnullvalid if needed */ + /* All good -- reset attnotnull if needed */ if (attForm->attnotnull) { attForm->attnotnull = false; - attForm->attnotnullvalid = false; CatalogTupleUpdate(attrel, &atttup->t_self, atttup); } @@ -19945,18 +19945,19 @@ PartConstraintImpliedByRelConstraint(Relation scanrel, for (i = 1; i <= natts; i++) { - Form_pg_attribute att = TupleDescAttr(scanrel->rd_att, i - 1); + CompactAttribute *att = TupleDescCompactAttr(scanrel->rd_att, i - 1); - /* invalid not-null constraint must be ignored */ - if (att->attnotnull && att->attnotnullvalid && !att->attisdropped) + /* invalid not-null constraint must be ignored here */ + if (att->attnullability == ATTNULLABLE_VALID && !att->attisdropped) { + Form_pg_attribute wholeatt = TupleDescAttr(scanrel->rd_att, i - 1); NullTest *ntest = makeNode(NullTest); ntest->arg = (Expr *) makeVar(1, i, - att->atttypid, - att->atttypmod, - att->attcollation, + wholeatt->atttypid, + wholeatt->atttypmod, + wholeatt->attcollation, 0); ntest->nulltesttype = IS_NOT_NULL; diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index 191a14b876e..b3d41596b87 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1352,17 +1352,18 @@ get_relation_constraints(PlannerInfo *root, for (i = 1; i <= natts; i++) { - Form_pg_attribute att = TupleDescAttr(relation->rd_att, i - 1); + CompactAttribute *att = TupleDescCompactAttr(relation->rd_att, i - 1); - if (att->attnotnull && att->attnotnullvalid && !att->attisdropped) + if (att->attnullability == ATTNULLABLE_VALID && !att->attisdropped) { + Form_pg_attribute wholeatt = TupleDescAttr(relation->rd_att, i - 1); NullTest *ntest = makeNode(NullTest); ntest->arg = (Expr *) makeVar(varno, i, - att->atttypid, - att->atttypmod, - att->attcollation, + wholeatt->atttypid, + wholeatt->atttypmod, + wholeatt->attcollation, 0); ntest->nulltesttype = IS_NOT_NULL; diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 70c11529b90..9ad7681f155 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1142,7 +1142,6 @@ CatalogCacheInitializeCache(CatCache *cache) keytype = attr->atttypid; /* cache key columns should always be NOT NULL */ Assert(attr->attnotnull); - Assert(attr->attnotnullvalid); } else { diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index c80e40cd47a..4128cfa8314 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -307,7 +307,7 @@ static TupleDesc GetPgClassDescriptor(void); static TupleDesc GetPgIndexDescriptor(void); static void AttrDefaultFetch(Relation relation, int ndef); static int AttrDefaultCmp(const void *a, const void *b); -static void CheckConstraintFetch(Relation relation); +static void CheckNNConstraintFetch(Relation relation); static int CheckConstraintCmp(const void *a, const void *b); static void InitIndexAmRoutine(Relation relation); static void IndexSupportInitialize(oidvector *indclass, @@ -592,7 +592,7 @@ RelationBuildTupleDesc(Relation relation) /* Update constraint/default info */ if (attp->attnotnull) - constr->has_not_null = true; /* invalid ones included */ + constr->has_not_null = true; if (attp->attgenerated == ATTRIBUTE_GENERATED_STORED) constr->has_generated_stored = true; if (attp->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL) @@ -693,9 +693,38 @@ RelationBuildTupleDesc(Relation relation) constr->missing = attrmiss; - if (relation->rd_rel->relchecks > 0) /* CHECKs */ - CheckConstraintFetch(relation); - else + /* CHECK and NOT NULLs */ + if (!IsCatalogRelation(relation) && + (relation->rd_rel->relchecks > 0 || constr->has_not_null)) + CheckNNConstraintFetch(relation); + + /* + * Any not-null constraint that wasn't marked invalid by + * CheckNNConstraintFetch must necessarily be valid; make it so in the + * CompactAttribute array. In catalog relations however, any not-null + * constraint is necessarily valid. + */ + for (int i = 0; i < relation->rd_rel->relnatts - 1; i++) + { + CompactAttribute *attr; + + attr = TupleDescCompactAttr(relation->rd_att, i); + + if (IsCatalogRelation(relation)) + { + if (attr->attnullability == ATTNULLABLE_UNKNOWN) + attr->attnullability = ATTNULLABLE_VALID; + continue; + } + + if (attr->attnullability == ATTNULLABLE_UNKNOWN) + attr->attnullability = ATTNULLABLE_VALID; + else + Assert(attr->attnullability == ATTNULLABLE_INVALID || + attr->attnullability == ATTNULLABLE_UNRESTRICTED); + } + + if (relation->rd_rel->relchecks == 0) constr->num_check = 0; } else @@ -3573,7 +3602,6 @@ RelationBuildLocalRelation(const char *relname, datt->attidentity = satt->attidentity; datt->attgenerated = satt->attgenerated; datt->attnotnull = satt->attnotnull; - datt->attnotnullvalid = satt->attnotnullvalid; has_not_null |= satt->attnotnull; populate_compact_attribute(rel->rd_att, i); } @@ -4534,13 +4562,14 @@ AttrDefaultCmp(const void *a, const void *b) } /* - * Load any check constraints for the relation. + * Load any check constraints for the relation, and update not-null validity + * of invalid constraints. * * As with defaults, if we don't find the expected number of them, just warn * here. The executor should throw an error if an INSERT/UPDATE is attempted. */ static void -CheckConstraintFetch(Relation relation) +CheckNNConstraintFetch(Relation relation) { ConstrCheck *check; int ncheck = relation->rd_rel->relchecks; @@ -4571,6 +4600,27 @@ CheckConstraintFetch(Relation relation) Datum val; bool isnull; + /* + * Consider only invalid not-null constraints and mark the TupleDesc + * entry invalid. + */ + if (!IsCatalogRelation(relation) && + conform->contype == CONSTRAINT_NOTNULL) + { + if (!conform->convalidated) + { + AttrNumber attnum; + + attnum = extractNotNullColumn(htup); + Assert(relation->rd_att->compact_attrs[attnum - 1].attnullability == + ATTNULLABLE_UNKNOWN); + relation->rd_att->compact_attrs[attnum - 1].attnullability = + ATTNULLABLE_INVALID; + } + + continue; + } + /* We want check constraints only */ if (conform->contype != CONSTRAINT_CHECK) continue; diff --git a/src/include/access/tupdesc.h b/src/include/access/tupdesc.h index 5523cfcf5aa..1600f967032 100644 --- a/src/include/access/tupdesc.h +++ b/src/include/access/tupdesc.h @@ -76,14 +76,15 @@ typedef struct CompactAttribute bool atthasmissing; /* as FormData_pg_attribute.atthasmissing */ bool attisdropped; /* as FormData_pg_attribute.attisdropped */ bool attgenerated; /* FormData_pg_attribute.attgenerated != '\0' */ - char attnullability; /* attnotnull + attnotnullvalid */ + char attnullability; /* status of not-null constraint, see below */ uint8 attalignby; /* alignment requirement in bytes */ } CompactAttribute; /* Valid values for CompactAttribute->attnullability */ -#define ATTNULLABLE_VALID 'v' /* valid constraint exists */ -#define ATTNULLABLE_INVALID 'i' /* constraint exists, marked invalid */ -#define ATTNULLABLE_NONE 'f' /* no constraint exists */ +#define ATTNULLABLE_UNRESTRICTED 'f' /* No constraint exists */ +#define ATTNULLABLE_UNKNOWN 'u' /* constraint exists, validity unknown */ +#define ATTNULLABLE_VALID 'v' /* valid constraint exists */ +#define ATTNULLABLE_INVALID 'i' /* constraint exists, marked invalid */ /* * This struct is passed around within the backend to describe the structure diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 9b1236e7a90..df7a77ee224 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -120,9 +120,6 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* Whether a not-null constraint exists for the column */ bool attnotnull; - /* Whether the not-null constraint, if it exists, is valid */ - bool attnotnullvalid; - /* Has DEFAULT value or not */ bool atthasdef BKI_DEFAULT(f); diff --git a/src/test/regress/expected/sanity_check.out b/src/test/regress/expected/sanity_check.out index 2e39037e3fb..8370c1561cc 100644 --- a/src/test/regress/expected/sanity_check.out +++ b/src/test/regress/expected/sanity_check.out @@ -25,23 +25,3 @@ SELECT relname, relkind ---------+--------- (0 rows) --- pg_attribute sanity check: attnotnullvalid can only be true when --- attnotnull is valid. -select pa.attrelid::regclass, pa.attname, pa.attnum, - pa.attnotnull, pa.attnotnullvalid -from pg_attribute pa -where not pa.attnotnull and pa.attnotnullvalid; - attrelid | attname | attnum | attnotnull | attnotnullvalid -----------+---------+--------+------------+----------------- -(0 rows) - --- pg_attribute's attnotnullvalid should match pg_constraint.convalidated -select att.attrelid::regclass, att.attname, att.attnotnull, - att.attnotnullvalid, con.convalidated - from pg_attribute att join pg_constraint con - on (att.attnum = con.conkey[1] and att.attrelid = con.conrelid) - where contype = 'n' and con.convalidated <> att.attnotnullvalid; - attrelid | attname | attnotnull | attnotnullvalid | convalidated -----------+---------+------------+-----------------+-------------- -(0 rows) - diff --git a/src/test/regress/sql/sanity_check.sql b/src/test/regress/sql/sanity_check.sql index 6ef36ab84b7..162e5324b5d 100644 --- a/src/test/regress/sql/sanity_check.sql +++ b/src/test/regress/sql/sanity_check.sql @@ -19,17 +19,3 @@ SELECT relname, relkind FROM pg_class WHERE relkind IN ('v', 'c', 'f', 'p', 'I') AND relfilenode <> 0; - --- pg_attribute sanity check: attnotnullvalid can only be true when --- attnotnull is valid. -select pa.attrelid::regclass, pa.attname, pa.attnum, - pa.attnotnull, pa.attnotnullvalid -from pg_attribute pa -where not pa.attnotnull and pa.attnotnullvalid; - --- pg_attribute's attnotnullvalid should match pg_constraint.convalidated -select att.attrelid::regclass, att.attname, att.attnotnull, - att.attnotnullvalid, con.convalidated - from pg_attribute att join pg_constraint con - on (att.attnum = con.conkey[1] and att.attrelid = con.conrelid) - where contype = 'n' and con.convalidated <> att.attnotnullvalid; -- 2.39.5 [text/x-diff] v8-0003-get_relation_info-assert-correct-attnullability.patch (984B, ../../[email protected]/4-v8-0003-get_relation_info-assert-correct-attnullability.patch) download | inline diff: From 70ecb5835cc71521d552d7ba176b815e58076b05 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=81lvaro=20Herrera?= <[email protected]> Date: Sun, 6 Apr 2025 19:47:22 +0200 Subject: [PATCH v8 3/3] get_relation_info: assert correct attnullability This fails immediately, proving we're missing some spot to set attnullability. --- src/backend/optimizer/util/plancat.c | 1 + 1 file changed, 1 insertion(+) diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index b3d41596b87..ba1702c993b 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -177,6 +177,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, { CompactAttribute *attr = TupleDescCompactAttr(relation->rd_att, i); + Assert(attr->attnullability != ATTNULLABLE_UNKNOWN); if (attr->attnullability == ATTNULLABLE_VALID) { rel->notnullattnums = bms_add_member(rel->notnullattnums, -- 2.39.5 ^ permalink raw reply [nested|flat] 13+ messages in thread
* Re: Support NOT VALID / VALIDATE constraint options for named NOT NULL constraints @ 2025-04-07 08:28 Rushabh Lathia <[email protected]> parent: Alvaro Herrera <[email protected]> 0 siblings, 0 replies; 13+ messages in thread From: Rushabh Lathia @ 2025-04-07 08:28 UTC (permalink / raw) To: Alvaro Herrera <[email protected]>; +Cc: jian he <[email protected]>; pgsql-hackers Thanks Alvaro. On Mon, Apr 7, 2025 at 2:02 AM Alvaro Herrera <[email protected]> wrote: > On 2025-Apr-05, jian he wrote: > > > hi. > > + /* FIXME use CompactAttribute */ > > Form_pg_attribute att = TupleDescAttr(relation->rd_att, > i - 1); > > if (att->attnotnull && att->attnotnullvalid && > > !att->attisdropped) > > { > > NullTest *ntest = makeNode(NullTest); > > ntest->arg = (Expr *) makeVar(varno, > > i, > > att->atttypid, > > att->atttypmod, > > att->attcollation, > > 0); > > ntest->nulltesttype = IS_NOT_NULL; > > > > CompactAttribute doesn't have {atttypmod, attcollation} information, > > now it is impossible to use CompactAttribute here, > > so I removed this FIXME in get_relation_constraints. > > Ah, good point. In this new patch, I now consult both > TupleDescCompactAttr (to consult attnullability) and then TupleDescAttr > (to get collation etc), because now the information we need is split > between those two places. It feels a bit nasty TBH. > > > i am uncomfortable with the change in > > 'CREATE TABLE dump_test.test_table_generated' > > so I only added 'CONSTRAINT NOT NULL / INVALID' tests in > > 002_pg_dump.pl. > > so I only added a test case 'CONSTRAINT NOT NULL / INVALID' > > to 002_pg_dump.pl. > > Yeah, good call. > > > v7-0001 commit message explains what kind of problem > > MergeWithExistingConstraint is trying to fix. > > Umm. I have done away with this again (and the parts in 0002 where you > handle the equivalent issue for not-null constraints), because it is > still not clear to me exactly what fails if you don't have them. I > suggest that we should deal with this separately, and that a patch to > deal with it should include a test case that fails if the code fix is > not present. > > > v7-0002 bullet points summary about NOT NULL NOT VALID added to the > > commit message. > > Thanks. > > > add a test for CREATE TABLE LIKE. > > CREATE TABLE LIKE will copy the invalid not-null constraint and will > become > > valid, i think this is what we want. > > Yep, thanks. > > I removed the postgres_fdw changes. What I wrote was untested, and > failed as soon as I added a trivial test. Needs more thought. > > > I also did some more polish, and as I said in another email, it seems to > me that this approach is also wrong, and that a better approach is to > determine the value of CompactAttribute->attnullability using the > pg_constraint scan at the time when the tupdesc is built. I coded this, > and it almost works, but there are some spots that fail because some > tuple descriptors are not built correctly. I'm not sure what to do with > this at this stage; I would love to see this patch across the finish > line, but it seems a little late now. > > > Note: naturally, patch 0002 in this series would be squashed with 0001 > for commit. Patch 0003 is not for commit, just to show what the > problems are. If you run the regression tests without 0003, there are > surprisingly very few failures, and it's very clear that they are > because of tuple descriptors that don't have the attnullability flag set > correctly. > I reviewed the patch and found that the regression was failing because the loop over attributes in RelationBuildTupleDesc() were not executed correctly. After applying the fix, I noticed one more test was failing, which turned out to be related to nullability handling for temporary tables. Please find attached the 0003 patch, which addresses both issues. Thanks, > > -- > Álvaro Herrera 48°01'N 7°57'E — > https://www.EnterpriseDB.com/ > "La grandeza es una experiencia transitoria. Nunca es consistente. > Depende en gran parte de la imaginación humana creadora de mitos" > (Irulan) > -- Rushabh Lathia Attachments: [application/octet-stream] 0002-Remove-attnotnullvalid-again.patch (23.0K, ../../CAGPqQf2AGLBC0_+P=S_-u0VwCuZ=FdGaP6miwyvxVWBTNsBuPw@mail.gmail.com/3-0002-Remove-attnotnullvalid-again.patch) download | inline diff: From f9b4cdeeb4420ad41a8ef965c5ed71a3fee3a961 Mon Sep 17 00:00:00 2001 From: Rushabh Lathia <[email protected]> Date: Mon, 7 Apr 2025 09:51:47 +0530 Subject: [PATCH 2/3] Remove attnotnullvalid again TupleDesc construction already has a pg_constraint scan. This introduces a bit of ugliness, but the benefit is that we don't need another pg_attribute column. However, some regression tests aren't passing, and it seems to be because we don't set attnullability correctly everywhere. --- doc/src/sgml/catalogs.sgml | 11 +--- src/backend/access/common/tupdesc.c | 17 ++---- src/backend/bootstrap/bootstrap.c | 3 - src/backend/catalog/genbki.pl | 3 - src/backend/catalog/heap.c | 8 --- src/backend/commands/tablecmds.c | 47 +++++++-------- src/backend/optimizer/util/plancat.c | 11 ++-- src/backend/utils/cache/catcache.c | 1 - src/backend/utils/cache/relcache.c | 66 +++++++++++++++++++--- src/include/access/tupdesc.h | 9 +-- src/include/catalog/pg_attribute.h | 3 - src/test/regress/expected/sanity_check.out | 20 ------- src/test/regress/sql/sanity_check.sql | 14 ----- 13 files changed, 100 insertions(+), 113 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 3ab7d7b68aa..cbd4e40a320 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1260,16 +1260,7 @@ <structfield>attnotnull</structfield> <type>bool</type> </para> <para> - This column has a (possibly unvalidated) not-null constraint. - </para></entry> - </row> - - <row> - <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>attnotnullvalid</structfield> <type>bool</type> - </para> - <para> - Whether the not-null constraint, if one exists, has been validated. + This column has a (possibly invalid) not-null constraint. </para></entry> </row> diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 320de2cbda0..5831788cbff 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -74,8 +74,8 @@ populate_compact_attribute_internal(Form_pg_attribute src, dst->atthasmissing = src->atthasmissing; dst->attisdropped = src->attisdropped; dst->attgenerated = (src->attgenerated != '\0'); - dst->attnullability = !src->attnotnull ? ATTNULLABLE_NONE : - src->attnotnullvalid ? ATTNULLABLE_VALID : ATTNULLABLE_INVALID; + dst->attnullability = !src->attnotnull ? ATTNULLABLE_UNRESTRICTED : + ATTNULLABLE_UNKNOWN; switch (src->attalign) { @@ -145,9 +145,10 @@ verify_compact_attribute(TupleDesc tupdesc, int attnum) /* * Make the attcacheoff match since it's been reset to -1 by - * populate_compact_attribute_internal. + * populate_compact_attribute_internal. Same with attnullability. */ tmp.attcacheoff = cattr->attcacheoff; + tmp.attnullability = cattr->attnullability; /* Check the freshly populated CompactAttribute matches the TupleDesc's */ Assert(memcmp(&tmp, cattr, sizeof(CompactAttribute)) == 0); @@ -253,7 +254,6 @@ CreateTupleDescCopy(TupleDesc tupdesc) Form_pg_attribute att = TupleDescAttr(desc, i); att->attnotnull = false; - att->attnotnullvalid = false; att->atthasdef = false; att->atthasmissing = false; att->attidentity = '\0'; @@ -300,7 +300,6 @@ CreateTupleDescTruncatedCopy(TupleDesc tupdesc, int natts) Form_pg_attribute att = TupleDescAttr(desc, i); att->attnotnull = false; - att->attnotnullvalid = false; att->atthasdef = false; att->atthasmissing = false; att->attidentity = '\0'; @@ -421,7 +420,6 @@ TupleDescCopy(TupleDesc dst, TupleDesc src) Form_pg_attribute att = TupleDescAttr(dst, i); att->attnotnull = false; - att->attnotnullvalid = false; att->atthasdef = false; att->atthasmissing = false; att->attidentity = '\0'; @@ -468,7 +466,6 @@ TupleDescCopyEntry(TupleDesc dst, AttrNumber dstAttno, /* since we're not copying constraints or defaults, clear these */ dstAtt->attnotnull = false; - dstAtt->attnotnullvalid = false; dstAtt->atthasdef = false; dstAtt->atthasmissing = false; dstAtt->attidentity = '\0'; @@ -567,6 +564,8 @@ DecrTupleDescRefCount(TupleDesc tupdesc) /* * Compare two TupleDesc structures for logical equality + * + * XXX should we compare CompactAttribute->attnullability here? */ bool equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) @@ -618,8 +617,6 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attnotnull != attr2->attnotnull) return false; - if (attr1->attnotnullvalid != attr2->attnotnullvalid) - return false; if (attr1->atthasdef != attr2->atthasdef) return false; if (attr1->attidentity != attr2->attidentity) @@ -848,7 +845,6 @@ TupleDescInitEntry(TupleDesc desc, att->attndims = attdim; att->attnotnull = false; - att->attnotnullvalid = false; att->atthasdef = false; att->atthasmissing = false; att->attidentity = '\0'; @@ -912,7 +908,6 @@ TupleDescInitBuiltinEntry(TupleDesc desc, att->attndims = attdim; att->attnotnull = false; - att->attnotnullvalid = false; att->atthasdef = false; att->atthasmissing = false; att->attidentity = '\0'; diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 44ec9e58520..6db864892d0 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -615,9 +615,6 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attnotnull = true; } } - - /* Not-null constraints on system catalogs are always valid. */ - attrtypes[attnum]->attnotnullvalid = attrtypes[attnum]->attnotnull; } diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index bdf55d7dc8d..df3231fcd41 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -986,9 +986,6 @@ sub morph_row_for_pgattr $row->{attnotnull} = 'f'; } - # Not-null constraints on system catalogs are always valid. - $row->{attnotnullvalid} = $row->{attnotnull}; - Catalog::AddDefaultValues($row, $pgattr_schema, 'pg_attribute'); return; } diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 503e7cb3f58..fbaed5359ad 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -151,7 +151,6 @@ static const FormData_pg_attribute a1 = { .attalign = TYPALIGN_SHORT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, - .attnotnullvalid = true, .attislocal = true, }; @@ -165,7 +164,6 @@ static const FormData_pg_attribute a2 = { .attalign = TYPALIGN_INT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, - .attnotnullvalid = true, .attislocal = true, }; @@ -179,7 +177,6 @@ static const FormData_pg_attribute a3 = { .attalign = TYPALIGN_INT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, - .attnotnullvalid = true, .attislocal = true, }; @@ -193,7 +190,6 @@ static const FormData_pg_attribute a4 = { .attalign = TYPALIGN_INT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, - .attnotnullvalid = true, .attislocal = true, }; @@ -207,7 +203,6 @@ static const FormData_pg_attribute a5 = { .attalign = TYPALIGN_INT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, - .attnotnullvalid = true, .attislocal = true, }; @@ -227,7 +222,6 @@ static const FormData_pg_attribute a6 = { .attalign = TYPALIGN_INT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, - .attnotnullvalid = true, .attislocal = true, }; @@ -759,7 +753,6 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attstorage - 1] = CharGetDatum(attrs->attstorage); slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); slot[slotCount]->tts_values[Anum_pg_attribute_attnotnull - 1] = BoolGetDatum(attrs->attnotnull); - slot[slotCount]->tts_values[Anum_pg_attribute_attnotnullvalid - 1] = BoolGetDatum(attrs->attnotnullvalid); slot[slotCount]->tts_values[Anum_pg_attribute_atthasdef - 1] = BoolGetDatum(attrs->atthasdef); slot[slotCount]->tts_values[Anum_pg_attribute_atthasmissing - 1] = BoolGetDatum(attrs->atthasmissing); slot[slotCount]->tts_values[Anum_pg_attribute_attidentity - 1] = CharGetDatum(attrs->attidentity); @@ -1721,7 +1714,6 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Remove any not-null constraint the column may have */ attStruct->attnotnull = false; - attStruct->attnotnullvalid = false; /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 0c7b1231c2e..b86c55c7430 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -1427,7 +1427,7 @@ BuildDescForRelation(const List *columns) TupleDescInitEntryCollation(desc, attnum, attcollation); /* Fill in additional stuff not handled by TupleDescInitEntry */ - att->attnotnull = att->attnotnullvalid = entry->is_not_null; + att->attnotnull = entry->is_not_null; att->attislocal = entry->is_local; att->attinhcount = entry->inhcount; att->attidentity = entry->identity; @@ -6222,16 +6222,18 @@ ATRewriteTable(AlteredTableInfo *tab, Oid OIDNewHeap) */ for (i = 0; i < newTupDesc->natts; i++) { - Form_pg_attribute attr = TupleDescAttr(newTupDesc, i); + CompactAttribute *attr = TupleDescCompactAttr(newTupDesc, i); - if (attr->attnotnull && attr->attnotnullvalid && + if (attr->attnullability != ATTNULLABLE_UNRESTRICTED && !attr->attisdropped) { - if (attr->attgenerated != ATTRIBUTE_GENERATED_VIRTUAL) - notnull_attrs = lappend_int(notnull_attrs, attr->attnum); + Form_pg_attribute wholeatt = TupleDescAttr(newTupDesc, i); + + if (wholeatt->attgenerated != ATTRIBUTE_GENERATED_VIRTUAL) + notnull_attrs = lappend_int(notnull_attrs, wholeatt->attnum); else notnull_virtual_attrs = lappend_int(notnull_virtual_attrs, - attr->attnum); + wholeatt->attnum); } } if (notnull_attrs || notnull_virtual_attrs) @@ -7795,7 +7797,7 @@ ATExecDropNotNull(Relation rel, const char *colName, bool recurse, /* * Find the constraint that makes this column NOT NULL, and drop it. - * dropconstraint_internal() resets attnotnull/attnotnullvalid. + * dropconstraint_internal() resets attnotnull. */ conTup = findNotNullConstraintAttnum(RelationGetRelid(rel), attnum); if (conTup == NULL) @@ -7820,10 +7822,10 @@ ATExecDropNotNull(Relation rel, const char *colName, bool recurse, * Helper to update/validate the pg_attribute status of a not-null * constraint * - * pg_attribute.attnotnull is set true, if it isn't already. If is_valid - * is true, also set pg_attribute.attnotnullvalid. If queue_validation is - * true, also set up wqueue to validate the constraint. wqueue may be given - * as NULL when validation is not needed (e.g., on table creation). + * pg_attribute.attnotnull is set true, if it isn't already. + * If queue_validation is true, also set up wqueue to validate the constraint. + * wqueue may be given as NULL when validation is not needed (e.g., on table + * creation). */ static void set_attnotnull(List **wqueue, Relation rel, AttrNumber attnum, @@ -7843,7 +7845,7 @@ set_attnotnull(List **wqueue, Relation rel, AttrNumber attnum, if (attr->attisdropped) return; - if (!attr->attnotnull || (is_valid && !attr->attnotnullvalid)) + if (!attr->attnotnull) { Relation attr_rel; HeapTuple tuple; @@ -7858,7 +7860,6 @@ set_attnotnull(List **wqueue, Relation rel, AttrNumber attnum, attr = (Form_pg_attribute) GETSTRUCT(tuple); attr->attnotnull = true; - attr->attnotnullvalid = is_valid; CatalogTupleUpdate(attr_rel, &tuple->t_self, tuple); /* @@ -9899,8 +9900,8 @@ ATAddCheckNNConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel, /* * If adding a valid not-null constraint, set the pg_attribute flag * and tell phase 3 to verify existing rows, if needed. For an - * invalid constraint, just set attnotnull and attnotnullvalid, - * without queueing verification. + * invalid constraint, just set attnotnull, without queueing + * verification. */ if (constr->contype == CONSTR_NOTNULL) set_attnotnull(wqueue, rel, ccon->attnum, @@ -14091,11 +14092,10 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha false), RelationGetRelationName(rel))); - /* All good -- reset attnotnull and attnotnullvalid if needed */ + /* All good -- reset attnotnull if needed */ if (attForm->attnotnull) { attForm->attnotnull = false; - attForm->attnotnullvalid = false; CatalogTupleUpdate(attrel, &atttup->t_self, atttup); } @@ -19945,18 +19945,19 @@ PartConstraintImpliedByRelConstraint(Relation scanrel, for (i = 1; i <= natts; i++) { - Form_pg_attribute att = TupleDescAttr(scanrel->rd_att, i - 1); + CompactAttribute *att = TupleDescCompactAttr(scanrel->rd_att, i - 1); - /* invalid not-null constraint must be ignored */ - if (att->attnotnull && att->attnotnullvalid && !att->attisdropped) + /* invalid not-null constraint must be ignored here */ + if (att->attnullability == ATTNULLABLE_VALID && !att->attisdropped) { + Form_pg_attribute wholeatt = TupleDescAttr(scanrel->rd_att, i - 1); NullTest *ntest = makeNode(NullTest); ntest->arg = (Expr *) makeVar(1, i, - att->atttypid, - att->atttypmod, - att->attcollation, + wholeatt->atttypid, + wholeatt->atttypmod, + wholeatt->attcollation, 0); ntest->nulltesttype = IS_NOT_NULL; diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index 191a14b876e..b3d41596b87 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1352,17 +1352,18 @@ get_relation_constraints(PlannerInfo *root, for (i = 1; i <= natts; i++) { - Form_pg_attribute att = TupleDescAttr(relation->rd_att, i - 1); + CompactAttribute *att = TupleDescCompactAttr(relation->rd_att, i - 1); - if (att->attnotnull && att->attnotnullvalid && !att->attisdropped) + if (att->attnullability == ATTNULLABLE_VALID && !att->attisdropped) { + Form_pg_attribute wholeatt = TupleDescAttr(relation->rd_att, i - 1); NullTest *ntest = makeNode(NullTest); ntest->arg = (Expr *) makeVar(varno, i, - att->atttypid, - att->atttypmod, - att->attcollation, + wholeatt->atttypid, + wholeatt->atttypmod, + wholeatt->attcollation, 0); ntest->nulltesttype = IS_NOT_NULL; diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 70c11529b90..9ad7681f155 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1142,7 +1142,6 @@ CatalogCacheInitializeCache(CatCache *cache) keytype = attr->atttypid; /* cache key columns should always be NOT NULL */ Assert(attr->attnotnull); - Assert(attr->attnotnullvalid); } else { diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index c80e40cd47a..4128cfa8314 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -307,7 +307,7 @@ static TupleDesc GetPgClassDescriptor(void); static TupleDesc GetPgIndexDescriptor(void); static void AttrDefaultFetch(Relation relation, int ndef); static int AttrDefaultCmp(const void *a, const void *b); -static void CheckConstraintFetch(Relation relation); +static void CheckNNConstraintFetch(Relation relation); static int CheckConstraintCmp(const void *a, const void *b); static void InitIndexAmRoutine(Relation relation); static void IndexSupportInitialize(oidvector *indclass, @@ -592,7 +592,7 @@ RelationBuildTupleDesc(Relation relation) /* Update constraint/default info */ if (attp->attnotnull) - constr->has_not_null = true; /* invalid ones included */ + constr->has_not_null = true; if (attp->attgenerated == ATTRIBUTE_GENERATED_STORED) constr->has_generated_stored = true; if (attp->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL) @@ -693,9 +693,38 @@ RelationBuildTupleDesc(Relation relation) constr->missing = attrmiss; - if (relation->rd_rel->relchecks > 0) /* CHECKs */ - CheckConstraintFetch(relation); - else + /* CHECK and NOT NULLs */ + if (!IsCatalogRelation(relation) && + (relation->rd_rel->relchecks > 0 || constr->has_not_null)) + CheckNNConstraintFetch(relation); + + /* + * Any not-null constraint that wasn't marked invalid by + * CheckNNConstraintFetch must necessarily be valid; make it so in the + * CompactAttribute array. In catalog relations however, any not-null + * constraint is necessarily valid. + */ + for (int i = 0; i < relation->rd_rel->relnatts - 1; i++) + { + CompactAttribute *attr; + + attr = TupleDescCompactAttr(relation->rd_att, i); + + if (IsCatalogRelation(relation)) + { + if (attr->attnullability == ATTNULLABLE_UNKNOWN) + attr->attnullability = ATTNULLABLE_VALID; + continue; + } + + if (attr->attnullability == ATTNULLABLE_UNKNOWN) + attr->attnullability = ATTNULLABLE_VALID; + else + Assert(attr->attnullability == ATTNULLABLE_INVALID || + attr->attnullability == ATTNULLABLE_UNRESTRICTED); + } + + if (relation->rd_rel->relchecks == 0) constr->num_check = 0; } else @@ -3573,7 +3602,6 @@ RelationBuildLocalRelation(const char *relname, datt->attidentity = satt->attidentity; datt->attgenerated = satt->attgenerated; datt->attnotnull = satt->attnotnull; - datt->attnotnullvalid = satt->attnotnullvalid; has_not_null |= satt->attnotnull; populate_compact_attribute(rel->rd_att, i); } @@ -4534,13 +4562,14 @@ AttrDefaultCmp(const void *a, const void *b) } /* - * Load any check constraints for the relation. + * Load any check constraints for the relation, and update not-null validity + * of invalid constraints. * * As with defaults, if we don't find the expected number of them, just warn * here. The executor should throw an error if an INSERT/UPDATE is attempted. */ static void -CheckConstraintFetch(Relation relation) +CheckNNConstraintFetch(Relation relation) { ConstrCheck *check; int ncheck = relation->rd_rel->relchecks; @@ -4571,6 +4600,27 @@ CheckConstraintFetch(Relation relation) Datum val; bool isnull; + /* + * Consider only invalid not-null constraints and mark the TupleDesc + * entry invalid. + */ + if (!IsCatalogRelation(relation) && + conform->contype == CONSTRAINT_NOTNULL) + { + if (!conform->convalidated) + { + AttrNumber attnum; + + attnum = extractNotNullColumn(htup); + Assert(relation->rd_att->compact_attrs[attnum - 1].attnullability == + ATTNULLABLE_UNKNOWN); + relation->rd_att->compact_attrs[attnum - 1].attnullability = + ATTNULLABLE_INVALID; + } + + continue; + } + /* We want check constraints only */ if (conform->contype != CONSTRAINT_CHECK) continue; diff --git a/src/include/access/tupdesc.h b/src/include/access/tupdesc.h index 5523cfcf5aa..1600f967032 100644 --- a/src/include/access/tupdesc.h +++ b/src/include/access/tupdesc.h @@ -76,14 +76,15 @@ typedef struct CompactAttribute bool atthasmissing; /* as FormData_pg_attribute.atthasmissing */ bool attisdropped; /* as FormData_pg_attribute.attisdropped */ bool attgenerated; /* FormData_pg_attribute.attgenerated != '\0' */ - char attnullability; /* attnotnull + attnotnullvalid */ + char attnullability; /* status of not-null constraint, see below */ uint8 attalignby; /* alignment requirement in bytes */ } CompactAttribute; /* Valid values for CompactAttribute->attnullability */ -#define ATTNULLABLE_VALID 'v' /* valid constraint exists */ -#define ATTNULLABLE_INVALID 'i' /* constraint exists, marked invalid */ -#define ATTNULLABLE_NONE 'f' /* no constraint exists */ +#define ATTNULLABLE_UNRESTRICTED 'f' /* No constraint exists */ +#define ATTNULLABLE_UNKNOWN 'u' /* constraint exists, validity unknown */ +#define ATTNULLABLE_VALID 'v' /* valid constraint exists */ +#define ATTNULLABLE_INVALID 'i' /* constraint exists, marked invalid */ /* * This struct is passed around within the backend to describe the structure diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 9b1236e7a90..df7a77ee224 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -120,9 +120,6 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* Whether a not-null constraint exists for the column */ bool attnotnull; - /* Whether the not-null constraint, if it exists, is valid */ - bool attnotnullvalid; - /* Has DEFAULT value or not */ bool atthasdef BKI_DEFAULT(f); diff --git a/src/test/regress/expected/sanity_check.out b/src/test/regress/expected/sanity_check.out index 2e39037e3fb..8370c1561cc 100644 --- a/src/test/regress/expected/sanity_check.out +++ b/src/test/regress/expected/sanity_check.out @@ -25,23 +25,3 @@ SELECT relname, relkind ---------+--------- (0 rows) --- pg_attribute sanity check: attnotnullvalid can only be true when --- attnotnull is valid. -select pa.attrelid::regclass, pa.attname, pa.attnum, - pa.attnotnull, pa.attnotnullvalid -from pg_attribute pa -where not pa.attnotnull and pa.attnotnullvalid; - attrelid | attname | attnum | attnotnull | attnotnullvalid -----------+---------+--------+------------+----------------- -(0 rows) - --- pg_attribute's attnotnullvalid should match pg_constraint.convalidated -select att.attrelid::regclass, att.attname, att.attnotnull, - att.attnotnullvalid, con.convalidated - from pg_attribute att join pg_constraint con - on (att.attnum = con.conkey[1] and att.attrelid = con.conrelid) - where contype = 'n' and con.convalidated <> att.attnotnullvalid; - attrelid | attname | attnotnull | attnotnullvalid | convalidated -----------+---------+------------+-----------------+-------------- -(0 rows) - diff --git a/src/test/regress/sql/sanity_check.sql b/src/test/regress/sql/sanity_check.sql index 6ef36ab84b7..162e5324b5d 100644 --- a/src/test/regress/sql/sanity_check.sql +++ b/src/test/regress/sql/sanity_check.sql @@ -19,17 +19,3 @@ SELECT relname, relkind FROM pg_class WHERE relkind IN ('v', 'c', 'f', 'p', 'I') AND relfilenode <> 0; - --- pg_attribute sanity check: attnotnullvalid can only be true when --- attnotnull is valid. -select pa.attrelid::regclass, pa.attname, pa.attnum, - pa.attnotnull, pa.attnotnullvalid -from pg_attribute pa -where not pa.attnotnull and pa.attnotnullvalid; - --- pg_attribute's attnotnullvalid should match pg_constraint.convalidated -select att.attrelid::regclass, att.attname, att.attnotnull, - att.attnotnullvalid, con.convalidated - from pg_attribute att join pg_constraint con - on (att.attnum = con.conkey[1] and att.attrelid = con.conrelid) - where contype = 'n' and con.convalidated <> att.attnotnullvalid; -- 2.43.0 [application/octet-stream] 0001-Allow-NOT-NULL-constraints-to-be-added-as-NOT-VALID.patch (86.2K, ../../CAGPqQf2AGLBC0_+P=S_-u0VwCuZ=FdGaP6miwyvxVWBTNsBuPw@mail.gmail.com/4-0001-Allow-NOT-NULL-constraints-to-be-added-as-NOT-VALID.patch) download | inline diff: From efa75dc7ab06c1484554f9161fdfb9021cdee7c1 Mon Sep 17 00:00:00 2001 From: Rushabh Lathia <[email protected]> Date: Mon, 7 Apr 2025 09:48:03 +0530 Subject: [PATCH 1/3] Allow NOT NULL constraints to be added as NOT VALID This required adding a new system column, pg_attribute.attnotnullvalid, in order to allow constructing a TupleDesc without having to scan pg_constraint; at the same time, we keep pg_attribute.attnotnull unchanged, to avoid breaking the countless applications that rely on that. Also add support for ALTER TABLE .. VALIDATE CONSTRAINT for them. --- doc/src/sgml/catalogs.sgml | 11 +- doc/src/sgml/ref/alter_table.sgml | 8 +- src/backend/access/common/tupdesc.c | 11 +- src/backend/bootstrap/bootstrap.c | 3 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 17 +- src/backend/catalog/pg_constraint.c | 46 ++-- src/backend/commands/tablecmds.c | 246 +++++++++++++++++--- src/backend/executor/execMain.c | 1 + src/backend/jit/llvm/llvmjit_deform.c | 10 +- src/backend/optimizer/util/plancat.c | 5 +- src/backend/parser/gram.y | 5 +- src/backend/utils/cache/catcache.c | 1 + src/backend/utils/cache/relcache.c | 3 +- src/bin/pg_dump/pg_dump.c | 186 +++++++++++++-- src/bin/pg_dump/pg_dump.h | 2 + src/bin/pg_dump/t/002_pg_dump.pl | 21 +- src/bin/psql/describe.c | 9 +- src/include/access/tupdesc.h | 9 +- src/include/catalog/catversion.h | 2 +- src/include/catalog/pg_attribute.h | 5 +- src/include/catalog/pg_constraint.h | 2 +- src/test/regress/expected/alter_table.out | 69 ++++++ src/test/regress/expected/constraints.out | 250 +++++++++++++++++++++ src/test/regress/expected/sanity_check.out | 20 ++ src/test/regress/sql/alter_table.sql | 14 ++ src/test/regress/sql/constraints.sql | 166 ++++++++++++++ src/test/regress/sql/sanity_check.sql | 14 ++ 28 files changed, 1047 insertions(+), 92 deletions(-) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 45ba9c5118f..3ab7d7b68aa 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1260,7 +1260,16 @@ <structfield>attnotnull</structfield> <type>bool</type> </para> <para> - This column has a not-null constraint. + This column has a (possibly unvalidated) not-null constraint. + </para></entry> + </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>attnotnullvalid</structfield> <type>bool</type> + </para> + <para> + Whether the not-null constraint, if one exists, has been validated. </para></entry> </row> diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml index ece438f0075..a75e75d800d 100644 --- a/doc/src/sgml/ref/alter_table.sgml +++ b/doc/src/sgml/ref/alter_table.sgml @@ -243,6 +243,8 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM entire table; however, if a valid <literal>CHECK</literal> constraint is found which proves no <literal>NULL</literal> can exist, then the table scan is skipped. + If a column has an invalid not-null constraint, + <literal>SET NOT NULL</literal> validates it. </para> <para> @@ -458,8 +460,8 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM <para> This form adds a new constraint to a table using the same constraint syntax as <link linkend="sql-createtable"><command>CREATE TABLE</command></link>, plus the option <literal>NOT - VALID</literal>, which is currently only allowed for foreign key - and CHECK constraints. + VALID</literal>, which is currently only allowed for foreign key, + <literal>CHECK</literal> constraints and not-null constraints. </para> <para> @@ -586,7 +588,7 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM <term><literal>VALIDATE CONSTRAINT</literal></term> <listitem> <para> - This form validates a foreign key or check constraint that was + This form validates a foreign key, check, or not-null constraint that was previously created as <literal>NOT VALID</literal>, by scanning the table to ensure there are no rows for which the constraint is not satisfied. If the constraint is not enforced, an error is thrown. diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index ed2195f14b2..320de2cbda0 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -74,7 +74,8 @@ populate_compact_attribute_internal(Form_pg_attribute src, dst->atthasmissing = src->atthasmissing; dst->attisdropped = src->attisdropped; dst->attgenerated = (src->attgenerated != '\0'); - dst->attnotnull = src->attnotnull; + dst->attnullability = !src->attnotnull ? ATTNULLABLE_NONE : + src->attnotnullvalid ? ATTNULLABLE_VALID : ATTNULLABLE_INVALID; switch (src->attalign) { @@ -252,6 +253,7 @@ CreateTupleDescCopy(TupleDesc tupdesc) Form_pg_attribute att = TupleDescAttr(desc, i); att->attnotnull = false; + att->attnotnullvalid = false; att->atthasdef = false; att->atthasmissing = false; att->attidentity = '\0'; @@ -298,6 +300,7 @@ CreateTupleDescTruncatedCopy(TupleDesc tupdesc, int natts) Form_pg_attribute att = TupleDescAttr(desc, i); att->attnotnull = false; + att->attnotnullvalid = false; att->atthasdef = false; att->atthasmissing = false; att->attidentity = '\0'; @@ -418,6 +421,7 @@ TupleDescCopy(TupleDesc dst, TupleDesc src) Form_pg_attribute att = TupleDescAttr(dst, i); att->attnotnull = false; + att->attnotnullvalid = false; att->atthasdef = false; att->atthasmissing = false; att->attidentity = '\0'; @@ -464,6 +468,7 @@ TupleDescCopyEntry(TupleDesc dst, AttrNumber dstAttno, /* since we're not copying constraints or defaults, clear these */ dstAtt->attnotnull = false; + dstAtt->attnotnullvalid = false; dstAtt->atthasdef = false; dstAtt->atthasmissing = false; dstAtt->attidentity = '\0'; @@ -613,6 +618,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attnotnull != attr2->attnotnull) return false; + if (attr1->attnotnullvalid != attr2->attnotnullvalid) + return false; if (attr1->atthasdef != attr2->atthasdef) return false; if (attr1->attidentity != attr2->attidentity) @@ -841,6 +848,7 @@ TupleDescInitEntry(TupleDesc desc, att->attndims = attdim; att->attnotnull = false; + att->attnotnullvalid = false; att->atthasdef = false; att->atthasmissing = false; att->attidentity = '\0'; @@ -904,6 +912,7 @@ TupleDescInitBuiltinEntry(TupleDesc desc, att->attndims = attdim; att->attnotnull = false; + att->attnotnullvalid = false; att->atthasdef = false; att->atthasmissing = false; att->attidentity = '\0'; diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6db864892d0..44ec9e58520 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -615,6 +615,9 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attnotnull = true; } } + + /* Not-null constraints on system catalogs are always valid. */ + attrtypes[attnum]->attnotnullvalid = attrtypes[attnum]->attnotnull; } diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index df3231fcd41..bdf55d7dc8d 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -986,6 +986,9 @@ sub morph_row_for_pgattr $row->{attnotnull} = 'f'; } + # Not-null constraints on system catalogs are always valid. + $row->{attnotnullvalid} = $row->{attnotnull}; + Catalog::AddDefaultValues($row, $pgattr_schema, 'pg_attribute'); return; } diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index b807ab66668..503e7cb3f58 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -151,6 +151,7 @@ static const FormData_pg_attribute a1 = { .attalign = TYPALIGN_SHORT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, + .attnotnullvalid = true, .attislocal = true, }; @@ -164,6 +165,7 @@ static const FormData_pg_attribute a2 = { .attalign = TYPALIGN_INT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, + .attnotnullvalid = true, .attislocal = true, }; @@ -177,6 +179,7 @@ static const FormData_pg_attribute a3 = { .attalign = TYPALIGN_INT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, + .attnotnullvalid = true, .attislocal = true, }; @@ -190,6 +193,7 @@ static const FormData_pg_attribute a4 = { .attalign = TYPALIGN_INT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, + .attnotnullvalid = true, .attislocal = true, }; @@ -203,6 +207,7 @@ static const FormData_pg_attribute a5 = { .attalign = TYPALIGN_INT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, + .attnotnullvalid = true, .attislocal = true, }; @@ -222,6 +227,7 @@ static const FormData_pg_attribute a6 = { .attalign = TYPALIGN_INT, .attstorage = TYPSTORAGE_PLAIN, .attnotnull = true, + .attnotnullvalid = true, .attislocal = true, }; @@ -753,6 +759,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attstorage - 1] = CharGetDatum(attrs->attstorage); slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); slot[slotCount]->tts_values[Anum_pg_attribute_attnotnull - 1] = BoolGetDatum(attrs->attnotnull); + slot[slotCount]->tts_values[Anum_pg_attribute_attnotnullvalid - 1] = BoolGetDatum(attrs->attnotnullvalid); slot[slotCount]->tts_values[Anum_pg_attribute_atthasdef - 1] = BoolGetDatum(attrs->atthasdef); slot[slotCount]->tts_values[Anum_pg_attribute_atthasmissing - 1] = BoolGetDatum(attrs->atthasmissing); slot[slotCount]->tts_values[Anum_pg_attribute_attidentity - 1] = CharGetDatum(attrs->attidentity); @@ -1714,6 +1721,7 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Remove any not-null constraint the column may have */ attStruct->attnotnull = false; + attStruct->attnotnullvalid = false; /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; @@ -2616,12 +2624,17 @@ AddRelationNewConstraints(Relation rel, errmsg("cannot add not-null constraint on system column \"%s\"", strVal(linitial(cdef->keys)))); + Assert(cdef->initially_valid != cdef->skip_validation); + /* * If the column already has a not-null constraint, we don't want - * to add another one; just adjust inheritance status as needed. + * to add another one; adjust inheritance status as needed. This + * also checks whether the existing constraint matches the + * requested validity. */ if (AdjustNotNullInheritance(RelationGetRelid(rel), colnum, - is_local, cdef->is_no_inherit)) + is_local, cdef->is_no_inherit, + cdef->skip_validation)) continue; /* diff --git a/src/backend/catalog/pg_constraint.c b/src/backend/catalog/pg_constraint.c index b97960d2766..2f73085961b 100644 --- a/src/backend/catalog/pg_constraint.c +++ b/src/backend/catalog/pg_constraint.c @@ -576,8 +576,8 @@ ChooseConstraintName(const char *name1, const char *name2, /* * Find and return a copy of the pg_constraint tuple that implements a - * validated not-null constraint for the given column of the given relation. - * If no such constraint exists, return NULL. + * (possibly not valid) not-null constraint for the given column of the + * given relation. If no such constraint exists, return NULL. * * XXX This would be easier if we had pg_attribute.notnullconstr with the OID * of the constraint that implements the not-null constraint for that column. @@ -606,13 +606,11 @@ findNotNullConstraintAttnum(Oid relid, AttrNumber attnum) AttrNumber conkey; /* - * We're looking for a NOTNULL constraint that's marked validated, - * with the column we're looking for as the sole element in conkey. + * We're looking for a NOTNULL constraint with the column we're + * looking for as the sole element in conkey. */ if (con->contype != CONSTRAINT_NOTNULL) continue; - if (!con->convalidated) - continue; conkey = extractNotNullColumn(conTup); if (conkey != attnum) @@ -630,9 +628,10 @@ findNotNullConstraintAttnum(Oid relid, AttrNumber attnum) } /* - * Find and return the pg_constraint tuple that implements a validated - * not-null constraint for the given column of the given relation. If - * no such column or no such constraint exists, return NULL. + * Find and return a copy of the pg_constraint tuple that implements a + * (possibly not valid) not-null constraint for the given column of the + * given relation. + * If no such column or no such constraint exists, return NULL. */ HeapTuple findNotNullConstraint(Oid relid, const char *colname) @@ -723,15 +722,19 @@ extractNotNullColumn(HeapTuple constrTup) * * If no not-null constraint is found for the column, return false. * Caller can create one. + * * If a constraint exists but the connoinherit flag is not what the caller - * wants, throw an error about the incompatibility. Otherwise, we adjust - * conislocal/coninhcount and return true. - * In the latter case, if is_local is true we flip conislocal true, or do - * nothing if it's already true; otherwise we increment coninhcount by 1. + * wants, throw an error about the incompatibility. If the desired + * constraint is valid but the existing constraint is not valid, also + * throw an error about that (the opposite case is acceptable). + * + * If everything checks out, we adjust conislocal/coninhcount and return + * true. If is_local is true we flip conislocal true, or do nothing if + * it's already true; otherwise we increment coninhcount by 1. */ bool AdjustNotNullInheritance(Oid relid, AttrNumber attnum, - bool is_local, bool is_no_inherit) + bool is_local, bool is_no_inherit, bool is_notvalid) { HeapTuple tup; @@ -755,6 +758,17 @@ AdjustNotNullInheritance(Oid relid, AttrNumber attnum, errmsg("cannot change NO INHERIT status of NOT NULL constraint \"%s\" on relation \"%s\"", NameStr(conform->conname), get_rel_name(relid))); + /* + * Throw an error if the existing constraint is NOT VALID and caller + * wants a valid one. + */ + if (!is_notvalid && !conform->convalidated) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("incompatible NOT VALID constraint \"%s\" on relation \"%s\"", + NameStr(conform->conname), get_rel_name(relid)), + errhint("You will need to use ALTER TABLE ... VALIDATE CONSTRAINT to validate it.")); + if (!is_local) { if (pg_add_s16_overflow(conform->coninhcount, 1, @@ -832,7 +846,7 @@ RelationGetNotNullConstraints(Oid relid, bool cooked, bool include_noinh) cooked->attnum = colnum; cooked->expr = NULL; cooked->is_enforced = true; - cooked->skip_validation = false; + cooked->skip_validation = !conForm->convalidated; cooked->is_local = true; cooked->inhcount = 0; cooked->is_no_inherit = conForm->connoinherit; @@ -852,7 +866,7 @@ RelationGetNotNullConstraints(Oid relid, bool cooked, bool include_noinh) constr->keys = list_make1(makeString(get_attname(relid, colnum, false))); constr->is_enforced = true; - constr->skip_validation = false; + constr->skip_validation = !conForm->convalidated; constr->initially_valid = true; constr->is_no_inherit = conForm->connoinherit; notnulls = lappend(notnulls, constr); diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 4397123398e..0c7b1231c2e 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -435,6 +435,9 @@ static void QueueFKConstraintValidation(List **wqueue, Relation conrel, Relation static void QueueCheckConstraintValidation(List **wqueue, Relation conrel, Relation rel, char *constrName, HeapTuple contuple, bool recurse, bool recursing, LOCKMODE lockmode); +static void QueueNNConstraintValidation(List **wqueue, Relation conrel, Relation rel, + HeapTuple contuple, bool recurse, bool recursing, + LOCKMODE lockmode); static int transformColumnNameList(Oid relId, List *colList, int16 *attnums, Oid *atttypids, Oid *attcollids); static int transformFkeyGetPrimaryKey(Relation pkrel, Oid *indexOid, @@ -498,7 +501,7 @@ static void add_column_collation_dependency(Oid relid, int32 attnum, Oid collid) static ObjectAddress ATExecDropNotNull(Relation rel, const char *colName, bool recurse, LOCKMODE lockmode); static void set_attnotnull(List **wqueue, Relation rel, AttrNumber attnum, - LOCKMODE lockmode); + bool is_valid, bool queue_validation); static ObjectAddress ATExecSetNotNull(List **wqueue, Relation rel, char *constrname, char *colName, bool recurse, bool recursing, @@ -1340,7 +1343,7 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, nncols = AddRelationNotNullConstraints(rel, stmt->nnconstraints, old_notnulls); foreach_int(attrnum, nncols) - set_attnotnull(NULL, rel, attrnum, NoLock); + set_attnotnull(NULL, rel, attrnum, true, false); ObjectAddressSet(address, RelationRelationId, relationId); @@ -1424,7 +1427,7 @@ BuildDescForRelation(const List *columns) TupleDescInitEntryCollation(desc, attnum, attcollation); /* Fill in additional stuff not handled by TupleDescInitEntry */ - att->attnotnull = entry->is_not_null; + att->attnotnull = att->attnotnullvalid = entry->is_not_null; att->attislocal = entry->is_local; att->attinhcount = entry->inhcount; att->attidentity = entry->identity; @@ -2738,7 +2741,7 @@ MergeAttributes(List *columns, const List *supers, char relpersistence, /* * Request attnotnull on columns that have a not-null constraint - * that's not marked NO INHERIT. + * that's not marked NO INHERIT (even if not valid). */ nnconstrs = RelationGetNotNullConstraints(RelationGetRelid(relation), true, false); @@ -6207,18 +6210,22 @@ ATRewriteTable(AlteredTableInfo *tab, Oid OIDNewHeap) { /* * If we are rebuilding the tuples OR if we added any new but not - * verified not-null constraints, check all not-null constraints. This - * is a bit of overkill but it minimizes risk of bugs. + * verified not-null constraints, check all valid not-null constraints. + * This is a bit of overkill but it minimizes risk of bugs. * * notnull_attrs does *not* collect attribute numbers for not-null * constraints over virtual generated columns; instead, they are * collected in notnull_virtual_attrs. + * + * But we don't need check invalid not-null constraint! this is aligned + * with check constraint behavior. */ for (i = 0; i < newTupDesc->natts; i++) { Form_pg_attribute attr = TupleDescAttr(newTupDesc, i); - if (attr->attnotnull && !attr->attisdropped) + if (attr->attnotnull && attr->attnotnullvalid && + !attr->attisdropped) { if (attr->attgenerated != ATTRIBUTE_GENERATED_VIRTUAL) notnull_attrs = lappend_int(notnull_attrs, attr->attnum); @@ -7788,7 +7795,7 @@ ATExecDropNotNull(Relation rel, const char *colName, bool recurse, /* * Find the constraint that makes this column NOT NULL, and drop it. - * dropconstraint_internal() resets attnotnull. + * dropconstraint_internal() resets attnotnull/attnotnullvalid. */ conTup = findNotNullConstraintAttnum(RelationGetRelid(rel), attnum); if (conTup == NULL) @@ -7809,19 +7816,23 @@ ATExecDropNotNull(Relation rel, const char *colName, bool recurse, } /* - * Helper to set pg_attribute.attnotnull if it isn't set, and to tell phase 3 - * to verify it. + * set_attnotnull + * Helper to update/validate the pg_attribute status of a not-null + * constraint * - * When called to alter an existing table, 'wqueue' must be given so that we - * can queue a check that existing tuples pass the constraint. When called - * from table creation, 'wqueue' should be passed as NULL. + * pg_attribute.attnotnull is set true, if it isn't already. If is_valid + * is true, also set pg_attribute.attnotnullvalid. If queue_validation is + * true, also set up wqueue to validate the constraint. wqueue may be given + * as NULL when validation is not needed (e.g., on table creation). */ static void set_attnotnull(List **wqueue, Relation rel, AttrNumber attnum, - LOCKMODE lockmode) + bool is_valid, bool queue_validation) { Form_pg_attribute attr; + Assert(!queue_validation || wqueue); + CheckAlterTableIsSafe(rel); /* @@ -7832,7 +7843,7 @@ set_attnotnull(List **wqueue, Relation rel, AttrNumber attnum, if (attr->attisdropped) return; - if (!attr->attnotnull) + if (!attr->attnotnull || (is_valid && !attr->attnotnullvalid)) { Relation attr_rel; HeapTuple tuple; @@ -7845,15 +7856,17 @@ set_attnotnull(List **wqueue, Relation rel, AttrNumber attnum, attnum, RelationGetRelid(rel)); attr = (Form_pg_attribute) GETSTRUCT(tuple); - Assert(!attr->attnotnull); + attr->attnotnull = true; + attr->attnotnullvalid = is_valid; CatalogTupleUpdate(attr_rel, &tuple->t_self, tuple); /* * If the nullness isn't already proven by validated constraints, have * ALTER TABLE phase 3 test for it. */ - if (wqueue && !NotNullImpliedByRelConstraints(rel, attr)) + if (queue_validation && wqueue && + !NotNullImpliedByRelConstraints(rel, attr)) { AlteredTableInfo *tab; @@ -7951,6 +7964,15 @@ ATExecSetNotNull(List **wqueue, Relation rel, char *conName, char *colName, conForm->conislocal = true; changed = true; } + else if (!conForm->convalidated) + { + /* + * Flip attnotnull and convalidated, and also validate the + * constraint. + */ + return ATExecValidateConstraint(wqueue, rel, NameStr(conForm->conname), + recurse, recursing, lockmode); + } if (changed) { @@ -8013,8 +8035,8 @@ ATExecSetNotNull(List **wqueue, Relation rel, char *conName, char *colName, InvokeObjectPostAlterHook(RelationRelationId, RelationGetRelid(rel), attnum); - /* Mark pg_attribute.attnotnull for the column */ - set_attnotnull(wqueue, rel, attnum, lockmode); + /* Mark pg_attribute.attnotnull for the column and queue validation */ + set_attnotnull(wqueue, rel, attnum, true, true); /* * Recurse to propagate the constraint to children that don't have one. @@ -9417,7 +9439,6 @@ ATPrepAddPrimaryKey(List **wqueue, Relation rel, AlterTableCmd *cmd, bool recurse, LOCKMODE lockmode, AlterTableUtilityContext *context) { - ListCell *lc; Constraint *pkconstr; pkconstr = castNode(Constraint, cmd->def); @@ -9436,33 +9457,73 @@ ATPrepAddPrimaryKey(List **wqueue, Relation rel, AlterTableCmd *cmd, lockmode); foreach_oid(childrelid, children) { - foreach(lc, pkconstr->keys) + foreach_node(String, attname, pkconstr->keys) { HeapTuple tup; Form_pg_attribute attrForm; - char *attname = strVal(lfirst(lc)); - tup = SearchSysCacheAttName(childrelid, attname); + tup = SearchSysCacheAttName(childrelid, strVal(attname)); if (!tup) elog(ERROR, "cache lookup failed for attribute %s of relation %u", - attname, childrelid); + strVal(attname), childrelid); attrForm = (Form_pg_attribute) GETSTRUCT(tup); if (!attrForm->attnotnull) ereport(ERROR, errmsg("column \"%s\" of table \"%s\" is not marked NOT NULL", - attname, get_rel_name(childrelid))); + strVal(attname), get_rel_name(childrelid))); ReleaseSysCache(tup); } } } - /* Insert not-null constraints in the queue for the PK columns */ - foreach(lc, pkconstr->keys) + /* Verify that columns are not-null, or request that they be made so */ + foreach_node(String, column, pkconstr->keys) { AlterTableCmd *newcmd; Constraint *nnconstr; + HeapTuple tuple; - nnconstr = makeNotNullConstraint(lfirst(lc)); + /* + * First check if a suitable constraint exists. If it does, we don't + * need to request another one. We do need to bail out if it's not + * valid, though. + */ + tuple = findNotNullConstraint(RelationGetRelid(rel), strVal(column)); + if (tuple != NULL) + { + Form_pg_constraint conForm = (Form_pg_constraint) GETSTRUCT(tuple); + + /* a NO INHERIT constraint is no good */ + if (conForm->connoinherit) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot create primary key on column \"%s\"", + strVal(column)), + /*- translator: third %s is a constraint characteristic such as NOT VALID */ + errdetail("The constraint \"%s\" on column \"%s\", marked %s, is incompatible with a primary key.", + NameStr(conForm->conname), strVal(column), "NO INHERIT"), + errhint("You will need to make it inheritable using %s.", + "ALTER TABLE ... ALTER CONSTRAINT ... INHERIT")); + + /* an unvalidated constraint is no good */ + if (!conForm->convalidated) + ereport(ERROR, + errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot create primary key on column \"%s\"", + strVal(column)), + /*- translator: third %s is a constraint characteristic such as NOT VALID */ + errdetail("The constraint \"%s\" on column \"%s\", marked %s, is incompatible with a primary key.", + NameStr(conForm->conname), strVal(column), "NOT VALID"), + errhint("You will need to validate it using %s.", + "ALTER TABLE ... VALIDATE CONSTRAINT")); + + /* All good with this one; don't request another */ + heap_freetuple(tuple); + continue; + } + + /* This column is not already not-null, so add it to the queue */ + nnconstr = makeNotNullConstraint(column); newcmd = makeNode(AlterTableCmd); newcmd->subtype = AT_AddConstraint; @@ -9836,11 +9897,15 @@ ATAddCheckNNConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel, constr->conname = ccon->name; /* - * If adding a not-null constraint, set the pg_attribute flag and tell - * phase 3 to verify existing rows, if needed. + * If adding a valid not-null constraint, set the pg_attribute flag + * and tell phase 3 to verify existing rows, if needed. For an + * invalid constraint, just set attnotnull and attnotnullvalid, + * without queueing verification. */ if (constr->contype == CONSTR_NOTNULL) - set_attnotnull(wqueue, rel, ccon->attnum, lockmode); + set_attnotnull(wqueue, rel, ccon->attnum, + !constr->skip_validation, + !constr->skip_validation); ObjectAddressSet(address, ConstraintRelationId, ccon->conoid); } @@ -12811,10 +12876,11 @@ ATExecValidateConstraint(List **wqueue, Relation rel, char *constrName, con = (Form_pg_constraint) GETSTRUCT(tuple); if (con->contype != CONSTRAINT_FOREIGN && - con->contype != CONSTRAINT_CHECK) + con->contype != CONSTRAINT_CHECK && + con->contype != CONSTRAINT_NOTNULL) ereport(ERROR, (errcode(ERRCODE_WRONG_OBJECT_TYPE), - errmsg("constraint \"%s\" of relation \"%s\" is not a foreign key or check constraint", + errmsg("constraint \"%s\" of relation \"%s\" is not a foreign key, check, or not-null constraint", constrName, RelationGetRelationName(rel)))); if (!con->conenforced) @@ -12833,6 +12899,11 @@ ATExecValidateConstraint(List **wqueue, Relation rel, char *constrName, QueueCheckConstraintValidation(wqueue, conrel, rel, constrName, tuple, recurse, recursing, lockmode); } + else if (con->contype == CONSTRAINT_NOTNULL) + { + QueueNNConstraintValidation(wqueue, conrel, rel, + tuple, recurse, recursing, lockmode); + } ObjectAddressSet(address, ConstraintRelationId, con->oid); } @@ -13049,6 +13120,109 @@ QueueCheckConstraintValidation(List **wqueue, Relation conrel, Relation rel, heap_freetuple(copyTuple); } +/* + * QueueNNConstraintValidation + * + * Add an entry to the wqueue to validate the given not-null constraint in + * Phase 3 and update the convalidated field in the pg_constraint catalog for + * the specified relation and all its inheriting children. + */ +static void +QueueNNConstraintValidation(List **wqueue, Relation conrel, Relation rel, + HeapTuple contuple, bool recurse, bool recursing, + LOCKMODE lockmode) +{ + Form_pg_constraint con; + AlteredTableInfo *tab; + HeapTuple copyTuple; + Form_pg_constraint copy_con; + List *children = NIL; + AttrNumber attnum; + char *colname; + + con = (Form_pg_constraint) GETSTRUCT(contuple); + Assert(con->contype == CONSTRAINT_NOTNULL); + + attnum = extractNotNullColumn(contuple); + + /* + * If we're recursing, we've already done this for parent, so skip it. + * Also, if the constraint is a NO INHERIT constraint, we shouldn't try to + * look for it in the children. + * + * We recurse before validating on the parent, to reduce risk of + * deadlocks. + */ + if (!recursing && !con->connoinherit) + children = find_all_inheritors(RelationGetRelid(rel), lockmode, NULL); + + colname = get_attname(RelationGetRelid(rel), attnum, false); + foreach_oid(childoid, children) + { + Relation childrel; + HeapTuple contup; + Form_pg_constraint childcon; + char *conname; + + if (childoid == RelationGetRelid(rel)) + continue; + + /* + * If we are told not to recurse, there had better not be any child + * tables, because we can't mark the constraint on the parent valid + * unless it is valid for all child tables. + */ + if (!recurse) + ereport(ERROR, + errcode(ERRCODE_INVALID_TABLE_DEFINITION), + errmsg("constraint must be validated on child tables too")); + + /* + * The column on child might have a different attnum, so search by + * column name. + */ + contup = findNotNullConstraint(childoid, colname); + if (!contup) + elog(ERROR, "cache lookup failed for not-null constraint on column \"%s\" of relation \"%s\"", + colname, get_rel_name(childoid)); + childcon = (Form_pg_constraint) GETSTRUCT(contup); + if (childcon->convalidated) + continue; + + /* find_all_inheritors already got lock */ + childrel = table_open(childoid, NoLock); + conname = pstrdup(NameStr(childcon->conname)); + + /* XXX improve ATExecValidateConstraint API to avoid double search */ + ATExecValidateConstraint(wqueue, childrel, conname, + false, true, lockmode); + table_close(childrel, NoLock); + } + + /* Set the flags appropriately without queueing another validation */ + set_attnotnull(NULL, rel, attnum, true, false); + + tab = ATGetQueueEntry(wqueue, rel); + tab->verify_new_notnull = true; + + /* + * Invalidate relcache so that others see the new validated constraint. + */ + CacheInvalidateRelcache(rel); + + /* + * Now update the catalogs, while we have the door open. + */ + copyTuple = heap_copytuple(contuple); + copy_con = (Form_pg_constraint) GETSTRUCT(copyTuple); + copy_con->convalidated = true; + CatalogTupleUpdate(conrel, ©Tuple->t_self, copyTuple); + + InvokeObjectPostAlterHook(ConstraintRelationId, con->oid, 0); + + heap_freetuple(copyTuple); +} + /* * transformColumnNameList - transform list of column names * @@ -13917,10 +14091,11 @@ dropconstraint_internal(Relation rel, HeapTuple constraintTup, DropBehavior beha false), RelationGetRelationName(rel))); - /* All good -- reset attnotnull if needed */ + /* All good -- reset attnotnull and attnotnullvalid if needed */ if (attForm->attnotnull) { attForm->attnotnull = false; + attForm->attnotnullvalid = false; CatalogTupleUpdate(attrel, &atttup->t_self, atttup); } @@ -19772,7 +19947,8 @@ PartConstraintImpliedByRelConstraint(Relation scanrel, { Form_pg_attribute att = TupleDescAttr(scanrel->rd_att, i - 1); - if (att->attnotnull && !att->attisdropped) + /* invalid not-null constraint must be ignored */ + if (att->attnotnull && att->attnotnullvalid && !att->attisdropped) { NullTest *ntest = makeNode(NullTest); diff --git a/src/backend/executor/execMain.c b/src/backend/executor/execMain.c index 2da848970be..c8d44e3086b 100644 --- a/src/backend/executor/execMain.c +++ b/src/backend/executor/execMain.c @@ -2074,6 +2074,7 @@ ExecConstraints(ResultRelInfo *resultRelInfo, { for (AttrNumber attnum = 1; attnum <= tupdesc->natts; attnum++) { + /* FIXME use CompactAttribute */ Form_pg_attribute att = TupleDescAttr(tupdesc, attnum - 1); if (att->attnotnull && att->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL) diff --git a/src/backend/jit/llvm/llvmjit_deform.c b/src/backend/jit/llvm/llvmjit_deform.c index 5d169c7a40b..c562edd094b 100644 --- a/src/backend/jit/llvm/llvmjit_deform.c +++ b/src/backend/jit/llvm/llvmjit_deform.c @@ -123,7 +123,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc, * combination of attisdropped && attnotnull combination shouldn't * exist. */ - if (att->attnotnull && + if (att->attnullability == ATTNULLABLE_VALID && !att->atthasmissing && !att->attisdropped) guaranteed_column_number = attnum; @@ -438,7 +438,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc, * into account, because if they're present the heaptuple's natts * would have indicated that a slot_getmissingattrs() is needed. */ - if (!att->attnotnull) + if (att->attnullability != ATTNULLABLE_VALID) { LLVMBasicBlockRef b_ifnotnull; LLVMBasicBlockRef b_ifnull; @@ -604,7 +604,8 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc, known_alignment = -1; attguaranteedalign = false; } - else if (att->attnotnull && attguaranteedalign && known_alignment >= 0) + else if (att->attnullability == ATTNULLABLE_VALID && + attguaranteedalign && known_alignment >= 0) { /* * If the offset to the column was previously known, a NOT NULL & @@ -614,7 +615,8 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc, Assert(att->attlen > 0); known_alignment += att->attlen; } - else if (att->attnotnull && (att->attlen % alignto) == 0) + else if (att->attnullability == ATTNULLABLE_VALID && + (att->attlen % alignto) == 0) { /* * After a NOT NULL fixed-width column with a length that is a diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index 67d879be8b8..191a14b876e 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -177,7 +177,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, { CompactAttribute *attr = TupleDescCompactAttr(relation->rd_att, i); - if (attr->attnotnull) + if (attr->attnullability == ATTNULLABLE_VALID) { rel->notnullattnums = bms_add_member(rel->notnullattnums, i + 1); @@ -1251,6 +1251,7 @@ get_relation_data_width(Oid relid, int32 *attr_widths) * get_relation_constraints * * Retrieve the applicable constraint expressions of the given relation. + * Only constraints that have been validated are considered. * * Returns a List (possibly empty) of constraint expressions. Each one * has been canonicalized, and its Vars are changed to have the varno @@ -1353,7 +1354,7 @@ get_relation_constraints(PlannerInfo *root, { Form_pg_attribute att = TupleDescAttr(relation->rd_att, i - 1); - if (att->attnotnull && !att->attisdropped) + if (att->attnotnull && att->attnotnullvalid && !att->attisdropped) { NullTest *ntest = makeNode(NullTest); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index f1156e2fca3..3c4268b271a 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -4217,11 +4217,10 @@ ConstraintElem: n->contype = CONSTR_NOTNULL; n->location = @1; n->keys = list_make1(makeString($3)); - /* no NOT VALID support yet */ processCASbits($4, @4, "NOT NULL", - NULL, NULL, NULL, NULL, + NULL, NULL, NULL, &n->skip_validation, &n->is_no_inherit, yyscanner); - n->initially_valid = true; + n->initially_valid = !n->skip_validation; $$ = (Node *) n; } | UNIQUE opt_unique_null_treatment '(' columnList opt_without_overlaps ')' opt_c_include opt_definition OptConsTableSpace diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 9ad7681f155..70c11529b90 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1142,6 +1142,7 @@ CatalogCacheInitializeCache(CatCache *cache) keytype = attr->atttypid; /* cache key columns should always be NOT NULL */ Assert(attr->attnotnull); + Assert(attr->attnotnullvalid); } else { diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index 18a14ae186e..c80e40cd47a 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -592,7 +592,7 @@ RelationBuildTupleDesc(Relation relation) /* Update constraint/default info */ if (attp->attnotnull) - constr->has_not_null = true; + constr->has_not_null = true; /* invalid ones included */ if (attp->attgenerated == ATTRIBUTE_GENERATED_STORED) constr->has_generated_stored = true; if (attp->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL) @@ -3573,6 +3573,7 @@ RelationBuildLocalRelation(const char *relname, datt->attidentity = satt->attidentity; datt->attgenerated = satt->attgenerated; datt->attnotnull = satt->attnotnull; + datt->attnotnullvalid = satt->attnotnullvalid; has_not_null |= satt->attnotnull; populate_compact_attribute(rel->rd_att, i); } diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 8e6364d32d7..17c2996faa7 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -350,8 +350,10 @@ static void buildMatViewRefreshDependencies(Archive *fout); static void getTableDataFKConstraints(void); static void determineNotNullFlags(Archive *fout, PGresult *res, int r, TableInfo *tbinfo, int j, - int i_notnull_name, int i_notnull_noinherit, - int i_notnull_islocal); + int i_notnull_name, int i_notnull_invalidoid, + int i_notnull_noinherit, + int i_notnull_islocal, + PQExpBuffer *invalidnotnulloids); static char *format_function_arguments(const FuncInfo *finfo, const char *funcargs, bool is_agg); static char *format_function_signature(Archive *fout, @@ -8984,6 +8986,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) PQExpBuffer q = createPQExpBuffer(); PQExpBuffer tbloids = createPQExpBuffer(); PQExpBuffer checkoids = createPQExpBuffer(); + PQExpBuffer invalidnotnulloids = NULL; PGresult *res; int ntups; int curtblindx; @@ -9003,6 +9006,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) int i_notnull_name; int i_notnull_noinherit; int i_notnull_islocal; + int i_notnull_invalidoid; int i_attoptions; int i_attcollation; int i_attcompression; @@ -9089,6 +9093,10 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) * attnotnull (this cues dumpTableSchema to print the NOT NULL clause * without a name); also, such cases are never NO INHERIT. * + * For invalid constraints, we need to store their OIDs for processing + * elsewhere, so we bring the pg_constraint.oid value when the constraint + * is invalid, and NULL otherwise. + * * We track in notnull_islocal whether the constraint was defined directly * in this table or via an ancestor, for binary upgrade. flagInhAttrs * might modify this later for servers older than 18; it's also in charge @@ -9097,11 +9105,14 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) if (fout->remoteVersion >= 180000) appendPQExpBufferStr(q, "co.conname AS notnull_name,\n" + "CASE WHEN NOT co.convalidated THEN co.oid " + "ELSE NULL END AS notnull_invalidoid,\n" "co.connoinherit AS notnull_noinherit,\n" "co.conislocal AS notnull_islocal,\n"); else appendPQExpBufferStr(q, "CASE WHEN a.attnotnull THEN '' ELSE NULL END AS notnull_name,\n" + "NULL AS notnull_invalidoid,\n" "false AS notnull_noinherit,\n" "a.attislocal AS notnull_islocal,\n"); @@ -9176,6 +9187,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) i_attalign = PQfnumber(res, "attalign"); i_attislocal = PQfnumber(res, "attislocal"); i_notnull_name = PQfnumber(res, "notnull_name"); + i_notnull_invalidoid = PQfnumber(res, "notnull_invalidoid"); i_notnull_noinherit = PQfnumber(res, "notnull_noinherit"); i_notnull_islocal = PQfnumber(res, "notnull_islocal"); i_attoptions = PQfnumber(res, "attoptions"); @@ -9272,8 +9284,11 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) /* Handle not-null constraint name and flags */ determineNotNullFlags(fout, res, r, tbinfo, j, - i_notnull_name, i_notnull_noinherit, - i_notnull_islocal); + i_notnull_name, + i_notnull_invalidoid, + i_notnull_noinherit, + i_notnull_islocal, + &invalidnotnulloids); tbinfo->attoptions[j] = pg_strdup(PQgetvalue(res, r, i_attoptions)); tbinfo->attcollation[j] = atooid(PQgetvalue(res, r, i_attcollation)); @@ -9294,6 +9309,10 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) } } + /* If invalidnotnulloids has any data, finalize it */ + if (invalidnotnulloids != NULL) + appendPQExpBufferChar(invalidnotnulloids, '}'); + PQclear(res); /* @@ -9426,6 +9445,103 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) PQclear(res); } + /* + * Get info about NOT NULL NOT VALID constraints. This is skipped for a + * data-only dump, as it is only needed for table schemas. + */ + if (dopt->dumpSchema && invalidnotnulloids) + { + ConstraintInfo *constrs; + int numConstrs; + int i_tableoid; + int i_oid; + int i_conrelid; + int i_conname; + int i_consrc; + int i_conislocal; + + pg_log_info("finding invalid not null constraints"); + + resetPQExpBuffer(q); + appendPQExpBuffer(q, + "SELECT c.tableoid, c.oid, conrelid, conname, " + "pg_catalog.pg_get_constraintdef(c.oid) AS consrc, " + "conislocal, convalidated " + "FROM unnest('%s'::pg_catalog.oid[]) AS src(conoid)\n" + "JOIN pg_catalog.pg_constraint c ON (src.conoid = c.oid)\n" + "ORDER BY c.conrelid, c.conname", + invalidnotnulloids->data); + + res = ExecuteSqlQuery(fout, q->data, PGRES_TUPLES_OK); + + numConstrs = PQntuples(res); + constrs = (ConstraintInfo *) pg_malloc(numConstrs * sizeof(ConstraintInfo)); + + i_tableoid = PQfnumber(res, "tableoid"); + i_oid = PQfnumber(res, "oid"); + i_conrelid = PQfnumber(res, "conrelid"); + i_conname = PQfnumber(res, "conname"); + i_consrc = PQfnumber(res, "consrc"); + i_conislocal = PQfnumber(res, "conislocal"); + + /* As above, this loop iterates once per table, not once per row */ + curtblindx = -1; + for (int j = 0; j < numConstrs;) + { + Oid conrelid = atooid(PQgetvalue(res, j, i_conrelid)); + TableInfo *tbinfo = NULL; + int numcons; + + /* Count rows for this table */ + for (numcons = 1; numcons < numConstrs - j; numcons++) + if (atooid(PQgetvalue(res, j + numcons, i_conrelid)) != conrelid) + break; + + /* + * Locate the associated TableInfo; we rely on tblinfo[] being in + * OID order. + */ + while (++curtblindx < numTables) + { + tbinfo = &tblinfo[curtblindx]; + if (tbinfo->dobj.catId.oid == conrelid) + break; + } + if (curtblindx >= numTables) + pg_fatal("unrecognized table OID %u", conrelid); + + for (int c = 0; c < numcons; c++, j++) + { + constrs[j].dobj.objType = DO_CONSTRAINT; + constrs[j].dobj.catId.tableoid = atooid(PQgetvalue(res, j, i_tableoid)); + constrs[j].dobj.catId.oid = atooid(PQgetvalue(res, j, i_oid)); + AssignDumpId(&constrs[j].dobj); + constrs[j].dobj.name = pg_strdup(PQgetvalue(res, j, i_conname)); + constrs[j].dobj.namespace = tbinfo->dobj.namespace; + constrs[j].contable = tbinfo; + constrs[j].condomain = NULL; + constrs[j].contype = 'n'; + constrs[j].condef = pg_strdup(PQgetvalue(res, j, i_consrc)); + constrs[j].confrelid = InvalidOid; + constrs[j].conindex = 0; + constrs[j].condeferrable = false; + constrs[j].condeferred = false; + constrs[j].conislocal = (PQgetvalue(res, j, i_conislocal)[0] == 't'); + + /* + * All invalid not-null constraints must be dumped separately, + * because CREATE TABLE would not create them as invalid, and + * also because they must be created after potentially + * violating data has been loaded. + */ + constrs[j].separate = true; + + constrs[j].dobj.dump = tbinfo->dobj.dump; + } + } + PQclear(res); + } + /* * Get info about table CHECK constraints. This is skipped for a * data-only dump, as it is only needed for table schemas. @@ -9570,18 +9686,23 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) * * Result row 'r' is for tbinfo's attribute 'j'. * - * There are three possibilities: + * There are four possibilities: * 1) the column has no not-null constraints. In that case, ->notnull_constrs * (the constraint name) remains NULL. * 2) The column has a constraint with no name (this is the case when * constraints come from pre-18 servers). In this case, ->notnull_constrs * is set to the empty string; dumpTableSchema will print just "NOT NULL". - * 3) The column has a constraint with a known name; in that case + * 3) The column has an invalid not-null constraint. This must be treated + * as a separate object (because it must be created after the table data + * is loaded). So we add its OID to invalidnotnulloids for processing + * elsewhere and do nothing further with it here. We distinguish this + * case because the "notnull_invalidoid" column has been set to a non-NULL + * value, which is the constraint OID. Valid constraints have a null OID. + * 4) The column has a constraint with a known name; in that case * notnull_constrs carries that name and dumpTableSchema will print * "CONSTRAINT the_name NOT NULL". However, if the name is the default * (table_column_not_null), there's no need to print that name in the dump, - * so notnull_constrs is set to the empty string and it behaves as the case - * above. + * so notnull_constrs is set to the empty string and it behaves as case 2. * * In a child table that inherits from a parent already containing NOT NULL * constraints and the columns in the child don't have their own NOT NULL @@ -9593,7 +9714,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) * Any of these constraints might have the NO INHERIT bit. If so we set * ->notnull_noinh and NO INHERIT will be printed by dumpTableSchema. * - * In case 3 above, the name comparison is a bit of a hack; it actually fails + * In case 4 above, the name comparison is a bit of a hack; it actually fails * to do the right thing in all but the trivial case. However, the downside * of getting it wrong is simply that the name is printed rather than * suppressed, so it's not a big deal. @@ -9601,11 +9722,41 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) static void determineNotNullFlags(Archive *fout, PGresult *res, int r, TableInfo *tbinfo, int j, - int i_notnull_name, int i_notnull_noinherit, - int i_notnull_islocal) + int i_notnull_name, + int i_notnull_invalidoid, + int i_notnull_noinherit, + int i_notnull_islocal, + PQExpBuffer *invalidnotnulloids) { DumpOptions *dopt = fout->dopt; + /* + * If this not-null constraint is not valid, list its OID in + * invalidnotnulloids and do nothing further. It'll be processed + * elsewhere later. + * + * Because invalid not-null constraints are rare, we don't want to malloc + * invalidnotnulloids until we're sure we're going it need it, which + * happens here. + */ + if (!PQgetisnull(res, r, i_notnull_invalidoid)) + { + char *constroid = PQgetvalue(res, r, i_notnull_invalidoid); + + if (*invalidnotnulloids == NULL) + { + *invalidnotnulloids = createPQExpBuffer(); + appendPQExpBufferChar(*invalidnotnulloids, '{'); + appendPQExpBuffer(*invalidnotnulloids, "%s", constroid); + } + else + appendPQExpBuffer(*invalidnotnulloids, ",%s", constroid); + + /* nothing else to do */ + tbinfo->notnull_constrs[j] = NULL; + return; + } + /* * notnull_noinh is straight from the query result. notnull_islocal also, * though flagInhAttrs may change that one later in versions < 18. @@ -18152,13 +18303,20 @@ dumpConstraint(Archive *fout, const ConstraintInfo *coninfo) .createStmt = q->data, .dropStmt = delq->data)); } - else if (coninfo->contype == 'c' && tbinfo) + else if ((coninfo->contype == 'c' || coninfo->contype == 'n') && tbinfo) { - /* CHECK constraint on a table */ + /* CHECK or invalid not-null constraint on a table */ /* Ignore if not to be dumped separately, or if it was inherited */ if (coninfo->separate && coninfo->conislocal) { + const char *keyword; + + if (coninfo->contype == 'c') + keyword = "CHECK CONSTRAINT"; + else + keyword = "CONSTRAINT"; + /* not ONLY since we want it to propagate to children */ appendPQExpBuffer(q, "ALTER %sTABLE %s\n", foreign, fmtQualifiedDumpable(tbinfo)); @@ -18178,7 +18336,7 @@ dumpConstraint(Archive *fout, const ConstraintInfo *coninfo) ARCHIVE_OPTS(.tag = tag, .namespace = tbinfo->dobj.namespace->dobj.name, .owner = tbinfo->rolname, - .description = "CHECK CONSTRAINT", + .description = keyword, .section = SECTION_POST_DATA, .createStmt = q->data, .dropStmt = delq->data)); diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index e6f0f86a459..b426b5e4736 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -498,6 +498,8 @@ typedef struct _evttriggerInfo * use a different objType for foreign key constraints, to make it easier * to sort them the way we want. * + * Not-null constraints don't need this, unless they are NOT VALID. + * * Note: condeferrable and condeferred are currently only valid for * unique/primary-key constraints. Otherwise that info is in condef. */ diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 576326daec7..6c03eca8e50 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -744,8 +744,8 @@ my %pgdump_runs = ( schema_only_with_statistics => { dump_cmd => [ 'pg_dump', '--no-sync', - "--file=$tempdir/schema_only_with_statistics.sql", '--schema-only', - '--with-statistics', 'postgres', + "--file=$tempdir/schema_only_with_statistics.sql", + '--schema-only', '--with-statistics', 'postgres', ], }, no_schema => { @@ -1118,6 +1118,23 @@ my %tests = ( }, }, + 'CONSTRAINT NOT NULL / INVALID' => { + create_sql => 'CREATE TABLE dump_test.test_table_nn ( + col1 int); + ALTER TABLE dump_test.test_table_nn ADD CONSTRAINT nn NOT NULL col1 NOT VALID;', + regexp => qr/^ + \QALTER TABLE dump_test.test_table_nn\E \n^\s+ + \QADD CONSTRAINT nn NOT NULL col1 NOT VALID;\E + /xm, + like => { + %full_runs, %dump_test_schema_runs, section_post_data => 1, + }, + unlike => { + exclude_dump_test_schema => 1, + only_dump_measurement => 1, + }, + }, + 'CONSTRAINT PRIMARY KEY / WITHOUT OVERLAPS' => { create_sql => 'CREATE TABLE dump_test.test_table_tpk ( col1 int4range, diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 8970677ac64..1d08268393e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -3106,7 +3106,8 @@ describeOneTableDetails(const char *schemaname, { printfPQExpBuffer(&buf, "SELECT c.conname, a.attname, c.connoinherit,\n" - " c.conislocal, c.coninhcount <> 0\n" + " c.conislocal, c.coninhcount <> 0,\n" + " c.convalidated\n" "FROM pg_catalog.pg_constraint c JOIN\n" " pg_catalog.pg_attribute a ON\n" " (a.attrelid = c.conrelid AND a.attnum = c.conkey[1])\n" @@ -3129,14 +3130,16 @@ describeOneTableDetails(const char *schemaname, { bool islocal = PQgetvalue(result, i, 3)[0] == 't'; bool inherited = PQgetvalue(result, i, 4)[0] == 't'; + bool validated = PQgetvalue(result, i, 5)[0] == 't'; - printfPQExpBuffer(&buf, " \"%s\" NOT NULL \"%s\"%s", + printfPQExpBuffer(&buf, " \"%s\" NOT NULL \"%s\"%s%s", PQgetvalue(result, i, 0), PQgetvalue(result, i, 1), PQgetvalue(result, i, 2)[0] == 't' ? " NO INHERIT" : islocal && inherited ? _(" (local, inherited)") : - inherited ? _(" (inherited)") : ""); + inherited ? _(" (inherited)") : "", + !validated ? " NOT VALID" : ""); printTableAddFooter(&cont, buf.data); } diff --git a/src/include/access/tupdesc.h b/src/include/access/tupdesc.h index 396eeb7a0bb..5523cfcf5aa 100644 --- a/src/include/access/tupdesc.h +++ b/src/include/access/tupdesc.h @@ -42,7 +42,7 @@ typedef struct TupleConstr struct AttrMissing *missing; /* missing attributes values, NULL if none */ uint16 num_defval; uint16 num_check; - bool has_not_null; + bool has_not_null; /* any not-null, including not valid ones */ bool has_generated_stored; bool has_generated_virtual; } TupleConstr; @@ -76,10 +76,15 @@ typedef struct CompactAttribute bool atthasmissing; /* as FormData_pg_attribute.atthasmissing */ bool attisdropped; /* as FormData_pg_attribute.attisdropped */ bool attgenerated; /* FormData_pg_attribute.attgenerated != '\0' */ - bool attnotnull; /* as FormData_pg_attribute.attnotnull */ + char attnullability; /* attnotnull + attnotnullvalid */ uint8 attalignby; /* alignment requirement in bytes */ } CompactAttribute; +/* Valid values for CompactAttribute->attnullability */ +#define ATTNULLABLE_VALID 'v' /* valid constraint exists */ +#define ATTNULLABLE_INVALID 'i' /* constraint exists, marked invalid */ +#define ATTNULLABLE_NONE 'f' /* no constraint exists */ + /* * This struct is passed around within the backend to describe the structure * of tuples. For tuples coming from on-disk relations, the information is diff --git a/src/include/catalog/catversion.h b/src/include/catalog/catversion.h index 208936962ef..b573a0a2828 100644 --- a/src/include/catalog/catversion.h +++ b/src/include/catalog/catversion.h @@ -57,6 +57,6 @@ */ /* yyyymmddN */ -#define CATALOG_VERSION_NO 202504041 +#define CATALOG_VERSION_NO 202504051 #endif diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index deaa515fe53..9b1236e7a90 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -117,9 +117,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, */ char attcompression BKI_DEFAULT('\0'); - /* This flag represents the "NOT NULL" constraint */ + /* Whether a not-null constraint exists for the column */ bool attnotnull; + /* Whether the not-null constraint, if it exists, is valid */ + bool attnotnullvalid; + /* Has DEFAULT value or not */ bool atthasdef BKI_DEFAULT(f); diff --git a/src/include/catalog/pg_constraint.h b/src/include/catalog/pg_constraint.h index 6da164e7e4d..4afceb5c692 100644 --- a/src/include/catalog/pg_constraint.h +++ b/src/include/catalog/pg_constraint.h @@ -264,7 +264,7 @@ extern HeapTuple findNotNullConstraint(Oid relid, const char *colname); extern HeapTuple findDomainNotNullConstraint(Oid typid); extern AttrNumber extractNotNullColumn(HeapTuple constrTup); extern bool AdjustNotNullInheritance(Oid relid, AttrNumber attnum, - bool is_local, bool is_no_inherit); + bool is_local, bool is_no_inherit, bool is_notvalid); extern List *RelationGetNotNullConstraints(Oid relid, bool cooked, bool include_noinh); diff --git a/src/test/regress/expected/alter_table.out b/src/test/regress/expected/alter_table.out index 362f38856d2..8a44321034b 100644 --- a/src/test/regress/expected/alter_table.out +++ b/src/test/regress/expected/alter_table.out @@ -1202,6 +1202,75 @@ alter table atacc1 alter test_a drop not null, alter test_b drop not null; alter table atacc1 add constraint atacc1_constr_b_valid check(test_b is not null); alter table atacc1 alter test_b set not null, alter test_a set not null; drop table atacc1; +-- not null not valid with partitions +CREATE TABLE atnnparted (id int, col1 int) PARTITION BY LIST (id); +ALTER TABLE atnnparted ADD CONSTRAINT dummy_constr NOT NULL id NOT VALID; +CREATE TABLE atnnpart1 (col1 int, id int); +ALTER TABLE atnnpart1 ADD CONSTRAINT another_constr NOT NULL id; +ALTER TABLE atnnpart1 ADD PRIMARY KEY (id); +ALTER TABLE atnnparted ATTACH PARTITION atnnpart1 FOR VALUES IN ('1'); +\d+ atnnpart* + Table "public.atnnpart1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+---------+-----------+----------+---------+---------+--------------+------------- + col1 | integer | | | | plain | | + id | integer | | not null | | plain | | +Partition of: atnnparted FOR VALUES IN (1) +Partition constraint: ((id IS NOT NULL) AND (id = 1)) +Indexes: + "atnnpart1_pkey" PRIMARY KEY, btree (id) +Not-null constraints: + "another_constr" NOT NULL "id" (inherited) + + Index "public.atnnpart1_pkey" + Column | Type | Key? | Definition | Storage | Stats target +--------+---------+------+------------+---------+-------------- + id | integer | yes | id | plain | +primary key, btree, for table "public.atnnpart1" + + Partitioned table "public.atnnparted" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+---------+-----------+----------+---------+---------+--------------+------------- + id | integer | | not null | | plain | | + col1 | integer | | | | plain | | +Partition key: LIST (id) +Not-null constraints: + "dummy_constr" NOT NULL "id" NOT VALID +Partitions: atnnpart1 FOR VALUES IN (1) + +BEGIN; +ALTER TABLE atnnparted VALIDATE CONSTRAINT dummy_constr; +\d+ atnnpart* + Table "public.atnnpart1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+---------+-----------+----------+---------+---------+--------------+------------- + col1 | integer | | | | plain | | + id | integer | | not null | | plain | | +Partition of: atnnparted FOR VALUES IN (1) +Partition constraint: ((id IS NOT NULL) AND (id = 1)) +Indexes: + "atnnpart1_pkey" PRIMARY KEY, btree (id) +Not-null constraints: + "another_constr" NOT NULL "id" (inherited) + + Index "public.atnnpart1_pkey" + Column | Type | Key? | Definition | Storage | Stats target +--------+---------+------+------------+---------+-------------- + id | integer | yes | id | plain | +primary key, btree, for table "public.atnnpart1" + + Partitioned table "public.atnnparted" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+---------+-----------+----------+---------+---------+--------------+------------- + id | integer | | not null | | plain | | + col1 | integer | | | | plain | | +Partition key: LIST (id) +Not-null constraints: + "dummy_constr" NOT NULL "id" +Partitions: atnnpart1 FOR VALUES IN (1) + +ROLLBACK; +-- leave a table in this state for the pg_upgrade test -- test inheritance create table parent (a int); create table child (b varchar(255)) inherits (parent); diff --git a/src/test/regress/expected/constraints.out b/src/test/regress/expected/constraints.out index a719d2f74e9..d8547a9bc81 100644 --- a/src/test/regress/expected/constraints.out +++ b/src/test/regress/expected/constraints.out @@ -1229,6 +1229,13 @@ alter table cnn_uq add unique using index cnn_uq_idx; Indexes: "cnn_uq_idx" UNIQUE CONSTRAINT, btree (a) +-- can't create a primary key on a noinherit not-null +create table cnn_pk (a int not null no inherit); +alter table cnn_pk add primary key (a); +ERROR: cannot create primary key on column "a" +DETAIL: The constraint "cnn_pk_a_not_null" on column "a", marked NO INHERIT, is incompatible with a primary key. +HINT: You will need to make it inheritable using ALTER TABLE ... ALTER CONSTRAINT ... INHERIT. +drop table cnn_pk; -- Ensure partitions are scanned for null values when adding a PK create table cnn2_parted(a int) partition by list (a); create table cnn_part1 partition of cnn2_parted for values in (1, null); @@ -1355,6 +1362,249 @@ Not-null constraints: "ann" NOT NULL "a" "bnn" NOT NULL "b" +-- NOT NULL NOT VALID +PREPARE get_nnconstraint_info(regclass[]) AS +SELECT conrelid::regclass as tabname, conname, convalidated, conislocal, coninhcount +FROM pg_constraint +WHERE conrelid = ANY($1) +ORDER BY conrelid::regclass::text, conname; +CREATE TABLE notnull_tbl1 (a int, b int); +INSERT INTO notnull_tbl1 VALUES (NULL, 1), (300, 3); +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nn NOT NULL a; -- error +ERROR: column "a" of relation "notnull_tbl1" contains null values +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nn NOT NULL a NOT VALID; -- ok +-- even an invalid not-null forbids new nulls +INSERT INTO notnull_tbl1 VALUES (NULL, 4); +ERROR: null value in column "a" of relation "notnull_tbl1" violates not-null constraint +DETAIL: Failing row contains (null, 4). +\d+ notnull_tbl1 + Table "public.notnull_tbl1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+---------+-----------+----------+---------+---------+--------------+------------- + a | integer | | not null | | plain | | + b | integer | | | | plain | | +Not-null constraints: + "nn" NOT NULL "a" NOT VALID + +-- If we have an invalid constraint, we can't have another +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nn1 NOT NULL a NOT VALID NO INHERIT; +ERROR: cannot change NO INHERIT status of NOT NULL constraint "nn" on relation "notnull_tbl1" +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nn NOT NULL a; +ERROR: incompatible NOT VALID constraint "nn" on relation "notnull_tbl1" +HINT: You will need to use ALTER TABLE ... VALIDATE CONSTRAINT to validate it. +-- cannot add primary key on a column with an invalid not-null +ALTER TABLE notnull_tbl1 ADD PRIMARY KEY (a); +ERROR: cannot create primary key on column "a" +DETAIL: The constraint "nn" on column "a", marked NOT VALID, is incompatible with a primary key. +HINT: You will need to validate it using ALTER TABLE ... VALIDATE CONSTRAINT. +-- ALTER column SET NOT NULL validates an invalid constraint (but this fails +-- because of rows with null values) +ALTER TABLE notnull_tbl1 ALTER a SET NOT NULL; +ERROR: column "a" of relation "notnull_tbl1" contains null values +\d+ notnull_tbl1 + Table "public.notnull_tbl1" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +--------+---------+-----------+----------+---------+---------+--------------+------------- + a | integer | | not null | | plain | | + b | integer | | | | plain | | +Not-null constraints: + "nn" NOT NULL "a" NOT VALID + +-- Creating a derived table using LIKE gets the constraint, but it's valid +CREATE TABLE notnull_tbl1_copy (LIKE notnull_tbl1); +EXECUTE get_nnconstraint_info('{notnull_tbl1_copy}'); + tabname | conname | convalidated | conislocal | coninhcount +-------------------+---------+--------------+------------+------------- + notnull_tbl1_copy | nn | t | t | 0 +(1 row) + +-- An inheritance child table gets the constraint, but it's valid +CREATE TABLE notnull_tbl1_child (a int, b int) INHERITS (notnull_tbl1); +NOTICE: merging column "a" with inherited definition +NOTICE: merging column "b" with inherited definition +EXECUTE get_nnconstraint_info('{notnull_tbl1_child, notnull_tbl1}'); + tabname | conname | convalidated | conislocal | coninhcount +--------------------+---------+--------------+------------+------------- + notnull_tbl1 | nn | f | t | 0 + notnull_tbl1_child | nn | t | f | 1 +(2 rows) + +-- Also try inheritance added after table creation +CREATE TABLE notnull_tbl1_child2 (c int, b int, a int); +ALTER TABLE notnull_tbl1_child2 INHERIT notnull_tbl1; -- nope +ERROR: column "a" in child table "notnull_tbl1_child2" must be marked NOT NULL +ALTER TABLE notnull_tbl1_child2 ADD NOT NULL a NOT VALID; +ALTER TABLE notnull_tbl1_child2 INHERIT notnull_tbl1; +EXECUTE get_nnconstraint_info('{notnull_tbl1_child2}'); + tabname | conname | convalidated | conislocal | coninhcount +---------------------+--------------------------------+--------------+------------+------------- + notnull_tbl1_child2 | notnull_tbl1_child2_a_not_null | f | t | 1 +(1 row) + +-- VALIDATE CONSTRAINT scans the table +ALTER TABLE notnull_tbl1 VALIDATE CONSTRAINT nn; -- error, nulls exist +ERROR: column "a" of relation "notnull_tbl1" contains null values +UPDATE notnull_tbl1 SET a = 100 WHERE b = 1; +ALTER TABLE notnull_tbl1 VALIDATE CONSTRAINT nn; -- now ok +EXECUTE get_nnconstraint_info('{notnull_tbl1}'); + tabname | conname | convalidated | conislocal | coninhcount +--------------+---------+--------------+------------+------------- + notnull_tbl1 | nn | t | t | 0 +(1 row) + +--- now we can add primary key +ALTER TABLE notnull_tbl1 ADD PRIMARY KEY (a); +DROP TABLE notnull_tbl1, notnull_tbl1_child, notnull_tbl1_child2; +-- dropping an invalid constraint is possible +CREATE TABLE notnull_tbl1 (a int, b int); +ALTER TABLE notnull_tbl1 ADD NOT NULL a NOT VALID, + ADD NOT NULL b NOT VALID; +ALTER TABLE notnull_tbl1 ALTER a DROP NOT NULL; +ALTER TABLE notnull_tbl1 DROP CONSTRAINT notnull_tbl1_b_not_null; +DROP TABLE notnull_tbl1; +-- ALTER .. NO INHERIT works for invalid constraints +CREATE TABLE notnull_tbl1 (a int); +CREATE TABLE notnull_tbl1_chld () INHERITS (notnull_tbl1); +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nntbl1_a NOT NULL a NOT VALID; +ALTER TABLE notnull_tbl1 ALTER CONSTRAINT nntbl1_a NO INHERIT; +-- DROP CONSTRAINT recurses correctly on invalid constraints +ALTER TABLE notnull_tbl1 ALTER CONSTRAINT nntbl1_a INHERIT; +ALTER TABLE notnull_tbl1 DROP CONSTRAINT nntbl1_a; +DROP TABLE notnull_tbl1, notnull_tbl1_chld; +-- if a parent has a valid not null constraint then a child table cannot +-- have an invalid one +CREATE TABLE notnull_tbl1 (a int); +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nn_parent NOT NULL a not valid; +CREATE TABLE notnull_chld0 (a int, CONSTRAINT nn_chld0 NOT NULL a); +ALTER TABLE notnull_tbl1 INHERIT notnull_chld0; --error +ERROR: constraint "nn_parent" conflicts with NOT VALID constraint on child table "notnull_tbl1" +ALTER TABLE notnull_chld0 DROP CONSTRAINT nn_chld0; +ALTER TABLE notnull_chld0 ADD CONSTRAINT nn_chld0 NOT NULL a not valid; +ALTER TABLE notnull_tbl1 INHERIT notnull_chld0; --now ok +-- parents and child not-null will all be validated. +ALTER TABLE notnull_tbl1 VALIDATE CONSTRAINT nn_parent; +EXECUTE get_nnconstraint_info('{notnull_tbl1, notnull_chld0}'); + tabname | conname | convalidated | conislocal | coninhcount +---------------+-----------+--------------+------------+------------- + notnull_chld0 | nn_chld0 | f | t | 0 + notnull_tbl1 | nn_parent | t | t | 1 +(2 rows) + +DROP TABLE notnull_tbl1, notnull_chld0; +-- Test invalid not null on inheritance table. +CREATE TABLE notnull_inhparent (i int); +CREATE TABLE notnull_inhchild (i int) INHERITS (notnull_inhparent); +NOTICE: merging column "i" with inherited definition +CREATE TABLE notnull_inhgrand () INHERITS (notnull_inhparent, notnull_inhchild); +NOTICE: merging multiple inherited definitions of column "i" +ALTER TABLE notnull_inhparent ADD CONSTRAINT nn NOT NULL i NOT VALID; +ALTER TABLE notnull_inhchild ADD CONSTRAINT nn1 NOT NULL i; -- error +ERROR: incompatible NOT VALID constraint "nn" on relation "notnull_inhchild" +HINT: You will need to use ALTER TABLE ... VALIDATE CONSTRAINT to validate it. +EXECUTE get_nnconstraint_info('{notnull_inhparent, notnull_inhchild, notnull_inhgrand}'); + tabname | conname | convalidated | conislocal | coninhcount +-------------------+---------+--------------+------------+------------- + notnull_inhchild | nn | f | f | 1 + notnull_inhgrand | nn | f | f | 2 + notnull_inhparent | nn | f | t | 0 +(3 rows) + +ALTER TABLE notnull_inhparent ALTER i SET NOT NULL; -- ok +EXECUTE get_nnconstraint_info('{notnull_inhparent, notnull_inhchild, notnull_inhgrand}'); + tabname | conname | convalidated | conislocal | coninhcount +-------------------+---------+--------------+------------+------------- + notnull_inhchild | nn | t | f | 1 + notnull_inhgrand | nn | t | f | 2 + notnull_inhparent | nn | t | t | 0 +(3 rows) + +DROP TABLE notnull_inhparent, notnull_inhchild, notnull_inhgrand; +-- Verify NOT NULL VALID/NOT VALID with partition table. +DROP TABLE notnull_tbl1; +ERROR: table "notnull_tbl1" does not exist +CREATE TABLE notnull_tbl1 (a int, b int) PARTITION BY LIST (a); +ALTER TABLE notnull_tbl1 ADD CONSTRAINT notnull_con NOT NULL a NOT VALID; --ok +CREATE TABLE notnull_tbl1_1 PARTITION OF notnull_tbl1 FOR VALUES IN (1,2); +CREATE TABLE notnull_tbl1_2(a int, CONSTRAINT nn2 NOT NULL a, b int); +ALTER TABLE notnull_tbl1 ATTACH PARTITION notnull_tbl1_2 FOR VALUES IN (3,4); +CREATE TABLE notnull_tbl1_3(a int, b int); +INSERT INTO notnull_tbl1_3 values(NULL,1); +ALTER TABLE notnull_tbl1_3 add CONSTRAINT nn3 NOT NULL a NOT VALID; +ALTER TABLE notnull_tbl1 ATTACH PARTITION notnull_tbl1_3 FOR VALUES IN (NULL,5); +EXECUTE get_nnconstraint_info('{notnull_tbl1, notnull_tbl1_1, notnull_tbl1_2, notnull_tbl1_3}'); + tabname | conname | convalidated | conislocal | coninhcount +----------------+-------------+--------------+------------+------------- + notnull_tbl1 | notnull_con | f | t | 0 + notnull_tbl1_1 | notnull_con | t | f | 1 + notnull_tbl1_2 | nn2 | t | f | 1 + notnull_tbl1_3 | nn3 | f | f | 1 +(4 rows) + +ALTER TABLE notnull_tbl1 ALTER COLUMN a SET NOT NULL; --error, notnull_tbl1_3 have null values +ERROR: column "a" of relation "notnull_tbl1_3" contains null values +ALTER TABLE notnull_tbl1_3 VALIDATE CONSTRAINT nn3; --error +ERROR: column "a" of relation "notnull_tbl1_3" contains null values +TRUNCATE notnull_tbl1; +ALTER TABLE notnull_tbl1 ALTER COLUMN a SET NOT NULL; --OK +EXECUTE get_nnconstraint_info('{notnull_tbl1, notnull_tbl1_1, notnull_tbl1_2, notnull_tbl1_3}'); + tabname | conname | convalidated | conislocal | coninhcount +----------------+-------------+--------------+------------+------------- + notnull_tbl1 | notnull_con | t | t | 0 + notnull_tbl1_1 | notnull_con | t | f | 1 + notnull_tbl1_2 | nn2 | t | f | 1 + notnull_tbl1_3 | nn3 | t | f | 1 +(4 rows) + +DROP TABLE notnull_tbl1; +-- partitioned table have not-null, then the partitions can not be NOT NULL NOT VALID. +CREATE TABLE pp_nn (a int, b int, NOT NULL a) PARTITION BY LIST (a); +CREATE TABLE pp_nn_1(a int, b int); +ALTER TABLE pp_nn_1 ADD CONSTRAINT nn1 NOT NULL a NOT VALID; +ALTER TABLE pp_nn ATTACH PARTITION pp_nn_1 FOR VALUES IN (NULL,5); --error +ERROR: constraint "nn1" conflicts with NOT VALID constraint on child table "pp_nn_1" +ALTER TABLE pp_nn_1 VALIDATE CONSTRAINT nn1; +ALTER TABLE pp_nn ATTACH PARTITION pp_nn_1 FOR VALUES IN (NULL,5); --ok +DROP TABLE pp_nn; +-- Create table with NOT NULL INVALID constraint, for pg_upgrade. +CREATE TABLE notnull_tbl1_upg (a int, b int); +INSERT INTO notnull_tbl1_upg VALUES (NULL, 1), (NULL, 2), (300, 3); +ALTER TABLE notnull_tbl1_upg ADD CONSTRAINT nn NOT NULL a NOT VALID; +-- Inherit test for pg_upgrade +CREATE TABLE notnull_parent_upg (a int); +CREATE TABLE notnull_child_upg () INHERITS (notnull_parent_upg); +ALTER TABLE notnull_child_upg ADD CONSTRAINT nn NOT NULL a; +ALTER TABLE notnull_parent_upg ADD CONSTRAINT nn NOT NULL a NOT VALID; +SELECT conrelid::regclass, contype, convalidated, conislocal +FROM pg_catalog.pg_constraint +WHERE conrelid in ('notnull_parent_upg'::regclass, 'notnull_child_upg'::regclass) +ORDER BY 1; + conrelid | contype | convalidated | conislocal +--------------------+---------+--------------+------------ + notnull_parent_upg | n | f | t + notnull_child_upg | n | t | t +(2 rows) + +-- Partition table test, for pg_upgrade +CREATE TABLE notnull_part1_upg (a int, b int) PARTITION BY LIST (a); +ALTER TABLE notnull_part1_upg ADD CONSTRAINT notnull_con NOT NULL a NOT VALID; --ok +CREATE TABLE notnull_part1_1_upg PARTITION OF notnull_part1_upg FOR VALUES IN (1,2); +CREATE TABLE notnull_part1_2_upg (a int, CONSTRAINT nn2 NOT NULL a, b int); +ALTER TABLE notnull_part1_upg ATTACH PARTITION notnull_part1_2_upg FOR VALUES IN (3,4); +CREATE TABLE notnull_part1_3_upg (a int, b int); +INSERT INTO notnull_part1_3_upg values(NULL,1); +ALTER TABLE notnull_part1_3_upg add CONSTRAINT nn3 NOT NULL a NOT VALID; +ALTER TABLE notnull_part1_upg ATTACH PARTITION notnull_part1_3_upg FOR VALUES IN (NULL,5); +EXECUTE get_nnconstraint_info('{notnull_part1_upg, notnull_part1_1_upg, notnull_part1_2_upg, notnull_part1_3_upg}'); + tabname | conname | convalidated | conislocal | coninhcount +---------------------+-------------+--------------+------------+------------- + notnull_part1_1_upg | notnull_con | t | f | 1 + notnull_part1_2_upg | nn2 | t | f | 1 + notnull_part1_3_upg | nn3 | f | f | 1 + notnull_part1_upg | notnull_con | f | t | 0 +(4 rows) + +DEALLOCATE get_nnconstraint_info; +-- end NOT NULL NOT VALID -- Comments -- Setup a low-level role to enforce non-superuser checks. CREATE ROLE regress_constraint_comments; diff --git a/src/test/regress/expected/sanity_check.out b/src/test/regress/expected/sanity_check.out index 8370c1561cc..2e39037e3fb 100644 --- a/src/test/regress/expected/sanity_check.out +++ b/src/test/regress/expected/sanity_check.out @@ -25,3 +25,23 @@ SELECT relname, relkind ---------+--------- (0 rows) +-- pg_attribute sanity check: attnotnullvalid can only be true when +-- attnotnull is valid. +select pa.attrelid::regclass, pa.attname, pa.attnum, + pa.attnotnull, pa.attnotnullvalid +from pg_attribute pa +where not pa.attnotnull and pa.attnotnullvalid; + attrelid | attname | attnum | attnotnull | attnotnullvalid +----------+---------+--------+------------+----------------- +(0 rows) + +-- pg_attribute's attnotnullvalid should match pg_constraint.convalidated +select att.attrelid::regclass, att.attname, att.attnotnull, + att.attnotnullvalid, con.convalidated + from pg_attribute att join pg_constraint con + on (att.attnum = con.conkey[1] and att.attrelid = con.conrelid) + where contype = 'n' and con.convalidated <> att.attnotnullvalid; + attrelid | attname | attnotnull | attnotnullvalid | convalidated +----------+---------+------------+-----------------+-------------- +(0 rows) + diff --git a/src/test/regress/sql/alter_table.sql b/src/test/regress/sql/alter_table.sql index 84e93ef575e..8432e8e3d54 100644 --- a/src/test/regress/sql/alter_table.sql +++ b/src/test/regress/sql/alter_table.sql @@ -910,6 +910,20 @@ alter table atacc1 add constraint atacc1_constr_b_valid check(test_b is not null alter table atacc1 alter test_b set not null, alter test_a set not null; drop table atacc1; +-- not null not valid with partitions +CREATE TABLE atnnparted (id int, col1 int) PARTITION BY LIST (id); +ALTER TABLE atnnparted ADD CONSTRAINT dummy_constr NOT NULL id NOT VALID; +CREATE TABLE atnnpart1 (col1 int, id int); +ALTER TABLE atnnpart1 ADD CONSTRAINT another_constr NOT NULL id; +ALTER TABLE atnnpart1 ADD PRIMARY KEY (id); +ALTER TABLE atnnparted ATTACH PARTITION atnnpart1 FOR VALUES IN ('1'); +\d+ atnnpart* +BEGIN; +ALTER TABLE atnnparted VALIDATE CONSTRAINT dummy_constr; +\d+ atnnpart* +ROLLBACK; +-- leave a table in this state for the pg_upgrade test + -- test inheritance create table parent (a int); create table child (b varchar(255)) inherits (parent); diff --git a/src/test/regress/sql/constraints.sql b/src/test/regress/sql/constraints.sql index 21ce4177de4..8f199c7ba34 100644 --- a/src/test/regress/sql/constraints.sql +++ b/src/test/regress/sql/constraints.sql @@ -761,6 +761,11 @@ create unique index cnn_uq_idx on cnn_uq (a); alter table cnn_uq add unique using index cnn_uq_idx; \d+ cnn_uq +-- can't create a primary key on a noinherit not-null +create table cnn_pk (a int not null no inherit); +alter table cnn_pk add primary key (a); +drop table cnn_pk; + -- Ensure partitions are scanned for null values when adding a PK create table cnn2_parted(a int) partition by list (a); create table cnn_part1 partition of cnn2_parted for values in (1, null); @@ -801,6 +806,167 @@ ALTER TABLE ONLY notnull_tbl6 DROP CONSTRAINT ann; ALTER TABLE ONLY notnull_tbl6 ALTER b DROP NOT NULL; \d+ notnull_tbl6_1 + +-- NOT NULL NOT VALID +PREPARE get_nnconstraint_info(regclass[]) AS +SELECT conrelid::regclass as tabname, conname, convalidated, conislocal, coninhcount +FROM pg_constraint +WHERE conrelid = ANY($1) +ORDER BY conrelid::regclass::text, conname; + +CREATE TABLE notnull_tbl1 (a int, b int); +INSERT INTO notnull_tbl1 VALUES (NULL, 1), (300, 3); +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nn NOT NULL a; -- error +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nn NOT NULL a NOT VALID; -- ok +-- even an invalid not-null forbids new nulls +INSERT INTO notnull_tbl1 VALUES (NULL, 4); +\d+ notnull_tbl1 + +-- If we have an invalid constraint, we can't have another +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nn1 NOT NULL a NOT VALID NO INHERIT; +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nn NOT NULL a; + +-- cannot add primary key on a column with an invalid not-null +ALTER TABLE notnull_tbl1 ADD PRIMARY KEY (a); + +-- ALTER column SET NOT NULL validates an invalid constraint (but this fails +-- because of rows with null values) +ALTER TABLE notnull_tbl1 ALTER a SET NOT NULL; +\d+ notnull_tbl1 + +-- Creating a derived table using LIKE gets the constraint, but it's valid +CREATE TABLE notnull_tbl1_copy (LIKE notnull_tbl1); +EXECUTE get_nnconstraint_info('{notnull_tbl1_copy}'); + +-- An inheritance child table gets the constraint, but it's valid +CREATE TABLE notnull_tbl1_child (a int, b int) INHERITS (notnull_tbl1); +EXECUTE get_nnconstraint_info('{notnull_tbl1_child, notnull_tbl1}'); + +-- Also try inheritance added after table creation +CREATE TABLE notnull_tbl1_child2 (c int, b int, a int); +ALTER TABLE notnull_tbl1_child2 INHERIT notnull_tbl1; -- nope +ALTER TABLE notnull_tbl1_child2 ADD NOT NULL a NOT VALID; +ALTER TABLE notnull_tbl1_child2 INHERIT notnull_tbl1; +EXECUTE get_nnconstraint_info('{notnull_tbl1_child2}'); + +-- VALIDATE CONSTRAINT scans the table +ALTER TABLE notnull_tbl1 VALIDATE CONSTRAINT nn; -- error, nulls exist +UPDATE notnull_tbl1 SET a = 100 WHERE b = 1; +ALTER TABLE notnull_tbl1 VALIDATE CONSTRAINT nn; -- now ok +EXECUTE get_nnconstraint_info('{notnull_tbl1}'); + +--- now we can add primary key +ALTER TABLE notnull_tbl1 ADD PRIMARY KEY (a); +DROP TABLE notnull_tbl1, notnull_tbl1_child, notnull_tbl1_child2; + +-- dropping an invalid constraint is possible +CREATE TABLE notnull_tbl1 (a int, b int); +ALTER TABLE notnull_tbl1 ADD NOT NULL a NOT VALID, + ADD NOT NULL b NOT VALID; +ALTER TABLE notnull_tbl1 ALTER a DROP NOT NULL; +ALTER TABLE notnull_tbl1 DROP CONSTRAINT notnull_tbl1_b_not_null; +DROP TABLE notnull_tbl1; + +-- ALTER .. NO INHERIT works for invalid constraints +CREATE TABLE notnull_tbl1 (a int); +CREATE TABLE notnull_tbl1_chld () INHERITS (notnull_tbl1); +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nntbl1_a NOT NULL a NOT VALID; +ALTER TABLE notnull_tbl1 ALTER CONSTRAINT nntbl1_a NO INHERIT; + +-- DROP CONSTRAINT recurses correctly on invalid constraints +ALTER TABLE notnull_tbl1 ALTER CONSTRAINT nntbl1_a INHERIT; +ALTER TABLE notnull_tbl1 DROP CONSTRAINT nntbl1_a; +DROP TABLE notnull_tbl1, notnull_tbl1_chld; + +-- if a parent has a valid not null constraint then a child table cannot +-- have an invalid one +CREATE TABLE notnull_tbl1 (a int); +ALTER TABLE notnull_tbl1 ADD CONSTRAINT nn_parent NOT NULL a not valid; +CREATE TABLE notnull_chld0 (a int, CONSTRAINT nn_chld0 NOT NULL a); +ALTER TABLE notnull_tbl1 INHERIT notnull_chld0; --error + +ALTER TABLE notnull_chld0 DROP CONSTRAINT nn_chld0; +ALTER TABLE notnull_chld0 ADD CONSTRAINT nn_chld0 NOT NULL a not valid; +ALTER TABLE notnull_tbl1 INHERIT notnull_chld0; --now ok + +-- parents and child not-null will all be validated. +ALTER TABLE notnull_tbl1 VALIDATE CONSTRAINT nn_parent; +EXECUTE get_nnconstraint_info('{notnull_tbl1, notnull_chld0}'); +DROP TABLE notnull_tbl1, notnull_chld0; + +-- Test invalid not null on inheritance table. +CREATE TABLE notnull_inhparent (i int); +CREATE TABLE notnull_inhchild (i int) INHERITS (notnull_inhparent); +CREATE TABLE notnull_inhgrand () INHERITS (notnull_inhparent, notnull_inhchild); +ALTER TABLE notnull_inhparent ADD CONSTRAINT nn NOT NULL i NOT VALID; +ALTER TABLE notnull_inhchild ADD CONSTRAINT nn1 NOT NULL i; -- error +EXECUTE get_nnconstraint_info('{notnull_inhparent, notnull_inhchild, notnull_inhgrand}'); +ALTER TABLE notnull_inhparent ALTER i SET NOT NULL; -- ok +EXECUTE get_nnconstraint_info('{notnull_inhparent, notnull_inhchild, notnull_inhgrand}'); +DROP TABLE notnull_inhparent, notnull_inhchild, notnull_inhgrand; + +-- Verify NOT NULL VALID/NOT VALID with partition table. +DROP TABLE notnull_tbl1; +CREATE TABLE notnull_tbl1 (a int, b int) PARTITION BY LIST (a); +ALTER TABLE notnull_tbl1 ADD CONSTRAINT notnull_con NOT NULL a NOT VALID; --ok +CREATE TABLE notnull_tbl1_1 PARTITION OF notnull_tbl1 FOR VALUES IN (1,2); +CREATE TABLE notnull_tbl1_2(a int, CONSTRAINT nn2 NOT NULL a, b int); +ALTER TABLE notnull_tbl1 ATTACH PARTITION notnull_tbl1_2 FOR VALUES IN (3,4); + +CREATE TABLE notnull_tbl1_3(a int, b int); +INSERT INTO notnull_tbl1_3 values(NULL,1); +ALTER TABLE notnull_tbl1_3 add CONSTRAINT nn3 NOT NULL a NOT VALID; +ALTER TABLE notnull_tbl1 ATTACH PARTITION notnull_tbl1_3 FOR VALUES IN (NULL,5); + +EXECUTE get_nnconstraint_info('{notnull_tbl1, notnull_tbl1_1, notnull_tbl1_2, notnull_tbl1_3}'); +ALTER TABLE notnull_tbl1 ALTER COLUMN a SET NOT NULL; --error, notnull_tbl1_3 have null values +ALTER TABLE notnull_tbl1_3 VALIDATE CONSTRAINT nn3; --error + +TRUNCATE notnull_tbl1; +ALTER TABLE notnull_tbl1 ALTER COLUMN a SET NOT NULL; --OK + +EXECUTE get_nnconstraint_info('{notnull_tbl1, notnull_tbl1_1, notnull_tbl1_2, notnull_tbl1_3}'); +DROP TABLE notnull_tbl1; + +-- partitioned table have not-null, then the partitions can not be NOT NULL NOT VALID. +CREATE TABLE pp_nn (a int, b int, NOT NULL a) PARTITION BY LIST (a); +CREATE TABLE pp_nn_1(a int, b int); +ALTER TABLE pp_nn_1 ADD CONSTRAINT nn1 NOT NULL a NOT VALID; +ALTER TABLE pp_nn ATTACH PARTITION pp_nn_1 FOR VALUES IN (NULL,5); --error +ALTER TABLE pp_nn_1 VALIDATE CONSTRAINT nn1; +ALTER TABLE pp_nn ATTACH PARTITION pp_nn_1 FOR VALUES IN (NULL,5); --ok +DROP TABLE pp_nn; + +-- Create table with NOT NULL INVALID constraint, for pg_upgrade. +CREATE TABLE notnull_tbl1_upg (a int, b int); +INSERT INTO notnull_tbl1_upg VALUES (NULL, 1), (NULL, 2), (300, 3); +ALTER TABLE notnull_tbl1_upg ADD CONSTRAINT nn NOT NULL a NOT VALID; +-- Inherit test for pg_upgrade +CREATE TABLE notnull_parent_upg (a int); +CREATE TABLE notnull_child_upg () INHERITS (notnull_parent_upg); +ALTER TABLE notnull_child_upg ADD CONSTRAINT nn NOT NULL a; +ALTER TABLE notnull_parent_upg ADD CONSTRAINT nn NOT NULL a NOT VALID; +SELECT conrelid::regclass, contype, convalidated, conislocal +FROM pg_catalog.pg_constraint +WHERE conrelid in ('notnull_parent_upg'::regclass, 'notnull_child_upg'::regclass) +ORDER BY 1; + +-- Partition table test, for pg_upgrade +CREATE TABLE notnull_part1_upg (a int, b int) PARTITION BY LIST (a); +ALTER TABLE notnull_part1_upg ADD CONSTRAINT notnull_con NOT NULL a NOT VALID; --ok +CREATE TABLE notnull_part1_1_upg PARTITION OF notnull_part1_upg FOR VALUES IN (1,2); +CREATE TABLE notnull_part1_2_upg (a int, CONSTRAINT nn2 NOT NULL a, b int); +ALTER TABLE notnull_part1_upg ATTACH PARTITION notnull_part1_2_upg FOR VALUES IN (3,4); +CREATE TABLE notnull_part1_3_upg (a int, b int); +INSERT INTO notnull_part1_3_upg values(NULL,1); +ALTER TABLE notnull_part1_3_upg add CONSTRAINT nn3 NOT NULL a NOT VALID; +ALTER TABLE notnull_part1_upg ATTACH PARTITION notnull_part1_3_upg FOR VALUES IN (NULL,5); +EXECUTE get_nnconstraint_info('{notnull_part1_upg, notnull_part1_1_upg, notnull_part1_2_upg, notnull_part1_3_upg}'); +DEALLOCATE get_nnconstraint_info; + +-- end NOT NULL NOT VALID + + -- Comments -- Setup a low-level role to enforce non-superuser checks. CREATE ROLE regress_constraint_comments; diff --git a/src/test/regress/sql/sanity_check.sql b/src/test/regress/sql/sanity_check.sql index 162e5324b5d..6ef36ab84b7 100644 --- a/src/test/regress/sql/sanity_check.sql +++ b/src/test/regress/sql/sanity_check.sql @@ -19,3 +19,17 @@ SELECT relname, relkind FROM pg_class WHERE relkind IN ('v', 'c', 'f', 'p', 'I') AND relfilenode <> 0; + +-- pg_attribute sanity check: attnotnullvalid can only be true when +-- attnotnull is valid. +select pa.attrelid::regclass, pa.attname, pa.attnum, + pa.attnotnull, pa.attnotnullvalid +from pg_attribute pa +where not pa.attnotnull and pa.attnotnullvalid; + +-- pg_attribute's attnotnullvalid should match pg_constraint.convalidated +select att.attrelid::regclass, att.attname, att.attnotnull, + att.attnotnullvalid, con.convalidated + from pg_attribute att join pg_constraint con + on (att.attnum = con.conkey[1] and att.attrelid = con.conrelid) + where contype = 'n' and con.convalidated <> att.attnotnullvalid; -- 2.43.0 [application/octet-stream] 0003-Fix-relation-attribute-loop-correctly-and-also-set-t.patch (2.0K, ../../CAGPqQf2AGLBC0_+P=S_-u0VwCuZ=FdGaP6miwyvxVWBTNsBuPw@mail.gmail.com/5-0003-Fix-relation-attribute-loop-correctly-and-also-set-t.patch) download | inline diff: From b2ea2c97eda39e4575da0f736553eecafd186ab8 Mon Sep 17 00:00:00 2001 From: Rushabh Lathia <[email protected]> Date: Mon, 7 Apr 2025 13:51:22 +0530 Subject: [PATCH 3/3] Fix relation attribute loop correctly and also set the nullability to ATTNULLABLE_VALID for the catalog/temp tables. --- src/backend/access/common/tupdesc.c | 9 +++++++-- src/backend/utils/cache/relcache.c | 2 +- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 5831788cbff..d481fadaf73 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -22,6 +22,7 @@ #include "access/htup_details.h" #include "access/toast_compression.h" #include "access/tupdesc_details.h" +#include "catalog/catalog.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -74,8 +75,12 @@ populate_compact_attribute_internal(Form_pg_attribute src, dst->atthasmissing = src->atthasmissing; dst->attisdropped = src->attisdropped; dst->attgenerated = (src->attgenerated != '\0'); - dst->attnullability = !src->attnotnull ? ATTNULLABLE_UNRESTRICTED : - ATTNULLABLE_UNKNOWN; + if (IsCatalogRelationOid(src->attrelid)) + dst->attnullability = !src->attnotnull ? ATTNULLABLE_UNRESTRICTED : + ATTNULLABLE_VALID; + else + dst->attnullability = !src->attnotnull ? ATTNULLABLE_UNRESTRICTED : + ATTNULLABLE_UNKNOWN; switch (src->attalign) { diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index 4128cfa8314..483f19711f0 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -704,7 +704,7 @@ RelationBuildTupleDesc(Relation relation) * CompactAttribute array. In catalog relations however, any not-null * constraint is necessarily valid. */ - for (int i = 0; i < relation->rd_rel->relnatts - 1; i++) + for (int i = 0; i < relation->rd_rel->relnatts; i++) { CompactAttribute *attr; -- 2.43.0 ^ permalink raw reply [nested|flat] 13+ messages in thread
end of thread, other threads:[~2025-04-07 08:28 UTC | newest] Thread overview: 13+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2016-07-27 06:47 [PATCH 6/7] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 06:47 [PATCH 6/7] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 06:47 [PATCH 6/7] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 06:47 [PATCH 6/7] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 07:59 [PATCH 7/8] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 07:59 [PATCH 7/8] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 07:59 [PATCH 7/8] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 07:59 [PATCH 7/8] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 07:59 [PATCH 7/8] Tuple routing for partitioned tables. amit <[email protected]> 2016-07-27 07:59 [PATCH 7/8] Tuple routing for partitioned tables. amit <[email protected]> 2020-09-12 13:07 [PATCH 6/6] Ignore correlation for new BRIN opclasses Tomas Vondra <[email protected]> 2025-04-06 18:02 Re: Support NOT VALID / VALIDATE constraint options for named NOT NULL constraints Alvaro Herrera <[email protected]> 2025-04-07 08:28 ` Re: Support NOT VALID / VALIDATE constraint options for named NOT NULL constraints Rushabh Lathia <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox