public inbox for [email protected]help / color / mirror / Atom feed
Re: Expanding HOT updates for expression and partial indexes 6+ messages / 2 participants [nested] [flat]
* Re: Expanding HOT updates for expression and partial indexes @ 2025-02-13 18:46 Burd, Greg <[email protected]> 0 siblings, 1 reply; 6+ messages in thread From: Burd, Greg @ 2025-02-13 18:46 UTC (permalink / raw) To: Matthias van de Meent <[email protected]>; +Cc: pgsql-hackers Attached find an updated patchset v5 that is an evolution of v4. Changes v4 to v5 are: * replaced GUC with table reloption called "expression_checks" (open to other name ideas) * minimal documentation updates to README.HOT to address changes * avoid, when possible, the expensive path that requires evaluating an estate using bitmaps * determines the set of summarized indexes requiring updates, only updates those * more tests in heap_hot_updates.sql (perhaps too many...) * rebased to master, formatted, and make check-world passes More comments in context below... -greg > On Feb 11, 2025, at 4:18 PM, Matthias van de Meent <[email protected]> wrote: > > > On Mon, 10 Feb 2025 at 20:11, Burd, Greg <[email protected]> wrote: >>> On Feb 10, 2025, at 12:17 PM, Matthias van de Meent <[email protected]> wrote: >>> >>>> >>>> I have a few concerns with the patch, things I’d greatly appreciate your thoughts on: >>>> >>>> First, I pass an EState along the update path to enable running the checks in heapam, this works but leaves me feeling as if I violated separation of concerns. If there is a better way to do this let me know or if you think the cost of creating one in the execIndexing.c ExecIndexesRequiringUpdates() is okay that’s another possibility. >>> >>> I think that doesn't have to be bad. >> >> Meaning that the approach I’ve taken is okay with you? > > If you mean "passing EState down through the AM so we can check if we > really need HOT updates", then Yes, that's OK. I don't think the basic > idea (executing the projections to check for differences) is bad per > se, however I do think we may need more design work on the exact shape > of how ExecIndexesRequiringUpdates() receives its information (which > includes the EState). > > E.g. we could pass an opaquely typed pointer that's passed from the > executor state through the table_update method into this > ExecIndexesRequiringUpdates(). That opaque struct would then contain > the important information for index update state checking, so that the > AM can't realistically break things without bypassing the separation > of concerns, and doesn't have to know about any executor nodes. I'm open to this idea and will attempt an implementation in v6, ideas welcome. >>>> Third, there is overhead to this patch, it is no longer a single simple bitmap test to choose HOT or not in heap_update(). >>> >>> Why can't it mostly be that simple in simple cases? >> >> It can remain that simple in the cases you mention. In relcache the hot blocking attributes are nearly the same, only the summarizing attributes are removed. The first test then in heap_update() is for overlap with the modified set. When there is none, the update will proceed on the HOT path. >> >> The presence of a summarizing index is determined in ExecIndexesRequiringUpdates() in execIndexing.c, so a slightly longer code path but not much new overhead. > > Yes, but that's only determined at an index-by-index level, rather > than determined all at once, and that's real bad when you have > hundreds of indexes to go through (however unlikely it might be, I've > heard of cases where there are 1000s of indexes on one table). So, I > prefer limiting any O(n_indexes) operations to only the most critical > cases. This makes sense, and I agree that avoiding O(n_indexes) operations is a good goal when possible. >>> I mean, it's clear that "updated indexed column's value == non-HOT >>> update". And that to determine whether an updated *projected* column's >>> value (i.e., expression index column's value) was actually updated we >>> need to calculate the previous and current index value, thus execute >>> the projection twice. But why would we have significant additional >>> overhead if there are no expression indexes, or when we can know by >>> bitmap overlap that the only interesting cases are summarizing >>> indexes? >> >> You’re right, there’s not a lot of new overhead in that case except what happens in ExecIndexesRequiringUpdates() to scan over the list of IndexInfo. It is really only when there are many expressions/predicates requiring examination that there is any significant cost to this approach AFAICT (but if you see something please point it out). > > See the attached approach. Evaluation of the expressions only has to > happen if there are any HOT-blocking attributes which are exclusively > hot-blockingly indexed through expressions, so if the updated > attribute numbers are a subset of hotblockingexprattrs. (substitute > hotblocking with summarizing for the summarizing approach) I believe I've incorporated the gist of your idea in this v5 patch, let me know if I missed something. >>> I would've implemented this with (1) two new bitmaps, one each for >>> normal and summarizing indexes, each containing which columns are >>> exclusively used in expression indexes (and which should thus be used >>> to trigger the (comparatively) expensive recalculation). >> >> That was one where I started, over time that became harder to work as the bitmaps contain the union of index attributes for the table not per-column. > > I think it's fairly easy to create, though. > >> Now there is one bitmap to cover the broadest case and then a function to find the modified set of indexes where each is examined against bitmaps that contain only attributes specific to the index in question. This helped in cases where there were both expression and non-expression indexes on the same attribute. > > Fair, but do we care about one expression index on (attr1->>'data')'s > value *not* changing when an index on (attr1) exists and attr1 has > changed? That index on att1 would block HOT updates regardless of the > (lack of) changes to the (att1->>'data') index, so doing those > expensive calculations seems quite wasteful. Agreed, when both a non-expression and an expression index exist on the same attribute then the expression checks are unnecessary and should be avoided. In this v5 patchset this case becomes two checks of bitmaps (first hot_attrs, then exclusively exp_attrs) before proceeding with a non-HOT update. > So, in my opinion, we should also keep track of those attributes only > included in expressions of indexes, and that's fairly easy: see > attached prototype.diff.txt (might need some work, the patch was > drafted on v16's codebase, but the idea is clear). Thank you for your patch, I've included and expanded it. > The resulting %exprattrs bitmap contains attributes that are used only > in expressions of those index types. > >>> The "new" output of these expression >>> evaluations would be stored to be used later as index datums, reducing >>> the number of per-expression evaluations down to 2 at most, rather >>> than 2+1 when the index needs an insertion but the expression itself >>> wasn't updated. >> >> Not reforming the new index tuples is also an interesting optimization. I wonder how that can be passed from within heapam’s call into a function in execIndexing up into nodeModifiyTable and back down into execIndexing and on to the index access method? I’ll have to think about that, ideas welcome. > > Note that index tuple forming happens only in the index AM, it's the > Datum construction (i.e. projection from attributes/tuple to indexed > value) that I'd like to deduplicate. Though looking at the current > code, I don't think it's reasonable to have that as a requirement for > this work. It'd be a nice-to-have for sure, but not as requirement. Agreed that it's a nice-to-have, but not a priority. >>> Do you have any documentation on the approaches used, and the specific >>> differences between v3 and v4? I don't see much of that in your >>> initial mail, and the patches themselves also don't show much of that >>> in their details. I'd like at least some documentation of the new >>> behaviour in src/backend/access/heap/README.HOT at some point before >>> this got marked as RFC in the commitfest app, though preferably sooner >>> rather than later. >> >> Good point, I should have updated README.HOT with the initial patchset. I’ll jump on that and update ASAP. > > Thanks in advance. > > Kind regards, > > Matthias van de Meent > Neon (https://neon.tech) > <prototype.diff.txt> Attachments: [application/octet-stream] v5-0001-Expand-HOT-update-path-to-include-expression-and-.patch (97.2K, 3-v5-0001-Expand-HOT-update-path-to-include-expression-and-.patch) download | inline diff: From 263bf11af1fd4d4cd30b5aa4a9a0692162141eb1 Mon Sep 17 00:00:00 2001 From: Gregory Burd <[email protected]> Date: Mon, 27 Jan 2025 13:28:59 -0500 Subject: [PATCH v5] Expand HOT update path to include expression and partial indexes. This patch extends the cases where HOT updates are possible in the heapam by examining expression indexes and determining if indexed values where mutated or not. Previously, any expression index on a column would disqualify it from the HOT update path. Also examines partial indexes to see if the values are within the predicate or not. This is a modified application of a patch proposed on the pgsql-hackers list: https://www.postgresql.org/message-id/flat/4d9928ee-a9e6-15f9-9c82-5981f13ffca6%40postgrespro.ru applied: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=c203d6cf8 reverted: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=05f84605dbeb9cf8279a157234b24bbb706c5256 Signed-off-by: Greg Burd <[email protected]> --- doc/src/sgml/ref/create_table.sgml | 17 + doc/src/sgml/storage.sgml | 21 + src/backend/access/common/reloptions.c | 12 +- src/backend/access/heap/README.HOT | 43 +- src/backend/access/heap/heapam.c | 115 ++-- src/backend/access/heap/heapam_handler.c | 27 +- src/backend/access/table/tableam.c | 5 +- src/backend/catalog/index.c | 15 + src/backend/catalog/indexing.c | 51 +- src/backend/executor/execIndexing.c | 238 ++++++- src/backend/executor/execReplication.c | 10 +- src/backend/executor/nodeModifyTable.c | 13 +- src/backend/utils/cache/relcache.c | 58 +- src/bin/psql/tab-complete.in.c | 2 +- src/include/access/heapam.h | 5 +- src/include/access/tableam.h | 28 +- src/include/executor/executor.h | 8 +- src/include/nodes/execnodes.h | 8 + src/include/utils/rel.h | 14 + src/include/utils/relcache.h | 1 + .../regress/expected/heap_hot_updates.out | 586 ++++++++++++++++++ src/test/regress/parallel_schedule | 5 + src/test/regress/sql/heap_hot_updates.sql | 440 +++++++++++++ 23 files changed, 1550 insertions(+), 172 deletions(-) create mode 100644 src/test/regress/expected/heap_hot_updates.out create mode 100644 src/test/regress/sql/heap_hot_updates.sql diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 0a3e520f215..e29c9ea45a7 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -1981,6 +1981,23 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry id="reloption-expression-checks" xreflabel="expression_checks"> + <term><literal>expression_checks</literal> (<type>boolean</type>) + <indexterm> + <primary><varname>expression_checks</varname> storage parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Enables or disables evaulation of predicate expressions on partial + indexes or expressions used to define indexes during updates. + If <literal>true</literal>, then these expressions are evaluated during + updates to data within the heap relation against the old and new values + and then compared to determine if <acronym>HOT</acronym> updates are + allowable or not. The default value is <literal>true</literal>. + </para> + </listitem> + </variablelist> </refsect2> diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml index 61250799ec0..f40ada9c989 100644 --- a/doc/src/sgml/storage.sgml +++ b/doc/src/sgml/storage.sgml @@ -1138,6 +1138,27 @@ data. Empty in ordinary tables.</entry> </itemizedlist> </para> + <para> + <acronym>HOT</acronym> updates can occur when the expression used to define + and index shows no changes to the indexed value. To determine this requires + that the expression be evaulated for the old and new values to be stored in + the index and then compared. This allows for <acronym>HOT</acronym> updates + when data indexed within JSONB columns is unchanged. To disable this + behavior and avoid the overhead of evaluating the expression during updates + set the <literal>expression_checks</literal> option to false for the table. + </para> + + <para> + <acronym>HOT</acronym> updates can also occur when updated values are not + within the predicate of a partial index. However, <acronym>HOT</acronym> + updates are not possible when the updated value and the current value differ + with regards to the predicate. To determin this requires that the predicate + expression be evaluated for the old and new values to be stored in the index + and then compared. To disable this behavior and avoid the overhead of + evaluating the expression during updates set + the <literal>expression_checks</literal> option to false for the table. + </para> + <para> You can increase the likelihood of sufficient page space for <acronym>HOT</acronym> updates by decreasing a table's <link diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c index 59fb53e7707..c081611926e 100644 --- a/src/backend/access/common/reloptions.c +++ b/src/backend/access/common/reloptions.c @@ -166,6 +166,15 @@ static relopt_bool boolRelOpts[] = }, true }, + { + { + "expression_checks", + "When disabled prevents checking expressions on indexes and predicates on partial indexes for changes that might influence heap-only tuple (HOT) updates.", + RELOPT_KIND_HEAP, + ShareUpdateExclusiveLock + }, + true + }, /* list terminator */ {{NULL}} }; @@ -1903,7 +1912,8 @@ default_reloptions(Datum reloptions, bool validate, relopt_kind kind) {"vacuum_truncate", RELOPT_TYPE_BOOL, offsetof(StdRdOptions, vacuum_truncate)}, {"vacuum_max_eager_freeze_failure_rate", RELOPT_TYPE_REAL, - offsetof(StdRdOptions, vacuum_max_eager_freeze_failure_rate)} + offsetof(StdRdOptions, vacuum_max_eager_freeze_failure_rate)}, + {"expression_checks", RELOPT_TYPE_BOOL, offsetof(StdRdOptions, expression_checks)} }; return (bytea *) build_reloptions(reloptions, validate, kind, diff --git a/src/backend/access/heap/README.HOT b/src/backend/access/heap/README.HOT index 74e407f375a..48aab81fdf0 100644 --- a/src/backend/access/heap/README.HOT +++ b/src/backend/access/heap/README.HOT @@ -36,7 +36,7 @@ HOT solves this problem for two restricted but useful special cases: First, where a tuple is repeatedly updated in ways that do not change its indexed columns. (Here, "indexed column" means any column referenced at all in an index definition, including for example columns that are -tested in a partial-index predicate but are not stored in the index.) +tested in a partial-index predicate, that has materially changed.) Second, where the modified columns are only used in indexes that do not contain tuple IDs, but maintain summaries of the indexed data by block. @@ -133,18 +133,26 @@ Note: we can use a "dead" line pointer for any DELETEd tuple, whether it was part of a HOT chain or not. This allows space reclamation in advance of running VACUUM for plain DELETEs as well as HOT updates. -The requirement for doing a HOT update is that indexes which point to -the root line pointer (and thus need to be cleaned up by VACUUM when the -tuple is dead) do not reference columns which are updated in that HOT -chain. Summarizing indexes (such as BRIN) are assumed to have no -references to individual tuples and thus are ignored when checking HOT -applicability. The updated columns are checked at execution time by -comparing the binary representation of the old and new values. We insist -on bitwise equality rather than using datatype-specific equality routines. -The main reason to avoid the latter is that there might be multiple -notions of equality for a datatype, and we don't know exactly which one -is relevant for the indexes at hand. We assume that bitwise equality -guarantees equality for all purposes. +The requirement for doing a HOT update is that indexes which point to the root +line pointer (and thus need to be cleaned up by VACUUM when the tuple is dead) +do not reference columns which are updated in that HOT chain. + +Summarizing indexes (such as BRIN) are assumed to have no references to +individual tuples and thus are ignored when checking HOT applicability. + +Expressions on indexes are evaluated and the results are used when check for +changes. This allows for the JSONB datatype to have HOT updates when the +indexed portion of the document are not modified. + +Partial index expressions are evaluated, HOT updates are allowed when the +updated index values do not satisfy the predicate. + +The updated columns are checked at execution time by comparing the binary +representation of the old and new values. We insist on bitwise equality rather +than using datatype-specific equality routines. The main reason to avoid the +latter is that there might be multiple notions of equality for a datatype, and +we don't know exactly which one is relevant for the indexes at hand. We assume +that bitwise equality guarantees equality for all purposes. If any columns that are included by non-summarizing indexes are updated, the HOT optimization is not applied, and the new tuple is inserted into @@ -152,9 +160,7 @@ all indexes of the table. If none of the updated columns are included in the table's indexes, the HOT optimization is applied and no indexes are updated. If instead the updated columns are only indexed by summarizing indexes, the HOT optimization is applied, but the update is propagated to -all summarizing indexes. (Realistically, we only need to propagate the -update to the indexes that contain the updated values, but that is yet to -be implemented.) +the summarizing indexes that have updated values. Abort Cases ----------- @@ -477,8 +483,9 @@ Heap-only tuple HOT-safe A proposed tuple update is said to be HOT-safe if it changes - none of the tuple's indexed columns. It will only become an - actual HOT update if we can find room on the same page for + none of the tuple's indexed columns or if the changes remain + outside of a partial index's predicate. It will only become + an actual HOT update if we can find room on the same page for the new tuple version. HOT update diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index fa7935a0ed3..a8bbfe1b2b5 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -3164,12 +3164,13 @@ TM_Result heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, CommandId cid, Snapshot crosscheck, bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode, - TU_UpdateIndexes *update_indexes) + Bitmapset **modified_indexes, struct EState *estate) { TM_Result result; TransactionId xid = GetCurrentTransactionId(); Bitmapset *hot_attrs; Bitmapset *sum_attrs; + Bitmapset *exp_attrs; Bitmapset *key_attrs; Bitmapset *id_attrs; Bitmapset *interesting_attrs; @@ -3192,7 +3193,6 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, bool have_tuple_lock = false; bool iscombo; bool use_hot_update = false; - bool summarized_update = false; bool key_intact; bool all_visible_cleared = false; bool all_visible_cleared_new = false; @@ -3246,6 +3246,8 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, INDEX_ATTR_BITMAP_HOT_BLOCKING); sum_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_SUMMARIZED); + exp_attrs = RelationGetIndexAttrBitmap(relation, + INDEX_ATTR_BITMAP_EXPRESSION); key_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_KEY); id_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_IDENTITY_KEY); @@ -3307,10 +3309,10 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, tmfd->ctid = *otid; tmfd->xmax = InvalidTransactionId; tmfd->cmax = InvalidCommandId; - *update_indexes = TU_None; bms_free(hot_attrs); bms_free(sum_attrs); + bms_free(exp_attrs); bms_free(key_attrs); bms_free(id_attrs); /* modified_attrs not yet initialized */ @@ -3608,10 +3610,10 @@ l2: UnlockTupleTuplock(relation, &(oldtup.t_self), *lockmode); if (vmbuffer != InvalidBuffer) ReleaseBuffer(vmbuffer); - *update_indexes = TU_None; bms_free(hot_attrs); bms_free(sum_attrs); + bms_free(exp_attrs); bms_free(key_attrs); bms_free(id_attrs); bms_free(modified_attrs); @@ -3926,30 +3928,91 @@ l2: * one pin is held. */ - if (newbuf == buffer) + if (modified_indexes && newbuf == buffer) { + bool only_summarizing = false; + bool already_checked = false; + int num_indexes = list_length(relation->rd_indexinfolist); + /* * Since the new tuple is going into the same page, we might be able - * to do a HOT update. Check if any of the index columns have been - * changed. + * to do a HOT update. As a reminder, hot_attrs includes attributes + * used in expressions, but not attributes used by summarizing + * indexes. */ if (!bms_overlap(modified_attrs, hot_attrs)) { + *modified_indexes = bms_add_member(*modified_indexes, num_indexes); use_hot_update = true; - + } + else + { /* - * If none of the columns that are used in hot-blocking indexes - * were updated, we can apply HOT, but we do still need to check - * if we need to update the summarizing indexes, and update those - * indexes if the columns were updated, or we may fail to detect - * e.g. value bound changes in BRIN minmax indexes. + * We may still be able to use the HOT update path if the reason + * for the overlap is an unchanged expression. */ - if (bms_overlap(modified_attrs, sum_attrs)) - summarized_update = true; + if (bms_overlap(modified_attrs, exp_attrs)) + { + *modified_indexes = + ExecIndexesRequiringUpdates(relation, modified_attrs, + estate, &oldtup, newtup, + &only_summarizing); + already_checked = true; + if (bms_is_empty(*modified_indexes) || only_summarizing) + { + /* + * When no indexes were updated, or the only indexes + * updated were summarizing indexes, we can use the HOT + * path. We ensure that the bitmapset isn't NULL by + * adding in an index that can't match later when we + * filter to determine which indexes to update and which + * to skip in execIndexing. We do that so as to signal + * the need to filter which indexes are updated. + */ + *modified_indexes = bms_add_member(*modified_indexes, num_indexes); + use_hot_update = true; + } + else + { + /* + * When any index requires updates, then all indexes must + * be updated and this update cannot be HOT. We signal + * that by setting modified_indexes to NULL which + * indicates on need to filter indexes out during + * execIndexing. + */ + bms_free(*modified_indexes); + *modified_indexes = NULL; + } + } + } + + /* + * Summarizing indexes need not prevent a HOT update when their + * attributes are modified like other index types, but their indexed + * values do need to be updated. Let's find out which indexes need to + * be updated. + */ + if (!already_checked && bms_overlap(modified_attrs, sum_attrs)) + { + bms_free(*modified_indexes); + *modified_indexes = + ExecIndexesRequiringUpdates(relation, modified_attrs, + estate, &oldtup, newtup, + &only_summarizing); + if (only_summarizing) + use_hot_update = true; } } else { + /* + * We're not able on the HOT update path. Setting modified_indexes to + * NULL is the signal to update all indexes. + */ + if (modified_indexes) + *modified_indexes = NULL; + /* Set a hint that the old page could use prune/defrag */ PageSetFull(page); } @@ -4106,27 +4169,10 @@ l2: heap_freetuple(heaptup); } - /* - * If it is a HOT update, the update may still need to update summarized - * indexes, lest we fail to update those summaries and get incorrect - * results (for example, minmax bounds of the block may change with this - * update). - */ - if (use_hot_update) - { - if (summarized_update) - *update_indexes = TU_Summarizing; - else - *update_indexes = TU_None; - } - else - *update_indexes = TU_All; - if (old_key_tuple != NULL && old_key_copied) heap_freetuple(old_key_tuple); bms_free(hot_attrs); - bms_free(sum_attrs); bms_free(key_attrs); bms_free(id_attrs); bms_free(modified_attrs); @@ -4403,8 +4449,7 @@ HeapDetermineColumnsInfo(Relation relation, * via ereport(). */ void -simple_heap_update(Relation relation, ItemPointer otid, HeapTuple tup, - TU_UpdateIndexes *update_indexes) +simple_heap_update(Relation relation, ItemPointer otid, HeapTuple tup) { TM_Result result; TM_FailureData tmfd; @@ -4413,7 +4458,7 @@ simple_heap_update(Relation relation, ItemPointer otid, HeapTuple tup, result = heap_update(relation, otid, tup, GetCurrentCommandId(true), InvalidSnapshot, true /* wait for commit */ , - &tmfd, &lockmode, update_indexes); + &tmfd, &lockmode, NULL, NULL); switch (result) { case TM_SelfModified: diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index c0bec014154..84f72b6ed33 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -312,8 +312,8 @@ heapam_tuple_delete(Relation relation, ItemPointer tid, CommandId cid, static TM_Result heapam_tuple_update(Relation relation, ItemPointer otid, TupleTableSlot *slot, CommandId cid, Snapshot snapshot, Snapshot crosscheck, - bool wait, TM_FailureData *tmfd, - LockTupleMode *lockmode, TU_UpdateIndexes *update_indexes) + bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode, + Bitmapset **modified_indexes, EState *estate) { bool shouldFree = true; HeapTuple tuple = ExecFetchSlotHeapTuple(slot, true, &shouldFree); @@ -324,30 +324,9 @@ heapam_tuple_update(Relation relation, ItemPointer otid, TupleTableSlot *slot, tuple->t_tableOid = slot->tts_tableOid; result = heap_update(relation, otid, tuple, cid, crosscheck, wait, - tmfd, lockmode, update_indexes); + tmfd, lockmode, modified_indexes, estate); ItemPointerCopy(&tuple->t_self, &slot->tts_tid); - /* - * Decide whether new index entries are needed for the tuple - * - * Note: heap_update returns the tid (location) of the new tuple in the - * t_self field. - * - * If the update is not HOT, we must update all indexes. If the update is - * HOT, it could be that we updated summarized columns, so we either - * update only summarized indexes, or none at all. - */ - if (result != TM_Ok) - { - Assert(*update_indexes == TU_None); - *update_indexes = TU_None; - } - else if (!HeapTupleIsHeapOnly(tuple)) - Assert(*update_indexes == TU_All); - else - Assert((*update_indexes == TU_Summarizing) || - (*update_indexes == TU_None)); - if (shouldFree) pfree(tuple); diff --git a/src/backend/access/table/tableam.c b/src/backend/access/table/tableam.c index e18a8f8250f..2b5fe33e43b 100644 --- a/src/backend/access/table/tableam.c +++ b/src/backend/access/table/tableam.c @@ -334,8 +334,7 @@ simple_table_tuple_delete(Relation rel, ItemPointer tid, Snapshot snapshot) void simple_table_tuple_update(Relation rel, ItemPointer otid, TupleTableSlot *slot, - Snapshot snapshot, - TU_UpdateIndexes *update_indexes) + Snapshot snapshot) { TM_Result result; TM_FailureData tmfd; @@ -345,7 +344,7 @@ simple_table_tuple_update(Relation rel, ItemPointer otid, GetCurrentCommandId(true), snapshot, InvalidSnapshot, true /* wait for commit */ , - &tmfd, &lockmode, update_indexes); + &tmfd, &lockmode, NULL, NULL); switch (result) { diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index cdabf780244..bf9d558a8c7 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -2455,7 +2455,20 @@ BuildIndexInfo(Relation index) /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) + { ii->ii_IndexAttrNumbers[i] = indexStruct->indkey.values[i]; + ii->ii_IndexAttrs = + bms_add_member(ii->ii_IndexAttrs, + indexStruct->indkey.values[i] - FirstLowInvalidHeapAttributeNumber); + } + + /* collect attributes used in the expression, if one is present */ + if (ii->ii_Expressions) + pull_varattnos((Node *) ii->ii_Expressions, 1, &ii->ii_ExpressionAttrs); + + /* collect attributes used in the predicate, if one is present */ + if (ii->ii_Predicate) + pull_varattnos((Node *) ii->ii_Predicate, 1, &ii->ii_PredicateAttrs); /* fetch exclusion constraint info if any */ if (indexStruct->indisexclusion) @@ -2466,6 +2479,8 @@ BuildIndexInfo(Relation index) &ii->ii_ExclusionStrats); } + ii->ii_OpClassDataTypes = index->rd_opcintype; + return ii; } diff --git a/src/backend/catalog/indexing.c b/src/backend/catalog/indexing.c index 25c4b6bdc87..3fdebba3f07 100644 --- a/src/backend/catalog/indexing.c +++ b/src/backend/catalog/indexing.c @@ -72,8 +72,7 @@ CatalogCloseIndexes(CatalogIndexState indstate) * This is effectively a cut-down version of ExecInsertIndexTuples. */ static void -CatalogIndexInsert(CatalogIndexState indstate, HeapTuple heapTuple, - TU_UpdateIndexes updateIndexes) +CatalogIndexInsert(CatalogIndexState indstate, HeapTuple heapTuple) { int i; int numIndexes; @@ -83,20 +82,9 @@ CatalogIndexInsert(CatalogIndexState indstate, HeapTuple heapTuple, IndexInfo **indexInfoArray; Datum values[INDEX_MAX_KEYS]; bool isnull[INDEX_MAX_KEYS]; - bool onlySummarized = (updateIndexes == TU_Summarizing); - /* - * HOT update does not require index inserts. But with asserts enabled we - * want to check that it'd be legal to currently insert into the - * table/index. - */ -#ifndef USE_ASSERT_CHECKING - if (HeapTupleIsHeapOnly(heapTuple) && !onlySummarized) - return; -#endif - - /* When only updating summarized indexes, the tuple has to be HOT. */ - Assert((!onlySummarized) || HeapTupleIsHeapOnly(heapTuple)); + /* The tuple never be HOT. */ + Assert(!HeapTupleIsHeapOnly(heapTuple)); /* * Get information from the state structure. Fall out if nothing to do. @@ -138,22 +126,6 @@ CatalogIndexInsert(CatalogIndexState indstate, HeapTuple heapTuple, Assert(index->rd_index->indimmediate); Assert(indexInfo->ii_NumIndexKeyAttrs != 0); - /* see earlier check above */ -#ifdef USE_ASSERT_CHECKING - if (HeapTupleIsHeapOnly(heapTuple) && !onlySummarized) - { - Assert(!ReindexIsProcessingIndex(RelationGetRelid(index))); - continue; - } -#endif /* USE_ASSERT_CHECKING */ - - /* - * Skip insertions into non-summarizing indexes if we only need to - * update summarizing indexes. - */ - if (onlySummarized && !indexInfo->ii_Summarizing) - continue; - /* * FormIndexDatum fills in its values and isnull parameters with the * appropriate values for the column(s) of the index. @@ -240,7 +212,7 @@ CatalogTupleInsert(Relation heapRel, HeapTuple tup) simple_heap_insert(heapRel, tup); - CatalogIndexInsert(indstate, tup, TU_All); + CatalogIndexInsert(indstate, tup); CatalogCloseIndexes(indstate); } @@ -260,7 +232,7 @@ CatalogTupleInsertWithInfo(Relation heapRel, HeapTuple tup, simple_heap_insert(heapRel, tup); - CatalogIndexInsert(indstate, tup, TU_All); + CatalogIndexInsert(indstate, tup); } /* @@ -291,7 +263,7 @@ CatalogTuplesMultiInsertWithInfo(Relation heapRel, TupleTableSlot **slot, tuple = ExecFetchSlotHeapTuple(slot[i], true, &should_free); tuple->t_tableOid = slot[i]->tts_tableOid; - CatalogIndexInsert(indstate, tuple, TU_All); + CatalogIndexInsert(indstate, tuple); if (should_free) heap_freetuple(tuple); @@ -313,15 +285,14 @@ void CatalogTupleUpdate(Relation heapRel, ItemPointer otid, HeapTuple tup) { CatalogIndexState indstate; - TU_UpdateIndexes updateIndexes = TU_All; CatalogTupleCheckConstraints(heapRel, tup); indstate = CatalogOpenIndexes(heapRel); - simple_heap_update(heapRel, otid, tup, &updateIndexes); + simple_heap_update(heapRel, otid, tup); - CatalogIndexInsert(indstate, tup, updateIndexes); + CatalogIndexInsert(indstate, tup); CatalogCloseIndexes(indstate); } @@ -337,13 +308,11 @@ void CatalogTupleUpdateWithInfo(Relation heapRel, ItemPointer otid, HeapTuple tup, CatalogIndexState indstate) { - TU_UpdateIndexes updateIndexes = TU_All; - CatalogTupleCheckConstraints(heapRel, tup); - simple_heap_update(heapRel, otid, tup, &updateIndexes); + simple_heap_update(heapRel, otid, tup); - CatalogIndexInsert(indstate, tup, updateIndexes); + CatalogIndexInsert(indstate, tup); } /* diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c index 7c87f012c30..094bf499028 100644 --- a/src/backend/executor/execIndexing.c +++ b/src/backend/executor/execIndexing.c @@ -117,6 +117,8 @@ #include "utils/multirangetypes.h" #include "utils/rangetypes.h" #include "utils/snapmgr.h" +#include "utils/datum.h" +#include "utils/lsyscache.h" /* waitMode argument to check_exclusion_or_unique_constraint() */ typedef enum @@ -168,6 +170,7 @@ ExecOpenIndices(ResultRelInfo *resultRelInfo, bool speculative) IndexInfo **indexInfoArray; resultRelInfo->ri_NumIndices = 0; + resultRelation->rd_indexinfolist = NIL; /* fast path if no indexes */ if (!RelationGetForm(resultRelation)->relhasindex) @@ -210,6 +213,10 @@ ExecOpenIndices(ResultRelInfo *resultRelInfo, bool speculative) /* extract index key information from the index's pg_index info */ ii = BuildIndexInfo(indexDesc); + /* used when determining if indexed values changed during update */ + resultRelation->rd_indexinfolist = + lappend(resultRelation->rd_indexinfolist, ii); + /* * If the indexes are to be used for speculative insertion or conflict * detection in logical replication, add extra information required by @@ -307,7 +314,7 @@ ExecInsertIndexTuples(ResultRelInfo *resultRelInfo, bool noDupErr, bool *specConflict, List *arbiterIndexes, - bool onlySummarizing) + Bitmapset *modified_indexes) { ItemPointer tupleid = &slot->tts_tid; List *result = NIL; @@ -364,10 +371,12 @@ ExecInsertIndexTuples(ResultRelInfo *resultRelInfo, continue; /* - * Skip processing of non-summarizing indexes if we only update - * summarizing indexes + * If modified_indexes is NULL, we're not in a HOT update, so we + * should update all indexes. If it's not NULL, we should only update + * the listed indexes. This list will include any summarizing indexes + * that require updates on the HOT path as well. */ - if (onlySummarizing && !indexInfo->ii_Summarizing) + if (update && modified_indexes && !bms_is_member(i, modified_indexes)) continue; /* Check for partial index */ @@ -1089,6 +1098,9 @@ index_unchanged_by_update(ResultRelInfo *resultRelInfo, EState *estate, if (hasexpression) { + if (indexInfo->ii_IndexUnchanged) + return true; + indexInfo->ii_IndexUnchanged = false; return false; } @@ -1166,3 +1178,221 @@ ExecWithoutOverlapsNotEmpty(Relation rel, NameData attname, Datum attval, char t errmsg("empty WITHOUT OVERLAPS value found in column \"%s\" in relation \"%s\"", NameStr(attname), RelationGetRelationName(rel)))); } + +/* + * Determine which indexes must be invoked during ExecIndexInsertTuple(). + * + * That will include: any index on a column where there is an attribute that was + * modified, any expression index where the expression's value updated, any + * index used in a constraint, and any summarizing index. + */ +Bitmapset * +ExecIndexesRequiringUpdates(Relation relation, + Bitmapset *modified_attrs, + EState *estate, + HeapTuple old_tuple, + HeapTuple new_tuple, + bool *only_summarizing) +{ + ListCell *lc; + List *indexinfolist = relation->rd_indexinfolist; + ExprContext *econtext = NULL; + int num_summarizing = 0; + TupleDesc tupledesc; + TupleTableSlot *old_tts, + *new_tts; + bool expression_checks = RelationGetExpressionChecks(relation); + Bitmapset *result = NULL; + + /* + * Examine each index on this relation relative to the changes between old + * and new tuples. + */ + foreach(lc, indexinfolist) + { + IndexInfo *indexInfo = (IndexInfo *) lfirst(lc); + + /* + * Summarizing indexes don't prevent HOT updates, but do require that + * they are updated when necessary so we include it in the set. + */ + if (indexInfo->ii_Summarizing) + { + if (bms_overlap(indexInfo->ii_IndexAttrs, modified_attrs)) + { + num_summarizing++; + result = bms_add_member(result, foreach_current_index(lc)); + } + continue; + } + + /* + * If this is a partial index it has a predicate, evaluate the + * expression to determine if we need to include it or not. + */ + if (expression_checks && estate != NULL && + bms_overlap(indexInfo->ii_PredicateAttrs, modified_attrs)) + { + ExprState *pstate; + bool old_tuple_qualifies, + new_tuple_qualifies; + + /* Create these once, only if necessary, then reuse them. */ + if (!econtext) + { + econtext = GetPerTupleExprContext(estate); + tupledesc = RelationGetDescr(relation); + old_tts = MakeSingleTupleTableSlot(tupledesc, + &TTSOpsHeapTuple); + new_tts = MakeSingleTupleTableSlot(tupledesc, + &TTSOpsHeapTuple); + + ExecStoreHeapTuple(old_tuple, old_tts, InvalidBuffer); + ExecStoreHeapTuple(new_tuple, new_tts, InvalidBuffer); + } + + pstate = ExecPrepareQual(indexInfo->ii_Predicate, estate); + + /* + * Here the term "qualifies" means "satisfies the predicate + * condition of the partial index". + */ + econtext->ecxt_scantuple = old_tts; + old_tuple_qualifies = ExecQual(pstate, econtext); + + econtext->ecxt_scantuple = new_tts; + new_tuple_qualifies = ExecQual(pstate, econtext); + + /* + * If neither the old nor the new tuples satisfy the predicate we + * can be sure that this index doesn't need updating, continue to + * the next index. + */ + if ((new_tuple_qualifies == false) && (old_tuple_qualifies == false)) + continue; + + /* + * If there is a transition between indexed and not indexed, + * that's enough to require that this index is updated. + */ + if (new_tuple_qualifies != old_tuple_qualifies) + { + result = bms_add_member(result, foreach_current_index(lc)); + continue; + } + + /* + * Otherwise the old and new values exist in the index, but did + * they get updated? We don't yet know, so proceed with the next + * statement in the loop to find out. + */ + } + + + /* + * Indexes with expressions may or may not have changed, it is + * impossible to know without exercising their expression and + * reviewing index tuple state for changes. This is a lot of work, + * but because all indexes on JSONB columns fall into this category it + * can be worth it to avoid index updates and remain on the HOT update + * path when possible. + */ + if (bms_overlap(indexInfo->ii_ExpressionAttrs, modified_attrs)) + { + Datum old_values[INDEX_MAX_KEYS]; + bool old_isnull[INDEX_MAX_KEYS]; + Datum new_values[INDEX_MAX_KEYS]; + bool new_isnull[INDEX_MAX_KEYS]; + bool changed = false; + + /* + * Assume the index is changed when we don't have an estate + * context to use or the reloption is disabled. + */ + if (!expression_checks || estate == NULL) + { + result = bms_add_member(result, foreach_current_index(lc)); + continue; + } + + /* Create these once, only if necessary, then reuse them. */ + if (!econtext) + { + econtext = GetPerTupleExprContext(estate); + tupledesc = RelationGetDescr(relation); + old_tts = MakeSingleTupleTableSlot(tupledesc, + &TTSOpsHeapTuple); + new_tts = MakeSingleTupleTableSlot(tupledesc, + &TTSOpsHeapTuple); + + ExecStoreHeapTuple(old_tuple, old_tts, InvalidBuffer); + ExecStoreHeapTuple(new_tuple, new_tts, InvalidBuffer); + } + + indexInfo->ii_ExpressionsState = NIL; + + econtext->ecxt_scantuple = old_tts; + FormIndexDatum(indexInfo, + old_tts, + estate, + old_values, + old_isnull); + + econtext->ecxt_scantuple = new_tts; + FormIndexDatum(indexInfo, + new_tts, + estate, + new_values, + new_isnull); + + for (int i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++) + { + if (old_isnull[i] != new_isnull[i]) + { + changed = true; + break; + } + else if (!old_isnull[i]) + { + int16 elmlen; + bool elmbyval; + Oid opcintyp = indexInfo->ii_OpClassDataTypes[i]; + + get_typlenbyval(opcintyp, &elmlen, &elmbyval); + if (!datum_image_eq(old_values[i], new_values[i], + elmbyval, elmlen)) + { + changed = true; + break; + } + } + } + + if (changed) + result = bms_add_member(result, foreach_current_index(lc)); + + /* Shortcut index_unchanged_by_update(), we know the answer. */ + indexInfo->ii_CheckedUnchanged = true; + indexInfo->ii_IndexUnchanged = !changed; + continue; + } + + /* + * If the index references modified attributes then it needs to be + * updated. + */ + if (bms_overlap(indexInfo->ii_IndexAttrs, modified_attrs)) + result = bms_add_member(result, foreach_current_index(lc)); + } + + if (econtext) + { + ExecDropSingleTupleTableSlot(old_tts); + ExecDropSingleTupleTableSlot(new_tts); + } + + *only_summarizing = (list_length(indexinfolist) > 0 && + num_summarizing == bms_num_members(result)); + + return result; +} diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c index 5f7613cc831..dd5108dc8f2 100644 --- a/src/backend/executor/execReplication.c +++ b/src/backend/executor/execReplication.c @@ -639,9 +639,9 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo, if (!skip_tuple) { List *recheckIndexes = NIL; - TU_UpdateIndexes update_indexes; List *conflictindexes; bool conflict = false; + Bitmapset *modified_indexes = NULL; /* Compute stored generated columns */ if (rel->rd_att->constr && @@ -655,17 +655,16 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo, if (rel->rd_rel->relispartition) ExecPartitionCheck(resultRelInfo, slot, estate, true); - simple_table_tuple_update(rel, tid, slot, estate->es_snapshot, - &update_indexes); + simple_table_tuple_update(rel, tid, slot, estate->es_snapshot); conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes; - if (resultRelInfo->ri_NumIndices > 0 && (update_indexes != TU_None)) + if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(resultRelInfo, slot, estate, true, conflictindexes ? true : false, &conflict, conflictindexes, - (update_indexes == TU_Summarizing)); + modified_indexes); /* * Refer to the comments above the call to CheckAndReportConflict() in @@ -683,6 +682,7 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo, recheckIndexes, NULL, false); list_free(recheckIndexes); + bms_free(modified_indexes); } } diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index a15e7863b0d..98ae771c8f7 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -120,8 +120,12 @@ typedef struct ModifyTableContext */ typedef struct UpdateContext { + Bitmapset *modifiedIndexes; /* Either NULL indicating that all indexes + * should be updated or a bitmap of + * IndexInfo array positions for the + * subset of modified indexes requiring + * updates. */ bool crossPartUpdate; /* was it a cross-partition update? */ - TU_UpdateIndexes updateIndexes; /* Which index updates are required? */ /* * Lock mode to acquire on the latest tuple version before performing @@ -2283,7 +2287,8 @@ lreplace: estate->es_crosscheck_snapshot, true /* wait for commit */ , &context->tmfd, &updateCxt->lockmode, - &updateCxt->updateIndexes); + &updateCxt->modifiedIndexes, + estate); return result; } @@ -2303,12 +2308,12 @@ ExecUpdateEpilogue(ModifyTableContext *context, UpdateContext *updateCxt, List *recheckIndexes = NIL; /* insert index entries for tuple if necessary */ - if (resultRelInfo->ri_NumIndices > 0 && (updateCxt->updateIndexes != TU_None)) + if (resultRelInfo->ri_NumIndices > 0) recheckIndexes = ExecInsertIndexTuples(resultRelInfo, slot, context->estate, true, false, NULL, NIL, - (updateCxt->updateIndexes == TU_Summarizing)); + updateCxt->modifiedIndexes); /* AFTER ROW UPDATE Triggers */ ExecARUpdateTriggers(context->estate, resultRelInfo, diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index 398114373e9..f1ddf1ea505 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -5229,7 +5229,11 @@ RelationGetIndexAttrBitmap(Relation relation, IndexAttrBitmapKind attrKind) Bitmapset *pkindexattrs; /* columns in the primary index */ Bitmapset *idindexattrs; /* columns in the replica identity */ Bitmapset *hotblockingattrs; /* columns with HOT blocking indexes */ + Bitmapset *hotblockingexprattrs; /* as above, but only those in + * expressions */ Bitmapset *summarizedattrs; /* columns with summarizing indexes */ + Bitmapset *summarizedexprattrs; /* as above, but only those in + * expressions */ List *indexoidlist; List *newindexoidlist; Oid relpkindex; @@ -5252,6 +5256,8 @@ RelationGetIndexAttrBitmap(Relation relation, IndexAttrBitmapKind attrKind) return bms_copy(relation->rd_hotblockingattr); case INDEX_ATTR_BITMAP_SUMMARIZED: return bms_copy(relation->rd_summarizedattr); + case INDEX_ATTR_BITMAP_EXPRESSION: + return bms_copy(relation->rd_expressionattr); default: elog(ERROR, "unknown attrKind %u", attrKind); } @@ -5295,7 +5301,9 @@ restart: pkindexattrs = NULL; idindexattrs = NULL; hotblockingattrs = NULL; + hotblockingexprattrs = NULL; summarizedattrs = NULL; + summarizedexprattrs = NULL; foreach(l, indexoidlist) { Oid indexOid = lfirst_oid(l); @@ -5309,6 +5317,7 @@ restart: bool isPK; /* primary key */ bool isIDKey; /* replica identity index */ Bitmapset **attrs; + Bitmapset **exprattrs; indexDesc = index_open(indexOid, AccessShareLock); @@ -5352,14 +5361,20 @@ restart: * decide which bitmap we'll update in the following loop. */ if (indexDesc->rd_indam->amsummarizing) + { attrs = &summarizedattrs; + exprattrs = &summarizedexprattrs; + } else + { attrs = &hotblockingattrs; + exprattrs = &hotblockingexprattrs; + } /* Collect simple attribute references */ for (i = 0; i < indexDesc->rd_index->indnatts; i++) { - int attrnum = indexDesc->rd_index->indkey.values[i]; + int attridx = indexDesc->rd_index->indkey.values[i]; /* * Since we have covering indexes with non-key columns, we must @@ -5375,30 +5390,28 @@ restart: * key or identity key. Hence we do not include them into * uindexattrs, pkindexattrs and idindexattrs bitmaps. */ - if (attrnum != 0) + if (attridx != 0) { - *attrs = bms_add_member(*attrs, - attrnum - FirstLowInvalidHeapAttributeNumber); + AttrNumber attrnum = attridx - FirstLowInvalidHeapAttributeNumber; + + *attrs = bms_add_member(*attrs, attrnum); if (isKey && i < indexDesc->rd_index->indnkeyatts) - uindexattrs = bms_add_member(uindexattrs, - attrnum - FirstLowInvalidHeapAttributeNumber); + uindexattrs = bms_add_member(uindexattrs, attrnum); if (isPK && i < indexDesc->rd_index->indnkeyatts) - pkindexattrs = bms_add_member(pkindexattrs, - attrnum - FirstLowInvalidHeapAttributeNumber); + pkindexattrs = bms_add_member(pkindexattrs, attrnum); if (isIDKey && i < indexDesc->rd_index->indnkeyatts) - idindexattrs = bms_add_member(idindexattrs, - attrnum - FirstLowInvalidHeapAttributeNumber); + idindexattrs = bms_add_member(idindexattrs, attrnum); } } /* Collect all attributes used in expressions, too */ - pull_varattnos(indexExpressions, 1, attrs); + pull_varattnos(indexExpressions, 1, exprattrs); /* Collect all attributes in the index predicate, too */ - pull_varattnos(indexPredicate, 1, attrs); + pull_varattnos(indexPredicate, 1, exprattrs); index_close(indexDesc, AccessShareLock); } @@ -5427,11 +5440,27 @@ restart: bms_free(pkindexattrs); bms_free(idindexattrs); bms_free(hotblockingattrs); + bms_free(hotblockingexprattrs); bms_free(summarizedattrs); + bms_free(summarizedexprattrs); goto restart; } + /* {expression-only columns} = {expression columns} - {direct columns} */ + hotblockingexprattrs = bms_del_members(hotblockingexprattrs, + hotblockingattrs); + /* {hot-blocking columns} = {direct columns} + {expression-only columns} */ + hotblockingattrs = bms_add_members(hotblockingattrs, + hotblockingexprattrs); + + /* {summarized-only columns} = {summarized columns} - {direct columns} */ + summarizedexprattrs = bms_del_members(summarizedexprattrs, + summarizedattrs); + /* {summarized columns} = {direct columns} + {summarized-only columns} */ + summarizedattrs = bms_add_members(summarizedattrs, + summarizedexprattrs); + /* Don't leak the old values of these bitmaps, if any */ relation->rd_attrsvalid = false; bms_free(relation->rd_keyattr); @@ -5444,6 +5473,8 @@ restart: relation->rd_hotblockingattr = NULL; bms_free(relation->rd_summarizedattr); relation->rd_summarizedattr = NULL; + bms_free(relation->rd_expressionattr); + relation->rd_expressionattr = NULL; /* * Now save copies of the bitmaps in the relcache entry. We intentionally @@ -5458,6 +5489,7 @@ restart: relation->rd_idattr = bms_copy(idindexattrs); relation->rd_hotblockingattr = bms_copy(hotblockingattrs); relation->rd_summarizedattr = bms_copy(summarizedattrs); + relation->rd_expressionattr = bms_copy(hotblockingexprattrs); relation->rd_attrsvalid = true; MemoryContextSwitchTo(oldcxt); @@ -5474,6 +5506,8 @@ restart: return hotblockingattrs; case INDEX_ATTR_BITMAP_SUMMARIZED: return summarizedattrs; + case INDEX_ATTR_BITMAP_EXPRESSION: + return hotblockingexprattrs; default: elog(ERROR, "unknown attrKind %u", attrKind); return NULL; diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index a9a81ab3c14..1c4cee75157 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -2967,7 +2967,7 @@ match_previous_words(int pattern_id, COMPLETE_WITH("("); /* ALTER TABLESPACE <foo> SET|RESET ( */ else if (Matches("ALTER", "TABLESPACE", MatchAny, "SET|RESET", "(")) - COMPLETE_WITH("seq_page_cost", "random_page_cost", + COMPLETE_WITH("seq_page_cost", "random_page_cost", "expression_checks", "effective_io_concurrency", "maintenance_io_concurrency"); /* ALTER TEXT SEARCH */ diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 1640d9c32f7..e12173eba42 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -21,6 +21,7 @@ #include "access/skey.h" #include "access/table.h" /* for backward compatibility */ #include "access/tableam.h" +#include "executor/executor.h" #include "nodes/lockoptions.h" #include "nodes/primnodes.h" #include "storage/bufpage.h" @@ -339,7 +340,7 @@ extern TM_Result heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, CommandId cid, Snapshot crosscheck, bool wait, struct TM_FailureData *tmfd, LockTupleMode *lockmode, - TU_UpdateIndexes *update_indexes); + Bitmapset **modified_indexes, struct EState *estate); extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple, CommandId cid, LockTupleMode mode, LockWaitPolicy wait_policy, bool follow_updates, @@ -374,7 +375,7 @@ extern bool heap_tuple_needs_eventual_freeze(HeapTupleHeader tuple); extern void simple_heap_insert(Relation relation, HeapTuple tup); extern void simple_heap_delete(Relation relation, ItemPointer tid); extern void simple_heap_update(Relation relation, ItemPointer otid, - HeapTuple tup, TU_UpdateIndexes *update_indexes); + HeapTuple tup); extern TransactionId heap_index_delete_tuples(Relation rel, TM_IndexDeleteOp *delstate); diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h index 131c050c15f..300226d44d5 100644 --- a/src/include/access/tableam.h +++ b/src/include/access/tableam.h @@ -38,6 +38,7 @@ struct IndexInfo; struct SampleScanState; struct VacuumParams; struct ValidateIndexState; +struct EState; /* * Bitmask values for the flags argument to the scan_begin callback. @@ -109,21 +110,6 @@ typedef enum TM_Result TM_WouldBlock, } TM_Result; -/* - * Result codes for table_update(..., update_indexes*..). - * Used to determine which indexes to update. - */ -typedef enum TU_UpdateIndexes -{ - /* No indexed columns were updated (incl. TID addressing of tuple) */ - TU_None, - - /* A non-summarizing indexed column was updated, or the TID has changed */ - TU_All, - - /* Only summarized columns were updated, TID is unchanged */ - TU_Summarizing, -} TU_UpdateIndexes; /* * When table_tuple_update, table_tuple_delete, or table_tuple_lock fail @@ -550,7 +536,8 @@ typedef struct TableAmRoutine bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode, - TU_UpdateIndexes *update_indexes); + Bitmapset **modified_indexes, + struct EState *estate); /* see table_tuple_lock() for reference about parameters */ TM_Result (*tuple_lock) (Relation rel, @@ -1541,12 +1528,12 @@ static inline TM_Result table_tuple_update(Relation rel, ItemPointer otid, TupleTableSlot *slot, CommandId cid, Snapshot snapshot, Snapshot crosscheck, bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode, - TU_UpdateIndexes *update_indexes) + Bitmapset **modified_indexes, struct EState *estate) { return rel->rd_tableam->tuple_update(rel, otid, slot, cid, snapshot, crosscheck, - wait, tmfd, - lockmode, update_indexes); + wait, tmfd, lockmode, + modified_indexes, estate); } /* @@ -2075,8 +2062,7 @@ extern void simple_table_tuple_insert(Relation rel, TupleTableSlot *slot); extern void simple_table_tuple_delete(Relation rel, ItemPointer tid, Snapshot snapshot); extern void simple_table_tuple_update(Relation rel, ItemPointer otid, - TupleTableSlot *slot, Snapshot snapshot, - TU_UpdateIndexes *update_indexes); + TupleTableSlot *slot, Snapshot snapshot); /* ---------------------------------------------------------------------------- diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 30e2a82346f..1018bb2afa5 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -650,7 +650,7 @@ extern List *ExecInsertIndexTuples(ResultRelInfo *resultRelInfo, bool update, bool noDupErr, bool *specConflict, List *arbiterIndexes, - bool onlySummarizing); + Bitmapset *modified_indexes); extern bool ExecCheckIndexConstraints(ResultRelInfo *resultRelInfo, TupleTableSlot *slot, EState *estate, ItemPointer conflictTid, @@ -661,6 +661,12 @@ extern void check_exclusion_constraint(Relation heap, Relation index, ItemPointer tupleid, const Datum *values, const bool *isnull, EState *estate, bool newIndex); +extern Bitmapset *ExecIndexesRequiringUpdates(Relation relation, + Bitmapset *modified_attrs, + EState *estate, + HeapTuple old_tuple, + HeapTuple new_tuple, + bool *only_summarizing); /* * prototypes from functions in execReplication.c diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e2d1dc1e067..54035aeb9f8 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -159,12 +159,15 @@ typedef struct ExprState * * NumIndexAttrs total number of columns in this index * NumIndexKeyAttrs number of key columns in index + * IndexAttrs bitmap of index attributes * IndexAttrNumbers underlying-rel attribute numbers used as keys * (zeroes indicate expressions). It also contains * info about included columns. * Expressions expr trees for expression entries, or NIL if none + * ExpressionAttrs bitmap of attributes used within the expression * ExpressionsState exec state for expressions, or NIL if none * Predicate partial-index predicate, or NIL if none + * PredicateAttrs bitmap of attributes used within the predicate * PredicateState exec state for predicate, or NIL if none * ExclusionOps Per-column exclusion operators, or NULL if none * ExclusionProcs Underlying function OIDs for ExclusionOps @@ -183,6 +186,7 @@ typedef struct ExprState * ParallelWorkers # of workers requested (excludes leader) * Am Oid of index AM * AmCache private cache area for index AM + * OpClassDataTypes operator class data types * Context memory context holding this IndexInfo * * ii_Concurrent, ii_BrokenHotChain, and ii_ParallelWorkers are used only @@ -194,10 +198,13 @@ typedef struct IndexInfo NodeTag type; int ii_NumIndexAttrs; /* total number of columns in index */ int ii_NumIndexKeyAttrs; /* number of key columns in index */ + Bitmapset *ii_IndexAttrs; AttrNumber ii_IndexAttrNumbers[INDEX_MAX_KEYS]; List *ii_Expressions; /* list of Expr */ + Bitmapset *ii_ExpressionAttrs; List *ii_ExpressionsState; /* list of ExprState */ List *ii_Predicate; /* list of Expr */ + Bitmapset *ii_PredicateAttrs; ExprState *ii_PredicateState; Oid *ii_ExclusionOps; /* array with one entry per column */ Oid *ii_ExclusionProcs; /* array with one entry per column */ @@ -217,6 +224,7 @@ typedef struct IndexInfo int ii_ParallelWorkers; Oid ii_Am; void *ii_AmCache; + Oid *ii_OpClassDataTypes; MemoryContext ii_Context; } IndexInfo; diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index db3e504c3d2..3067379b036 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -154,6 +154,10 @@ typedef struct RelationData bool rd_ispkdeferrable; /* is rd_pkindex a deferrable PK? */ Oid rd_replidindex; /* OID of replica identity index, if any */ + /* list of IndexInfo for this relation */ + List *rd_indexinfolist; /* an ordered list of IndexInfo for + * indexes on relation */ + /* data managed by RelationGetStatExtList: */ List *rd_statlist; /* list of OIDs of extended stats */ @@ -164,6 +168,7 @@ typedef struct RelationData Bitmapset *rd_idattr; /* included in replica identity index */ Bitmapset *rd_hotblockingattr; /* cols blocking HOT update */ Bitmapset *rd_summarizedattr; /* cols indexed by summarizing indexes */ + Bitmapset *rd_expressionattr; /* indexed cols referenced by expressions */ PublicationDesc *rd_pubdesc; /* publication descriptor, or NULL */ @@ -344,6 +349,7 @@ typedef struct StdRdOptions int parallel_workers; /* max number of parallel workers */ StdRdOptIndexCleanup vacuum_index_cleanup; /* controls index vacuuming */ bool vacuum_truncate; /* enables vacuum to truncate a relation */ + bool expression_checks; /* use expression to checks for changes */ /* * Fraction of pages in a relation that vacuum can eagerly scan and fail @@ -405,6 +411,14 @@ typedef struct StdRdOptions ((relation)->rd_options ? \ ((StdRdOptions *) (relation)->rd_options)->parallel_workers : (defaultpw)) +/* + * RelationGetExpressionChecks + * Returns the relation's expression_checks reloption setting. + */ +#define RelationGetExpressionChecks(relation) \ + ((relation)->rd_options ? \ + ((StdRdOptions *) (relation)->rd_options)->expression_checks : true) + /* ViewOptions->check_option values */ typedef enum ViewOptCheckOption { diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h index a7c55db339e..0cc28cb97e2 100644 --- a/src/include/utils/relcache.h +++ b/src/include/utils/relcache.h @@ -63,6 +63,7 @@ typedef enum IndexAttrBitmapKind INDEX_ATTR_BITMAP_IDENTITY_KEY, INDEX_ATTR_BITMAP_HOT_BLOCKING, INDEX_ATTR_BITMAP_SUMMARIZED, + INDEX_ATTR_BITMAP_EXPRESSION, } IndexAttrBitmapKind; extern Bitmapset *RelationGetIndexAttrBitmap(Relation relation, diff --git a/src/test/regress/expected/heap_hot_updates.out b/src/test/regress/expected/heap_hot_updates.out new file mode 100644 index 00000000000..2ad6e5418e9 --- /dev/null +++ b/src/test/regress/expected/heap_hot_updates.out @@ -0,0 +1,586 @@ +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table will have two columns and two indexes, one on the primary key +-- id and one on the expression (info->>'name'). That means that the indexed +-- attributes are 'id' and 'info'. +create table keyvalue(id integer primary key, info jsonb); +create index nameindex on keyvalue((info->>'name')); +insert into keyvalue values (1, '{"name": "john", "data": "some data"}'); +-- Disable expression checks. +ALTER TABLE keyvalue SET (expression_checks = false); +SELECT reloptions FROM pg_class WHERE relname = 'keyvalue'; + reloptions +--------------------------- + {expression_checks=false} +(1 row) + +-- While the indexed attribute "name" is unchanged we've disabled expression +-- checks so this update should not go HOT as the system can't determine if +-- the indexed attribute has changed without evaluating the expression. +update keyvalue set info='{"name": "john", "data": "something else"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 0 row + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Re-enable expression checks. +ALTER TABLE keyvalue SET (expression_checks = true); +SELECT reloptions FROM pg_class WHERE relname = 'keyvalue'; + reloptions +-------------------------- + {expression_checks=true} +(1 row) + +-- The indexed attribute "name" with value "john" is unchanged, expect a HOT update. +update keyvalue set info='{"name": "john", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 1 row + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- The following update changes the indexed attribute "name", this should not be a HOT update. +update keyvalue set info='{"name": "smith", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Now, this update does not change the indexed attribute "name" from "smith", this should be HOT. +update keyvalue set info='{"name": "smith", "data": "some more data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 2 rows now + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 2 +(1 row) + +drop table keyvalue; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table is the same as the previous one but it has a third index. The +-- index 'colindex' isn't an expression index, it indexes the entire value +-- in the info column. There are still only two indexed attributes for this +-- relation, the same two as before. The presence of an index on the entire +-- value of the info column should prevent HOT updates for any updates to any +-- portion of JSONB content in that column. +create table keyvalue(id integer primary key, info jsonb); +create index nameindex on keyvalue((info->>'name')); +create index colindex on keyvalue(info); +insert into keyvalue values (1, '{"name": "john", "data": "some data"}'); +-- This update doesn't change the value of the expression index, but it does +-- change the content of the info column and so should not be HOT because the +-- indexed value changed as a result of the update. +update keyvalue set info='{"name": "john", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 0 rows + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +drop table keyvalue; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- The table has one column docs and two indexes. They are both expression +-- indexes referencing the same column attribute (docs) but one is a partial +-- index. +CREATE TABLE ex (docs JSONB) WITH (fillfactor = 60); +INSERT INTO ex (docs) VALUES ('{"a": 0, "b": 0}'); +INSERT INTO ex (docs) SELECT jsonb_build_object('b', n) FROM generate_series(100, 10000) as n; +CREATE INDEX idx_ex_a ON ex ((docs->>'a')); +CREATE INDEX idx_ex_b ON ex ((docs->>'b')) WHERE (docs->>'b')::numeric > 9; +-- We're using BTREE indexes and for this test we want to make sure that they remain +-- in sync with changes to our relation. Force the choice of index scans below so +-- that we know we're checking the index's understanding of what values should be +-- in the index or not. +SET SESSION enable_seqscan = OFF; +SET SESSION enable_bitmapscan = OFF; +-- Leave 'a' unchanged but modify 'b' to a value outside of the index predicate. +-- This should be a HOT update because neither index is changed. +UPDATE ex SET docs = jsonb_build_object('a', 0, 'b', 1) WHERE (docs->>'a')::numeric = 0; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 row + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Let's check to make sure that the index does not contain a value for 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + QUERY PLAN +-------------------------------------------------------------- + Index Scan using idx_ex_b on ex + Filter: (((docs ->> 'b'::text))::numeric < '100'::numeric) +(2 rows) + +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + docs +------ +(0 rows) + +-- Leave 'a' unchanged but modify 'b' to a value within the index predicate. +-- This represents a change for field 'b' from unindexed to indexed and so +-- this should not take the HOT path. +UPDATE ex SET docs = jsonb_build_object('a', 0, 'b', 10) WHERE (docs->>'a')::numeric = 0; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Let's check to make sure that the index contains the new value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + QUERY PLAN +-------------------------------------------------------------- + Index Scan using idx_ex_b on ex + Filter: (((docs ->> 'b'::text))::numeric < '100'::numeric) +(2 rows) + +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + docs +------------------- + {"a": 0, "b": 10} +(1 row) + +-- This update modifies the value of 'a', an indexed field, so it also cannot +-- be a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 1, 'b', 10) WHERE (docs->>'b')::numeric = 10; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- This update changes both 'a' and 'b' to new values that require index updates, +-- this cannot use the HOT path. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 12) WHERE (docs->>'b')::numeric = 10; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Let's check to make sure that the index contains the new value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + QUERY PLAN +-------------------------------------------------------------- + Index Scan using idx_ex_b on ex + Filter: (((docs ->> 'b'::text))::numeric < '100'::numeric) +(2 rows) + +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + docs +------------------- + {"a": 2, "b": 12} +(1 row) + +-- This update changes 'b' to a value outside its predicate requiring that +-- we remove it from the index. That's a transition that can't be done +-- during a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 1) WHERE (docs->>'b')::numeric = 12; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Let's check to make sure that the index no longer contains the value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + QUERY PLAN +-------------------------------------------------------------- + Index Scan using idx_ex_b on ex + Filter: (((docs ->> 'b'::text))::numeric < '100'::numeric) +(2 rows) + +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + docs +------ +(0 rows) + +-- Disable expression checks. +ALTER TABLE ex SET (expression_checks = false); +SELECT reloptions FROM pg_class WHERE relname = 'ex'; + reloptions +----------------------------------------- + {fillfactor=60,expression_checks=false} +(1 row) + +-- This update changes 'b' to a value within its predicate just like it +-- previous value, which would allow for a HOT update but with expression +-- checks disabled we can't determine that so this should not be a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 2) WHERE (docs->>'b')::numeric = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Let's make sure we're recording HOT updates for our 'ex' relation properly in the system +-- table pg_stat_user_tables. Note that statistics are stored within a transaction context +-- first (xact) and then later into the global statistics for a relation, so first we need +-- to ensure pending stats are flushed. +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT + c.relname AS table_name, + -- Transaction statistics + pg_stat_get_xact_tuples_updated(c.oid) AS xact_updates, + pg_stat_get_xact_tuples_hot_updated(c.oid) AS xact_hot_updates, + ROUND(( + pg_stat_get_xact_tuples_hot_updated(c.oid)::float / + NULLIF(pg_stat_get_xact_tuples_updated(c.oid), 0) * 100 + )::numeric, 2) AS xact_hot_update_percentage, + -- Cumulative statistics + s.n_tup_upd AS total_updates, + s.n_tup_hot_upd AS hot_updates, + ROUND(( + s.n_tup_hot_upd::float / + NULLIF(s.n_tup_upd, 0) * 100 + )::numeric, 2) AS total_hot_update_percentage +FROM pg_class c +LEFT JOIN pg_stat_user_tables s ON c.relname = s.relname +WHERE c.relname = 'ex' +AND c.relnamespace = 'public'::regnamespace; + table_name | xact_updates | xact_hot_updates | xact_hot_update_percentage | total_updates | hot_updates | total_hot_update_percentage +------------+--------------+------------------+----------------------------+---------------+-------------+----------------------------- + ex | 0 | 0 | | 6 | 1 | 16.67 +(1 row) + +-- expect: 5 xact updates with 1 xact hot update and no cumulative updates as yet +SET SESSION enable_seqscan = ON; +SET SESSION enable_bitmapscan = ON; +DROP TABLE ex; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table has a single column 'email' and a unique constraint on it that +-- should preclude HOT updates. +CREATE TABLE users ( + user_id serial primary key, + name VARCHAR(255) NOT NULL, + email VARCHAR(255) NOT NULL, + EXCLUDE USING btree (lower(email) WITH =) +); +-- Add some data to the table and then update it in ways that should and should +-- not be HOT updates. +INSERT INTO users (name, email) VALUES +('user1', '[email protected]'), +('user2', '[email protected]'), +('taken', '[email protected]'), +('you', '[email protected]'), +('taken', '[email protected]'); +-- Should fail because of the unique constraint on the email column. +UPDATE users SET email = '[email protected]' WHERE email = '[email protected]'; +ERROR: conflicting key value violates exclusion constraint "users_lower_excl" +DETAIL: Key (lower(email::text))=([email protected]) conflicts with existing key (lower(email::text))=([email protected]). +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 0 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Should succeed because the email column is not being updated and should go HOT. +UPDATE users SET name = 'foo' WHERE email = '[email protected]'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 a single new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Create a partial index on the email column, updates +CREATE INDEX idx_users_email_no_example ON users (lower(email)) WHERE lower(email) LIKE '%@example.com%'; +-- An update that changes the email column but not the indexed portion of it and falls outside the constraint. +-- Shouldn't be a HOT update because of the exclusion constraint. +UPDATE users SET email = '[email protected]' WHERE name = 'you'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- An update that changes the email column but not the indexed portion of it and falls within the constraint. +-- Again, should fail constraint and fail to be a HOT update. +UPDATE users SET email = '[email protected]' WHERE name = 'you'; +ERROR: conflicting key value violates exclusion constraint "users_lower_excl" +DETAIL: Key (lower(email::text))=([email protected]) conflicts with existing key (lower(email::text))=([email protected]). +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +DROP TABLE users; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- Another test of constraints spoiling HOT updates, this time with a range. +CREATE TABLE events ( + id serial primary key, + name VARCHAR(255) NOT NULL, + event_time tstzrange, + constraint no_screening_time_overlap exclude using gist ( + event_time WITH && + ) +); +-- Add two non-overlapping events. +INSERT INTO events (id, event_time, name) +VALUES + (1, '["2023-01-01 19:00:00", "2023-01-01 20:45:00"]', 'event1'), + (2, '["2023-01-01 21:00:00", "2023-01-01 21:45:00"]', 'event2'); +-- Update the first event to overlap with the second, should fail the constraint and not be HOT. +UPDATE events SET event_time = '["2023-01-01 20:00:00", "2023-01-01 21:45:00"]' WHERE id = 1; +ERROR: conflicting key value violates exclusion constraint "no_screening_time_overlap" +DETAIL: Key (event_time)=(["Sun Jan 01 20:00:00 2023 PST","Sun Jan 01 21:45:00 2023 PST"]) conflicts with existing key (event_time)=(["Sun Jan 01 21:00:00 2023 PST","Sun Jan 01 21:45:00 2023 PST"]). +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 0 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Update the first event to not overlap with the second, again not HOT due to the constraint. +UPDATE events SET event_time = '["2023-01-01 22:00:00", "2023-01-01 22:45:00"]' WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 0 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Update the first event to not overlap with the second, this time we're HOT because we don't overlap with the constraint. +UPDATE events SET name = 'new name here' WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 1 one new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +DROP TABLE events; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- A test to ensure that summarizing indexes are not updated when they don't +-- change, but are updated when they do while not prefent HOT updates. +CREATE TABLE ex (att1 JSONB, att2 text) WITH (fillfactor = 60); +CREATE INDEX expression ON ex USING btree((att1->'data')); +CREATE INDEX summarizing ON ex USING BRIN(att2); +INSERT INTO ex VALUES ('{"data": []}', 'nothing special'); +-- Update the unindexed value of att1, this should be a HOT update and not +-- update the summarizing index. +UPDATE ex SET att1 = att1 || '{"status": "checkmate"}'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 one new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Update the indexed value of att2, a summarized value, this is a summarized +-- only update and should use the HOT path while still triggering an update to +-- the summarizing BRIN index. +UPDATE ex SET att2 = 'special indeed'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 2 one new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 2 +(1 row) + +-- Update to att1 doesn't change the indexed value while the update to att2 does, +-- this again is a summarized only update and should use the HOT path as well as +-- trigger an update to the BRIN index. +UPDATE ex SET att1 = att1 || '{"status": "checkmate!"}', att2 = 'special, so special'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 3 one new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 3 +(1 row) + +-- This updates both indexes, the expression index on att1 and the summarizing +-- index on att2. This should not be a HOT update because there are modified +-- indexes and only some are summarized, not all. This should force all +-- indexes to be updated. +UPDATE ex SET att1 = att1 || '{"data": [1,2,3]}', att2 = 'do you want to play a game?'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 3 both indexes updated + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 3 +(1 row) + +DROP TABLE ex; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This test is for a table with a custom type and a custom operators on +-- the BTREE index. The question is, when comparing values for equality +-- to determine if there are changes on the index or not... shouldn't we +-- be using the custom operators? +-- Create a type +CREATE TYPE my_custom_type AS (val int); +-- Comparison functions (returns boolean) +CREATE FUNCTION my_custom_lt(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val < b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +CREATE FUNCTION my_custom_le(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val <= b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +CREATE FUNCTION my_custom_eq(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val = b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +CREATE FUNCTION my_custom_ge(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val >= b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +CREATE FUNCTION my_custom_gt(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val > b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +CREATE FUNCTION my_custom_ne(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val != b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +-- Comparison function (returns -1, 0, 1) +CREATE FUNCTION my_custom_cmp(a my_custom_type, b my_custom_type) RETURNS int AS $$ +BEGIN + IF a.val < b.val THEN + RETURN -1; + ELSIF a.val > b.val THEN + RETURN 1; + ELSE + RETURN 0; + END IF; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +-- Create the operators +CREATE OPERATOR < ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_lt, + COMMUTATOR = >, + NEGATOR = >= +); +CREATE OPERATOR <= ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_le, + COMMUTATOR = >=, + NEGATOR = > +); +CREATE OPERATOR = ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_eq, + COMMUTATOR = =, + NEGATOR = <> +); +CREATE OPERATOR >= ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_ge, + COMMUTATOR = <=, + NEGATOR = < +); +CREATE OPERATOR > ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_gt, + COMMUTATOR = <, + NEGATOR = <= +); +CREATE OPERATOR <> ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_ne, + COMMUTATOR = <>, + NEGATOR = = +); +-- Create the operator class (including the support function) +CREATE OPERATOR CLASS my_custom_ops + DEFAULT FOR TYPE my_custom_type USING btree AS + OPERATOR 1 <, + OPERATOR 2 <=, + OPERATOR 3 =, + OPERATOR 4 >=, + OPERATOR 5 >, + FUNCTION 1 my_custom_cmp(my_custom_type, my_custom_type); +-- Create the table +CREATE TABLE my_table ( + id int, + custom_val my_custom_type +); +-- Insert some data +INSERT INTO my_table (id, custom_val) VALUES +(1, ROW(3)::my_custom_type), +(2, ROW(1)::my_custom_type), +(3, ROW(4)::my_custom_type), +(4, ROW(2)::my_custom_type); +-- Create a function to use when indexing +CREATE OR REPLACE FUNCTION abs_val(val my_custom_type) RETURNS int AS $$ +BEGIN + RETURN abs(val.val); +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +-- Create the index +CREATE INDEX idx_custom_val_abs ON my_table (abs_val(custom_val)); +-- Update 1 +UPDATE my_table SET custom_val = ROW(5)::my_custom_type WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Update 2 +UPDATE my_table SET custom_val = ROW(0)::my_custom_type WHERE custom_val < ROW(3)::my_custom_type; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Update 3 +UPDATE my_table SET custom_val = ROW(6)::my_custom_type WHERE id = 3; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Update 4 +UPDATE my_table SET id = 5 WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Query using the index +EXPLAIN (COSTS OFF) SELECT * FROM my_table WHERE abs_val(custom_val) = 6; + QUERY PLAN +------------------------------------- + Seq Scan on my_table + Filter: (abs_val(custom_val) = 6) +(2 rows) + +SELECT * FROM my_table WHERE abs_val(custom_val) = 6; + id | custom_val +----+------------ + 3 | (6) +(1 row) + +-- Clean up +DROP TABLE my_table CASCADE; +DROP OPERATOR CLASS my_custom_ops USING btree CASCADE; +DROP OPERATOR < (my_custom_type, my_custom_type); +DROP OPERATOR <= (my_custom_type, my_custom_type); +DROP OPERATOR = (my_custom_type, my_custom_type); +DROP OPERATOR >= (my_custom_type, my_custom_type); +DROP OPERATOR > (my_custom_type, my_custom_type); +DROP OPERATOR <> (my_custom_type, my_custom_type); +DROP FUNCTION my_custom_lt(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_le(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_eq(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_ge(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_gt(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_ne(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_cmp(my_custom_type, my_custom_type); +DROP FUNCTION abs_val(my_custom_type); +DROP TYPE my_custom_type CASCADE; diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e63ee2cf2bb..f9def3d93aa 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -68,6 +68,11 @@ test: select_into select_distinct select_distinct_on select_implicit select_havi # ---------- test: brin gin gist spgist privileges init_privs security_label collate matview lock replica_identity rowsecurity object_address tablesample groupingsets drop_operator password identity generated_stored join_hash +# ---------- +# Another group of parallel tests +# ---------- +test: heap_hot_updates + # ---------- # Additional BRIN tests # ---------- diff --git a/src/test/regress/sql/heap_hot_updates.sql b/src/test/regress/sql/heap_hot_updates.sql new file mode 100644 index 00000000000..7728075a7ae --- /dev/null +++ b/src/test/regress/sql/heap_hot_updates.sql @@ -0,0 +1,440 @@ +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table will have two columns and two indexes, one on the primary key +-- id and one on the expression (info->>'name'). That means that the indexed +-- attributes are 'id' and 'info'. +create table keyvalue(id integer primary key, info jsonb); +create index nameindex on keyvalue((info->>'name')); +insert into keyvalue values (1, '{"name": "john", "data": "some data"}'); + +-- Disable expression checks. +ALTER TABLE keyvalue SET (expression_checks = false); +SELECT reloptions FROM pg_class WHERE relname = 'keyvalue'; + +-- While the indexed attribute "name" is unchanged we've disabled expression +-- checks so this update should not go HOT as the system can't determine if +-- the indexed attribute has changed without evaluating the expression. +update keyvalue set info='{"name": "john", "data": "something else"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 0 row + +-- Re-enable expression checks. +ALTER TABLE keyvalue SET (expression_checks = true); +SELECT reloptions FROM pg_class WHERE relname = 'keyvalue'; + +-- The indexed attribute "name" with value "john" is unchanged, expect a HOT update. +update keyvalue set info='{"name": "john", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 1 row + +-- The following update changes the indexed attribute "name", this should not be a HOT update. +update keyvalue set info='{"name": "smith", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 1 no new HOT updates + +-- Now, this update does not change the indexed attribute "name" from "smith", this should be HOT. +update keyvalue set info='{"name": "smith", "data": "some more data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 2 rows now +drop table keyvalue; + + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table is the same as the previous one but it has a third index. The +-- index 'colindex' isn't an expression index, it indexes the entire value +-- in the info column. There are still only two indexed attributes for this +-- relation, the same two as before. The presence of an index on the entire +-- value of the info column should prevent HOT updates for any updates to any +-- portion of JSONB content in that column. +create table keyvalue(id integer primary key, info jsonb); +create index nameindex on keyvalue((info->>'name')); +create index colindex on keyvalue(info); +insert into keyvalue values (1, '{"name": "john", "data": "some data"}'); + +-- This update doesn't change the value of the expression index, but it does +-- change the content of the info column and so should not be HOT because the +-- indexed value changed as a result of the update. +update keyvalue set info='{"name": "john", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 0 rows +drop table keyvalue; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- The table has one column docs and two indexes. They are both expression +-- indexes referencing the same column attribute (docs) but one is a partial +-- index. +CREATE TABLE ex (docs JSONB) WITH (fillfactor = 60); +INSERT INTO ex (docs) VALUES ('{"a": 0, "b": 0}'); +INSERT INTO ex (docs) SELECT jsonb_build_object('b', n) FROM generate_series(100, 10000) as n; +CREATE INDEX idx_ex_a ON ex ((docs->>'a')); +CREATE INDEX idx_ex_b ON ex ((docs->>'b')) WHERE (docs->>'b')::numeric > 9; + +-- We're using BTREE indexes and for this test we want to make sure that they remain +-- in sync with changes to our relation. Force the choice of index scans below so +-- that we know we're checking the index's understanding of what values should be +-- in the index or not. +SET SESSION enable_seqscan = OFF; +SET SESSION enable_bitmapscan = OFF; + +-- Leave 'a' unchanged but modify 'b' to a value outside of the index predicate. +-- This should be a HOT update because neither index is changed. +UPDATE ex SET docs = jsonb_build_object('a', 0, 'b', 1) WHERE (docs->>'a')::numeric = 0; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 row +-- Let's check to make sure that the index does not contain a value for 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + +-- Leave 'a' unchanged but modify 'b' to a value within the index predicate. +-- This represents a change for field 'b' from unindexed to indexed and so +-- this should not take the HOT path. +UPDATE ex SET docs = jsonb_build_object('a', 0, 'b', 10) WHERE (docs->>'a')::numeric = 0; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates +-- Let's check to make sure that the index contains the new value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + +-- This update modifies the value of 'a', an indexed field, so it also cannot +-- be a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 1, 'b', 10) WHERE (docs->>'b')::numeric = 10; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + +-- This update changes both 'a' and 'b' to new values that require index updates, +-- this cannot use the HOT path. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 12) WHERE (docs->>'b')::numeric = 10; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates +-- Let's check to make sure that the index contains the new value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + +-- This update changes 'b' to a value outside its predicate requiring that +-- we remove it from the index. That's a transition that can't be done +-- during a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 1) WHERE (docs->>'b')::numeric = 12; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates +-- Let's check to make sure that the index no longer contains the value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + +-- Disable expression checks. +ALTER TABLE ex SET (expression_checks = false); +SELECT reloptions FROM pg_class WHERE relname = 'ex'; + +-- This update changes 'b' to a value within its predicate just like it +-- previous value, which would allow for a HOT update but with expression +-- checks disabled we can't determine that so this should not be a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 2) WHERE (docs->>'b')::numeric = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + + +-- Let's make sure we're recording HOT updates for our 'ex' relation properly in the system +-- table pg_stat_user_tables. Note that statistics are stored within a transaction context +-- first (xact) and then later into the global statistics for a relation, so first we need +-- to ensure pending stats are flushed. +SELECT pg_stat_force_next_flush(); +SELECT + c.relname AS table_name, + -- Transaction statistics + pg_stat_get_xact_tuples_updated(c.oid) AS xact_updates, + pg_stat_get_xact_tuples_hot_updated(c.oid) AS xact_hot_updates, + ROUND(( + pg_stat_get_xact_tuples_hot_updated(c.oid)::float / + NULLIF(pg_stat_get_xact_tuples_updated(c.oid), 0) * 100 + )::numeric, 2) AS xact_hot_update_percentage, + -- Cumulative statistics + s.n_tup_upd AS total_updates, + s.n_tup_hot_upd AS hot_updates, + ROUND(( + s.n_tup_hot_upd::float / + NULLIF(s.n_tup_upd, 0) * 100 + )::numeric, 2) AS total_hot_update_percentage +FROM pg_class c +LEFT JOIN pg_stat_user_tables s ON c.relname = s.relname +WHERE c.relname = 'ex' +AND c.relnamespace = 'public'::regnamespace; +-- expect: 5 xact updates with 1 xact hot update and no cumulative updates as yet + +SET SESSION enable_seqscan = ON; +SET SESSION enable_bitmapscan = ON; + +DROP TABLE ex; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table has a single column 'email' and a unique constraint on it that +-- should preclude HOT updates. +CREATE TABLE users ( + user_id serial primary key, + name VARCHAR(255) NOT NULL, + email VARCHAR(255) NOT NULL, + EXCLUDE USING btree (lower(email) WITH =) +); + +-- Add some data to the table and then update it in ways that should and should +-- not be HOT updates. +INSERT INTO users (name, email) VALUES +('user1', '[email protected]'), +('user2', '[email protected]'), +('taken', '[email protected]'), +('you', '[email protected]'), +('taken', '[email protected]'); + +-- Should fail because of the unique constraint on the email column. +UPDATE users SET email = '[email protected]' WHERE email = '[email protected]'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 0 no new HOT updates + +-- Should succeed because the email column is not being updated and should go HOT. +UPDATE users SET name = 'foo' WHERE email = '[email protected]'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 a single new HOT update + +-- Create a partial index on the email column, updates +CREATE INDEX idx_users_email_no_example ON users (lower(email)) WHERE lower(email) LIKE '%@example.com%'; + +-- An update that changes the email column but not the indexed portion of it and falls outside the constraint. +-- Shouldn't be a HOT update because of the exclusion constraint. +UPDATE users SET email = '[email protected]' WHERE name = 'you'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 no new HOT updates + +-- An update that changes the email column but not the indexed portion of it and falls within the constraint. +-- Again, should fail constraint and fail to be a HOT update. +UPDATE users SET email = '[email protected]' WHERE name = 'you'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 no new HOT updates + +DROP TABLE users; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- Another test of constraints spoiling HOT updates, this time with a range. +CREATE TABLE events ( + id serial primary key, + name VARCHAR(255) NOT NULL, + event_time tstzrange, + constraint no_screening_time_overlap exclude using gist ( + event_time WITH && + ) +); + +-- Add two non-overlapping events. +INSERT INTO events (id, event_time, name) +VALUES + (1, '["2023-01-01 19:00:00", "2023-01-01 20:45:00"]', 'event1'), + (2, '["2023-01-01 21:00:00", "2023-01-01 21:45:00"]', 'event2'); + +-- Update the first event to overlap with the second, should fail the constraint and not be HOT. +UPDATE events SET event_time = '["2023-01-01 20:00:00", "2023-01-01 21:45:00"]' WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 0 no new HOT updates + +-- Update the first event to not overlap with the second, again not HOT due to the constraint. +UPDATE events SET event_time = '["2023-01-01 22:00:00", "2023-01-01 22:45:00"]' WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 0 no new HOT updates + +-- Update the first event to not overlap with the second, this time we're HOT because we don't overlap with the constraint. +UPDATE events SET name = 'new name here' WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 1 one new HOT update + +DROP TABLE events; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- A test to ensure that summarizing indexes are not updated when they don't +-- change, but are updated when they do while not prefent HOT updates. +CREATE TABLE ex (att1 JSONB, att2 text) WITH (fillfactor = 60); +CREATE INDEX expression ON ex USING btree((att1->'data')); +CREATE INDEX summarizing ON ex USING BRIN(att2); +INSERT INTO ex VALUES ('{"data": []}', 'nothing special'); + +-- Update the unindexed value of att1, this should be a HOT update and not +-- update the summarizing index. +UPDATE ex SET att1 = att1 || '{"status": "checkmate"}'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 one new HOT update + +-- Update the indexed value of att2, a summarized value, this is a summarized +-- only update and should use the HOT path while still triggering an update to +-- the summarizing BRIN index. +UPDATE ex SET att2 = 'special indeed'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 2 one new HOT update + +-- Update to att1 doesn't change the indexed value while the update to att2 does, +-- this again is a summarized only update and should use the HOT path as well as +-- trigger an update to the BRIN index. +UPDATE ex SET att1 = att1 || '{"status": "checkmate!"}', att2 = 'special, so special'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 3 one new HOT update + +-- This updates both indexes, the expression index on att1 and the summarizing +-- index on att2. This should not be a HOT update because there are modified +-- indexes and only some are summarized, not all. This should force all +-- indexes to be updated. +UPDATE ex SET att1 = att1 || '{"data": [1,2,3]}', att2 = 'do you want to play a game?'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 3 both indexes updated + +DROP TABLE ex; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This test is for a table with a custom type and a custom operators on +-- the BTREE index. The question is, when comparing values for equality +-- to determine if there are changes on the index or not... shouldn't we +-- be using the custom operators? + +-- Create a type +CREATE TYPE my_custom_type AS (val int); + +-- Comparison functions (returns boolean) +CREATE FUNCTION my_custom_lt(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val < b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE FUNCTION my_custom_le(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val <= b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE FUNCTION my_custom_eq(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val = b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE FUNCTION my_custom_ge(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val >= b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE FUNCTION my_custom_gt(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val > b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE FUNCTION my_custom_ne(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val != b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +-- Comparison function (returns -1, 0, 1) +CREATE FUNCTION my_custom_cmp(a my_custom_type, b my_custom_type) RETURNS int AS $$ +BEGIN + IF a.val < b.val THEN + RETURN -1; + ELSIF a.val > b.val THEN + RETURN 1; + ELSE + RETURN 0; + END IF; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +-- Create the operators +CREATE OPERATOR < ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_lt, + COMMUTATOR = >, + NEGATOR = >= +); + +CREATE OPERATOR <= ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_le, + COMMUTATOR = >=, + NEGATOR = > +); + +CREATE OPERATOR = ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_eq, + COMMUTATOR = =, + NEGATOR = <> +); + +CREATE OPERATOR >= ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_ge, + COMMUTATOR = <=, + NEGATOR = < +); + +CREATE OPERATOR > ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_gt, + COMMUTATOR = <, + NEGATOR = <= +); + +CREATE OPERATOR <> ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_ne, + COMMUTATOR = <>, + NEGATOR = = +); + +-- Create the operator class (including the support function) +CREATE OPERATOR CLASS my_custom_ops + DEFAULT FOR TYPE my_custom_type USING btree AS + OPERATOR 1 <, + OPERATOR 2 <=, + OPERATOR 3 =, + OPERATOR 4 >=, + OPERATOR 5 >, + FUNCTION 1 my_custom_cmp(my_custom_type, my_custom_type); + +-- Create the table +CREATE TABLE my_table ( + id int, + custom_val my_custom_type +); + +-- Insert some data +INSERT INTO my_table (id, custom_val) VALUES +(1, ROW(3)::my_custom_type), +(2, ROW(1)::my_custom_type), +(3, ROW(4)::my_custom_type), +(4, ROW(2)::my_custom_type); + +-- Create a function to use when indexing +CREATE OR REPLACE FUNCTION abs_val(val my_custom_type) RETURNS int AS $$ +BEGIN + RETURN abs(val.val); +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +-- Create the index +CREATE INDEX idx_custom_val_abs ON my_table (abs_val(custom_val)); + +-- Update 1 +UPDATE my_table SET custom_val = ROW(5)::my_custom_type WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + +-- Update 2 +UPDATE my_table SET custom_val = ROW(0)::my_custom_type WHERE custom_val < ROW(3)::my_custom_type; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + +-- Update 3 +UPDATE my_table SET custom_val = ROW(6)::my_custom_type WHERE id = 3; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + +-- Update 4 +UPDATE my_table SET id = 5 WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + +-- Query using the index +EXPLAIN (COSTS OFF) SELECT * FROM my_table WHERE abs_val(custom_val) = 6; +SELECT * FROM my_table WHERE abs_val(custom_val) = 6; + +-- Clean up +DROP TABLE my_table CASCADE; +DROP OPERATOR CLASS my_custom_ops USING btree CASCADE; +DROP OPERATOR < (my_custom_type, my_custom_type); +DROP OPERATOR <= (my_custom_type, my_custom_type); +DROP OPERATOR = (my_custom_type, my_custom_type); +DROP OPERATOR >= (my_custom_type, my_custom_type); +DROP OPERATOR > (my_custom_type, my_custom_type); +DROP OPERATOR <> (my_custom_type, my_custom_type); +DROP FUNCTION my_custom_lt(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_le(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_eq(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_ge(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_gt(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_ne(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_cmp(my_custom_type, my_custom_type); +DROP FUNCTION abs_val(my_custom_type); +DROP TYPE my_custom_type CASCADE; -- 2.42.0 ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Expanding HOT updates for expression and partial indexes @ 2025-02-15 10:49 Matthias van de Meent <[email protected]> parent: Burd, Greg <[email protected]> 0 siblings, 1 reply; 6+ messages in thread From: Matthias van de Meent @ 2025-02-15 10:49 UTC (permalink / raw) To: Burd, Greg <[email protected]>; +Cc: pgsql-hackers On Thu, 13 Feb 2025 at 19:46, Burd, Greg <[email protected]> wrote: > > Attached find an updated patchset v5 that is an evolution of v4. > > Changes v4 to v5 are: > * replaced GUC with table reloption called "expression_checks" (open to other name ideas) > * minimal documentation updates to README.HOT to address changes > * avoid, when possible, the expensive path that requires evaluating an estate using bitmaps > * determines the set of summarized indexes requiring updates, only updates those > * more tests in heap_hot_updates.sql (perhaps too many...) > * rebased to master, formatted, and make check-world passes Thank you for the update. Below some comments, in no particular order. ----- I'm not a fan of how you replaced TU_UpdateIndexes with a bitmap. It seems unergonomic and a waste of performance. In HEAD, we don't do any expensive computations for the fast path of "indexed attribute updated" - at most we do the bitmap compare and then set a pointer. With this patch, the way we signal that is by allocating a bitmap of size O(n_indexes). That's potentially quite expensive (given 1000s of indexes), and definitely more expensive than only a pointer assignment. In HEAD, we have a clear indication of which classes of indexes to update, with TU_UpdateIndexes. With this patch, we have to derive that from the (lack of) bits in the bitmap that might be output by the table_update procedure. I think we can do with an additional parameter for which indexes would be updated (or store that info in the parameter which also will hold EState et al). I think it's cheaper that way, too - only when update_indexes could be TU_SUMMARIZING we might need the exact information for which indexes to insert new tuples into, and it only really needs to be sized to the number of summarizing indexes (usually small/nonexistent, but potentially huge). ----- I think your patch design came from trying to include at least two distinct optimizations: 1) to make the HOT-or-not check include whether the expressions of indexes were updated, and 2) to only insert index tuples into indexes that got updated values when table_tuple_update returns update_indexes=TU_Summarizing. While they touch similar code (clearly seen here), I think those should be implemented in different patches. For (1), the current API surface is good enough when the EState is passed down. For (2), you'll indeed also need an additional argument we can use to fill with the right summarizing indexes, but I don't think that can nor should replace the function of TU_UpdateIndexes. If you agree with my observation of those being distinct optimizations, could you split this patch into parts (but still within the same series) so that these are separately reviewable? ----- I notice that ExecIndexesRequiringUpdates() does work on all indexes, rather than just indexes relevant to this exact phase of checking. I think that is a waste of time, so if we sort the indexes in order of [hotblocking without expressions, hotblocking with expressions, summarizing], then (with stored start/end indexes) we can save time in cases where there are comparatively few of the types we're not going to look at. As an extreme example: we shouldn't do the (comparatively) expensive work evaluating expressions to determine which of 1000s of summarizing indexes has been updated when we're still not sure if we can apply HOT at all. (Sidenote: Though, arguably, we could be smarter by skipping index insertions into unmodified summarizing indexes altogether regardless of HOT status, as long as the update is on the same page - but that's getting ahead of ourselves and not relevant to this discussion.) ----- I noticed you've disabled any passing of "HOT or not" in the simple_update cases, and have done away with the various checks that are in place to prevent corruption. I don't think that's a great idea, it's quite likely to cause bugs. ----- You're extracting type info from the opclass, to use in datum_image_eq(). Couldn't you instead use the index relation's TupleDesc and its stored attribute information instead? That saves us from having to do further catalog lookups during execution. I'm also fairly sure that that information is supposed to be a more accurate representation of attributes' expression output types than the opclass' type information (though, they probably should match). ----- The operations applied in ExecIndexesRequiringUpdates partially duplicate those done in index_unchanged_by_update. Can we (partially) unify this, and pass which indexes were updated through the IndexInfo, rather than the current bitmap? ----- I don't see a good reason to add IndexInfo to Relation, by way of rd_indexInfoList. It seems like an ad-hoc way of passing data around, and I don't think that's the right way. >>>> I mean, it's clear that "updated indexed column's value == non-HOT >>>> update". And that to determine whether an updated *projected* column's >>>> value (i.e., expression index column's value) was actually updated we >>>> need to calculate the previous and current index value, thus execute >>>> the projection twice. But why would we have significant additional >>>> overhead if there are no expression indexes, or when we can know by >>>> bitmap overlap that the only interesting cases are summarizing >>>> indexes? >> >> See the attached approach. Evaluation of the expressions only has to >> happen if there are any HOT-blocking attributes which are exclusively >> hot-blockingly indexed through expressions, so if the updated >> attribute numbers are a subset of hotblockingexprattrs. (substitute >> hotblocking with summarizing for the summarizing approach) > > I believe I've incorporated the gist of your idea in this v5 patch, let me know if I missed something. Seems about accurate. >>>> I would've implemented this with (1) two new bitmaps, one each for >>>> normal and summarizing indexes, each containing which columns are >>>> exclusively used in expression indexes (and which should thus be used >>>> to trigger the (comparatively) expensive recalculation). >>> >>> That was one where I started, over time that became harder to work as the bitmaps contain the union of index attributes for the table not per-column. >> >> I think it's fairly easy to create, though. >> >>> Now there is one bitmap to cover the broadest case and then a function to find the modified set of indexes where each is examined against bitmaps that contain only attributes specific to the index in question. This helped in cases where there were both expression and non-expression indexes on the same attribute. >> >> Fair, but do we care about one expression index on (attr1->>'data')'s >> value *not* changing when an index on (attr1) exists and attr1 has >> changed? That index on att1 would block HOT updates regardless of the >> (lack of) changes to the (att1->>'data') index, so doing those >> expensive calculations seems quite wasteful. > > Agreed, when both a non-expression and an expression index exist on the same attribute then the expression checks are unnecessary and should be avoided. In this v5 patchset this case becomes two checks of bitmaps (first hot_attrs, then exclusively exp_attrs) before proceeding with a non-HOT update. > > So, in my opinion, we should also keep track of those attributes only > > included in expressions of indexes, and that's fairly easy: see > > attached prototype.diff.txt (might need some work, the patch was > > drafted on v16's codebase, but the idea is clear). > > Thank you for your patch, I've included and expanded it. ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Expanding HOT updates for expression and partial indexes @ 2025-02-17 19:53 Burd, Greg <[email protected]> parent: Matthias van de Meent <[email protected]> 0 siblings, 2 replies; 6+ messages in thread From: Burd, Greg @ 2025-02-17 19:53 UTC (permalink / raw) To: Matthias van de Meent <[email protected]>; +Cc: pgsql-hackers Matthias, First off, I can't thank you enough for taking the time to review in detail the patch. I appreciate and value your time and excellent feedback. Second, I think that I should admit to the fact that I've also been working on making PHOT functional again. I have it rebased against master, however it is still at the proof-of-concept phase and there are some larger issues to flesh out. One of the changes in this patch would have enabled that future with PHOT, specifically the bitmap as the method for conveying what indexes need updating. That said, I agree that this patch should simply focus on expanding HOT to expression indexes and partial indexes. With that in mind I will return to the TU_UpdateIndexes approach, even though I'm not a fan of it, and later propose the bitmap approach (or something better) as part of PHOT if and when that's ready. Changes v5 to v6: * reverted to TU_UpdateIndexes rather than Bitmapset * renamed ExecIndexesRequiringUpdates to ExecIndexesExpressionsWereNotUpdated * ExecIndexesExpressionsWereNotUpdated returns bool, exits early if possible * simple_update paths can be HOT once again * removed efforts to determine the subset of updated summarizing indexes * create filtered IndexInfo list in relcache containing only indexes with expressions * now using index TupleDesc CompactAttributes for arguments to datum_is_equal() <- did I get this one right, I'm not sure it is what you had in mind best. -greg > On Feb 15, 2025, at 5:49 AM, Matthias van de Meent <[email protected]> wrote: > > On Thu, 13 Feb 2025 at 19:46, Burd, Greg <[email protected]> wrote: > > ----- > > I'm not a fan of how you replaced TU_UpdateIndexes with a bitmap. It > seems unergonomic and a waste of performance. > > In HEAD, we don't do any expensive computations for the fast path of > "indexed attribute updated" - at most we do the bitmap compare and > then set a pointer. With this patch, the way we signal that is by > allocating a bitmap of size O(n_indexes). That's potentially quite > expensive (given 1000s of indexes), and definitely more expensive than > only a pointer assignment. Fair point. I'm not a fan of the TU_UpdateIndexes enum, but I appreciate your argument against the bitmapset. Under the conditions you describe (1000s of indexes) it could grow unwieldy and impact performance. > In HEAD, we have a clear indication of which classes of indexes to > update, with TU_UpdateIndexes. With this patch, we have to derive that > from the (lack of) bits in the bitmap that might be output by the > table_update procedure. Yes, but... that "clear indication" is lacking the ability to convey more detailed information. It doesn't tell you which summarizing indexes really need updating just that as a result of being on the HOT path all summarizing indexes require updates. > I think we can do with an additional parameter for which indexes would > be updated (or store that info in the parameter which also will hold > EState et al). I think it's cheaper that way, too - only when > update_indexes could be TU_SUMMARIZING we might need the exact > information for which indexes to insert new tuples into, and it only > really needs to be sized to the number of summarizing indexes (usually > small/nonexistent, but potentially huge). Okay, yes with this patch we need only concern ourselves with all, none, or some subset of summarizing as before. I'll work on the opaque parameter next iteration. > ----- > > I think your patch design came from trying to include at least two > distinct optimizations: > 1) to make the HOT-or-not check include whether the expressions of > indexes were updated, and > 2) to only insert index tuples into indexes that got updated values > when table_tuple_update returns update_indexes=TU_Summarizing. It did. (1) for sure, but the second is more related to the PHOT work (as mentioned above). With that work almost all updates are on the HOT/PHOT path and so the bitmap of changed indexes is small. It would take an update of a table with 1000s of indexes where almost all those were modified to create a bitmap that was large, which certainly could (and somewhere likely does) exist, but it's not the common case (I'd imagine). In that case the work to update all those indexes will likely dwarf the work to build that bitmap, but I could be wrong. But you're fundamentally right, I'm conflating two ideas and I shouldn't. I will focus the changes required for this idea without pulling in changes useful to a future ones. > While they touch similar code (clearly seen here), I think those > should be implemented in different patches. For (1), the current API > surface is good enough when the EState is passed down. For (2), you'll > indeed also need an additional argument we can use to fill with the > right summarizing indexes, but I don't think that can nor should > replace the function of TU_UpdateIndexes. Okay, I'll combine the earlier v3 patch with the changes in v5. That should leave TU_UpdateIndexes in place and allow for HOT with expression indexes. I'll change the ExecIndexesRequiringUpdates() a bit (and rename it) so that it is just a boolean test for any index that spoils the HOT path and can exit early in that case potentially avoiding extra work. > If you agree with my observation of those being distinct > optimizations, could you split this patch into parts (but still within > the same series) so that these are separately reviewable? I agree, but I think that a single simple more focused patch will suffice. > ----- > > I notice that ExecIndexesRequiringUpdates() does work on all indexes, > rather than just indexes relevant to this exact phase of checking. I > think that is a waste of time, so if we sort the indexes in order of > [hotblocking without expressions, hotblocking with expressions, > summarizing], then (with stored start/end indexes) we can save time in > cases where there are comparatively few of the types we're not going > to look at. If I plan on just having ExecIndexesRequiringUpdates() return a bool rather than a bitmap then sorting, or even just filtering the list of IndexInfo to only include indexes with expressions, makes sense. That way the only indexes in question in that function's loop will be those that may spoil the HOT path. When that list is length 0, we can skip the tests entirely. > As an extreme example: we shouldn't do the (comparatively) expensive > work evaluating expressions to determine which of 1000s of summarizing > indexes has been updated when we're still not sure if we can apply HOT > at all. That makes sense, and your sorting idea would inform that kind of work. I'll keep that in mind if I reintroduce code that aims to only update changed summarized indexes. > (Sidenote: Though, arguably, we could be smarter by skipping index > insertions into unmodified summarizing indexes altogether regardless > of HOT status, as long as the update is on the same page - but that's > getting ahead of ourselves and not relevant to this discussion.) > > ----- > > I noticed you've disabled any passing of "HOT or not" in the > simple_update cases, and have done away with the various checks that > are in place to prevent corruption. I don't think that's a great idea, > it's quite likely to cause bugs. Yes. I'll resurrect that. > ----- > > You're extracting type info from the opclass, to use in > datum_image_eq(). Couldn't you instead use the index relation's > TupleDesc and its stored attribute information instead? That saves us > from having to do further catalog lookups during execution. I'm also > fairly sure that that information is supposed to be a more accurate > representation of attributes' expression output types than the > opclass' type information (though, they probably should match). I hadn't thought of that, I think it's a valid idea and I'll update accordingly. I think I understand what you are suggesting. > > ----- > > The operations applied in ExecIndexesRequiringUpdates partially > duplicate those done in index_unchanged_by_update. Can we (partially) > unify this, and pass which indexes were updated through the IndexInfo, > rather than the current bitmap? I think I do that now, feel free to say otherwise. When the expression is checked in ExecIndexesExpressionsWereNotUpdated() I set: /* Shortcut index_unchanged_by_update(), we know the answer. */ indexInfo->ii_CheckedUnchanged = true; indexInfo->ii_IndexUnchanged = !changed; That prevents duplicate effort in index_unchanged_by_update(). > ----- > > I don't see a good reason to add IndexInfo to Relation, by way of > rd_indexInfoList. It seems like an ad-hoc way of passing data around, > and I don't think that's the right way. At one point I'd created a way to get this set via relcache, I will resurrect that approach but I'm not sure it is what you were hinting at. The current method avoids pulling a the lock on the index to build the list, but doing that once in relcache isn't horrible. Maybe you were suggesting using that opaque struct to pass around the list of IndexInfo? Let me know on this one if you had a specific idea. The swap I've made in v6 really just moves the IndexInfo list to a filtered list with a new name created in relcache. Attachments: [application/octet-stream] v6-0001-Expand-HOT-update-path-to-include-expression-and-.patch (86.8K, 2-v6-0001-Expand-HOT-update-path-to-include-expression-and-.patch) download | inline diff: From 47aa9640223d2b95dbb074ffc780fa109053191b Mon Sep 17 00:00:00 2001 From: Gregory Burd <[email protected]> Date: Mon, 27 Jan 2025 13:28:59 -0500 Subject: [PATCH v6] Expand HOT update path to include expression and partial indexes. This patch extends the cases where HOT updates are possible in the heapam by examining expression indexes and determining if indexed values where mutated or not. Previously, any expression index on a column would disqualify it from the HOT update path. Also examines partial indexes to see if the values are within the predicate or not. This is a modified application of a patch proposed on the pgsql-hackers list: https://www.postgresql.org/message-id/flat/4d9928ee-a9e6-15f9-9c82-5981f13ffca6%40postgrespro.ru applied: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=c203d6cf8 reverted: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=05f84605dbeb9cf8279a157234b24bbb706c5256 Signed-off-by: Greg Burd <[email protected]> --- doc/src/sgml/ref/create_table.sgml | 17 + doc/src/sgml/storage.sgml | 21 + src/backend/access/common/reloptions.c | 12 +- src/backend/access/heap/README.HOT | 43 +- src/backend/access/heap/heapam.c | 43 +- src/backend/access/heap/heapam_handler.c | 6 +- src/backend/access/table/tableam.c | 2 +- src/backend/catalog/index.c | 15 + src/backend/executor/execIndexing.c | 200 ++++++ src/backend/executor/nodeModifyTable.c | 3 +- src/backend/utils/cache/relcache.c | 181 +++++- src/bin/psql/tab-complete.in.c | 2 +- src/include/access/genam.h | 2 +- src/include/access/heapam.h | 3 +- src/include/access/tableam.h | 10 +- src/include/executor/executor.h | 5 + src/include/nodes/execnodes.h | 8 + src/include/utils/rel.h | 14 + src/include/utils/relcache.h | 2 + .../regress/expected/heap_hot_updates.out | 586 ++++++++++++++++++ src/test/regress/parallel_schedule | 5 + src/test/regress/sql/heap_hot_updates.sql | 440 +++++++++++++ 22 files changed, 1560 insertions(+), 60 deletions(-) create mode 100644 src/test/regress/expected/heap_hot_updates.out create mode 100644 src/test/regress/sql/heap_hot_updates.sql diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 0a3e520f215..e29c9ea45a7 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -1981,6 +1981,23 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry id="reloption-expression-checks" xreflabel="expression_checks"> + <term><literal>expression_checks</literal> (<type>boolean</type>) + <indexterm> + <primary><varname>expression_checks</varname> storage parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Enables or disables evaulation of predicate expressions on partial + indexes or expressions used to define indexes during updates. + If <literal>true</literal>, then these expressions are evaluated during + updates to data within the heap relation against the old and new values + and then compared to determine if <acronym>HOT</acronym> updates are + allowable or not. The default value is <literal>true</literal>. + </para> + </listitem> + </variablelist> </refsect2> diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml index 61250799ec0..f40ada9c989 100644 --- a/doc/src/sgml/storage.sgml +++ b/doc/src/sgml/storage.sgml @@ -1138,6 +1138,27 @@ data. Empty in ordinary tables.</entry> </itemizedlist> </para> + <para> + <acronym>HOT</acronym> updates can occur when the expression used to define + and index shows no changes to the indexed value. To determine this requires + that the expression be evaulated for the old and new values to be stored in + the index and then compared. This allows for <acronym>HOT</acronym> updates + when data indexed within JSONB columns is unchanged. To disable this + behavior and avoid the overhead of evaluating the expression during updates + set the <literal>expression_checks</literal> option to false for the table. + </para> + + <para> + <acronym>HOT</acronym> updates can also occur when updated values are not + within the predicate of a partial index. However, <acronym>HOT</acronym> + updates are not possible when the updated value and the current value differ + with regards to the predicate. To determin this requires that the predicate + expression be evaluated for the old and new values to be stored in the index + and then compared. To disable this behavior and avoid the overhead of + evaluating the expression during updates set + the <literal>expression_checks</literal> option to false for the table. + </para> + <para> You can increase the likelihood of sufficient page space for <acronym>HOT</acronym> updates by decreasing a table's <link diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c index 59fb53e7707..c081611926e 100644 --- a/src/backend/access/common/reloptions.c +++ b/src/backend/access/common/reloptions.c @@ -166,6 +166,15 @@ static relopt_bool boolRelOpts[] = }, true }, + { + { + "expression_checks", + "When disabled prevents checking expressions on indexes and predicates on partial indexes for changes that might influence heap-only tuple (HOT) updates.", + RELOPT_KIND_HEAP, + ShareUpdateExclusiveLock + }, + true + }, /* list terminator */ {{NULL}} }; @@ -1903,7 +1912,8 @@ default_reloptions(Datum reloptions, bool validate, relopt_kind kind) {"vacuum_truncate", RELOPT_TYPE_BOOL, offsetof(StdRdOptions, vacuum_truncate)}, {"vacuum_max_eager_freeze_failure_rate", RELOPT_TYPE_REAL, - offsetof(StdRdOptions, vacuum_max_eager_freeze_failure_rate)} + offsetof(StdRdOptions, vacuum_max_eager_freeze_failure_rate)}, + {"expression_checks", RELOPT_TYPE_BOOL, offsetof(StdRdOptions, expression_checks)} }; return (bytea *) build_reloptions(reloptions, validate, kind, diff --git a/src/backend/access/heap/README.HOT b/src/backend/access/heap/README.HOT index 74e407f375a..48aab81fdf0 100644 --- a/src/backend/access/heap/README.HOT +++ b/src/backend/access/heap/README.HOT @@ -36,7 +36,7 @@ HOT solves this problem for two restricted but useful special cases: First, where a tuple is repeatedly updated in ways that do not change its indexed columns. (Here, "indexed column" means any column referenced at all in an index definition, including for example columns that are -tested in a partial-index predicate but are not stored in the index.) +tested in a partial-index predicate, that has materially changed.) Second, where the modified columns are only used in indexes that do not contain tuple IDs, but maintain summaries of the indexed data by block. @@ -133,18 +133,26 @@ Note: we can use a "dead" line pointer for any DELETEd tuple, whether it was part of a HOT chain or not. This allows space reclamation in advance of running VACUUM for plain DELETEs as well as HOT updates. -The requirement for doing a HOT update is that indexes which point to -the root line pointer (and thus need to be cleaned up by VACUUM when the -tuple is dead) do not reference columns which are updated in that HOT -chain. Summarizing indexes (such as BRIN) are assumed to have no -references to individual tuples and thus are ignored when checking HOT -applicability. The updated columns are checked at execution time by -comparing the binary representation of the old and new values. We insist -on bitwise equality rather than using datatype-specific equality routines. -The main reason to avoid the latter is that there might be multiple -notions of equality for a datatype, and we don't know exactly which one -is relevant for the indexes at hand. We assume that bitwise equality -guarantees equality for all purposes. +The requirement for doing a HOT update is that indexes which point to the root +line pointer (and thus need to be cleaned up by VACUUM when the tuple is dead) +do not reference columns which are updated in that HOT chain. + +Summarizing indexes (such as BRIN) are assumed to have no references to +individual tuples and thus are ignored when checking HOT applicability. + +Expressions on indexes are evaluated and the results are used when check for +changes. This allows for the JSONB datatype to have HOT updates when the +indexed portion of the document are not modified. + +Partial index expressions are evaluated, HOT updates are allowed when the +updated index values do not satisfy the predicate. + +The updated columns are checked at execution time by comparing the binary +representation of the old and new values. We insist on bitwise equality rather +than using datatype-specific equality routines. The main reason to avoid the +latter is that there might be multiple notions of equality for a datatype, and +we don't know exactly which one is relevant for the indexes at hand. We assume +that bitwise equality guarantees equality for all purposes. If any columns that are included by non-summarizing indexes are updated, the HOT optimization is not applied, and the new tuple is inserted into @@ -152,9 +160,7 @@ all indexes of the table. If none of the updated columns are included in the table's indexes, the HOT optimization is applied and no indexes are updated. If instead the updated columns are only indexed by summarizing indexes, the HOT optimization is applied, but the update is propagated to -all summarizing indexes. (Realistically, we only need to propagate the -update to the indexes that contain the updated values, but that is yet to -be implemented.) +the summarizing indexes that have updated values. Abort Cases ----------- @@ -477,8 +483,9 @@ Heap-only tuple HOT-safe A proposed tuple update is said to be HOT-safe if it changes - none of the tuple's indexed columns. It will only become an - actual HOT update if we can find room on the same page for + none of the tuple's indexed columns or if the changes remain + outside of a partial index's predicate. It will only become + an actual HOT update if we can find room on the same page for the new tuple version. HOT update diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index fa7935a0ed3..46ce824c4ea 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -3164,12 +3164,13 @@ TM_Result heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, CommandId cid, Snapshot crosscheck, bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode, - TU_UpdateIndexes *update_indexes) + TU_UpdateIndexes *update_indexes, struct EState *estate) { TM_Result result; TransactionId xid = GetCurrentTransactionId(); Bitmapset *hot_attrs; Bitmapset *sum_attrs; + Bitmapset *exp_attrs; Bitmapset *key_attrs; Bitmapset *id_attrs; Bitmapset *interesting_attrs; @@ -3246,6 +3247,8 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, INDEX_ATTR_BITMAP_HOT_BLOCKING); sum_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_SUMMARIZED); + exp_attrs = RelationGetIndexAttrBitmap(relation, + INDEX_ATTR_BITMAP_EXPRESSION); key_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_KEY); id_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_IDENTITY_KEY); @@ -3311,6 +3314,7 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, bms_free(hot_attrs); bms_free(sum_attrs); + bms_free(exp_attrs); bms_free(key_attrs); bms_free(id_attrs); /* modified_attrs not yet initialized */ @@ -3612,6 +3616,7 @@ l2: bms_free(hot_attrs); bms_free(sum_attrs); + bms_free(exp_attrs); bms_free(key_attrs); bms_free(id_attrs); bms_free(modified_attrs); @@ -3928,25 +3933,30 @@ l2: if (newbuf == buffer) { + bool expression_checks = RelationGetExpressionChecks(relation); + /* * Since the new tuple is going into the same page, we might be able - * to do a HOT update. Check if any of the index columns have been - * changed. + * to do a HOT update. As a reminder, hot_attrs includes attributes + * used in expressions, but not attributes used by summarizing + * indexes. */ - if (!bms_overlap(modified_attrs, hot_attrs)) - { + if (!bms_overlap(modified_attrs, hot_attrs) || + (expression_checks == true && estate != NULL && exp_attrs && + bms_overlap(modified_attrs, exp_attrs) && + ExecIndexesExpressionsWereNotUpdated(relation, modified_attrs, + estate, &oldtup, newtup))) use_hot_update = true; - /* - * If none of the columns that are used in hot-blocking indexes - * were updated, we can apply HOT, but we do still need to check - * if we need to update the summarizing indexes, and update those - * indexes if the columns were updated, or we may fail to detect - * e.g. value bound changes in BRIN minmax indexes. - */ - if (bms_overlap(modified_attrs, sum_attrs)) - summarized_update = true; - } + /* + * If none of the columns that are used in hot-blocking indexes were + * updated, we can apply HOT, but we do still need to check if we need + * to update the summarizing indexes, and update those indexes if the + * columns were updated, or we may fail to detect e.g. value bound + * changes in BRIN minmax indexes. + */ + if (use_hot_update && bms_overlap(modified_attrs, sum_attrs)) + summarized_update = true; } else { @@ -4126,7 +4136,6 @@ l2: heap_freetuple(old_key_tuple); bms_free(hot_attrs); - bms_free(sum_attrs); bms_free(key_attrs); bms_free(id_attrs); bms_free(modified_attrs); @@ -4413,7 +4422,7 @@ simple_heap_update(Relation relation, ItemPointer otid, HeapTuple tup, result = heap_update(relation, otid, tup, GetCurrentCommandId(true), InvalidSnapshot, true /* wait for commit */ , - &tmfd, &lockmode, update_indexes); + &tmfd, &lockmode, update_indexes, NULL); switch (result) { case TM_SelfModified: diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index c0bec014154..8e9ab892d73 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -312,8 +312,8 @@ heapam_tuple_delete(Relation relation, ItemPointer tid, CommandId cid, static TM_Result heapam_tuple_update(Relation relation, ItemPointer otid, TupleTableSlot *slot, CommandId cid, Snapshot snapshot, Snapshot crosscheck, - bool wait, TM_FailureData *tmfd, - LockTupleMode *lockmode, TU_UpdateIndexes *update_indexes) + bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode, + TU_UpdateIndexes *update_indexes, EState *estate) { bool shouldFree = true; HeapTuple tuple = ExecFetchSlotHeapTuple(slot, true, &shouldFree); @@ -324,7 +324,7 @@ heapam_tuple_update(Relation relation, ItemPointer otid, TupleTableSlot *slot, tuple->t_tableOid = slot->tts_tableOid; result = heap_update(relation, otid, tuple, cid, crosscheck, wait, - tmfd, lockmode, update_indexes); + tmfd, lockmode, update_indexes, estate); ItemPointerCopy(&tuple->t_self, &slot->tts_tid); /* diff --git a/src/backend/access/table/tableam.c b/src/backend/access/table/tableam.c index e18a8f8250f..9feca8a296f 100644 --- a/src/backend/access/table/tableam.c +++ b/src/backend/access/table/tableam.c @@ -345,7 +345,7 @@ simple_table_tuple_update(Relation rel, ItemPointer otid, GetCurrentCommandId(true), snapshot, InvalidSnapshot, true /* wait for commit */ , - &tmfd, &lockmode, update_indexes); + &tmfd, &lockmode, update_indexes, NULL); switch (result) { diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index cdabf780244..c28857a0943 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -2455,7 +2455,22 @@ BuildIndexInfo(Relation index) /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) + { ii->ii_IndexAttrNumbers[i] = indexStruct->indkey.values[i]; + ii->ii_IndexAttrs = + bms_add_member(ii->ii_IndexAttrs, + indexStruct->indkey.values[i] - FirstLowInvalidHeapAttributeNumber); + + ii->ii_CompactAttr[i] = TupleDescCompactAttr(RelationGetDescr(index), i); + } + + /* collect attributes used in the expression, if one is present */ + if (ii->ii_Expressions) + pull_varattnos((Node *) ii->ii_Expressions, 1, &ii->ii_ExpressionAttrs); + + /* collect attributes used in the predicate, if one is present */ + if (ii->ii_Predicate) + pull_varattnos((Node *) ii->ii_Predicate, 1, &ii->ii_PredicateAttrs); /* fetch exclusion constraint info if any */ if (indexStruct->indisexclusion) diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c index 7c87f012c30..7dce38dd08a 100644 --- a/src/backend/executor/execIndexing.c +++ b/src/backend/executor/execIndexing.c @@ -117,6 +117,8 @@ #include "utils/multirangetypes.h" #include "utils/rangetypes.h" #include "utils/snapmgr.h" +#include "utils/datum.h" +#include "utils/lsyscache.h" /* waitMode argument to check_exclusion_or_unique_constraint() */ typedef enum @@ -1089,6 +1091,9 @@ index_unchanged_by_update(ResultRelInfo *resultRelInfo, EState *estate, if (hasexpression) { + if (indexInfo->ii_IndexUnchanged) + return true; + indexInfo->ii_IndexUnchanged = false; return false; } @@ -1166,3 +1171,198 @@ ExecWithoutOverlapsNotEmpty(Relation rel, NameData attname, Datum attval, char t errmsg("empty WITHOUT OVERLAPS value found in column \"%s\" in relation \"%s\"", NameStr(attname), RelationGetRelationName(rel)))); } + +/* + * Determine if any index with an expression requires updating. + * + * This function will review all indexes with expressions to determine if the + * update from old to new tuple requires that the index be updated. This is + * used to determine if a HOT update is possible or not. + * + * Returns true iff none of the expression indexes require updating due to the + * change from old to new tuple or when both the old and new tuples do not + * satisfy the predicate of the index. + */ +bool +ExecIndexesExpressionsWereNotUpdated(Relation relation, + Bitmapset *modified_attrs, + EState *estate, + HeapTuple old_tuple, + HeapTuple new_tuple) +{ + ListCell *lc; + List *indexinfolist = RelationGetExprIndexInfoList(relation); + ExprContext *econtext = NULL; + TupleDesc tupledesc; + TupleTableSlot *old_tts = NULL; + TupleTableSlot *new_tts = NULL; + bool expression_checks = RelationGetExpressionChecks(relation); + bool changed = false; + +#ifndef USE_ASSERT_CHECKING + + /* + * Assume the index is changed when we don't have an estate context to use + * or the reloption is disabled. + */ + if (!expression_checks || estate == NULL) + return changed; +#endif + + Assert(expression_checks == true); + Assert(estate != NULL); + + /* + * Examine expression and partial indexs on this relation relative to the + * changes between old and new tuples. + */ + foreach(lc, indexinfolist) + { + IndexInfo *indexInfo = (IndexInfo *) lfirst(lc); + + /* + * If this is a partial index it has a predicate, evaluate the + * expression to determine if we need to include it or not. + */ + if (bms_overlap(indexInfo->ii_PredicateAttrs, modified_attrs)) + { + ExprState *pstate; + bool old_tuple_qualifies, + new_tuple_qualifies; + + /* Create these once, only if necessary, then reuse them. */ + if (!econtext) + { + econtext = GetPerTupleExprContext(estate); + tupledesc = RelationGetDescr(relation); + old_tts = MakeSingleTupleTableSlot(tupledesc, + &TTSOpsHeapTuple); + new_tts = MakeSingleTupleTableSlot(tupledesc, + &TTSOpsHeapTuple); + + ExecStoreHeapTuple(old_tuple, old_tts, InvalidBuffer); + ExecStoreHeapTuple(new_tuple, new_tts, InvalidBuffer); + } + + pstate = ExecPrepareQual(indexInfo->ii_Predicate, estate); + + /* + * Here the term "qualifies" means "satisfies the predicate + * condition of the partial index". + */ + econtext->ecxt_scantuple = old_tts; + old_tuple_qualifies = ExecQual(pstate, econtext); + + econtext->ecxt_scantuple = new_tts; + new_tuple_qualifies = ExecQual(pstate, econtext); + + /* + * If neither the old nor the new tuples satisfy the predicate we + * can be sure that this index doesn't need updating, continue to + * the next index. + */ + if ((new_tuple_qualifies == false) && (old_tuple_qualifies == false)) + continue; + + /* + * If there is a transition between indexed and not indexed, + * that's enough to require that this index is updated. + */ + if (new_tuple_qualifies != old_tuple_qualifies) + { + changed = true; + break; + } + + /* + * Otherwise the old and new values exist in the index, but did + * they get updated? We don't yet know, so proceed with the next + * statement in the loop to find out. + */ + } + + + /* + * Indexes with expressions may or may not have changed, it is + * impossible to know without exercising their expression and + * reviewing index tuple state for changes. This is a lot of work, + * but because all indexes on JSONB columns fall into this category it + * can be worth it to avoid index updates and remain on the HOT update + * path when possible. + */ + if (bms_overlap(indexInfo->ii_ExpressionAttrs, modified_attrs)) + { + Datum old_values[INDEX_MAX_KEYS]; + bool old_isnull[INDEX_MAX_KEYS]; + Datum new_values[INDEX_MAX_KEYS]; + bool new_isnull[INDEX_MAX_KEYS]; + + /* Create these once, only if necessary, then reuse them. */ + if (!econtext) + { + econtext = GetPerTupleExprContext(estate); + tupledesc = RelationGetDescr(relation); + old_tts = MakeSingleTupleTableSlot(tupledesc, + &TTSOpsHeapTuple); + new_tts = MakeSingleTupleTableSlot(tupledesc, + &TTSOpsHeapTuple); + + ExecStoreHeapTuple(old_tuple, old_tts, InvalidBuffer); + ExecStoreHeapTuple(new_tuple, new_tts, InvalidBuffer); + } + + indexInfo->ii_ExpressionsState = NIL; + + econtext->ecxt_scantuple = old_tts; + FormIndexDatum(indexInfo, + old_tts, + estate, + old_values, + old_isnull); + + econtext->ecxt_scantuple = new_tts; + FormIndexDatum(indexInfo, + new_tts, + estate, + new_values, + new_isnull); + + for (int i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++) + { + if (old_isnull[i] != new_isnull[i]) + { + changed = true; + break; + } + else if (!old_isnull[i]) + { + int16 elmlen = indexInfo->ii_CompactAttr[i]->attlen; + bool elmbyval = indexInfo->ii_CompactAttr[i]->attbyval; + + if (!datum_image_eq(old_values[i], new_values[i], + elmbyval, elmlen)) + { + changed = true; + break; + } + } + } + + if (changed) + { + /* Shortcut index_unchanged_by_update(), we know the answer. */ + indexInfo->ii_CheckedUnchanged = true; + indexInfo->ii_IndexUnchanged = !changed; + break; + } + } + } + + if (econtext) + { + ExecDropSingleTupleTableSlot(old_tts); + ExecDropSingleTupleTableSlot(new_tts); + } + + return !changed; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index b0fe50075ad..d68f40cb998 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2283,7 +2283,8 @@ lreplace: estate->es_crosscheck_snapshot, true /* wait for commit */ , &context->tmfd, &updateCxt->lockmode, - &updateCxt->updateIndexes); + &updateCxt->updateIndexes, + estate); return result; } diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index 398114373e9..b5e47b3521e 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -64,6 +64,7 @@ #include "catalog/pg_type.h" #include "catalog/schemapg.h" #include "catalog/storage.h" +#include "catalog/index.h" #include "commands/policy.h" #include "commands/publicationcmds.h" #include "commands/trigger.h" @@ -5188,6 +5189,128 @@ RelationGetIndexPredicate(Relation relation) return result; } + /* + * RelationGetExprIndexInfoList -- get a list of IndexInfo for expression + * indexes on this relation + */ +List * +RelationGetExprIndexInfoList(Relation relation) +{ + ListCell *lc; + List *indexoidlist; + List *newindexoidlist; + List *result = NULL; + List *oldlist; + Oid relpkindex; + Oid relreplindex; + MemoryContext oldcxt; + + if (relation->rd_iiexprlist) + return list_copy(relation->rd_iiexprlist); + + /* Fast path if definitely no indexes */ + if (!RelationGetForm(relation)->relhasindex) + return NIL; + +restart: + indexoidlist = RelationGetIndexList(relation); + + /* Fall out if no indexes (but relhasindex was set) */ + if (indexoidlist == NIL) + return NIL; + + /* + * Copy the rd_pkindex and rd_replidindex values computed by + * RelationGetIndexList before proceeding. This is needed because a + * relcache flush could occur inside index_open below, resetting the + * fields managed by RelationGetIndexList. We need to do the work with + * stable values of these fields. + */ + relpkindex = relation->rd_pkindex; + relreplindex = relation->rd_replidindex; + + foreach(lc, indexoidlist) + { + Oid oid = lfirst_oid(lc); + Datum datum; + bool isnull; + Node *indexExpressions; + Node *indexPredicate; + Relation indexDesc; + IndexInfo *indexInfo; + + /* + * Extract index expressions and index predicate. Note: Don't use + * RelationGetIndexExpressions()/RelationGetIndexPredicate(), because + * those might run constant expressions evaluation, which needs a + * snapshot, which we might not have here. (Also, it's probably more + * sound to collect the bitmaps before any transformations that might + * eliminate columns, but the practical impact of this is limited.) + */ + indexDesc = index_open(oid, AccessShareLock); + datum = heap_getattr(indexDesc->rd_indextuple, Anum_pg_index_indexprs, + GetPgIndexDescriptor(), &isnull); + if (!isnull) + indexExpressions = stringToNode(TextDatumGetCString(datum)); + else + indexExpressions = NULL; + + datum = heap_getattr(indexDesc->rd_indextuple, Anum_pg_index_indpred, + GetPgIndexDescriptor(), &isnull); + if (!isnull) + indexPredicate = stringToNode(TextDatumGetCString(datum)); + else + indexPredicate = NULL; + + if (indexExpressions || indexPredicate) + { + oldcxt = MemoryContextSwitchTo(CacheMemoryContext); + indexInfo = BuildIndexInfo(indexDesc); + MemoryContextSwitchTo(oldcxt); + result = lappend(result, indexInfo); + } + + index_close(indexDesc, AccessShareLock); + } + + /* + * During one of the index_opens in the above loop, we might have received + * a relcache flush event on this relcache entry, which might have been + * signaling a change in the rel's index list. If so, we'd better start + * over to ensure we deliver up-to-date IndexInfo. + */ + newindexoidlist = RelationGetIndexList(relation); + if (equal(indexoidlist, newindexoidlist) && + relpkindex == relation->rd_pkindex && + relreplindex == relation->rd_replidindex) + { + /* Still the same index set, so proceed */ + list_free(newindexoidlist); + list_free(indexoidlist); + } + else + { + /* Gotta do it over ... might as well not leak memory */ + list_free(newindexoidlist); + list_free(indexoidlist); + list_free(result); + + goto restart; + } + + /* Now save a copy of the completed list in the relcache entry. */ + oldcxt = MemoryContextSwitchTo(CacheMemoryContext); + oldlist = relation->rd_iiexprlist; + relation->rd_iiexprlist = list_copy(result); + MemoryContextSwitchTo(oldcxt); + + /* Don't leak the old list, if there is one */ + if (oldlist) + list_free(oldlist); + + return result; +} + /* * RelationGetIndexAttrBitmap -- get a bitmap of index attribute numbers * @@ -5229,7 +5352,11 @@ RelationGetIndexAttrBitmap(Relation relation, IndexAttrBitmapKind attrKind) Bitmapset *pkindexattrs; /* columns in the primary index */ Bitmapset *idindexattrs; /* columns in the replica identity */ Bitmapset *hotblockingattrs; /* columns with HOT blocking indexes */ + Bitmapset *hotblockingexprattrs; /* as above, but only those in + * expressions */ Bitmapset *summarizedattrs; /* columns with summarizing indexes */ + Bitmapset *summarizedexprattrs; /* as above, but only those in + * expressions */ List *indexoidlist; List *newindexoidlist; Oid relpkindex; @@ -5252,6 +5379,8 @@ RelationGetIndexAttrBitmap(Relation relation, IndexAttrBitmapKind attrKind) return bms_copy(relation->rd_hotblockingattr); case INDEX_ATTR_BITMAP_SUMMARIZED: return bms_copy(relation->rd_summarizedattr); + case INDEX_ATTR_BITMAP_EXPRESSION: + return bms_copy(relation->rd_expressionattr); default: elog(ERROR, "unknown attrKind %u", attrKind); } @@ -5295,7 +5424,9 @@ restart: pkindexattrs = NULL; idindexattrs = NULL; hotblockingattrs = NULL; + hotblockingexprattrs = NULL; summarizedattrs = NULL; + summarizedexprattrs = NULL; foreach(l, indexoidlist) { Oid indexOid = lfirst_oid(l); @@ -5309,6 +5440,7 @@ restart: bool isPK; /* primary key */ bool isIDKey; /* replica identity index */ Bitmapset **attrs; + Bitmapset **exprattrs; indexDesc = index_open(indexOid, AccessShareLock); @@ -5352,14 +5484,20 @@ restart: * decide which bitmap we'll update in the following loop. */ if (indexDesc->rd_indam->amsummarizing) + { attrs = &summarizedattrs; + exprattrs = &summarizedexprattrs; + } else + { attrs = &hotblockingattrs; + exprattrs = &hotblockingexprattrs; + } /* Collect simple attribute references */ for (i = 0; i < indexDesc->rd_index->indnatts; i++) { - int attrnum = indexDesc->rd_index->indkey.values[i]; + int attridx = indexDesc->rd_index->indkey.values[i]; /* * Since we have covering indexes with non-key columns, we must @@ -5375,30 +5513,28 @@ restart: * key or identity key. Hence we do not include them into * uindexattrs, pkindexattrs and idindexattrs bitmaps. */ - if (attrnum != 0) + if (attridx != 0) { - *attrs = bms_add_member(*attrs, - attrnum - FirstLowInvalidHeapAttributeNumber); + AttrNumber attrnum = attridx - FirstLowInvalidHeapAttributeNumber; + + *attrs = bms_add_member(*attrs, attrnum); if (isKey && i < indexDesc->rd_index->indnkeyatts) - uindexattrs = bms_add_member(uindexattrs, - attrnum - FirstLowInvalidHeapAttributeNumber); + uindexattrs = bms_add_member(uindexattrs, attrnum); if (isPK && i < indexDesc->rd_index->indnkeyatts) - pkindexattrs = bms_add_member(pkindexattrs, - attrnum - FirstLowInvalidHeapAttributeNumber); + pkindexattrs = bms_add_member(pkindexattrs, attrnum); if (isIDKey && i < indexDesc->rd_index->indnkeyatts) - idindexattrs = bms_add_member(idindexattrs, - attrnum - FirstLowInvalidHeapAttributeNumber); + idindexattrs = bms_add_member(idindexattrs, attrnum); } } /* Collect all attributes used in expressions, too */ - pull_varattnos(indexExpressions, 1, attrs); + pull_varattnos(indexExpressions, 1, exprattrs); /* Collect all attributes in the index predicate, too */ - pull_varattnos(indexPredicate, 1, attrs); + pull_varattnos(indexPredicate, 1, exprattrs); index_close(indexDesc, AccessShareLock); } @@ -5427,11 +5563,27 @@ restart: bms_free(pkindexattrs); bms_free(idindexattrs); bms_free(hotblockingattrs); + bms_free(hotblockingexprattrs); bms_free(summarizedattrs); + bms_free(summarizedexprattrs); goto restart; } + /* {expression-only columns} = {expression columns} - {direct columns} */ + hotblockingexprattrs = bms_del_members(hotblockingexprattrs, + hotblockingattrs); + /* {hot-blocking columns} = {direct columns} + {expression-only columns} */ + hotblockingattrs = bms_add_members(hotblockingattrs, + hotblockingexprattrs); + + /* {summarized-only columns} = {summarized columns} - {direct columns} */ + summarizedexprattrs = bms_del_members(summarizedexprattrs, + summarizedattrs); + /* {summarized columns} = {direct columns} + {summarized-only columns} */ + summarizedattrs = bms_add_members(summarizedattrs, + summarizedexprattrs); + /* Don't leak the old values of these bitmaps, if any */ relation->rd_attrsvalid = false; bms_free(relation->rd_keyattr); @@ -5444,6 +5596,8 @@ restart: relation->rd_hotblockingattr = NULL; bms_free(relation->rd_summarizedattr); relation->rd_summarizedattr = NULL; + bms_free(relation->rd_expressionattr); + relation->rd_expressionattr = NULL; /* * Now save copies of the bitmaps in the relcache entry. We intentionally @@ -5458,6 +5612,7 @@ restart: relation->rd_idattr = bms_copy(idindexattrs); relation->rd_hotblockingattr = bms_copy(hotblockingattrs); relation->rd_summarizedattr = bms_copy(summarizedattrs); + relation->rd_expressionattr = bms_copy(hotblockingexprattrs); relation->rd_attrsvalid = true; MemoryContextSwitchTo(oldcxt); @@ -5474,6 +5629,8 @@ restart: return hotblockingattrs; case INDEX_ATTR_BITMAP_SUMMARIZED: return summarizedattrs; + case INDEX_ATTR_BITMAP_EXPRESSION: + return hotblockingexprattrs; default: elog(ERROR, "unknown attrKind %u", attrKind); return NULL; diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index eb8bc128720..f0a8286a25b 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -2992,7 +2992,7 @@ match_previous_words(int pattern_id, COMPLETE_WITH("("); /* ALTER TABLESPACE <foo> SET|RESET ( */ else if (Matches("ALTER", "TABLESPACE", MatchAny, "SET|RESET", "(")) - COMPLETE_WITH("seq_page_cost", "random_page_cost", + COMPLETE_WITH("seq_page_cost", "random_page_cost", "expression_checks", "effective_io_concurrency", "maintenance_io_concurrency"); /* ALTER TEXT SEARCH */ diff --git a/src/include/access/genam.h b/src/include/access/genam.h index 1be8739573f..91af1e8238d 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -194,7 +194,7 @@ extern bool index_can_return(Relation indexRelation, int attno); extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum, uint16 procnum); extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum, - uint16 procnum); + uint16 procnum); extern void index_store_float8_orderby_distances(IndexScanDesc scan, Oid *orderByTypes, IndexOrderByDistance *distances, diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 1640d9c32f7..e12da1934e9 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -21,6 +21,7 @@ #include "access/skey.h" #include "access/table.h" /* for backward compatibility */ #include "access/tableam.h" +#include "executor/executor.h" #include "nodes/lockoptions.h" #include "nodes/primnodes.h" #include "storage/bufpage.h" @@ -339,7 +340,7 @@ extern TM_Result heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, CommandId cid, Snapshot crosscheck, bool wait, struct TM_FailureData *tmfd, LockTupleMode *lockmode, - TU_UpdateIndexes *update_indexes); + TU_UpdateIndexes *update_indexes, struct EState *estate); extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple, CommandId cid, LockTupleMode mode, LockWaitPolicy wait_policy, bool follow_updates, diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h index 131c050c15f..62ed6592b85 100644 --- a/src/include/access/tableam.h +++ b/src/include/access/tableam.h @@ -38,6 +38,7 @@ struct IndexInfo; struct SampleScanState; struct VacuumParams; struct ValidateIndexState; +struct EState; /* * Bitmask values for the flags argument to the scan_begin callback. @@ -550,7 +551,8 @@ typedef struct TableAmRoutine bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode, - TU_UpdateIndexes *update_indexes); + TU_UpdateIndexes *update_indexes, + struct EState *estate); /* see table_tuple_lock() for reference about parameters */ TM_Result (*tuple_lock) (Relation rel, @@ -1541,12 +1543,12 @@ static inline TM_Result table_tuple_update(Relation rel, ItemPointer otid, TupleTableSlot *slot, CommandId cid, Snapshot snapshot, Snapshot crosscheck, bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode, - TU_UpdateIndexes *update_indexes) + TU_UpdateIndexes *update_indexes, struct EState *estate) { return rel->rd_tableam->tuple_update(rel, otid, slot, cid, snapshot, crosscheck, - wait, tmfd, - lockmode, update_indexes); + wait, tmfd, lockmode, + update_indexes, estate); } /* diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 30e2a82346f..343d04fff57 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -661,6 +661,11 @@ extern void check_exclusion_constraint(Relation heap, Relation index, ItemPointer tupleid, const Datum *values, const bool *isnull, EState *estate, bool newIndex); +extern bool ExecIndexesExpressionsWereNotUpdated(Relation relation, + Bitmapset *modified_attrs, + EState *estate, + HeapTuple old_tuple, + HeapTuple new_tuple); /* * prototypes from functions in execReplication.c diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index 2625d7e8222..9bfdd0d1464 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -159,12 +159,15 @@ typedef struct ExprState * * NumIndexAttrs total number of columns in this index * NumIndexKeyAttrs number of key columns in index + * IndexAttrs bitmap of index attributes * IndexAttrNumbers underlying-rel attribute numbers used as keys * (zeroes indicate expressions). It also contains * info about included columns. * Expressions expr trees for expression entries, or NIL if none + * ExpressionAttrs bitmap of attributes used within the expression * ExpressionsState exec state for expressions, or NIL if none * Predicate partial-index predicate, or NIL if none + * PredicateAttrs bitmap of attributes used within the predicate * PredicateState exec state for predicate, or NIL if none * ExclusionOps Per-column exclusion operators, or NULL if none * ExclusionProcs Underlying function OIDs for ExclusionOps @@ -183,6 +186,7 @@ typedef struct ExprState * ParallelWorkers # of workers requested (excludes leader) * Am Oid of index AM * AmCache private cache area for index AM + * OpClassDataTypes operator class data types * Context memory context holding this IndexInfo * * ii_Concurrent, ii_BrokenHotChain, and ii_ParallelWorkers are used only @@ -194,10 +198,14 @@ typedef struct IndexInfo NodeTag type; int ii_NumIndexAttrs; /* total number of columns in index */ int ii_NumIndexKeyAttrs; /* number of key columns in index */ + Bitmapset *ii_IndexAttrs; AttrNumber ii_IndexAttrNumbers[INDEX_MAX_KEYS]; + CompactAttribute *ii_CompactAttr[INDEX_MAX_KEYS]; List *ii_Expressions; /* list of Expr */ + Bitmapset *ii_ExpressionAttrs; List *ii_ExpressionsState; /* list of ExprState */ List *ii_Predicate; /* list of Expr */ + Bitmapset *ii_PredicateAttrs; ExprState *ii_PredicateState; Oid *ii_ExclusionOps; /* array with one entry per column */ Oid *ii_ExclusionProcs; /* array with one entry per column */ diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index db3e504c3d2..3eba254a366 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -164,6 +164,11 @@ typedef struct RelationData Bitmapset *rd_idattr; /* included in replica identity index */ Bitmapset *rd_hotblockingattr; /* cols blocking HOT update */ Bitmapset *rd_summarizedattr; /* cols indexed by summarizing indexes */ + Bitmapset *rd_expressionattr; /* indexed cols referenced by expressions */ + + /* data managed by RelationGetExprIndexInfoList: */ + List *rd_iiexprlist; /* list of IndexInfo for indexes with + * expressions */ PublicationDesc *rd_pubdesc; /* publication descriptor, or NULL */ @@ -344,6 +349,7 @@ typedef struct StdRdOptions int parallel_workers; /* max number of parallel workers */ StdRdOptIndexCleanup vacuum_index_cleanup; /* controls index vacuuming */ bool vacuum_truncate; /* enables vacuum to truncate a relation */ + bool expression_checks; /* use expression to checks for changes */ /* * Fraction of pages in a relation that vacuum can eagerly scan and fail @@ -405,6 +411,14 @@ typedef struct StdRdOptions ((relation)->rd_options ? \ ((StdRdOptions *) (relation)->rd_options)->parallel_workers : (defaultpw)) +/* + * RelationGetExpressionChecks + * Returns the relation's expression_checks reloption setting. + */ +#define RelationGetExpressionChecks(relation) \ + ((relation)->rd_options ? \ + ((StdRdOptions *) (relation)->rd_options)->expression_checks : true) + /* ViewOptions->check_option values */ typedef enum ViewOptCheckOption { diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h index a7c55db339e..56524a73399 100644 --- a/src/include/utils/relcache.h +++ b/src/include/utils/relcache.h @@ -52,6 +52,7 @@ extern List *RelationGetIndexExpressions(Relation relation); extern List *RelationGetDummyIndexExpressions(Relation relation); extern List *RelationGetIndexPredicate(Relation relation); extern bytea **RelationGetIndexAttOptions(Relation relation, bool copy); +extern List *RelationGetExprIndexInfoList(Relation relation); /* * Which set of columns to return by RelationGetIndexAttrBitmap. @@ -63,6 +64,7 @@ typedef enum IndexAttrBitmapKind INDEX_ATTR_BITMAP_IDENTITY_KEY, INDEX_ATTR_BITMAP_HOT_BLOCKING, INDEX_ATTR_BITMAP_SUMMARIZED, + INDEX_ATTR_BITMAP_EXPRESSION, } IndexAttrBitmapKind; extern Bitmapset *RelationGetIndexAttrBitmap(Relation relation, diff --git a/src/test/regress/expected/heap_hot_updates.out b/src/test/regress/expected/heap_hot_updates.out new file mode 100644 index 00000000000..2ad6e5418e9 --- /dev/null +++ b/src/test/regress/expected/heap_hot_updates.out @@ -0,0 +1,586 @@ +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table will have two columns and two indexes, one on the primary key +-- id and one on the expression (info->>'name'). That means that the indexed +-- attributes are 'id' and 'info'. +create table keyvalue(id integer primary key, info jsonb); +create index nameindex on keyvalue((info->>'name')); +insert into keyvalue values (1, '{"name": "john", "data": "some data"}'); +-- Disable expression checks. +ALTER TABLE keyvalue SET (expression_checks = false); +SELECT reloptions FROM pg_class WHERE relname = 'keyvalue'; + reloptions +--------------------------- + {expression_checks=false} +(1 row) + +-- While the indexed attribute "name" is unchanged we've disabled expression +-- checks so this update should not go HOT as the system can't determine if +-- the indexed attribute has changed without evaluating the expression. +update keyvalue set info='{"name": "john", "data": "something else"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 0 row + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Re-enable expression checks. +ALTER TABLE keyvalue SET (expression_checks = true); +SELECT reloptions FROM pg_class WHERE relname = 'keyvalue'; + reloptions +-------------------------- + {expression_checks=true} +(1 row) + +-- The indexed attribute "name" with value "john" is unchanged, expect a HOT update. +update keyvalue set info='{"name": "john", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 1 row + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- The following update changes the indexed attribute "name", this should not be a HOT update. +update keyvalue set info='{"name": "smith", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Now, this update does not change the indexed attribute "name" from "smith", this should be HOT. +update keyvalue set info='{"name": "smith", "data": "some more data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 2 rows now + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 2 +(1 row) + +drop table keyvalue; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table is the same as the previous one but it has a third index. The +-- index 'colindex' isn't an expression index, it indexes the entire value +-- in the info column. There are still only two indexed attributes for this +-- relation, the same two as before. The presence of an index on the entire +-- value of the info column should prevent HOT updates for any updates to any +-- portion of JSONB content in that column. +create table keyvalue(id integer primary key, info jsonb); +create index nameindex on keyvalue((info->>'name')); +create index colindex on keyvalue(info); +insert into keyvalue values (1, '{"name": "john", "data": "some data"}'); +-- This update doesn't change the value of the expression index, but it does +-- change the content of the info column and so should not be HOT because the +-- indexed value changed as a result of the update. +update keyvalue set info='{"name": "john", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 0 rows + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +drop table keyvalue; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- The table has one column docs and two indexes. They are both expression +-- indexes referencing the same column attribute (docs) but one is a partial +-- index. +CREATE TABLE ex (docs JSONB) WITH (fillfactor = 60); +INSERT INTO ex (docs) VALUES ('{"a": 0, "b": 0}'); +INSERT INTO ex (docs) SELECT jsonb_build_object('b', n) FROM generate_series(100, 10000) as n; +CREATE INDEX idx_ex_a ON ex ((docs->>'a')); +CREATE INDEX idx_ex_b ON ex ((docs->>'b')) WHERE (docs->>'b')::numeric > 9; +-- We're using BTREE indexes and for this test we want to make sure that they remain +-- in sync with changes to our relation. Force the choice of index scans below so +-- that we know we're checking the index's understanding of what values should be +-- in the index or not. +SET SESSION enable_seqscan = OFF; +SET SESSION enable_bitmapscan = OFF; +-- Leave 'a' unchanged but modify 'b' to a value outside of the index predicate. +-- This should be a HOT update because neither index is changed. +UPDATE ex SET docs = jsonb_build_object('a', 0, 'b', 1) WHERE (docs->>'a')::numeric = 0; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 row + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Let's check to make sure that the index does not contain a value for 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + QUERY PLAN +-------------------------------------------------------------- + Index Scan using idx_ex_b on ex + Filter: (((docs ->> 'b'::text))::numeric < '100'::numeric) +(2 rows) + +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + docs +------ +(0 rows) + +-- Leave 'a' unchanged but modify 'b' to a value within the index predicate. +-- This represents a change for field 'b' from unindexed to indexed and so +-- this should not take the HOT path. +UPDATE ex SET docs = jsonb_build_object('a', 0, 'b', 10) WHERE (docs->>'a')::numeric = 0; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Let's check to make sure that the index contains the new value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + QUERY PLAN +-------------------------------------------------------------- + Index Scan using idx_ex_b on ex + Filter: (((docs ->> 'b'::text))::numeric < '100'::numeric) +(2 rows) + +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + docs +------------------- + {"a": 0, "b": 10} +(1 row) + +-- This update modifies the value of 'a', an indexed field, so it also cannot +-- be a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 1, 'b', 10) WHERE (docs->>'b')::numeric = 10; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- This update changes both 'a' and 'b' to new values that require index updates, +-- this cannot use the HOT path. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 12) WHERE (docs->>'b')::numeric = 10; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Let's check to make sure that the index contains the new value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + QUERY PLAN +-------------------------------------------------------------- + Index Scan using idx_ex_b on ex + Filter: (((docs ->> 'b'::text))::numeric < '100'::numeric) +(2 rows) + +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + docs +------------------- + {"a": 2, "b": 12} +(1 row) + +-- This update changes 'b' to a value outside its predicate requiring that +-- we remove it from the index. That's a transition that can't be done +-- during a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 1) WHERE (docs->>'b')::numeric = 12; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Let's check to make sure that the index no longer contains the value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + QUERY PLAN +-------------------------------------------------------------- + Index Scan using idx_ex_b on ex + Filter: (((docs ->> 'b'::text))::numeric < '100'::numeric) +(2 rows) + +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + docs +------ +(0 rows) + +-- Disable expression checks. +ALTER TABLE ex SET (expression_checks = false); +SELECT reloptions FROM pg_class WHERE relname = 'ex'; + reloptions +----------------------------------------- + {fillfactor=60,expression_checks=false} +(1 row) + +-- This update changes 'b' to a value within its predicate just like it +-- previous value, which would allow for a HOT update but with expression +-- checks disabled we can't determine that so this should not be a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 2) WHERE (docs->>'b')::numeric = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Let's make sure we're recording HOT updates for our 'ex' relation properly in the system +-- table pg_stat_user_tables. Note that statistics are stored within a transaction context +-- first (xact) and then later into the global statistics for a relation, so first we need +-- to ensure pending stats are flushed. +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT + c.relname AS table_name, + -- Transaction statistics + pg_stat_get_xact_tuples_updated(c.oid) AS xact_updates, + pg_stat_get_xact_tuples_hot_updated(c.oid) AS xact_hot_updates, + ROUND(( + pg_stat_get_xact_tuples_hot_updated(c.oid)::float / + NULLIF(pg_stat_get_xact_tuples_updated(c.oid), 0) * 100 + )::numeric, 2) AS xact_hot_update_percentage, + -- Cumulative statistics + s.n_tup_upd AS total_updates, + s.n_tup_hot_upd AS hot_updates, + ROUND(( + s.n_tup_hot_upd::float / + NULLIF(s.n_tup_upd, 0) * 100 + )::numeric, 2) AS total_hot_update_percentage +FROM pg_class c +LEFT JOIN pg_stat_user_tables s ON c.relname = s.relname +WHERE c.relname = 'ex' +AND c.relnamespace = 'public'::regnamespace; + table_name | xact_updates | xact_hot_updates | xact_hot_update_percentage | total_updates | hot_updates | total_hot_update_percentage +------------+--------------+------------------+----------------------------+---------------+-------------+----------------------------- + ex | 0 | 0 | | 6 | 1 | 16.67 +(1 row) + +-- expect: 5 xact updates with 1 xact hot update and no cumulative updates as yet +SET SESSION enable_seqscan = ON; +SET SESSION enable_bitmapscan = ON; +DROP TABLE ex; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table has a single column 'email' and a unique constraint on it that +-- should preclude HOT updates. +CREATE TABLE users ( + user_id serial primary key, + name VARCHAR(255) NOT NULL, + email VARCHAR(255) NOT NULL, + EXCLUDE USING btree (lower(email) WITH =) +); +-- Add some data to the table and then update it in ways that should and should +-- not be HOT updates. +INSERT INTO users (name, email) VALUES +('user1', '[email protected]'), +('user2', '[email protected]'), +('taken', '[email protected]'), +('you', '[email protected]'), +('taken', '[email protected]'); +-- Should fail because of the unique constraint on the email column. +UPDATE users SET email = '[email protected]' WHERE email = '[email protected]'; +ERROR: conflicting key value violates exclusion constraint "users_lower_excl" +DETAIL: Key (lower(email::text))=([email protected]) conflicts with existing key (lower(email::text))=([email protected]). +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 0 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Should succeed because the email column is not being updated and should go HOT. +UPDATE users SET name = 'foo' WHERE email = '[email protected]'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 a single new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Create a partial index on the email column, updates +CREATE INDEX idx_users_email_no_example ON users (lower(email)) WHERE lower(email) LIKE '%@example.com%'; +-- An update that changes the email column but not the indexed portion of it and falls outside the constraint. +-- Shouldn't be a HOT update because of the exclusion constraint. +UPDATE users SET email = '[email protected]' WHERE name = 'you'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- An update that changes the email column but not the indexed portion of it and falls within the constraint. +-- Again, should fail constraint and fail to be a HOT update. +UPDATE users SET email = '[email protected]' WHERE name = 'you'; +ERROR: conflicting key value violates exclusion constraint "users_lower_excl" +DETAIL: Key (lower(email::text))=([email protected]) conflicts with existing key (lower(email::text))=([email protected]). +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +DROP TABLE users; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- Another test of constraints spoiling HOT updates, this time with a range. +CREATE TABLE events ( + id serial primary key, + name VARCHAR(255) NOT NULL, + event_time tstzrange, + constraint no_screening_time_overlap exclude using gist ( + event_time WITH && + ) +); +-- Add two non-overlapping events. +INSERT INTO events (id, event_time, name) +VALUES + (1, '["2023-01-01 19:00:00", "2023-01-01 20:45:00"]', 'event1'), + (2, '["2023-01-01 21:00:00", "2023-01-01 21:45:00"]', 'event2'); +-- Update the first event to overlap with the second, should fail the constraint and not be HOT. +UPDATE events SET event_time = '["2023-01-01 20:00:00", "2023-01-01 21:45:00"]' WHERE id = 1; +ERROR: conflicting key value violates exclusion constraint "no_screening_time_overlap" +DETAIL: Key (event_time)=(["Sun Jan 01 20:00:00 2023 PST","Sun Jan 01 21:45:00 2023 PST"]) conflicts with existing key (event_time)=(["Sun Jan 01 21:00:00 2023 PST","Sun Jan 01 21:45:00 2023 PST"]). +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 0 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Update the first event to not overlap with the second, again not HOT due to the constraint. +UPDATE events SET event_time = '["2023-01-01 22:00:00", "2023-01-01 22:45:00"]' WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 0 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Update the first event to not overlap with the second, this time we're HOT because we don't overlap with the constraint. +UPDATE events SET name = 'new name here' WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 1 one new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +DROP TABLE events; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- A test to ensure that summarizing indexes are not updated when they don't +-- change, but are updated when they do while not prefent HOT updates. +CREATE TABLE ex (att1 JSONB, att2 text) WITH (fillfactor = 60); +CREATE INDEX expression ON ex USING btree((att1->'data')); +CREATE INDEX summarizing ON ex USING BRIN(att2); +INSERT INTO ex VALUES ('{"data": []}', 'nothing special'); +-- Update the unindexed value of att1, this should be a HOT update and not +-- update the summarizing index. +UPDATE ex SET att1 = att1 || '{"status": "checkmate"}'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 one new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Update the indexed value of att2, a summarized value, this is a summarized +-- only update and should use the HOT path while still triggering an update to +-- the summarizing BRIN index. +UPDATE ex SET att2 = 'special indeed'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 2 one new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 2 +(1 row) + +-- Update to att1 doesn't change the indexed value while the update to att2 does, +-- this again is a summarized only update and should use the HOT path as well as +-- trigger an update to the BRIN index. +UPDATE ex SET att1 = att1 || '{"status": "checkmate!"}', att2 = 'special, so special'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 3 one new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 3 +(1 row) + +-- This updates both indexes, the expression index on att1 and the summarizing +-- index on att2. This should not be a HOT update because there are modified +-- indexes and only some are summarized, not all. This should force all +-- indexes to be updated. +UPDATE ex SET att1 = att1 || '{"data": [1,2,3]}', att2 = 'do you want to play a game?'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 3 both indexes updated + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 3 +(1 row) + +DROP TABLE ex; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This test is for a table with a custom type and a custom operators on +-- the BTREE index. The question is, when comparing values for equality +-- to determine if there are changes on the index or not... shouldn't we +-- be using the custom operators? +-- Create a type +CREATE TYPE my_custom_type AS (val int); +-- Comparison functions (returns boolean) +CREATE FUNCTION my_custom_lt(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val < b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +CREATE FUNCTION my_custom_le(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val <= b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +CREATE FUNCTION my_custom_eq(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val = b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +CREATE FUNCTION my_custom_ge(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val >= b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +CREATE FUNCTION my_custom_gt(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val > b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +CREATE FUNCTION my_custom_ne(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val != b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +-- Comparison function (returns -1, 0, 1) +CREATE FUNCTION my_custom_cmp(a my_custom_type, b my_custom_type) RETURNS int AS $$ +BEGIN + IF a.val < b.val THEN + RETURN -1; + ELSIF a.val > b.val THEN + RETURN 1; + ELSE + RETURN 0; + END IF; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +-- Create the operators +CREATE OPERATOR < ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_lt, + COMMUTATOR = >, + NEGATOR = >= +); +CREATE OPERATOR <= ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_le, + COMMUTATOR = >=, + NEGATOR = > +); +CREATE OPERATOR = ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_eq, + COMMUTATOR = =, + NEGATOR = <> +); +CREATE OPERATOR >= ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_ge, + COMMUTATOR = <=, + NEGATOR = < +); +CREATE OPERATOR > ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_gt, + COMMUTATOR = <, + NEGATOR = <= +); +CREATE OPERATOR <> ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_ne, + COMMUTATOR = <>, + NEGATOR = = +); +-- Create the operator class (including the support function) +CREATE OPERATOR CLASS my_custom_ops + DEFAULT FOR TYPE my_custom_type USING btree AS + OPERATOR 1 <, + OPERATOR 2 <=, + OPERATOR 3 =, + OPERATOR 4 >=, + OPERATOR 5 >, + FUNCTION 1 my_custom_cmp(my_custom_type, my_custom_type); +-- Create the table +CREATE TABLE my_table ( + id int, + custom_val my_custom_type +); +-- Insert some data +INSERT INTO my_table (id, custom_val) VALUES +(1, ROW(3)::my_custom_type), +(2, ROW(1)::my_custom_type), +(3, ROW(4)::my_custom_type), +(4, ROW(2)::my_custom_type); +-- Create a function to use when indexing +CREATE OR REPLACE FUNCTION abs_val(val my_custom_type) RETURNS int AS $$ +BEGIN + RETURN abs(val.val); +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +-- Create the index +CREATE INDEX idx_custom_val_abs ON my_table (abs_val(custom_val)); +-- Update 1 +UPDATE my_table SET custom_val = ROW(5)::my_custom_type WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Update 2 +UPDATE my_table SET custom_val = ROW(0)::my_custom_type WHERE custom_val < ROW(3)::my_custom_type; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Update 3 +UPDATE my_table SET custom_val = ROW(6)::my_custom_type WHERE id = 3; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Update 4 +UPDATE my_table SET id = 5 WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Query using the index +EXPLAIN (COSTS OFF) SELECT * FROM my_table WHERE abs_val(custom_val) = 6; + QUERY PLAN +------------------------------------- + Seq Scan on my_table + Filter: (abs_val(custom_val) = 6) +(2 rows) + +SELECT * FROM my_table WHERE abs_val(custom_val) = 6; + id | custom_val +----+------------ + 3 | (6) +(1 row) + +-- Clean up +DROP TABLE my_table CASCADE; +DROP OPERATOR CLASS my_custom_ops USING btree CASCADE; +DROP OPERATOR < (my_custom_type, my_custom_type); +DROP OPERATOR <= (my_custom_type, my_custom_type); +DROP OPERATOR = (my_custom_type, my_custom_type); +DROP OPERATOR >= (my_custom_type, my_custom_type); +DROP OPERATOR > (my_custom_type, my_custom_type); +DROP OPERATOR <> (my_custom_type, my_custom_type); +DROP FUNCTION my_custom_lt(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_le(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_eq(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_ge(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_gt(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_ne(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_cmp(my_custom_type, my_custom_type); +DROP FUNCTION abs_val(my_custom_type); +DROP TYPE my_custom_type CASCADE; diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e63ee2cf2bb..f9def3d93aa 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -68,6 +68,11 @@ test: select_into select_distinct select_distinct_on select_implicit select_havi # ---------- test: brin gin gist spgist privileges init_privs security_label collate matview lock replica_identity rowsecurity object_address tablesample groupingsets drop_operator password identity generated_stored join_hash +# ---------- +# Another group of parallel tests +# ---------- +test: heap_hot_updates + # ---------- # Additional BRIN tests # ---------- diff --git a/src/test/regress/sql/heap_hot_updates.sql b/src/test/regress/sql/heap_hot_updates.sql new file mode 100644 index 00000000000..7728075a7ae --- /dev/null +++ b/src/test/regress/sql/heap_hot_updates.sql @@ -0,0 +1,440 @@ +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table will have two columns and two indexes, one on the primary key +-- id and one on the expression (info->>'name'). That means that the indexed +-- attributes are 'id' and 'info'. +create table keyvalue(id integer primary key, info jsonb); +create index nameindex on keyvalue((info->>'name')); +insert into keyvalue values (1, '{"name": "john", "data": "some data"}'); + +-- Disable expression checks. +ALTER TABLE keyvalue SET (expression_checks = false); +SELECT reloptions FROM pg_class WHERE relname = 'keyvalue'; + +-- While the indexed attribute "name" is unchanged we've disabled expression +-- checks so this update should not go HOT as the system can't determine if +-- the indexed attribute has changed without evaluating the expression. +update keyvalue set info='{"name": "john", "data": "something else"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 0 row + +-- Re-enable expression checks. +ALTER TABLE keyvalue SET (expression_checks = true); +SELECT reloptions FROM pg_class WHERE relname = 'keyvalue'; + +-- The indexed attribute "name" with value "john" is unchanged, expect a HOT update. +update keyvalue set info='{"name": "john", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 1 row + +-- The following update changes the indexed attribute "name", this should not be a HOT update. +update keyvalue set info='{"name": "smith", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 1 no new HOT updates + +-- Now, this update does not change the indexed attribute "name" from "smith", this should be HOT. +update keyvalue set info='{"name": "smith", "data": "some more data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 2 rows now +drop table keyvalue; + + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table is the same as the previous one but it has a third index. The +-- index 'colindex' isn't an expression index, it indexes the entire value +-- in the info column. There are still only two indexed attributes for this +-- relation, the same two as before. The presence of an index on the entire +-- value of the info column should prevent HOT updates for any updates to any +-- portion of JSONB content in that column. +create table keyvalue(id integer primary key, info jsonb); +create index nameindex on keyvalue((info->>'name')); +create index colindex on keyvalue(info); +insert into keyvalue values (1, '{"name": "john", "data": "some data"}'); + +-- This update doesn't change the value of the expression index, but it does +-- change the content of the info column and so should not be HOT because the +-- indexed value changed as a result of the update. +update keyvalue set info='{"name": "john", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 0 rows +drop table keyvalue; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- The table has one column docs and two indexes. They are both expression +-- indexes referencing the same column attribute (docs) but one is a partial +-- index. +CREATE TABLE ex (docs JSONB) WITH (fillfactor = 60); +INSERT INTO ex (docs) VALUES ('{"a": 0, "b": 0}'); +INSERT INTO ex (docs) SELECT jsonb_build_object('b', n) FROM generate_series(100, 10000) as n; +CREATE INDEX idx_ex_a ON ex ((docs->>'a')); +CREATE INDEX idx_ex_b ON ex ((docs->>'b')) WHERE (docs->>'b')::numeric > 9; + +-- We're using BTREE indexes and for this test we want to make sure that they remain +-- in sync with changes to our relation. Force the choice of index scans below so +-- that we know we're checking the index's understanding of what values should be +-- in the index or not. +SET SESSION enable_seqscan = OFF; +SET SESSION enable_bitmapscan = OFF; + +-- Leave 'a' unchanged but modify 'b' to a value outside of the index predicate. +-- This should be a HOT update because neither index is changed. +UPDATE ex SET docs = jsonb_build_object('a', 0, 'b', 1) WHERE (docs->>'a')::numeric = 0; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 row +-- Let's check to make sure that the index does not contain a value for 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + +-- Leave 'a' unchanged but modify 'b' to a value within the index predicate. +-- This represents a change for field 'b' from unindexed to indexed and so +-- this should not take the HOT path. +UPDATE ex SET docs = jsonb_build_object('a', 0, 'b', 10) WHERE (docs->>'a')::numeric = 0; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates +-- Let's check to make sure that the index contains the new value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + +-- This update modifies the value of 'a', an indexed field, so it also cannot +-- be a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 1, 'b', 10) WHERE (docs->>'b')::numeric = 10; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + +-- This update changes both 'a' and 'b' to new values that require index updates, +-- this cannot use the HOT path. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 12) WHERE (docs->>'b')::numeric = 10; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates +-- Let's check to make sure that the index contains the new value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + +-- This update changes 'b' to a value outside its predicate requiring that +-- we remove it from the index. That's a transition that can't be done +-- during a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 1) WHERE (docs->>'b')::numeric = 12; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates +-- Let's check to make sure that the index no longer contains the value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + +-- Disable expression checks. +ALTER TABLE ex SET (expression_checks = false); +SELECT reloptions FROM pg_class WHERE relname = 'ex'; + +-- This update changes 'b' to a value within its predicate just like it +-- previous value, which would allow for a HOT update but with expression +-- checks disabled we can't determine that so this should not be a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 2) WHERE (docs->>'b')::numeric = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + + +-- Let's make sure we're recording HOT updates for our 'ex' relation properly in the system +-- table pg_stat_user_tables. Note that statistics are stored within a transaction context +-- first (xact) and then later into the global statistics for a relation, so first we need +-- to ensure pending stats are flushed. +SELECT pg_stat_force_next_flush(); +SELECT + c.relname AS table_name, + -- Transaction statistics + pg_stat_get_xact_tuples_updated(c.oid) AS xact_updates, + pg_stat_get_xact_tuples_hot_updated(c.oid) AS xact_hot_updates, + ROUND(( + pg_stat_get_xact_tuples_hot_updated(c.oid)::float / + NULLIF(pg_stat_get_xact_tuples_updated(c.oid), 0) * 100 + )::numeric, 2) AS xact_hot_update_percentage, + -- Cumulative statistics + s.n_tup_upd AS total_updates, + s.n_tup_hot_upd AS hot_updates, + ROUND(( + s.n_tup_hot_upd::float / + NULLIF(s.n_tup_upd, 0) * 100 + )::numeric, 2) AS total_hot_update_percentage +FROM pg_class c +LEFT JOIN pg_stat_user_tables s ON c.relname = s.relname +WHERE c.relname = 'ex' +AND c.relnamespace = 'public'::regnamespace; +-- expect: 5 xact updates with 1 xact hot update and no cumulative updates as yet + +SET SESSION enable_seqscan = ON; +SET SESSION enable_bitmapscan = ON; + +DROP TABLE ex; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table has a single column 'email' and a unique constraint on it that +-- should preclude HOT updates. +CREATE TABLE users ( + user_id serial primary key, + name VARCHAR(255) NOT NULL, + email VARCHAR(255) NOT NULL, + EXCLUDE USING btree (lower(email) WITH =) +); + +-- Add some data to the table and then update it in ways that should and should +-- not be HOT updates. +INSERT INTO users (name, email) VALUES +('user1', '[email protected]'), +('user2', '[email protected]'), +('taken', '[email protected]'), +('you', '[email protected]'), +('taken', '[email protected]'); + +-- Should fail because of the unique constraint on the email column. +UPDATE users SET email = '[email protected]' WHERE email = '[email protected]'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 0 no new HOT updates + +-- Should succeed because the email column is not being updated and should go HOT. +UPDATE users SET name = 'foo' WHERE email = '[email protected]'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 a single new HOT update + +-- Create a partial index on the email column, updates +CREATE INDEX idx_users_email_no_example ON users (lower(email)) WHERE lower(email) LIKE '%@example.com%'; + +-- An update that changes the email column but not the indexed portion of it and falls outside the constraint. +-- Shouldn't be a HOT update because of the exclusion constraint. +UPDATE users SET email = '[email protected]' WHERE name = 'you'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 no new HOT updates + +-- An update that changes the email column but not the indexed portion of it and falls within the constraint. +-- Again, should fail constraint and fail to be a HOT update. +UPDATE users SET email = '[email protected]' WHERE name = 'you'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 no new HOT updates + +DROP TABLE users; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- Another test of constraints spoiling HOT updates, this time with a range. +CREATE TABLE events ( + id serial primary key, + name VARCHAR(255) NOT NULL, + event_time tstzrange, + constraint no_screening_time_overlap exclude using gist ( + event_time WITH && + ) +); + +-- Add two non-overlapping events. +INSERT INTO events (id, event_time, name) +VALUES + (1, '["2023-01-01 19:00:00", "2023-01-01 20:45:00"]', 'event1'), + (2, '["2023-01-01 21:00:00", "2023-01-01 21:45:00"]', 'event2'); + +-- Update the first event to overlap with the second, should fail the constraint and not be HOT. +UPDATE events SET event_time = '["2023-01-01 20:00:00", "2023-01-01 21:45:00"]' WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 0 no new HOT updates + +-- Update the first event to not overlap with the second, again not HOT due to the constraint. +UPDATE events SET event_time = '["2023-01-01 22:00:00", "2023-01-01 22:45:00"]' WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 0 no new HOT updates + +-- Update the first event to not overlap with the second, this time we're HOT because we don't overlap with the constraint. +UPDATE events SET name = 'new name here' WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 1 one new HOT update + +DROP TABLE events; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- A test to ensure that summarizing indexes are not updated when they don't +-- change, but are updated when they do while not prefent HOT updates. +CREATE TABLE ex (att1 JSONB, att2 text) WITH (fillfactor = 60); +CREATE INDEX expression ON ex USING btree((att1->'data')); +CREATE INDEX summarizing ON ex USING BRIN(att2); +INSERT INTO ex VALUES ('{"data": []}', 'nothing special'); + +-- Update the unindexed value of att1, this should be a HOT update and not +-- update the summarizing index. +UPDATE ex SET att1 = att1 || '{"status": "checkmate"}'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 one new HOT update + +-- Update the indexed value of att2, a summarized value, this is a summarized +-- only update and should use the HOT path while still triggering an update to +-- the summarizing BRIN index. +UPDATE ex SET att2 = 'special indeed'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 2 one new HOT update + +-- Update to att1 doesn't change the indexed value while the update to att2 does, +-- this again is a summarized only update and should use the HOT path as well as +-- trigger an update to the BRIN index. +UPDATE ex SET att1 = att1 || '{"status": "checkmate!"}', att2 = 'special, so special'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 3 one new HOT update + +-- This updates both indexes, the expression index on att1 and the summarizing +-- index on att2. This should not be a HOT update because there are modified +-- indexes and only some are summarized, not all. This should force all +-- indexes to be updated. +UPDATE ex SET att1 = att1 || '{"data": [1,2,3]}', att2 = 'do you want to play a game?'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 3 both indexes updated + +DROP TABLE ex; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This test is for a table with a custom type and a custom operators on +-- the BTREE index. The question is, when comparing values for equality +-- to determine if there are changes on the index or not... shouldn't we +-- be using the custom operators? + +-- Create a type +CREATE TYPE my_custom_type AS (val int); + +-- Comparison functions (returns boolean) +CREATE FUNCTION my_custom_lt(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val < b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE FUNCTION my_custom_le(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val <= b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE FUNCTION my_custom_eq(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val = b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE FUNCTION my_custom_ge(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val >= b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE FUNCTION my_custom_gt(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val > b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE FUNCTION my_custom_ne(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val != b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +-- Comparison function (returns -1, 0, 1) +CREATE FUNCTION my_custom_cmp(a my_custom_type, b my_custom_type) RETURNS int AS $$ +BEGIN + IF a.val < b.val THEN + RETURN -1; + ELSIF a.val > b.val THEN + RETURN 1; + ELSE + RETURN 0; + END IF; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +-- Create the operators +CREATE OPERATOR < ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_lt, + COMMUTATOR = >, + NEGATOR = >= +); + +CREATE OPERATOR <= ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_le, + COMMUTATOR = >=, + NEGATOR = > +); + +CREATE OPERATOR = ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_eq, + COMMUTATOR = =, + NEGATOR = <> +); + +CREATE OPERATOR >= ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_ge, + COMMUTATOR = <=, + NEGATOR = < +); + +CREATE OPERATOR > ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_gt, + COMMUTATOR = <, + NEGATOR = <= +); + +CREATE OPERATOR <> ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_ne, + COMMUTATOR = <>, + NEGATOR = = +); + +-- Create the operator class (including the support function) +CREATE OPERATOR CLASS my_custom_ops + DEFAULT FOR TYPE my_custom_type USING btree AS + OPERATOR 1 <, + OPERATOR 2 <=, + OPERATOR 3 =, + OPERATOR 4 >=, + OPERATOR 5 >, + FUNCTION 1 my_custom_cmp(my_custom_type, my_custom_type); + +-- Create the table +CREATE TABLE my_table ( + id int, + custom_val my_custom_type +); + +-- Insert some data +INSERT INTO my_table (id, custom_val) VALUES +(1, ROW(3)::my_custom_type), +(2, ROW(1)::my_custom_type), +(3, ROW(4)::my_custom_type), +(4, ROW(2)::my_custom_type); + +-- Create a function to use when indexing +CREATE OR REPLACE FUNCTION abs_val(val my_custom_type) RETURNS int AS $$ +BEGIN + RETURN abs(val.val); +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +-- Create the index +CREATE INDEX idx_custom_val_abs ON my_table (abs_val(custom_val)); + +-- Update 1 +UPDATE my_table SET custom_val = ROW(5)::my_custom_type WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + +-- Update 2 +UPDATE my_table SET custom_val = ROW(0)::my_custom_type WHERE custom_val < ROW(3)::my_custom_type; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + +-- Update 3 +UPDATE my_table SET custom_val = ROW(6)::my_custom_type WHERE id = 3; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + +-- Update 4 +UPDATE my_table SET id = 5 WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + +-- Query using the index +EXPLAIN (COSTS OFF) SELECT * FROM my_table WHERE abs_val(custom_val) = 6; +SELECT * FROM my_table WHERE abs_val(custom_val) = 6; + +-- Clean up +DROP TABLE my_table CASCADE; +DROP OPERATOR CLASS my_custom_ops USING btree CASCADE; +DROP OPERATOR < (my_custom_type, my_custom_type); +DROP OPERATOR <= (my_custom_type, my_custom_type); +DROP OPERATOR = (my_custom_type, my_custom_type); +DROP OPERATOR >= (my_custom_type, my_custom_type); +DROP OPERATOR > (my_custom_type, my_custom_type); +DROP OPERATOR <> (my_custom_type, my_custom_type); +DROP FUNCTION my_custom_lt(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_le(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_eq(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_ge(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_gt(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_ne(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_cmp(my_custom_type, my_custom_type); +DROP FUNCTION abs_val(my_custom_type); +DROP TYPE my_custom_type CASCADE; -- 2.42.0 ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Expanding HOT updates for expression and partial indexes @ 2025-02-18 18:09 Burd, Greg <[email protected]> parent: Burd, Greg <[email protected]> 1 sibling, 1 reply; 6+ messages in thread From: Burd, Greg @ 2025-02-18 18:09 UTC (permalink / raw) To: pgsql-hackers; +Cc: Matthias van de Meent <[email protected]> Changes v6 to v7: * Fixed documentation oversight causing build failure * Changed how I convey attribute len/by-val in IndexInfo * Fixed method to shortcut index_unchanged_by_update() when possible -greg Attachments: [application/octet-stream] v7-0001-Expand-HOT-update-path-to-include-expression-and-.patch (87.0K, 3-v7-0001-Expand-HOT-update-path-to-include-expression-and-.patch) download | inline diff: From c675c674644b7a521712913b5cfc53f899e435ce Mon Sep 17 00:00:00 2001 From: Gregory Burd <[email protected]> Date: Mon, 27 Jan 2025 13:28:59 -0500 Subject: [PATCH v7] Expand HOT update path to include expression and partial indexes. This patch extends the cases where HOT updates are possible in the heapam by examining expression indexes and determining if indexed values where mutated or not. Previously, any expression index on a column would disqualify it from the HOT update path. Also examines partial indexes to see if the values are within the predicate or not. This is a modified application of a patch proposed on the pgsql-hackers list: https://www.postgresql.org/message-id/flat/4d9928ee-a9e6-15f9-9c82-5981f13ffca6%40postgrespro.ru applied: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=c203d6cf8 reverted: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=05f84605dbeb9cf8279a157234b24bbb706c5256 Signed-off-by: Greg Burd <[email protected]> --- doc/src/sgml/ref/create_table.sgml | 18 + doc/src/sgml/storage.sgml | 21 + src/backend/access/common/reloptions.c | 12 +- src/backend/access/heap/README.HOT | 45 +- src/backend/access/heap/heapam.c | 43 +- src/backend/access/heap/heapam_handler.c | 6 +- src/backend/access/table/tableam.c | 2 +- src/backend/catalog/index.c | 19 + src/backend/executor/execIndexing.c | 199 ++++++ src/backend/executor/nodeModifyTable.c | 3 +- src/backend/utils/cache/relcache.c | 181 +++++- src/bin/psql/tab-complete.in.c | 2 +- src/include/access/genam.h | 2 +- src/include/access/heapam.h | 3 +- src/include/access/tableam.h | 10 +- src/include/executor/executor.h | 5 + src/include/nodes/execnodes.h | 9 + src/include/utils/rel.h | 14 + src/include/utils/relcache.h | 2 + .../regress/expected/heap_hot_updates.out | 586 ++++++++++++++++++ src/test/regress/parallel_schedule | 5 + src/test/regress/sql/heap_hot_updates.sql | 440 +++++++++++++ 22 files changed, 1566 insertions(+), 61 deletions(-) create mode 100644 src/test/regress/expected/heap_hot_updates.out create mode 100644 src/test/regress/sql/heap_hot_updates.sql diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 0a3e520f215..8ee2ac523ee 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -1981,6 +1981,24 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry id="reloption-expression-checks" xreflabel="expression_checks"> + <term><literal>expression_checks</literal> (<type>boolean</type>) + <indexterm> + <primary><varname>expression_checks</varname> storage parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Enables or disables evaulation of predicate expressions on partial + indexes or expressions used to define indexes during updates. + If <literal>true</literal>, then these expressions are evaluated during + updates to data within the heap relation against the old and new values + and then compared to determine if <acronym>HOT</acronym> updates are + allowable or not. The default value is <literal>true</literal>. + </para> + </listitem> + </varlistentry> + </variablelist> </refsect2> diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml index 61250799ec0..f40ada9c989 100644 --- a/doc/src/sgml/storage.sgml +++ b/doc/src/sgml/storage.sgml @@ -1138,6 +1138,27 @@ data. Empty in ordinary tables.</entry> </itemizedlist> </para> + <para> + <acronym>HOT</acronym> updates can occur when the expression used to define + and index shows no changes to the indexed value. To determine this requires + that the expression be evaulated for the old and new values to be stored in + the index and then compared. This allows for <acronym>HOT</acronym> updates + when data indexed within JSONB columns is unchanged. To disable this + behavior and avoid the overhead of evaluating the expression during updates + set the <literal>expression_checks</literal> option to false for the table. + </para> + + <para> + <acronym>HOT</acronym> updates can also occur when updated values are not + within the predicate of a partial index. However, <acronym>HOT</acronym> + updates are not possible when the updated value and the current value differ + with regards to the predicate. To determin this requires that the predicate + expression be evaluated for the old and new values to be stored in the index + and then compared. To disable this behavior and avoid the overhead of + evaluating the expression during updates set + the <literal>expression_checks</literal> option to false for the table. + </para> + <para> You can increase the likelihood of sufficient page space for <acronym>HOT</acronym> updates by decreasing a table's <link diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c index 59fb53e7707..c081611926e 100644 --- a/src/backend/access/common/reloptions.c +++ b/src/backend/access/common/reloptions.c @@ -166,6 +166,15 @@ static relopt_bool boolRelOpts[] = }, true }, + { + { + "expression_checks", + "When disabled prevents checking expressions on indexes and predicates on partial indexes for changes that might influence heap-only tuple (HOT) updates.", + RELOPT_KIND_HEAP, + ShareUpdateExclusiveLock + }, + true + }, /* list terminator */ {{NULL}} }; @@ -1903,7 +1912,8 @@ default_reloptions(Datum reloptions, bool validate, relopt_kind kind) {"vacuum_truncate", RELOPT_TYPE_BOOL, offsetof(StdRdOptions, vacuum_truncate)}, {"vacuum_max_eager_freeze_failure_rate", RELOPT_TYPE_REAL, - offsetof(StdRdOptions, vacuum_max_eager_freeze_failure_rate)} + offsetof(StdRdOptions, vacuum_max_eager_freeze_failure_rate)}, + {"expression_checks", RELOPT_TYPE_BOOL, offsetof(StdRdOptions, expression_checks)} }; return (bytea *) build_reloptions(reloptions, validate, kind, diff --git a/src/backend/access/heap/README.HOT b/src/backend/access/heap/README.HOT index 74e407f375a..5bd61bd153c 100644 --- a/src/backend/access/heap/README.HOT +++ b/src/backend/access/heap/README.HOT @@ -36,7 +36,7 @@ HOT solves this problem for two restricted but useful special cases: First, where a tuple is repeatedly updated in ways that do not change its indexed columns. (Here, "indexed column" means any column referenced at all in an index definition, including for example columns that are -tested in a partial-index predicate but are not stored in the index.) +tested in a partial-index predicate, that has materially changed.) Second, where the modified columns are only used in indexes that do not contain tuple IDs, but maintain summaries of the indexed data by block. @@ -133,28 +133,34 @@ Note: we can use a "dead" line pointer for any DELETEd tuple, whether it was part of a HOT chain or not. This allows space reclamation in advance of running VACUUM for plain DELETEs as well as HOT updates. -The requirement for doing a HOT update is that indexes which point to -the root line pointer (and thus need to be cleaned up by VACUUM when the -tuple is dead) do not reference columns which are updated in that HOT -chain. Summarizing indexes (such as BRIN) are assumed to have no -references to individual tuples and thus are ignored when checking HOT -applicability. The updated columns are checked at execution time by -comparing the binary representation of the old and new values. We insist -on bitwise equality rather than using datatype-specific equality routines. -The main reason to avoid the latter is that there might be multiple -notions of equality for a datatype, and we don't know exactly which one -is relevant for the indexes at hand. We assume that bitwise equality -guarantees equality for all purposes. +The requirement for doing a HOT update is that indexes which point to the root +line pointer (and thus need to be cleaned up by VACUUM when the tuple is dead) +do not reference columns which are updated in that HOT chain. + +Summarizing indexes (such as BRIN) are assumed to have no references to +individual tuples and thus are ignored when checking HOT applicability. + +Expressions on indexes are evaluated and the results are used when check for +changes. This allows for the JSONB datatype to have HOT updates when the +indexed portion of the document are not modified. + +Partial index expressions are evaluated, HOT updates are allowed when the +updated index values do not satisfy the predicate. + +The updated columns are checked at execution time by comparing the binary +representation of the old and new values. We insist on bitwise equality rather +than using datatype-specific equality routines. The main reason to avoid the +latter is that there might be multiple notions of equality for a datatype, and +we don't know exactly which one is relevant for the indexes at hand. We assume +that bitwise equality guarantees equality for all purposes. If any columns that are included by non-summarizing indexes are updated, the HOT optimization is not applied, and the new tuple is inserted into all indexes of the table. If none of the updated columns are included in the table's indexes, the HOT optimization is applied and no indexes are updated. If instead the updated columns are only indexed by summarizing -indexes, the HOT optimization is applied, but the update is propagated to -all summarizing indexes. (Realistically, we only need to propagate the -update to the indexes that contain the updated values, but that is yet to -be implemented.) +indexes, the HOT optimization is applied, and the update is propagated to +all of the summarizing indexes. Abort Cases ----------- @@ -477,8 +483,9 @@ Heap-only tuple HOT-safe A proposed tuple update is said to be HOT-safe if it changes - none of the tuple's indexed columns. It will only become an - actual HOT update if we can find room on the same page for + none of the tuple's indexed columns or if the changes remain + outside of a partial index's predicate. It will only become + an actual HOT update if we can find room on the same page for the new tuple version. HOT update diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index fa7935a0ed3..46ce824c4ea 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -3164,12 +3164,13 @@ TM_Result heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, CommandId cid, Snapshot crosscheck, bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode, - TU_UpdateIndexes *update_indexes) + TU_UpdateIndexes *update_indexes, struct EState *estate) { TM_Result result; TransactionId xid = GetCurrentTransactionId(); Bitmapset *hot_attrs; Bitmapset *sum_attrs; + Bitmapset *exp_attrs; Bitmapset *key_attrs; Bitmapset *id_attrs; Bitmapset *interesting_attrs; @@ -3246,6 +3247,8 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, INDEX_ATTR_BITMAP_HOT_BLOCKING); sum_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_SUMMARIZED); + exp_attrs = RelationGetIndexAttrBitmap(relation, + INDEX_ATTR_BITMAP_EXPRESSION); key_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_KEY); id_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_IDENTITY_KEY); @@ -3311,6 +3314,7 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, bms_free(hot_attrs); bms_free(sum_attrs); + bms_free(exp_attrs); bms_free(key_attrs); bms_free(id_attrs); /* modified_attrs not yet initialized */ @@ -3612,6 +3616,7 @@ l2: bms_free(hot_attrs); bms_free(sum_attrs); + bms_free(exp_attrs); bms_free(key_attrs); bms_free(id_attrs); bms_free(modified_attrs); @@ -3928,25 +3933,30 @@ l2: if (newbuf == buffer) { + bool expression_checks = RelationGetExpressionChecks(relation); + /* * Since the new tuple is going into the same page, we might be able - * to do a HOT update. Check if any of the index columns have been - * changed. + * to do a HOT update. As a reminder, hot_attrs includes attributes + * used in expressions, but not attributes used by summarizing + * indexes. */ - if (!bms_overlap(modified_attrs, hot_attrs)) - { + if (!bms_overlap(modified_attrs, hot_attrs) || + (expression_checks == true && estate != NULL && exp_attrs && + bms_overlap(modified_attrs, exp_attrs) && + ExecIndexesExpressionsWereNotUpdated(relation, modified_attrs, + estate, &oldtup, newtup))) use_hot_update = true; - /* - * If none of the columns that are used in hot-blocking indexes - * were updated, we can apply HOT, but we do still need to check - * if we need to update the summarizing indexes, and update those - * indexes if the columns were updated, or we may fail to detect - * e.g. value bound changes in BRIN minmax indexes. - */ - if (bms_overlap(modified_attrs, sum_attrs)) - summarized_update = true; - } + /* + * If none of the columns that are used in hot-blocking indexes were + * updated, we can apply HOT, but we do still need to check if we need + * to update the summarizing indexes, and update those indexes if the + * columns were updated, or we may fail to detect e.g. value bound + * changes in BRIN minmax indexes. + */ + if (use_hot_update && bms_overlap(modified_attrs, sum_attrs)) + summarized_update = true; } else { @@ -4126,7 +4136,6 @@ l2: heap_freetuple(old_key_tuple); bms_free(hot_attrs); - bms_free(sum_attrs); bms_free(key_attrs); bms_free(id_attrs); bms_free(modified_attrs); @@ -4413,7 +4422,7 @@ simple_heap_update(Relation relation, ItemPointer otid, HeapTuple tup, result = heap_update(relation, otid, tup, GetCurrentCommandId(true), InvalidSnapshot, true /* wait for commit */ , - &tmfd, &lockmode, update_indexes); + &tmfd, &lockmode, update_indexes, NULL); switch (result) { case TM_SelfModified: diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index c0bec014154..8e9ab892d73 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -312,8 +312,8 @@ heapam_tuple_delete(Relation relation, ItemPointer tid, CommandId cid, static TM_Result heapam_tuple_update(Relation relation, ItemPointer otid, TupleTableSlot *slot, CommandId cid, Snapshot snapshot, Snapshot crosscheck, - bool wait, TM_FailureData *tmfd, - LockTupleMode *lockmode, TU_UpdateIndexes *update_indexes) + bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode, + TU_UpdateIndexes *update_indexes, EState *estate) { bool shouldFree = true; HeapTuple tuple = ExecFetchSlotHeapTuple(slot, true, &shouldFree); @@ -324,7 +324,7 @@ heapam_tuple_update(Relation relation, ItemPointer otid, TupleTableSlot *slot, tuple->t_tableOid = slot->tts_tableOid; result = heap_update(relation, otid, tuple, cid, crosscheck, wait, - tmfd, lockmode, update_indexes); + tmfd, lockmode, update_indexes, estate); ItemPointerCopy(&tuple->t_self, &slot->tts_tid); /* diff --git a/src/backend/access/table/tableam.c b/src/backend/access/table/tableam.c index e18a8f8250f..9feca8a296f 100644 --- a/src/backend/access/table/tableam.c +++ b/src/backend/access/table/tableam.c @@ -345,7 +345,7 @@ simple_table_tuple_update(Relation rel, ItemPointer otid, GetCurrentCommandId(true), snapshot, InvalidSnapshot, true /* wait for commit */ , - &tmfd, &lockmode, update_indexes); + &tmfd, &lockmode, update_indexes, NULL); switch (result) { diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index cdabf780244..78b4310e05f 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -2455,7 +2455,26 @@ BuildIndexInfo(Relation index) /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) + { + CompactAttribute *attr = TupleDescCompactAttr(RelationGetDescr(index), i); + ii->ii_IndexAttrNumbers[i] = indexStruct->indkey.values[i]; + ii->ii_IndexAttrs = + bms_add_member(ii->ii_IndexAttrs, + indexStruct->indkey.values[i] - FirstLowInvalidHeapAttributeNumber); + + ii->ii_IndexAttrLen[i] = attr->attlen; + if (attr->attbyval) + ii->ii_IndexAttrByVal = bms_add_member(ii->ii_IndexAttrByVal, i); + } + + /* collect attributes used in the expression, if one is present */ + if (ii->ii_Expressions) + pull_varattnos((Node *) ii->ii_Expressions, 1, &ii->ii_ExpressionAttrs); + + /* collect attributes used in the predicate, if one is present */ + if (ii->ii_Predicate) + pull_varattnos((Node *) ii->ii_Predicate, 1, &ii->ii_PredicateAttrs); /* fetch exclusion constraint info if any */ if (indexStruct->indisexclusion) diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c index 7c87f012c30..679124d1dfa 100644 --- a/src/backend/executor/execIndexing.c +++ b/src/backend/executor/execIndexing.c @@ -117,6 +117,8 @@ #include "utils/multirangetypes.h" #include "utils/rangetypes.h" #include "utils/snapmgr.h" +#include "utils/datum.h" +#include "utils/lsyscache.h" /* waitMode argument to check_exclusion_or_unique_constraint() */ typedef enum @@ -1089,6 +1091,9 @@ index_unchanged_by_update(ResultRelInfo *resultRelInfo, EState *estate, if (hasexpression) { + if (indexInfo->ii_IndexUnchanged) + return true; + indexInfo->ii_IndexUnchanged = false; return false; } @@ -1166,3 +1171,197 @@ ExecWithoutOverlapsNotEmpty(Relation rel, NameData attname, Datum attval, char t errmsg("empty WITHOUT OVERLAPS value found in column \"%s\" in relation \"%s\"", NameStr(attname), RelationGetRelationName(rel)))); } + +/* + * Determine if any index with an expression requires updating. + * + * This function will review all indexes with expressions to determine if the + * update from old to new tuple requires that the index be updated. This is + * used to determine if a HOT update is possible or not. + * + * Returns true iff none of the expression indexes require updating due to the + * change from old to new tuple or when both the old and new tuples do not + * satisfy the predicate of the index. + */ +bool +ExecIndexesExpressionsWereNotUpdated(Relation relation, + Bitmapset *modified_attrs, + EState *estate, + HeapTuple old_tuple, + HeapTuple new_tuple) +{ + ListCell *lc; + List *indexinfolist = RelationGetExprIndexInfoList(relation); + ExprContext *econtext = NULL; + TupleDesc tupledesc; + TupleTableSlot *old_tts = NULL; + TupleTableSlot *new_tts = NULL; + bool expression_checks = RelationGetExpressionChecks(relation); + bool changed = false; + +#ifndef USE_ASSERT_CHECKING + + /* + * Assume the index is changed when we don't have an estate context to use + * or the reloption is disabled. + */ + if (!expression_checks || estate == NULL) + return changed; +#endif + + Assert(expression_checks == true); + Assert(estate != NULL); + + /* + * Examine expression and partial indexs on this relation relative to the + * changes between old and new tuples. + */ + foreach(lc, indexinfolist) + { + IndexInfo *indexInfo = (IndexInfo *) lfirst(lc); + + /* + * If this is a partial index it has a predicate, evaluate the + * expression to determine if we need to include it or not. + */ + if (bms_overlap(indexInfo->ii_PredicateAttrs, modified_attrs)) + { + ExprState *pstate; + bool old_tuple_qualifies, + new_tuple_qualifies; + + /* Create these once, only if necessary, then reuse them. */ + if (!econtext) + { + econtext = GetPerTupleExprContext(estate); + tupledesc = RelationGetDescr(relation); + old_tts = MakeSingleTupleTableSlot(tupledesc, + &TTSOpsHeapTuple); + new_tts = MakeSingleTupleTableSlot(tupledesc, + &TTSOpsHeapTuple); + + ExecStoreHeapTuple(old_tuple, old_tts, InvalidBuffer); + ExecStoreHeapTuple(new_tuple, new_tts, InvalidBuffer); + } + + pstate = ExecPrepareQual(indexInfo->ii_Predicate, estate); + + /* + * Here the term "qualifies" means "satisfies the predicate + * condition of the partial index". + */ + econtext->ecxt_scantuple = old_tts; + old_tuple_qualifies = ExecQual(pstate, econtext); + + econtext->ecxt_scantuple = new_tts; + new_tuple_qualifies = ExecQual(pstate, econtext); + + /* + * If neither the old nor the new tuples satisfy the predicate we + * can be sure that this index doesn't need updating, continue to + * the next index. + */ + if ((new_tuple_qualifies == false) && (old_tuple_qualifies == false)) + continue; + + /* + * If there is a transition between indexed and not indexed, + * that's enough to require that this index is updated. + */ + if (new_tuple_qualifies != old_tuple_qualifies) + { + changed = true; + break; + } + + /* + * Otherwise the old and new values exist in the index, but did + * they get updated? We don't yet know, so proceed with the next + * statement in the loop to find out. + */ + } + + + /* + * Indexes with expressions may or may not have changed, it is + * impossible to know without exercising their expression and + * reviewing index tuple state for changes. This is a lot of work, + * but because all indexes on JSONB columns fall into this category it + * can be worth it to avoid index updates and remain on the HOT update + * path when possible. + */ + if (bms_overlap(indexInfo->ii_ExpressionAttrs, modified_attrs)) + { + Datum old_values[INDEX_MAX_KEYS]; + bool old_isnull[INDEX_MAX_KEYS]; + Datum new_values[INDEX_MAX_KEYS]; + bool new_isnull[INDEX_MAX_KEYS]; + + /* Create these once, only if necessary, then reuse them. */ + if (!econtext) + { + econtext = GetPerTupleExprContext(estate); + tupledesc = RelationGetDescr(relation); + old_tts = MakeSingleTupleTableSlot(tupledesc, + &TTSOpsHeapTuple); + new_tts = MakeSingleTupleTableSlot(tupledesc, + &TTSOpsHeapTuple); + + ExecStoreHeapTuple(old_tuple, old_tts, InvalidBuffer); + ExecStoreHeapTuple(new_tuple, new_tts, InvalidBuffer); + } + + indexInfo->ii_ExpressionsState = NIL; + + econtext->ecxt_scantuple = old_tts; + FormIndexDatum(indexInfo, + old_tts, + estate, + old_values, + old_isnull); + + econtext->ecxt_scantuple = new_tts; + FormIndexDatum(indexInfo, + new_tts, + estate, + new_values, + new_isnull); + + for (int i = 0; i < indexInfo->ii_NumIndexKeyAttrs; i++) + { + if (old_isnull[i] != new_isnull[i]) + { + changed = true; + break; + } + else if (!old_isnull[i]) + { + int16 elmlen = indexInfo->ii_IndexAttrLen[i]; + bool elmbyval = bms_is_member(i, indexInfo->ii_IndexAttrByVal); + + if (!datum_image_eq(old_values[i], new_values[i], + elmbyval, elmlen)) + { + changed = true; + break; + } + } + } + + /* Shortcut index_unchanged_by_update(), we know the answer. */ + indexInfo->ii_CheckedUnchanged = true; + indexInfo->ii_IndexUnchanged = !changed; + + if (changed) + break; + } + } + + if (econtext) + { + ExecDropSingleTupleTableSlot(old_tts); + ExecDropSingleTupleTableSlot(new_tts); + } + + return !changed; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index b0fe50075ad..d68f40cb998 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2283,7 +2283,8 @@ lreplace: estate->es_crosscheck_snapshot, true /* wait for commit */ , &context->tmfd, &updateCxt->lockmode, - &updateCxt->updateIndexes); + &updateCxt->updateIndexes, + estate); return result; } diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index 398114373e9..b5e47b3521e 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -64,6 +64,7 @@ #include "catalog/pg_type.h" #include "catalog/schemapg.h" #include "catalog/storage.h" +#include "catalog/index.h" #include "commands/policy.h" #include "commands/publicationcmds.h" #include "commands/trigger.h" @@ -5188,6 +5189,128 @@ RelationGetIndexPredicate(Relation relation) return result; } + /* + * RelationGetExprIndexInfoList -- get a list of IndexInfo for expression + * indexes on this relation + */ +List * +RelationGetExprIndexInfoList(Relation relation) +{ + ListCell *lc; + List *indexoidlist; + List *newindexoidlist; + List *result = NULL; + List *oldlist; + Oid relpkindex; + Oid relreplindex; + MemoryContext oldcxt; + + if (relation->rd_iiexprlist) + return list_copy(relation->rd_iiexprlist); + + /* Fast path if definitely no indexes */ + if (!RelationGetForm(relation)->relhasindex) + return NIL; + +restart: + indexoidlist = RelationGetIndexList(relation); + + /* Fall out if no indexes (but relhasindex was set) */ + if (indexoidlist == NIL) + return NIL; + + /* + * Copy the rd_pkindex and rd_replidindex values computed by + * RelationGetIndexList before proceeding. This is needed because a + * relcache flush could occur inside index_open below, resetting the + * fields managed by RelationGetIndexList. We need to do the work with + * stable values of these fields. + */ + relpkindex = relation->rd_pkindex; + relreplindex = relation->rd_replidindex; + + foreach(lc, indexoidlist) + { + Oid oid = lfirst_oid(lc); + Datum datum; + bool isnull; + Node *indexExpressions; + Node *indexPredicate; + Relation indexDesc; + IndexInfo *indexInfo; + + /* + * Extract index expressions and index predicate. Note: Don't use + * RelationGetIndexExpressions()/RelationGetIndexPredicate(), because + * those might run constant expressions evaluation, which needs a + * snapshot, which we might not have here. (Also, it's probably more + * sound to collect the bitmaps before any transformations that might + * eliminate columns, but the practical impact of this is limited.) + */ + indexDesc = index_open(oid, AccessShareLock); + datum = heap_getattr(indexDesc->rd_indextuple, Anum_pg_index_indexprs, + GetPgIndexDescriptor(), &isnull); + if (!isnull) + indexExpressions = stringToNode(TextDatumGetCString(datum)); + else + indexExpressions = NULL; + + datum = heap_getattr(indexDesc->rd_indextuple, Anum_pg_index_indpred, + GetPgIndexDescriptor(), &isnull); + if (!isnull) + indexPredicate = stringToNode(TextDatumGetCString(datum)); + else + indexPredicate = NULL; + + if (indexExpressions || indexPredicate) + { + oldcxt = MemoryContextSwitchTo(CacheMemoryContext); + indexInfo = BuildIndexInfo(indexDesc); + MemoryContextSwitchTo(oldcxt); + result = lappend(result, indexInfo); + } + + index_close(indexDesc, AccessShareLock); + } + + /* + * During one of the index_opens in the above loop, we might have received + * a relcache flush event on this relcache entry, which might have been + * signaling a change in the rel's index list. If so, we'd better start + * over to ensure we deliver up-to-date IndexInfo. + */ + newindexoidlist = RelationGetIndexList(relation); + if (equal(indexoidlist, newindexoidlist) && + relpkindex == relation->rd_pkindex && + relreplindex == relation->rd_replidindex) + { + /* Still the same index set, so proceed */ + list_free(newindexoidlist); + list_free(indexoidlist); + } + else + { + /* Gotta do it over ... might as well not leak memory */ + list_free(newindexoidlist); + list_free(indexoidlist); + list_free(result); + + goto restart; + } + + /* Now save a copy of the completed list in the relcache entry. */ + oldcxt = MemoryContextSwitchTo(CacheMemoryContext); + oldlist = relation->rd_iiexprlist; + relation->rd_iiexprlist = list_copy(result); + MemoryContextSwitchTo(oldcxt); + + /* Don't leak the old list, if there is one */ + if (oldlist) + list_free(oldlist); + + return result; +} + /* * RelationGetIndexAttrBitmap -- get a bitmap of index attribute numbers * @@ -5229,7 +5352,11 @@ RelationGetIndexAttrBitmap(Relation relation, IndexAttrBitmapKind attrKind) Bitmapset *pkindexattrs; /* columns in the primary index */ Bitmapset *idindexattrs; /* columns in the replica identity */ Bitmapset *hotblockingattrs; /* columns with HOT blocking indexes */ + Bitmapset *hotblockingexprattrs; /* as above, but only those in + * expressions */ Bitmapset *summarizedattrs; /* columns with summarizing indexes */ + Bitmapset *summarizedexprattrs; /* as above, but only those in + * expressions */ List *indexoidlist; List *newindexoidlist; Oid relpkindex; @@ -5252,6 +5379,8 @@ RelationGetIndexAttrBitmap(Relation relation, IndexAttrBitmapKind attrKind) return bms_copy(relation->rd_hotblockingattr); case INDEX_ATTR_BITMAP_SUMMARIZED: return bms_copy(relation->rd_summarizedattr); + case INDEX_ATTR_BITMAP_EXPRESSION: + return bms_copy(relation->rd_expressionattr); default: elog(ERROR, "unknown attrKind %u", attrKind); } @@ -5295,7 +5424,9 @@ restart: pkindexattrs = NULL; idindexattrs = NULL; hotblockingattrs = NULL; + hotblockingexprattrs = NULL; summarizedattrs = NULL; + summarizedexprattrs = NULL; foreach(l, indexoidlist) { Oid indexOid = lfirst_oid(l); @@ -5309,6 +5440,7 @@ restart: bool isPK; /* primary key */ bool isIDKey; /* replica identity index */ Bitmapset **attrs; + Bitmapset **exprattrs; indexDesc = index_open(indexOid, AccessShareLock); @@ -5352,14 +5484,20 @@ restart: * decide which bitmap we'll update in the following loop. */ if (indexDesc->rd_indam->amsummarizing) + { attrs = &summarizedattrs; + exprattrs = &summarizedexprattrs; + } else + { attrs = &hotblockingattrs; + exprattrs = &hotblockingexprattrs; + } /* Collect simple attribute references */ for (i = 0; i < indexDesc->rd_index->indnatts; i++) { - int attrnum = indexDesc->rd_index->indkey.values[i]; + int attridx = indexDesc->rd_index->indkey.values[i]; /* * Since we have covering indexes with non-key columns, we must @@ -5375,30 +5513,28 @@ restart: * key or identity key. Hence we do not include them into * uindexattrs, pkindexattrs and idindexattrs bitmaps. */ - if (attrnum != 0) + if (attridx != 0) { - *attrs = bms_add_member(*attrs, - attrnum - FirstLowInvalidHeapAttributeNumber); + AttrNumber attrnum = attridx - FirstLowInvalidHeapAttributeNumber; + + *attrs = bms_add_member(*attrs, attrnum); if (isKey && i < indexDesc->rd_index->indnkeyatts) - uindexattrs = bms_add_member(uindexattrs, - attrnum - FirstLowInvalidHeapAttributeNumber); + uindexattrs = bms_add_member(uindexattrs, attrnum); if (isPK && i < indexDesc->rd_index->indnkeyatts) - pkindexattrs = bms_add_member(pkindexattrs, - attrnum - FirstLowInvalidHeapAttributeNumber); + pkindexattrs = bms_add_member(pkindexattrs, attrnum); if (isIDKey && i < indexDesc->rd_index->indnkeyatts) - idindexattrs = bms_add_member(idindexattrs, - attrnum - FirstLowInvalidHeapAttributeNumber); + idindexattrs = bms_add_member(idindexattrs, attrnum); } } /* Collect all attributes used in expressions, too */ - pull_varattnos(indexExpressions, 1, attrs); + pull_varattnos(indexExpressions, 1, exprattrs); /* Collect all attributes in the index predicate, too */ - pull_varattnos(indexPredicate, 1, attrs); + pull_varattnos(indexPredicate, 1, exprattrs); index_close(indexDesc, AccessShareLock); } @@ -5427,11 +5563,27 @@ restart: bms_free(pkindexattrs); bms_free(idindexattrs); bms_free(hotblockingattrs); + bms_free(hotblockingexprattrs); bms_free(summarizedattrs); + bms_free(summarizedexprattrs); goto restart; } + /* {expression-only columns} = {expression columns} - {direct columns} */ + hotblockingexprattrs = bms_del_members(hotblockingexprattrs, + hotblockingattrs); + /* {hot-blocking columns} = {direct columns} + {expression-only columns} */ + hotblockingattrs = bms_add_members(hotblockingattrs, + hotblockingexprattrs); + + /* {summarized-only columns} = {summarized columns} - {direct columns} */ + summarizedexprattrs = bms_del_members(summarizedexprattrs, + summarizedattrs); + /* {summarized columns} = {direct columns} + {summarized-only columns} */ + summarizedattrs = bms_add_members(summarizedattrs, + summarizedexprattrs); + /* Don't leak the old values of these bitmaps, if any */ relation->rd_attrsvalid = false; bms_free(relation->rd_keyattr); @@ -5444,6 +5596,8 @@ restart: relation->rd_hotblockingattr = NULL; bms_free(relation->rd_summarizedattr); relation->rd_summarizedattr = NULL; + bms_free(relation->rd_expressionattr); + relation->rd_expressionattr = NULL; /* * Now save copies of the bitmaps in the relcache entry. We intentionally @@ -5458,6 +5612,7 @@ restart: relation->rd_idattr = bms_copy(idindexattrs); relation->rd_hotblockingattr = bms_copy(hotblockingattrs); relation->rd_summarizedattr = bms_copy(summarizedattrs); + relation->rd_expressionattr = bms_copy(hotblockingexprattrs); relation->rd_attrsvalid = true; MemoryContextSwitchTo(oldcxt); @@ -5474,6 +5629,8 @@ restart: return hotblockingattrs; case INDEX_ATTR_BITMAP_SUMMARIZED: return summarizedattrs; + case INDEX_ATTR_BITMAP_EXPRESSION: + return hotblockingexprattrs; default: elog(ERROR, "unknown attrKind %u", attrKind); return NULL; diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index eb8bc128720..f0a8286a25b 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -2992,7 +2992,7 @@ match_previous_words(int pattern_id, COMPLETE_WITH("("); /* ALTER TABLESPACE <foo> SET|RESET ( */ else if (Matches("ALTER", "TABLESPACE", MatchAny, "SET|RESET", "(")) - COMPLETE_WITH("seq_page_cost", "random_page_cost", + COMPLETE_WITH("seq_page_cost", "random_page_cost", "expression_checks", "effective_io_concurrency", "maintenance_io_concurrency"); /* ALTER TEXT SEARCH */ diff --git a/src/include/access/genam.h b/src/include/access/genam.h index 1be8739573f..91af1e8238d 100644 --- a/src/include/access/genam.h +++ b/src/include/access/genam.h @@ -194,7 +194,7 @@ extern bool index_can_return(Relation indexRelation, int attno); extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum, uint16 procnum); extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum, - uint16 procnum); + uint16 procnum); extern void index_store_float8_orderby_distances(IndexScanDesc scan, Oid *orderByTypes, IndexOrderByDistance *distances, diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 1640d9c32f7..e12da1934e9 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -21,6 +21,7 @@ #include "access/skey.h" #include "access/table.h" /* for backward compatibility */ #include "access/tableam.h" +#include "executor/executor.h" #include "nodes/lockoptions.h" #include "nodes/primnodes.h" #include "storage/bufpage.h" @@ -339,7 +340,7 @@ extern TM_Result heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, CommandId cid, Snapshot crosscheck, bool wait, struct TM_FailureData *tmfd, LockTupleMode *lockmode, - TU_UpdateIndexes *update_indexes); + TU_UpdateIndexes *update_indexes, struct EState *estate); extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple, CommandId cid, LockTupleMode mode, LockWaitPolicy wait_policy, bool follow_updates, diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h index 131c050c15f..62ed6592b85 100644 --- a/src/include/access/tableam.h +++ b/src/include/access/tableam.h @@ -38,6 +38,7 @@ struct IndexInfo; struct SampleScanState; struct VacuumParams; struct ValidateIndexState; +struct EState; /* * Bitmask values for the flags argument to the scan_begin callback. @@ -550,7 +551,8 @@ typedef struct TableAmRoutine bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode, - TU_UpdateIndexes *update_indexes); + TU_UpdateIndexes *update_indexes, + struct EState *estate); /* see table_tuple_lock() for reference about parameters */ TM_Result (*tuple_lock) (Relation rel, @@ -1541,12 +1543,12 @@ static inline TM_Result table_tuple_update(Relation rel, ItemPointer otid, TupleTableSlot *slot, CommandId cid, Snapshot snapshot, Snapshot crosscheck, bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode, - TU_UpdateIndexes *update_indexes) + TU_UpdateIndexes *update_indexes, struct EState *estate) { return rel->rd_tableam->tuple_update(rel, otid, slot, cid, snapshot, crosscheck, - wait, tmfd, - lockmode, update_indexes); + wait, tmfd, lockmode, + update_indexes, estate); } /* diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 30e2a82346f..343d04fff57 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -661,6 +661,11 @@ extern void check_exclusion_constraint(Relation heap, Relation index, ItemPointer tupleid, const Datum *values, const bool *isnull, EState *estate, bool newIndex); +extern bool ExecIndexesExpressionsWereNotUpdated(Relation relation, + Bitmapset *modified_attrs, + EState *estate, + HeapTuple old_tuple, + HeapTuple new_tuple); /* * prototypes from functions in execReplication.c diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index 2625d7e8222..aca9371dccd 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -159,12 +159,15 @@ typedef struct ExprState * * NumIndexAttrs total number of columns in this index * NumIndexKeyAttrs number of key columns in index + * IndexAttrs bitmap of index attributes * IndexAttrNumbers underlying-rel attribute numbers used as keys * (zeroes indicate expressions). It also contains * info about included columns. * Expressions expr trees for expression entries, or NIL if none + * ExpressionAttrs bitmap of attributes used within the expression * ExpressionsState exec state for expressions, or NIL if none * Predicate partial-index predicate, or NIL if none + * PredicateAttrs bitmap of attributes used within the predicate * PredicateState exec state for predicate, or NIL if none * ExclusionOps Per-column exclusion operators, or NULL if none * ExclusionProcs Underlying function OIDs for ExclusionOps @@ -183,6 +186,7 @@ typedef struct ExprState * ParallelWorkers # of workers requested (excludes leader) * Am Oid of index AM * AmCache private cache area for index AM + * OpClassDataTypes operator class data types * Context memory context holding this IndexInfo * * ii_Concurrent, ii_BrokenHotChain, and ii_ParallelWorkers are used only @@ -194,10 +198,15 @@ typedef struct IndexInfo NodeTag type; int ii_NumIndexAttrs; /* total number of columns in index */ int ii_NumIndexKeyAttrs; /* number of key columns in index */ + Bitmapset *ii_IndexAttrs; AttrNumber ii_IndexAttrNumbers[INDEX_MAX_KEYS]; + uint16 ii_IndexAttrLen[INDEX_MAX_KEYS]; + Bitmapset *ii_IndexAttrByVal; List *ii_Expressions; /* list of Expr */ + Bitmapset *ii_ExpressionAttrs; List *ii_ExpressionsState; /* list of ExprState */ List *ii_Predicate; /* list of Expr */ + Bitmapset *ii_PredicateAttrs; ExprState *ii_PredicateState; Oid *ii_ExclusionOps; /* array with one entry per column */ Oid *ii_ExclusionProcs; /* array with one entry per column */ diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index db3e504c3d2..3eba254a366 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -164,6 +164,11 @@ typedef struct RelationData Bitmapset *rd_idattr; /* included in replica identity index */ Bitmapset *rd_hotblockingattr; /* cols blocking HOT update */ Bitmapset *rd_summarizedattr; /* cols indexed by summarizing indexes */ + Bitmapset *rd_expressionattr; /* indexed cols referenced by expressions */ + + /* data managed by RelationGetExprIndexInfoList: */ + List *rd_iiexprlist; /* list of IndexInfo for indexes with + * expressions */ PublicationDesc *rd_pubdesc; /* publication descriptor, or NULL */ @@ -344,6 +349,7 @@ typedef struct StdRdOptions int parallel_workers; /* max number of parallel workers */ StdRdOptIndexCleanup vacuum_index_cleanup; /* controls index vacuuming */ bool vacuum_truncate; /* enables vacuum to truncate a relation */ + bool expression_checks; /* use expression to checks for changes */ /* * Fraction of pages in a relation that vacuum can eagerly scan and fail @@ -405,6 +411,14 @@ typedef struct StdRdOptions ((relation)->rd_options ? \ ((StdRdOptions *) (relation)->rd_options)->parallel_workers : (defaultpw)) +/* + * RelationGetExpressionChecks + * Returns the relation's expression_checks reloption setting. + */ +#define RelationGetExpressionChecks(relation) \ + ((relation)->rd_options ? \ + ((StdRdOptions *) (relation)->rd_options)->expression_checks : true) + /* ViewOptions->check_option values */ typedef enum ViewOptCheckOption { diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h index a7c55db339e..56524a73399 100644 --- a/src/include/utils/relcache.h +++ b/src/include/utils/relcache.h @@ -52,6 +52,7 @@ extern List *RelationGetIndexExpressions(Relation relation); extern List *RelationGetDummyIndexExpressions(Relation relation); extern List *RelationGetIndexPredicate(Relation relation); extern bytea **RelationGetIndexAttOptions(Relation relation, bool copy); +extern List *RelationGetExprIndexInfoList(Relation relation); /* * Which set of columns to return by RelationGetIndexAttrBitmap. @@ -63,6 +64,7 @@ typedef enum IndexAttrBitmapKind INDEX_ATTR_BITMAP_IDENTITY_KEY, INDEX_ATTR_BITMAP_HOT_BLOCKING, INDEX_ATTR_BITMAP_SUMMARIZED, + INDEX_ATTR_BITMAP_EXPRESSION, } IndexAttrBitmapKind; extern Bitmapset *RelationGetIndexAttrBitmap(Relation relation, diff --git a/src/test/regress/expected/heap_hot_updates.out b/src/test/regress/expected/heap_hot_updates.out new file mode 100644 index 00000000000..2ad6e5418e9 --- /dev/null +++ b/src/test/regress/expected/heap_hot_updates.out @@ -0,0 +1,586 @@ +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table will have two columns and two indexes, one on the primary key +-- id and one on the expression (info->>'name'). That means that the indexed +-- attributes are 'id' and 'info'. +create table keyvalue(id integer primary key, info jsonb); +create index nameindex on keyvalue((info->>'name')); +insert into keyvalue values (1, '{"name": "john", "data": "some data"}'); +-- Disable expression checks. +ALTER TABLE keyvalue SET (expression_checks = false); +SELECT reloptions FROM pg_class WHERE relname = 'keyvalue'; + reloptions +--------------------------- + {expression_checks=false} +(1 row) + +-- While the indexed attribute "name" is unchanged we've disabled expression +-- checks so this update should not go HOT as the system can't determine if +-- the indexed attribute has changed without evaluating the expression. +update keyvalue set info='{"name": "john", "data": "something else"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 0 row + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Re-enable expression checks. +ALTER TABLE keyvalue SET (expression_checks = true); +SELECT reloptions FROM pg_class WHERE relname = 'keyvalue'; + reloptions +-------------------------- + {expression_checks=true} +(1 row) + +-- The indexed attribute "name" with value "john" is unchanged, expect a HOT update. +update keyvalue set info='{"name": "john", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 1 row + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- The following update changes the indexed attribute "name", this should not be a HOT update. +update keyvalue set info='{"name": "smith", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Now, this update does not change the indexed attribute "name" from "smith", this should be HOT. +update keyvalue set info='{"name": "smith", "data": "some more data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 2 rows now + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 2 +(1 row) + +drop table keyvalue; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table is the same as the previous one but it has a third index. The +-- index 'colindex' isn't an expression index, it indexes the entire value +-- in the info column. There are still only two indexed attributes for this +-- relation, the same two as before. The presence of an index on the entire +-- value of the info column should prevent HOT updates for any updates to any +-- portion of JSONB content in that column. +create table keyvalue(id integer primary key, info jsonb); +create index nameindex on keyvalue((info->>'name')); +create index colindex on keyvalue(info); +insert into keyvalue values (1, '{"name": "john", "data": "some data"}'); +-- This update doesn't change the value of the expression index, but it does +-- change the content of the info column and so should not be HOT because the +-- indexed value changed as a result of the update. +update keyvalue set info='{"name": "john", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 0 rows + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +drop table keyvalue; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- The table has one column docs and two indexes. They are both expression +-- indexes referencing the same column attribute (docs) but one is a partial +-- index. +CREATE TABLE ex (docs JSONB) WITH (fillfactor = 60); +INSERT INTO ex (docs) VALUES ('{"a": 0, "b": 0}'); +INSERT INTO ex (docs) SELECT jsonb_build_object('b', n) FROM generate_series(100, 10000) as n; +CREATE INDEX idx_ex_a ON ex ((docs->>'a')); +CREATE INDEX idx_ex_b ON ex ((docs->>'b')) WHERE (docs->>'b')::numeric > 9; +-- We're using BTREE indexes and for this test we want to make sure that they remain +-- in sync with changes to our relation. Force the choice of index scans below so +-- that we know we're checking the index's understanding of what values should be +-- in the index or not. +SET SESSION enable_seqscan = OFF; +SET SESSION enable_bitmapscan = OFF; +-- Leave 'a' unchanged but modify 'b' to a value outside of the index predicate. +-- This should be a HOT update because neither index is changed. +UPDATE ex SET docs = jsonb_build_object('a', 0, 'b', 1) WHERE (docs->>'a')::numeric = 0; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 row + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Let's check to make sure that the index does not contain a value for 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + QUERY PLAN +-------------------------------------------------------------- + Index Scan using idx_ex_b on ex + Filter: (((docs ->> 'b'::text))::numeric < '100'::numeric) +(2 rows) + +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + docs +------ +(0 rows) + +-- Leave 'a' unchanged but modify 'b' to a value within the index predicate. +-- This represents a change for field 'b' from unindexed to indexed and so +-- this should not take the HOT path. +UPDATE ex SET docs = jsonb_build_object('a', 0, 'b', 10) WHERE (docs->>'a')::numeric = 0; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Let's check to make sure that the index contains the new value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + QUERY PLAN +-------------------------------------------------------------- + Index Scan using idx_ex_b on ex + Filter: (((docs ->> 'b'::text))::numeric < '100'::numeric) +(2 rows) + +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + docs +------------------- + {"a": 0, "b": 10} +(1 row) + +-- This update modifies the value of 'a', an indexed field, so it also cannot +-- be a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 1, 'b', 10) WHERE (docs->>'b')::numeric = 10; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- This update changes both 'a' and 'b' to new values that require index updates, +-- this cannot use the HOT path. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 12) WHERE (docs->>'b')::numeric = 10; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Let's check to make sure that the index contains the new value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + QUERY PLAN +-------------------------------------------------------------- + Index Scan using idx_ex_b on ex + Filter: (((docs ->> 'b'::text))::numeric < '100'::numeric) +(2 rows) + +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + docs +------------------- + {"a": 2, "b": 12} +(1 row) + +-- This update changes 'b' to a value outside its predicate requiring that +-- we remove it from the index. That's a transition that can't be done +-- during a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 1) WHERE (docs->>'b')::numeric = 12; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Let's check to make sure that the index no longer contains the value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + QUERY PLAN +-------------------------------------------------------------- + Index Scan using idx_ex_b on ex + Filter: (((docs ->> 'b'::text))::numeric < '100'::numeric) +(2 rows) + +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + docs +------ +(0 rows) + +-- Disable expression checks. +ALTER TABLE ex SET (expression_checks = false); +SELECT reloptions FROM pg_class WHERE relname = 'ex'; + reloptions +----------------------------------------- + {fillfactor=60,expression_checks=false} +(1 row) + +-- This update changes 'b' to a value within its predicate just like it +-- previous value, which would allow for a HOT update but with expression +-- checks disabled we can't determine that so this should not be a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 2) WHERE (docs->>'b')::numeric = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Let's make sure we're recording HOT updates for our 'ex' relation properly in the system +-- table pg_stat_user_tables. Note that statistics are stored within a transaction context +-- first (xact) and then later into the global statistics for a relation, so first we need +-- to ensure pending stats are flushed. +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT + c.relname AS table_name, + -- Transaction statistics + pg_stat_get_xact_tuples_updated(c.oid) AS xact_updates, + pg_stat_get_xact_tuples_hot_updated(c.oid) AS xact_hot_updates, + ROUND(( + pg_stat_get_xact_tuples_hot_updated(c.oid)::float / + NULLIF(pg_stat_get_xact_tuples_updated(c.oid), 0) * 100 + )::numeric, 2) AS xact_hot_update_percentage, + -- Cumulative statistics + s.n_tup_upd AS total_updates, + s.n_tup_hot_upd AS hot_updates, + ROUND(( + s.n_tup_hot_upd::float / + NULLIF(s.n_tup_upd, 0) * 100 + )::numeric, 2) AS total_hot_update_percentage +FROM pg_class c +LEFT JOIN pg_stat_user_tables s ON c.relname = s.relname +WHERE c.relname = 'ex' +AND c.relnamespace = 'public'::regnamespace; + table_name | xact_updates | xact_hot_updates | xact_hot_update_percentage | total_updates | hot_updates | total_hot_update_percentage +------------+--------------+------------------+----------------------------+---------------+-------------+----------------------------- + ex | 0 | 0 | | 6 | 1 | 16.67 +(1 row) + +-- expect: 5 xact updates with 1 xact hot update and no cumulative updates as yet +SET SESSION enable_seqscan = ON; +SET SESSION enable_bitmapscan = ON; +DROP TABLE ex; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table has a single column 'email' and a unique constraint on it that +-- should preclude HOT updates. +CREATE TABLE users ( + user_id serial primary key, + name VARCHAR(255) NOT NULL, + email VARCHAR(255) NOT NULL, + EXCLUDE USING btree (lower(email) WITH =) +); +-- Add some data to the table and then update it in ways that should and should +-- not be HOT updates. +INSERT INTO users (name, email) VALUES +('user1', '[email protected]'), +('user2', '[email protected]'), +('taken', '[email protected]'), +('you', '[email protected]'), +('taken', '[email protected]'); +-- Should fail because of the unique constraint on the email column. +UPDATE users SET email = '[email protected]' WHERE email = '[email protected]'; +ERROR: conflicting key value violates exclusion constraint "users_lower_excl" +DETAIL: Key (lower(email::text))=([email protected]) conflicts with existing key (lower(email::text))=([email protected]). +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 0 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Should succeed because the email column is not being updated and should go HOT. +UPDATE users SET name = 'foo' WHERE email = '[email protected]'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 a single new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Create a partial index on the email column, updates +CREATE INDEX idx_users_email_no_example ON users (lower(email)) WHERE lower(email) LIKE '%@example.com%'; +-- An update that changes the email column but not the indexed portion of it and falls outside the constraint. +-- Shouldn't be a HOT update because of the exclusion constraint. +UPDATE users SET email = '[email protected]' WHERE name = 'you'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- An update that changes the email column but not the indexed portion of it and falls within the constraint. +-- Again, should fail constraint and fail to be a HOT update. +UPDATE users SET email = '[email protected]' WHERE name = 'you'; +ERROR: conflicting key value violates exclusion constraint "users_lower_excl" +DETAIL: Key (lower(email::text))=([email protected]) conflicts with existing key (lower(email::text))=([email protected]). +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +DROP TABLE users; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- Another test of constraints spoiling HOT updates, this time with a range. +CREATE TABLE events ( + id serial primary key, + name VARCHAR(255) NOT NULL, + event_time tstzrange, + constraint no_screening_time_overlap exclude using gist ( + event_time WITH && + ) +); +-- Add two non-overlapping events. +INSERT INTO events (id, event_time, name) +VALUES + (1, '["2023-01-01 19:00:00", "2023-01-01 20:45:00"]', 'event1'), + (2, '["2023-01-01 21:00:00", "2023-01-01 21:45:00"]', 'event2'); +-- Update the first event to overlap with the second, should fail the constraint and not be HOT. +UPDATE events SET event_time = '["2023-01-01 20:00:00", "2023-01-01 21:45:00"]' WHERE id = 1; +ERROR: conflicting key value violates exclusion constraint "no_screening_time_overlap" +DETAIL: Key (event_time)=(["Sun Jan 01 20:00:00 2023 PST","Sun Jan 01 21:45:00 2023 PST"]) conflicts with existing key (event_time)=(["Sun Jan 01 21:00:00 2023 PST","Sun Jan 01 21:45:00 2023 PST"]). +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 0 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Update the first event to not overlap with the second, again not HOT due to the constraint. +UPDATE events SET event_time = '["2023-01-01 22:00:00", "2023-01-01 22:45:00"]' WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 0 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Update the first event to not overlap with the second, this time we're HOT because we don't overlap with the constraint. +UPDATE events SET name = 'new name here' WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 1 one new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +DROP TABLE events; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- A test to ensure that summarizing indexes are not updated when they don't +-- change, but are updated when they do while not prefent HOT updates. +CREATE TABLE ex (att1 JSONB, att2 text) WITH (fillfactor = 60); +CREATE INDEX expression ON ex USING btree((att1->'data')); +CREATE INDEX summarizing ON ex USING BRIN(att2); +INSERT INTO ex VALUES ('{"data": []}', 'nothing special'); +-- Update the unindexed value of att1, this should be a HOT update and not +-- update the summarizing index. +UPDATE ex SET att1 = att1 || '{"status": "checkmate"}'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 one new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Update the indexed value of att2, a summarized value, this is a summarized +-- only update and should use the HOT path while still triggering an update to +-- the summarizing BRIN index. +UPDATE ex SET att2 = 'special indeed'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 2 one new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 2 +(1 row) + +-- Update to att1 doesn't change the indexed value while the update to att2 does, +-- this again is a summarized only update and should use the HOT path as well as +-- trigger an update to the BRIN index. +UPDATE ex SET att1 = att1 || '{"status": "checkmate!"}', att2 = 'special, so special'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 3 one new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 3 +(1 row) + +-- This updates both indexes, the expression index on att1 and the summarizing +-- index on att2. This should not be a HOT update because there are modified +-- indexes and only some are summarized, not all. This should force all +-- indexes to be updated. +UPDATE ex SET att1 = att1 || '{"data": [1,2,3]}', att2 = 'do you want to play a game?'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 3 both indexes updated + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 3 +(1 row) + +DROP TABLE ex; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This test is for a table with a custom type and a custom operators on +-- the BTREE index. The question is, when comparing values for equality +-- to determine if there are changes on the index or not... shouldn't we +-- be using the custom operators? +-- Create a type +CREATE TYPE my_custom_type AS (val int); +-- Comparison functions (returns boolean) +CREATE FUNCTION my_custom_lt(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val < b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +CREATE FUNCTION my_custom_le(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val <= b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +CREATE FUNCTION my_custom_eq(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val = b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +CREATE FUNCTION my_custom_ge(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val >= b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +CREATE FUNCTION my_custom_gt(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val > b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +CREATE FUNCTION my_custom_ne(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val != b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +-- Comparison function (returns -1, 0, 1) +CREATE FUNCTION my_custom_cmp(a my_custom_type, b my_custom_type) RETURNS int AS $$ +BEGIN + IF a.val < b.val THEN + RETURN -1; + ELSIF a.val > b.val THEN + RETURN 1; + ELSE + RETURN 0; + END IF; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +-- Create the operators +CREATE OPERATOR < ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_lt, + COMMUTATOR = >, + NEGATOR = >= +); +CREATE OPERATOR <= ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_le, + COMMUTATOR = >=, + NEGATOR = > +); +CREATE OPERATOR = ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_eq, + COMMUTATOR = =, + NEGATOR = <> +); +CREATE OPERATOR >= ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_ge, + COMMUTATOR = <=, + NEGATOR = < +); +CREATE OPERATOR > ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_gt, + COMMUTATOR = <, + NEGATOR = <= +); +CREATE OPERATOR <> ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_ne, + COMMUTATOR = <>, + NEGATOR = = +); +-- Create the operator class (including the support function) +CREATE OPERATOR CLASS my_custom_ops + DEFAULT FOR TYPE my_custom_type USING btree AS + OPERATOR 1 <, + OPERATOR 2 <=, + OPERATOR 3 =, + OPERATOR 4 >=, + OPERATOR 5 >, + FUNCTION 1 my_custom_cmp(my_custom_type, my_custom_type); +-- Create the table +CREATE TABLE my_table ( + id int, + custom_val my_custom_type +); +-- Insert some data +INSERT INTO my_table (id, custom_val) VALUES +(1, ROW(3)::my_custom_type), +(2, ROW(1)::my_custom_type), +(3, ROW(4)::my_custom_type), +(4, ROW(2)::my_custom_type); +-- Create a function to use when indexing +CREATE OR REPLACE FUNCTION abs_val(val my_custom_type) RETURNS int AS $$ +BEGIN + RETURN abs(val.val); +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +-- Create the index +CREATE INDEX idx_custom_val_abs ON my_table (abs_val(custom_val)); +-- Update 1 +UPDATE my_table SET custom_val = ROW(5)::my_custom_type WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Update 2 +UPDATE my_table SET custom_val = ROW(0)::my_custom_type WHERE custom_val < ROW(3)::my_custom_type; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Update 3 +UPDATE my_table SET custom_val = ROW(6)::my_custom_type WHERE id = 3; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Update 4 +UPDATE my_table SET id = 5 WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Query using the index +EXPLAIN (COSTS OFF) SELECT * FROM my_table WHERE abs_val(custom_val) = 6; + QUERY PLAN +------------------------------------- + Seq Scan on my_table + Filter: (abs_val(custom_val) = 6) +(2 rows) + +SELECT * FROM my_table WHERE abs_val(custom_val) = 6; + id | custom_val +----+------------ + 3 | (6) +(1 row) + +-- Clean up +DROP TABLE my_table CASCADE; +DROP OPERATOR CLASS my_custom_ops USING btree CASCADE; +DROP OPERATOR < (my_custom_type, my_custom_type); +DROP OPERATOR <= (my_custom_type, my_custom_type); +DROP OPERATOR = (my_custom_type, my_custom_type); +DROP OPERATOR >= (my_custom_type, my_custom_type); +DROP OPERATOR > (my_custom_type, my_custom_type); +DROP OPERATOR <> (my_custom_type, my_custom_type); +DROP FUNCTION my_custom_lt(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_le(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_eq(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_ge(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_gt(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_ne(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_cmp(my_custom_type, my_custom_type); +DROP FUNCTION abs_val(my_custom_type); +DROP TYPE my_custom_type CASCADE; diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e63ee2cf2bb..f9def3d93aa 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -68,6 +68,11 @@ test: select_into select_distinct select_distinct_on select_implicit select_havi # ---------- test: brin gin gist spgist privileges init_privs security_label collate matview lock replica_identity rowsecurity object_address tablesample groupingsets drop_operator password identity generated_stored join_hash +# ---------- +# Another group of parallel tests +# ---------- +test: heap_hot_updates + # ---------- # Additional BRIN tests # ---------- diff --git a/src/test/regress/sql/heap_hot_updates.sql b/src/test/regress/sql/heap_hot_updates.sql new file mode 100644 index 00000000000..7728075a7ae --- /dev/null +++ b/src/test/regress/sql/heap_hot_updates.sql @@ -0,0 +1,440 @@ +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table will have two columns and two indexes, one on the primary key +-- id and one on the expression (info->>'name'). That means that the indexed +-- attributes are 'id' and 'info'. +create table keyvalue(id integer primary key, info jsonb); +create index nameindex on keyvalue((info->>'name')); +insert into keyvalue values (1, '{"name": "john", "data": "some data"}'); + +-- Disable expression checks. +ALTER TABLE keyvalue SET (expression_checks = false); +SELECT reloptions FROM pg_class WHERE relname = 'keyvalue'; + +-- While the indexed attribute "name" is unchanged we've disabled expression +-- checks so this update should not go HOT as the system can't determine if +-- the indexed attribute has changed without evaluating the expression. +update keyvalue set info='{"name": "john", "data": "something else"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 0 row + +-- Re-enable expression checks. +ALTER TABLE keyvalue SET (expression_checks = true); +SELECT reloptions FROM pg_class WHERE relname = 'keyvalue'; + +-- The indexed attribute "name" with value "john" is unchanged, expect a HOT update. +update keyvalue set info='{"name": "john", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 1 row + +-- The following update changes the indexed attribute "name", this should not be a HOT update. +update keyvalue set info='{"name": "smith", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 1 no new HOT updates + +-- Now, this update does not change the indexed attribute "name" from "smith", this should be HOT. +update keyvalue set info='{"name": "smith", "data": "some more data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 2 rows now +drop table keyvalue; + + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table is the same as the previous one but it has a third index. The +-- index 'colindex' isn't an expression index, it indexes the entire value +-- in the info column. There are still only two indexed attributes for this +-- relation, the same two as before. The presence of an index on the entire +-- value of the info column should prevent HOT updates for any updates to any +-- portion of JSONB content in that column. +create table keyvalue(id integer primary key, info jsonb); +create index nameindex on keyvalue((info->>'name')); +create index colindex on keyvalue(info); +insert into keyvalue values (1, '{"name": "john", "data": "some data"}'); + +-- This update doesn't change the value of the expression index, but it does +-- change the content of the info column and so should not be HOT because the +-- indexed value changed as a result of the update. +update keyvalue set info='{"name": "john", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 0 rows +drop table keyvalue; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- The table has one column docs and two indexes. They are both expression +-- indexes referencing the same column attribute (docs) but one is a partial +-- index. +CREATE TABLE ex (docs JSONB) WITH (fillfactor = 60); +INSERT INTO ex (docs) VALUES ('{"a": 0, "b": 0}'); +INSERT INTO ex (docs) SELECT jsonb_build_object('b', n) FROM generate_series(100, 10000) as n; +CREATE INDEX idx_ex_a ON ex ((docs->>'a')); +CREATE INDEX idx_ex_b ON ex ((docs->>'b')) WHERE (docs->>'b')::numeric > 9; + +-- We're using BTREE indexes and for this test we want to make sure that they remain +-- in sync with changes to our relation. Force the choice of index scans below so +-- that we know we're checking the index's understanding of what values should be +-- in the index or not. +SET SESSION enable_seqscan = OFF; +SET SESSION enable_bitmapscan = OFF; + +-- Leave 'a' unchanged but modify 'b' to a value outside of the index predicate. +-- This should be a HOT update because neither index is changed. +UPDATE ex SET docs = jsonb_build_object('a', 0, 'b', 1) WHERE (docs->>'a')::numeric = 0; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 row +-- Let's check to make sure that the index does not contain a value for 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + +-- Leave 'a' unchanged but modify 'b' to a value within the index predicate. +-- This represents a change for field 'b' from unindexed to indexed and so +-- this should not take the HOT path. +UPDATE ex SET docs = jsonb_build_object('a', 0, 'b', 10) WHERE (docs->>'a')::numeric = 0; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates +-- Let's check to make sure that the index contains the new value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + +-- This update modifies the value of 'a', an indexed field, so it also cannot +-- be a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 1, 'b', 10) WHERE (docs->>'b')::numeric = 10; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + +-- This update changes both 'a' and 'b' to new values that require index updates, +-- this cannot use the HOT path. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 12) WHERE (docs->>'b')::numeric = 10; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates +-- Let's check to make sure that the index contains the new value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + +-- This update changes 'b' to a value outside its predicate requiring that +-- we remove it from the index. That's a transition that can't be done +-- during a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 1) WHERE (docs->>'b')::numeric = 12; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates +-- Let's check to make sure that the index no longer contains the value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + +-- Disable expression checks. +ALTER TABLE ex SET (expression_checks = false); +SELECT reloptions FROM pg_class WHERE relname = 'ex'; + +-- This update changes 'b' to a value within its predicate just like it +-- previous value, which would allow for a HOT update but with expression +-- checks disabled we can't determine that so this should not be a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 2) WHERE (docs->>'b')::numeric = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + + +-- Let's make sure we're recording HOT updates for our 'ex' relation properly in the system +-- table pg_stat_user_tables. Note that statistics are stored within a transaction context +-- first (xact) and then later into the global statistics for a relation, so first we need +-- to ensure pending stats are flushed. +SELECT pg_stat_force_next_flush(); +SELECT + c.relname AS table_name, + -- Transaction statistics + pg_stat_get_xact_tuples_updated(c.oid) AS xact_updates, + pg_stat_get_xact_tuples_hot_updated(c.oid) AS xact_hot_updates, + ROUND(( + pg_stat_get_xact_tuples_hot_updated(c.oid)::float / + NULLIF(pg_stat_get_xact_tuples_updated(c.oid), 0) * 100 + )::numeric, 2) AS xact_hot_update_percentage, + -- Cumulative statistics + s.n_tup_upd AS total_updates, + s.n_tup_hot_upd AS hot_updates, + ROUND(( + s.n_tup_hot_upd::float / + NULLIF(s.n_tup_upd, 0) * 100 + )::numeric, 2) AS total_hot_update_percentage +FROM pg_class c +LEFT JOIN pg_stat_user_tables s ON c.relname = s.relname +WHERE c.relname = 'ex' +AND c.relnamespace = 'public'::regnamespace; +-- expect: 5 xact updates with 1 xact hot update and no cumulative updates as yet + +SET SESSION enable_seqscan = ON; +SET SESSION enable_bitmapscan = ON; + +DROP TABLE ex; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table has a single column 'email' and a unique constraint on it that +-- should preclude HOT updates. +CREATE TABLE users ( + user_id serial primary key, + name VARCHAR(255) NOT NULL, + email VARCHAR(255) NOT NULL, + EXCLUDE USING btree (lower(email) WITH =) +); + +-- Add some data to the table and then update it in ways that should and should +-- not be HOT updates. +INSERT INTO users (name, email) VALUES +('user1', '[email protected]'), +('user2', '[email protected]'), +('taken', '[email protected]'), +('you', '[email protected]'), +('taken', '[email protected]'); + +-- Should fail because of the unique constraint on the email column. +UPDATE users SET email = '[email protected]' WHERE email = '[email protected]'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 0 no new HOT updates + +-- Should succeed because the email column is not being updated and should go HOT. +UPDATE users SET name = 'foo' WHERE email = '[email protected]'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 a single new HOT update + +-- Create a partial index on the email column, updates +CREATE INDEX idx_users_email_no_example ON users (lower(email)) WHERE lower(email) LIKE '%@example.com%'; + +-- An update that changes the email column but not the indexed portion of it and falls outside the constraint. +-- Shouldn't be a HOT update because of the exclusion constraint. +UPDATE users SET email = '[email protected]' WHERE name = 'you'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 no new HOT updates + +-- An update that changes the email column but not the indexed portion of it and falls within the constraint. +-- Again, should fail constraint and fail to be a HOT update. +UPDATE users SET email = '[email protected]' WHERE name = 'you'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 no new HOT updates + +DROP TABLE users; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- Another test of constraints spoiling HOT updates, this time with a range. +CREATE TABLE events ( + id serial primary key, + name VARCHAR(255) NOT NULL, + event_time tstzrange, + constraint no_screening_time_overlap exclude using gist ( + event_time WITH && + ) +); + +-- Add two non-overlapping events. +INSERT INTO events (id, event_time, name) +VALUES + (1, '["2023-01-01 19:00:00", "2023-01-01 20:45:00"]', 'event1'), + (2, '["2023-01-01 21:00:00", "2023-01-01 21:45:00"]', 'event2'); + +-- Update the first event to overlap with the second, should fail the constraint and not be HOT. +UPDATE events SET event_time = '["2023-01-01 20:00:00", "2023-01-01 21:45:00"]' WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 0 no new HOT updates + +-- Update the first event to not overlap with the second, again not HOT due to the constraint. +UPDATE events SET event_time = '["2023-01-01 22:00:00", "2023-01-01 22:45:00"]' WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 0 no new HOT updates + +-- Update the first event to not overlap with the second, this time we're HOT because we don't overlap with the constraint. +UPDATE events SET name = 'new name here' WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 1 one new HOT update + +DROP TABLE events; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- A test to ensure that summarizing indexes are not updated when they don't +-- change, but are updated when they do while not prefent HOT updates. +CREATE TABLE ex (att1 JSONB, att2 text) WITH (fillfactor = 60); +CREATE INDEX expression ON ex USING btree((att1->'data')); +CREATE INDEX summarizing ON ex USING BRIN(att2); +INSERT INTO ex VALUES ('{"data": []}', 'nothing special'); + +-- Update the unindexed value of att1, this should be a HOT update and not +-- update the summarizing index. +UPDATE ex SET att1 = att1 || '{"status": "checkmate"}'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 one new HOT update + +-- Update the indexed value of att2, a summarized value, this is a summarized +-- only update and should use the HOT path while still triggering an update to +-- the summarizing BRIN index. +UPDATE ex SET att2 = 'special indeed'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 2 one new HOT update + +-- Update to att1 doesn't change the indexed value while the update to att2 does, +-- this again is a summarized only update and should use the HOT path as well as +-- trigger an update to the BRIN index. +UPDATE ex SET att1 = att1 || '{"status": "checkmate!"}', att2 = 'special, so special'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 3 one new HOT update + +-- This updates both indexes, the expression index on att1 and the summarizing +-- index on att2. This should not be a HOT update because there are modified +-- indexes and only some are summarized, not all. This should force all +-- indexes to be updated. +UPDATE ex SET att1 = att1 || '{"data": [1,2,3]}', att2 = 'do you want to play a game?'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 3 both indexes updated + +DROP TABLE ex; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This test is for a table with a custom type and a custom operators on +-- the BTREE index. The question is, when comparing values for equality +-- to determine if there are changes on the index or not... shouldn't we +-- be using the custom operators? + +-- Create a type +CREATE TYPE my_custom_type AS (val int); + +-- Comparison functions (returns boolean) +CREATE FUNCTION my_custom_lt(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val < b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE FUNCTION my_custom_le(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val <= b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE FUNCTION my_custom_eq(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val = b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE FUNCTION my_custom_ge(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val >= b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE FUNCTION my_custom_gt(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val > b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE FUNCTION my_custom_ne(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val != b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +-- Comparison function (returns -1, 0, 1) +CREATE FUNCTION my_custom_cmp(a my_custom_type, b my_custom_type) RETURNS int AS $$ +BEGIN + IF a.val < b.val THEN + RETURN -1; + ELSIF a.val > b.val THEN + RETURN 1; + ELSE + RETURN 0; + END IF; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +-- Create the operators +CREATE OPERATOR < ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_lt, + COMMUTATOR = >, + NEGATOR = >= +); + +CREATE OPERATOR <= ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_le, + COMMUTATOR = >=, + NEGATOR = > +); + +CREATE OPERATOR = ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_eq, + COMMUTATOR = =, + NEGATOR = <> +); + +CREATE OPERATOR >= ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_ge, + COMMUTATOR = <=, + NEGATOR = < +); + +CREATE OPERATOR > ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_gt, + COMMUTATOR = <, + NEGATOR = <= +); + +CREATE OPERATOR <> ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_ne, + COMMUTATOR = <>, + NEGATOR = = +); + +-- Create the operator class (including the support function) +CREATE OPERATOR CLASS my_custom_ops + DEFAULT FOR TYPE my_custom_type USING btree AS + OPERATOR 1 <, + OPERATOR 2 <=, + OPERATOR 3 =, + OPERATOR 4 >=, + OPERATOR 5 >, + FUNCTION 1 my_custom_cmp(my_custom_type, my_custom_type); + +-- Create the table +CREATE TABLE my_table ( + id int, + custom_val my_custom_type +); + +-- Insert some data +INSERT INTO my_table (id, custom_val) VALUES +(1, ROW(3)::my_custom_type), +(2, ROW(1)::my_custom_type), +(3, ROW(4)::my_custom_type), +(4, ROW(2)::my_custom_type); + +-- Create a function to use when indexing +CREATE OR REPLACE FUNCTION abs_val(val my_custom_type) RETURNS int AS $$ +BEGIN + RETURN abs(val.val); +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +-- Create the index +CREATE INDEX idx_custom_val_abs ON my_table (abs_val(custom_val)); + +-- Update 1 +UPDATE my_table SET custom_val = ROW(5)::my_custom_type WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + +-- Update 2 +UPDATE my_table SET custom_val = ROW(0)::my_custom_type WHERE custom_val < ROW(3)::my_custom_type; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + +-- Update 3 +UPDATE my_table SET custom_val = ROW(6)::my_custom_type WHERE id = 3; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + +-- Update 4 +UPDATE my_table SET id = 5 WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + +-- Query using the index +EXPLAIN (COSTS OFF) SELECT * FROM my_table WHERE abs_val(custom_val) = 6; +SELECT * FROM my_table WHERE abs_val(custom_val) = 6; + +-- Clean up +DROP TABLE my_table CASCADE; +DROP OPERATOR CLASS my_custom_ops USING btree CASCADE; +DROP OPERATOR < (my_custom_type, my_custom_type); +DROP OPERATOR <= (my_custom_type, my_custom_type); +DROP OPERATOR = (my_custom_type, my_custom_type); +DROP OPERATOR >= (my_custom_type, my_custom_type); +DROP OPERATOR > (my_custom_type, my_custom_type); +DROP OPERATOR <> (my_custom_type, my_custom_type); +DROP FUNCTION my_custom_lt(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_le(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_eq(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_ge(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_gt(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_ne(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_cmp(my_custom_type, my_custom_type); +DROP FUNCTION abs_val(my_custom_type); +DROP TYPE my_custom_type CASCADE; -- 2.42.0 ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Expanding HOT updates for expression and partial indexes @ 2025-03-05 17:20 Burd, Greg <[email protected]> parent: Burd, Greg <[email protected]> 0 siblings, 0 replies; 6+ messages in thread From: Burd, Greg @ 2025-03-05 17:20 UTC (permalink / raw) To: pgsql-hackers Hello, I've rebased and updated the patch a bit. The biggest change is that the performance penalty measured with v1 of this patch is essentially gone in v10. The overhead was due to re-creating IndexInfo information unnecessarily, which I found existed in the estate. I've added a few fields in IndexInfo that are not populated by default but necessary when checking expression indexes, those fields are populated on demand and only once limiting their overhead. Here's what you'll find if you look into execIndexing.c where the majority of changes happened. * assumes estate->es_result_relations[0] is the ResultRelInfo being updated * uses ri_IndexRelationInfo[] from within estate rather than re-creating it * augments IndexInfo only when needed for testing expressions and only once * only creates a local old/new TupleTableSlot when not present in estate * retains existing summarized index HOT update logic One remaining concern stems from the assumption that estate->es_result_relations[0] is always going to be the relation being updated. This is guarded by assert()'s in the patch. It seems this is safe, all tests are passing (including TAP) and my review of the code seems to line up with that assumption. That said... opinions? Another lingering question is under what conditions the old/new TupleTableSlots are not created and available via the ResultRelInfo found in estate. I've only seen this happen when there is an INSERT ... ON CONFLICT UPDATE ... with expression indexes. I was hopeful that in all cases I could avoid re-creating those when checking expression indexes to avoid that repeated overhead. I still avoid it when possible in this patch. When you have time I'd appreciate any feedback. -greg Amazon Web Services: https://aws.amazon.com Attachments: [application/octet-stream] v10-0001-Expand-HOT-update-path-to-include-expression-and.patch (90.9K, 2-v10-0001-Expand-HOT-update-path-to-include-expression-and.patch) download | inline diff: From 9422526a14e9d48150eb1d54c030351221d087a5 Mon Sep 17 00:00:00 2001 From: Gregory Burd <[email protected]> Date: Mon, 27 Jan 2025 13:28:59 -0500 Subject: [PATCH v10] Expand HOT update path to include expression and partial indexes. This patch extends the cases where HOT updates are possible in the heapam by examining expression indexes and determining if indexed values where mutated or not. Previously, any expression index on a column would disqualify it from the HOT update path. Also examines partial indexes to see if the values are within the predicate or not. For historical context, this is a rewrite of a patch first proposed on the pgsql-hackers list: https://www.postgresql.org/message-id/flat/4d9928ee-a9e6-15f9-9c82-5981f13ffca6%40postgrespro.ru applied: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=c203d6cf8 reverted: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=05f84605dbeb9cf8279a157234b24bbb706c5256 Signed-off-by: Greg Burd <[email protected]> --- doc/src/sgml/ref/create_table.sgml | 18 + doc/src/sgml/storage.sgml | 21 + src/backend/access/common/reloptions.c | 12 +- src/backend/access/heap/README.HOT | 45 +- src/backend/access/heap/heapam.c | 23 +- src/backend/access/heap/heapam_handler.c | 6 +- src/backend/access/table/tableam.c | 2 +- src/backend/catalog/index.c | 19 + src/backend/executor/execIndexing.c | 255 ++++++- src/backend/executor/execReplication.c | 2 +- src/backend/executor/nodeModifyTable.c | 7 +- src/backend/utils/cache/relcache.c | 70 +- src/bin/psql/tab-complete.in.c | 2 +- src/include/access/heapam.h | 3 +- src/include/access/tableam.h | 10 +- src/include/executor/executor.h | 5 + src/include/nodes/execnodes.h | 9 + src/include/utils/rel.h | 10 + src/include/utils/relcache.h | 1 + .../regress/expected/heap_hot_updates.out | 665 ++++++++++++++++++ src/test/regress/parallel_schedule | 5 + src/test/regress/sql/heap_hot_updates.sql | 481 +++++++++++++ 22 files changed, 1619 insertions(+), 52 deletions(-) create mode 100644 src/test/regress/expected/heap_hot_updates.out create mode 100644 src/test/regress/sql/heap_hot_updates.sql diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 0a3e520f215..8ee2ac523ee 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -1981,6 +1981,24 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </listitem> </varlistentry> + <varlistentry id="reloption-expression-checks" xreflabel="expression_checks"> + <term><literal>expression_checks</literal> (<type>boolean</type>) + <indexterm> + <primary><varname>expression_checks</varname> storage parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Enables or disables evaulation of predicate expressions on partial + indexes or expressions used to define indexes during updates. + If <literal>true</literal>, then these expressions are evaluated during + updates to data within the heap relation against the old and new values + and then compared to determine if <acronym>HOT</acronym> updates are + allowable or not. The default value is <literal>true</literal>. + </para> + </listitem> + </varlistentry> + </variablelist> </refsect2> diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml index 61250799ec0..64a6264d8fd 100644 --- a/doc/src/sgml/storage.sgml +++ b/doc/src/sgml/storage.sgml @@ -1138,6 +1138,27 @@ data. Empty in ordinary tables.</entry> </itemizedlist> </para> + <para> + <acronym>HOT</acronym> updates can occur when the expression used to define + an index shows no changes to the indexed value. To determine this requires + that the expression be evaulated for the old and new values to be stored in + the index and then compared. This allows for <acronym>HOT</acronym> updates + when data indexed within JSONB columns is unchanged. To disable this + behavior and avoid the overhead of evaluating the expression during updates + set the <literal>expression_checks</literal> option to false for the table. + </para> + + <para> + <acronym>HOT</acronym> updates can also occur when updated values are not + within the predicate of a partial index. However, <acronym>HOT</acronym> + updates are not possible when the updated value and the current value differ + with regards to the predicate. To determin this requires that the predicate + expression be evaluated for the old and new values to be stored in the index + and then compared. To disable this behavior and avoid the overhead of + evaluating the expression during updates set + the <literal>expression_checks</literal> option to false for the table. + </para> + <para> You can increase the likelihood of sufficient page space for <acronym>HOT</acronym> updates by decreasing a table's <link diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c index 59fb53e7707..c081611926e 100644 --- a/src/backend/access/common/reloptions.c +++ b/src/backend/access/common/reloptions.c @@ -166,6 +166,15 @@ static relopt_bool boolRelOpts[] = }, true }, + { + { + "expression_checks", + "When disabled prevents checking expressions on indexes and predicates on partial indexes for changes that might influence heap-only tuple (HOT) updates.", + RELOPT_KIND_HEAP, + ShareUpdateExclusiveLock + }, + true + }, /* list terminator */ {{NULL}} }; @@ -1903,7 +1912,8 @@ default_reloptions(Datum reloptions, bool validate, relopt_kind kind) {"vacuum_truncate", RELOPT_TYPE_BOOL, offsetof(StdRdOptions, vacuum_truncate)}, {"vacuum_max_eager_freeze_failure_rate", RELOPT_TYPE_REAL, - offsetof(StdRdOptions, vacuum_max_eager_freeze_failure_rate)} + offsetof(StdRdOptions, vacuum_max_eager_freeze_failure_rate)}, + {"expression_checks", RELOPT_TYPE_BOOL, offsetof(StdRdOptions, expression_checks)} }; return (bytea *) build_reloptions(reloptions, validate, kind, diff --git a/src/backend/access/heap/README.HOT b/src/backend/access/heap/README.HOT index 74e407f375a..2340d82053f 100644 --- a/src/backend/access/heap/README.HOT +++ b/src/backend/access/heap/README.HOT @@ -36,7 +36,7 @@ HOT solves this problem for two restricted but useful special cases: First, where a tuple is repeatedly updated in ways that do not change its indexed columns. (Here, "indexed column" means any column referenced at all in an index definition, including for example columns that are -tested in a partial-index predicate but are not stored in the index.) +tested in a partial-index predicate, that has materially changed.) Second, where the modified columns are only used in indexes that do not contain tuple IDs, but maintain summaries of the indexed data by block. @@ -133,28 +133,34 @@ Note: we can use a "dead" line pointer for any DELETEd tuple, whether it was part of a HOT chain or not. This allows space reclamation in advance of running VACUUM for plain DELETEs as well as HOT updates. -The requirement for doing a HOT update is that indexes which point to -the root line pointer (and thus need to be cleaned up by VACUUM when the -tuple is dead) do not reference columns which are updated in that HOT -chain. Summarizing indexes (such as BRIN) are assumed to have no -references to individual tuples and thus are ignored when checking HOT -applicability. The updated columns are checked at execution time by -comparing the binary representation of the old and new values. We insist -on bitwise equality rather than using datatype-specific equality routines. -The main reason to avoid the latter is that there might be multiple -notions of equality for a datatype, and we don't know exactly which one -is relevant for the indexes at hand. We assume that bitwise equality -guarantees equality for all purposes. +The requirement for doing a HOT update is that indexes which point to the root +line pointer (and thus need to be cleaned up by VACUUM when the tuple is dead) +do not reference columns which are updated in that HOT chain. + +Summarizing indexes (such as BRIN) are assumed to have no references to +individual tuples and thus are ignored when checking HOT applicability. + +Expressions on indexes are evaluated and the results used when checking for +changes. This allows for the JSONB datatype to have HOT updates when the +indexed portion of the document are not modified. + +Partial index expressions are evaluated, HOT updates are allowed when the +updated index values do not satisfy the predicate. + +The updated columns are checked at execution time by comparing the binary +representation of the old and new values. We insist on bitwise equality rather +than using datatype-specific equality routines. The main reason to avoid the +latter is that there might be multiple notions of equality for a datatype, and +we don't know exactly which one is relevant for the indexes at hand. We assume +that bitwise equality guarantees equality for all purposes. If any columns that are included by non-summarizing indexes are updated, the HOT optimization is not applied, and the new tuple is inserted into all indexes of the table. If none of the updated columns are included in the table's indexes, the HOT optimization is applied and no indexes are updated. If instead the updated columns are only indexed by summarizing -indexes, the HOT optimization is applied, but the update is propagated to -all summarizing indexes. (Realistically, we only need to propagate the -update to the indexes that contain the updated values, but that is yet to -be implemented.) +indexes, the HOT optimization is applied and the update is propagated to +all of the summarizing indexes. Abort Cases ----------- @@ -477,8 +483,9 @@ Heap-only tuple HOT-safe A proposed tuple update is said to be HOT-safe if it changes - none of the tuple's indexed columns. It will only become an - actual HOT update if we can find room on the same page for + none of the tuple's indexed columns or if the changes remain + outside of a partial index's predicate. It will only become + an actual HOT update if we can find room on the same page for the new tuple version. HOT update diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index fa7935a0ed3..32b3cc1d4ef 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -3164,12 +3164,13 @@ TM_Result heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, CommandId cid, Snapshot crosscheck, bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode, - TU_UpdateIndexes *update_indexes) + TU_UpdateIndexes *update_indexes, struct EState *estate) { TM_Result result; TransactionId xid = GetCurrentTransactionId(); Bitmapset *hot_attrs; Bitmapset *sum_attrs; + Bitmapset *exp_attrs; Bitmapset *key_attrs; Bitmapset *id_attrs; Bitmapset *interesting_attrs; @@ -3246,6 +3247,8 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, INDEX_ATTR_BITMAP_HOT_BLOCKING); sum_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_SUMMARIZED); + exp_attrs = RelationGetIndexAttrBitmap(relation, + INDEX_ATTR_BITMAP_EXPRESSION); key_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_KEY); id_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_IDENTITY_KEY); @@ -3311,6 +3314,7 @@ heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, bms_free(hot_attrs); bms_free(sum_attrs); + bms_free(exp_attrs); bms_free(key_attrs); bms_free(id_attrs); /* modified_attrs not yet initialized */ @@ -3612,6 +3616,7 @@ l2: bms_free(hot_attrs); bms_free(sum_attrs); + bms_free(exp_attrs); bms_free(key_attrs); bms_free(id_attrs); bms_free(modified_attrs); @@ -3928,12 +3933,19 @@ l2: if (newbuf == buffer) { + bool expression_checks = RelationGetExpressionChecks(relation); + /* * Since the new tuple is going into the same page, we might be able - * to do a HOT update. Check if any of the index columns have been - * changed. + * to do a HOT update. As a reminder, hot_attrs includes attributes + * used by indexes including within expressions and predicates, but + * not attributes only used by summarizing indexes. */ - if (!bms_overlap(modified_attrs, hot_attrs)) + if (!bms_overlap(modified_attrs, hot_attrs) || + (expression_checks && + bms_overlap(modified_attrs, exp_attrs) && + !ExecExpressionIndexesUpdated(relation, modified_attrs, estate, + &oldtup, newtup))) { use_hot_update = true; @@ -4126,7 +4138,6 @@ l2: heap_freetuple(old_key_tuple); bms_free(hot_attrs); - bms_free(sum_attrs); bms_free(key_attrs); bms_free(id_attrs); bms_free(modified_attrs); @@ -4413,7 +4424,7 @@ simple_heap_update(Relation relation, ItemPointer otid, HeapTuple tup, result = heap_update(relation, otid, tup, GetCurrentCommandId(true), InvalidSnapshot, true /* wait for commit */ , - &tmfd, &lockmode, update_indexes); + &tmfd, &lockmode, update_indexes, NULL); switch (result) { case TM_SelfModified: diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index e78682c3cef..a8710f29ffc 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -312,8 +312,8 @@ heapam_tuple_delete(Relation relation, ItemPointer tid, CommandId cid, static TM_Result heapam_tuple_update(Relation relation, ItemPointer otid, TupleTableSlot *slot, CommandId cid, Snapshot snapshot, Snapshot crosscheck, - bool wait, TM_FailureData *tmfd, - LockTupleMode *lockmode, TU_UpdateIndexes *update_indexes) + bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode, + TU_UpdateIndexes *update_indexes, EState *estate) { bool shouldFree = true; HeapTuple tuple = ExecFetchSlotHeapTuple(slot, true, &shouldFree); @@ -324,7 +324,7 @@ heapam_tuple_update(Relation relation, ItemPointer otid, TupleTableSlot *slot, tuple->t_tableOid = slot->tts_tableOid; result = heap_update(relation, otid, tuple, cid, crosscheck, wait, - tmfd, lockmode, update_indexes); + tmfd, lockmode, update_indexes, estate); ItemPointerCopy(&tuple->t_self, &slot->tts_tid); /* diff --git a/src/backend/access/table/tableam.c b/src/backend/access/table/tableam.c index a56c5eceb14..a62df7f4bce 100644 --- a/src/backend/access/table/tableam.c +++ b/src/backend/access/table/tableam.c @@ -346,7 +346,7 @@ simple_table_tuple_update(Relation rel, ItemPointer otid, GetCurrentCommandId(true), snapshot, InvalidSnapshot, true /* wait for commit */ , - &tmfd, &lockmode, update_indexes); + &tmfd, &lockmode, update_indexes, NULL); switch (result) { diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 8e1741c81f5..d9bd64bfe96 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -2455,7 +2455,26 @@ BuildIndexInfo(Relation index) /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) + { + CompactAttribute *attr = TupleDescCompactAttr(RelationGetDescr(index), i); + ii->ii_IndexAttrNumbers[i] = indexStruct->indkey.values[i]; + ii->ii_IndexAttrs = + bms_add_member(ii->ii_IndexAttrs, + indexStruct->indkey.values[i] - FirstLowInvalidHeapAttributeNumber); + + ii->ii_IndexAttrLen[i] = attr->attlen; + if (attr->attbyval) + ii->ii_IndexAttrByVal = bms_add_member(ii->ii_IndexAttrByVal, i); + } + + /* collect attributes used in the expression, if one is present */ + if (ii->ii_Expressions) + pull_varattnos((Node *) ii->ii_Expressions, 1, &ii->ii_ExpressionAttrs); + + /* collect attributes used in the predicate, if one is present */ + if (ii->ii_Predicate) + pull_varattnos((Node *) ii->ii_Predicate, 1, &ii->ii_PredicateAttrs); /* fetch exclusion constraint info if any */ if (indexStruct->indisexclusion) diff --git a/src/backend/executor/execIndexing.c b/src/backend/executor/execIndexing.c index 742f3f8c08d..2b61147ad88 100644 --- a/src/backend/executor/execIndexing.c +++ b/src/backend/executor/execIndexing.c @@ -113,10 +113,13 @@ #include "catalog/index.h" #include "executor/executor.h" #include "nodes/nodeFuncs.h" +#include "optimizer/optimizer.h" #include "storage/lmgr.h" #include "utils/multirangetypes.h" #include "utils/rangetypes.h" #include "utils/snapmgr.h" +#include "utils/datum.h" +#include "utils/lsyscache.h" /* waitMode argument to check_exclusion_or_unique_constraint() */ typedef enum @@ -372,7 +375,7 @@ ExecInsertIndexTuples(ResultRelInfo *resultRelInfo, /* * Skip processing of non-summarizing indexes if we only update - * summarizing indexes + * summarizing indexes. */ if (onlySummarizing && !indexInfo->ii_Summarizing) continue; @@ -1096,6 +1099,9 @@ index_unchanged_by_update(ResultRelInfo *resultRelInfo, EState *estate, if (hasexpression) { + if (indexInfo->ii_IndexUnchanged) + return true; + indexInfo->ii_IndexUnchanged = false; return false; } @@ -1173,3 +1179,250 @@ ExecWithoutOverlapsNotEmpty(Relation rel, NameData attname, Datum attval, char t errmsg("empty WITHOUT OVERLAPS value found in column \"%s\" in relation \"%s\"", NameStr(attname), RelationGetRelationName(rel)))); } + + + /* + * AttributeIndexInfo -- adds attribute information to IndexInfo + */ +static IndexInfo * +AttributeIndexInfo(Relation index, IndexInfo *indexInfo) +{ + if (indexInfo->ii_IndexAttrByVal) + return indexInfo; + + /* + * collect attributes used by the index, their len and if they are by + * value + */ + for (int i = 0; i < indexInfo->ii_NumIndexAttrs; i++) + { + indexInfo->ii_IndexAttrs = + bms_add_member(indexInfo->ii_IndexAttrs, + indexInfo->ii_IndexAttrNumbers[i] - FirstLowInvalidHeapAttributeNumber); + + if (i < indexInfo->ii_NumIndexKeyAttrs) + { + int16 elmlen; + bool elmbyval; + + get_typlenbyval(index->rd_opcintype[i], &elmlen, &elmbyval); + indexInfo->ii_IndexAttrLen[i] = elmlen; + if (elmbyval) + indexInfo->ii_IndexAttrByVal = bms_add_member(indexInfo->ii_IndexAttrByVal, i); + } + } + + /* collect attributes used in the expression */ + if (indexInfo->ii_Expressions) + pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &indexInfo->ii_ExpressionAttrs); + + /* collect attributes used in the predicate */ + if (indexInfo->ii_Predicate) + pull_varattnos((Node *) indexInfo->ii_Predicate, 1, &indexInfo->ii_PredicateAttrs); + + return indexInfo; +} + +/* + * Determine if indexes that have expressions or predicates require updates + * for the purposes of allowing HOT updates. Returns false iff all indexed + * values are unchanged. + */ +bool +ExecExpressionIndexesUpdated(Relation relation, + Bitmapset *modified_attrs, + EState *estate, + HeapTuple old_tuple, + HeapTuple new_tuple) +{ + bool expression_checks = RelationGetExpressionChecks(relation); + bool result = false; + ResultRelInfo *resultRelInfo; + Relation index; + IndexInfo *indexInfo; + ExprContext *econtext = NULL; + TupleDesc tupledesc; + TupleTableSlot *old_tts; + TupleTableSlot *new_tts; + + /* + * We're not able to access the IndexInfo array without an estate. + */ + if (estate == NULL || modified_attrs == NULL) + return true; + +#ifndef USE_ASSERT_CHECKING + if (estate->es_result_relations[0]->ri_RelationDesc != relation) + return true; +#endif + + Assert(estate->es_result_relations[0]->ri_RelationDesc == relation); + + if (expression_checks == false) + return true; + + resultRelInfo = estate->es_result_relations[0]; + old_tts = resultRelInfo->ri_oldTupleSlot; + new_tts = resultRelInfo->ri_newTupleSlot; + econtext = GetPerTupleExprContext(estate); + tupledesc = RelationGetDescr(relation); + + /* + * Examine each index on this relation relative to the changes between old + * and new tuples. + */ + for (int i = 0; i < resultRelInfo->ri_NumIndices; i++) + { + index = (Relation) resultRelInfo->ri_IndexRelationDescs[i]; + indexInfo = AttributeIndexInfo(index, + resultRelInfo->ri_IndexRelationInfo[i]); + + /* + * If this is a partial index it has a predicate, evaluate the + * expression to determine if we need to include it or not. + */ + if (bms_overlap(indexInfo->ii_PredicateAttrs, modified_attrs)) + { + ExprState *pstate; + bool old_tuple_qualifies, + new_tuple_qualifies; + + /* Create these once, only if necessary, then reuse them. */ + if (old_tts == NULL) + { + old_tts = MakeSingleTupleTableSlot(tupledesc, + &TTSOpsHeapTuple); + ExecStoreHeapTuple(old_tuple, old_tts, InvalidBuffer); + } + + if (new_tts == NULL) + { + new_tts = MakeSingleTupleTableSlot(tupledesc, + &TTSOpsHeapTuple); + ExecStoreHeapTuple(new_tuple, new_tts, InvalidBuffer); + } + + pstate = ExecPrepareQual(indexInfo->ii_Predicate, estate); + + /* + * Here the term "qualifies" means "satisfies the predicate + * condition of the partial index". + */ + econtext->ecxt_scantuple = old_tts; + old_tuple_qualifies = ExecQual(pstate, econtext); + + econtext->ecxt_scantuple = new_tts; + new_tuple_qualifies = ExecQual(pstate, econtext); + + /* + * If neither the old nor the new tuples satisfy the predicate we + * can be sure that this index doesn't need updating, continue to + * the next index. + */ + if ((new_tuple_qualifies == false) && (old_tuple_qualifies == false)) + continue; + + /* + * If there is a transition between indexed and not indexed, + * that's enough to require that this index is updated. If any + * single non-summarizing index requires updates then they all + * should be updated. + */ + if (new_tuple_qualifies != old_tuple_qualifies) + { + result = true; + break; + } + + /* + * Otherwise the old and new values exist in the index, but did + * they get updated? We don't yet know, so proceed with the next + * statement in the loop to find out. + */ + } + + /* + * Indexes with expressions may or may not have changed, it is + * impossible to know without exercising their expression and + * reviewing index tuple state for changes. This is a lot of work, + * but because all indexes on JSONB columns fall into this category it + * can be worth it to avoid index updates and remain on the HOT update + * path when possible. + */ + if (bms_overlap(indexInfo->ii_ExpressionAttrs, modified_attrs)) + { + Datum old_values[INDEX_MAX_KEYS]; + bool old_isnull[INDEX_MAX_KEYS]; + Datum new_values[INDEX_MAX_KEYS]; + bool new_isnull[INDEX_MAX_KEYS]; + bool changed = false; + + /* Create these once, only if necessary, then reuse them. */ + if (old_tts == NULL) + { + old_tts = MakeSingleTupleTableSlot(tupledesc, + &TTSOpsHeapTuple); + ExecStoreHeapTuple(old_tuple, old_tts, InvalidBuffer); + } + + if (new_tts == NULL) + { + new_tts = MakeSingleTupleTableSlot(tupledesc, + &TTSOpsHeapTuple); + ExecStoreHeapTuple(new_tuple, new_tts, InvalidBuffer); + } + + econtext->ecxt_scantuple = old_tts; + FormIndexDatum(indexInfo, + old_tts, + estate, + old_values, + old_isnull); + + econtext->ecxt_scantuple = new_tts; + FormIndexDatum(indexInfo, + new_tts, + estate, + new_values, + new_isnull); + + for (int j = 0; j < indexInfo->ii_NumIndexKeyAttrs; j++) + { + if (old_isnull[j] != new_isnull[j]) + { + changed = true; + break; + } + else if (!old_isnull[j]) + { + int16 elmlen = indexInfo->ii_IndexAttrLen[j]; + bool elmbyval = bms_is_member(j, indexInfo->ii_IndexAttrByVal); + + if (!datum_image_eq(old_values[j], new_values[j], + elmbyval, elmlen)) + { + changed = true; + break; + } + } + } + + indexInfo->ii_CheckedUnchanged = true; + indexInfo->ii_IndexUnchanged = !changed; + + if (changed) + { + result = true; + break; + } + } + } + + if (resultRelInfo->ri_oldTupleSlot == NULL) + ExecDropSingleTupleTableSlot(old_tts); + + if (resultRelInfo->ri_newTupleSlot == NULL) + ExecDropSingleTupleTableSlot(new_tts); + + return result; +} diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.c index 5cef54f00ed..b3cba354d8a 100644 --- a/src/backend/executor/execReplication.c +++ b/src/backend/executor/execReplication.c @@ -639,8 +639,8 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo, if (!skip_tuple) { List *recheckIndexes = NIL; - TU_UpdateIndexes update_indexes; List *conflictindexes; + TU_UpdateIndexes update_indexes; bool conflict = false; /* Compute stored generated columns */ diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index b0fe50075ad..ef4e8d2499d 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -2283,7 +2283,8 @@ lreplace: estate->es_crosscheck_snapshot, true /* wait for commit */ , &context->tmfd, &updateCxt->lockmode, - &updateCxt->updateIndexes); + &updateCxt->updateIndexes, + estate); return result; } @@ -2301,6 +2302,7 @@ ExecUpdateEpilogue(ModifyTableContext *context, UpdateContext *updateCxt, { ModifyTableState *mtstate = context->mtstate; List *recheckIndexes = NIL; + bool onlySummarizing = updateCxt->updateIndexes == TU_Summarizing; /* insert index entries for tuple if necessary */ if (resultRelInfo->ri_NumIndices > 0 && (updateCxt->updateIndexes != TU_None)) @@ -2308,7 +2310,7 @@ ExecUpdateEpilogue(ModifyTableContext *context, UpdateContext *updateCxt, slot, context->estate, true, false, NULL, NIL, - (updateCxt->updateIndexes == TU_Summarizing)); + onlySummarizing); /* AFTER ROW UPDATE Triggers */ ExecARUpdateTriggers(context->estate, resultRelInfo, @@ -4362,6 +4364,7 @@ ExecModifyTable(PlanState *pstate) /* Now apply the update. */ slot = ExecUpdate(&context, resultRelInfo, tupleid, oldtuple, oldSlot, slot, node->canSetTag); + if (tuplock) UnlockTuple(resultRelInfo->ri_RelationDesc, tupleid, InplaceUpdateTupleLock); diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index d1ae761b3f6..21ccefeefa1 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -64,6 +64,7 @@ #include "catalog/pg_type.h" #include "catalog/schemapg.h" #include "catalog/storage.h" +#include "catalog/index.h" #include "commands/policy.h" #include "commands/publicationcmds.h" #include "commands/trigger.h" @@ -5208,6 +5209,7 @@ RelationGetIndexPredicate(Relation relation) * index (empty if FULL) * INDEX_ATTR_BITMAP_HOT_BLOCKING Columns that block updates from being HOT * INDEX_ATTR_BITMAP_SUMMARIZED Columns included in summarizing indexes + * INDEX_ATTR_BITMAP_EXPRESSION Columns included in expresion indexes * * Attribute numbers are offset by FirstLowInvalidHeapAttributeNumber so that * we can include system attributes (e.g., OID) in the bitmap representation. @@ -5231,7 +5233,11 @@ RelationGetIndexAttrBitmap(Relation relation, IndexAttrBitmapKind attrKind) Bitmapset *pkindexattrs; /* columns in the primary index */ Bitmapset *idindexattrs; /* columns in the replica identity */ Bitmapset *hotblockingattrs; /* columns with HOT blocking indexes */ + Bitmapset *hotblockingexprattrs; /* as above, but only those in + * expressions */ Bitmapset *summarizedattrs; /* columns with summarizing indexes */ + Bitmapset *summarizedexprattrs; /* as above, but only those in + * expressions */ List *indexoidlist; List *newindexoidlist; Oid relpkindex; @@ -5254,6 +5260,8 @@ RelationGetIndexAttrBitmap(Relation relation, IndexAttrBitmapKind attrKind) return bms_copy(relation->rd_hotblockingattr); case INDEX_ATTR_BITMAP_SUMMARIZED: return bms_copy(relation->rd_summarizedattr); + case INDEX_ATTR_BITMAP_EXPRESSION: + return bms_copy(relation->rd_expressionattr); default: elog(ERROR, "unknown attrKind %u", attrKind); } @@ -5297,7 +5305,9 @@ restart: pkindexattrs = NULL; idindexattrs = NULL; hotblockingattrs = NULL; + hotblockingexprattrs = NULL; summarizedattrs = NULL; + summarizedexprattrs = NULL; foreach(l, indexoidlist) { Oid indexOid = lfirst_oid(l); @@ -5311,6 +5321,7 @@ restart: bool isPK; /* primary key */ bool isIDKey; /* replica identity index */ Bitmapset **attrs; + Bitmapset **exprattrs; indexDesc = index_open(indexOid, AccessShareLock); @@ -5354,14 +5365,20 @@ restart: * decide which bitmap we'll update in the following loop. */ if (indexDesc->rd_indam->amsummarizing) + { attrs = &summarizedattrs; + exprattrs = &summarizedexprattrs; + } else + { attrs = &hotblockingattrs; + exprattrs = &hotblockingexprattrs; + } /* Collect simple attribute references */ for (i = 0; i < indexDesc->rd_index->indnatts; i++) { - int attrnum = indexDesc->rd_index->indkey.values[i]; + int attridx = indexDesc->rd_index->indkey.values[i]; /* * Since we have covering indexes with non-key columns, we must @@ -5377,30 +5394,28 @@ restart: * key or identity key. Hence we do not include them into * uindexattrs, pkindexattrs and idindexattrs bitmaps. */ - if (attrnum != 0) + if (attridx != 0) { - *attrs = bms_add_member(*attrs, - attrnum - FirstLowInvalidHeapAttributeNumber); + AttrNumber attrnum = attridx - FirstLowInvalidHeapAttributeNumber; + + *attrs = bms_add_member(*attrs, attrnum); if (isKey && i < indexDesc->rd_index->indnkeyatts) - uindexattrs = bms_add_member(uindexattrs, - attrnum - FirstLowInvalidHeapAttributeNumber); + uindexattrs = bms_add_member(uindexattrs, attrnum); if (isPK && i < indexDesc->rd_index->indnkeyatts) - pkindexattrs = bms_add_member(pkindexattrs, - attrnum - FirstLowInvalidHeapAttributeNumber); + pkindexattrs = bms_add_member(pkindexattrs, attrnum); if (isIDKey && i < indexDesc->rd_index->indnkeyatts) - idindexattrs = bms_add_member(idindexattrs, - attrnum - FirstLowInvalidHeapAttributeNumber); + idindexattrs = bms_add_member(idindexattrs, attrnum); } } /* Collect all attributes used in expressions, too */ - pull_varattnos(indexExpressions, 1, attrs); + pull_varattnos(indexExpressions, 1, exprattrs); /* Collect all attributes in the index predicate, too */ - pull_varattnos(indexPredicate, 1, attrs); + pull_varattnos(indexPredicate, 1, exprattrs); index_close(indexDesc, AccessShareLock); } @@ -5429,11 +5444,37 @@ restart: bms_free(pkindexattrs); bms_free(idindexattrs); bms_free(hotblockingattrs); + bms_free(hotblockingexprattrs); bms_free(summarizedattrs); + bms_free(summarizedexprattrs); goto restart; } + /* + * HOT-blocking attributes (columns) should include all columns that are + * part of the index, but not part of the index expressions from partial + * and expression indexes. So, we need to remove the expression-only + * columns from the HOT-blocking columns bitmap as those will be checked + * separately. This is true for both summarizing and non-summarizing + * indexes which we've separated above, so we have to do this for both + * bitmaps. + */ + + /* {expression-only columns} = {expression columns} - {direct columns} */ + hotblockingexprattrs = bms_del_members(hotblockingexprattrs, + hotblockingattrs); + /* {hot-blocking columns} = {direct columns} + {expression-only columns} */ + hotblockingattrs = bms_add_members(hotblockingattrs, + hotblockingexprattrs); + + /* {summarized-only columns} = {all summarized columns} - {direct columns} */ + summarizedexprattrs = bms_del_members(summarizedexprattrs, + summarizedattrs); + /* {summarized columns} = {all direct columns} + {summarized-only columns} */ + summarizedattrs = bms_add_members(summarizedattrs, + summarizedexprattrs); + /* Don't leak the old values of these bitmaps, if any */ relation->rd_attrsvalid = false; bms_free(relation->rd_keyattr); @@ -5446,6 +5487,8 @@ restart: relation->rd_hotblockingattr = NULL; bms_free(relation->rd_summarizedattr); relation->rd_summarizedattr = NULL; + bms_free(relation->rd_expressionattr); + relation->rd_expressionattr = NULL; /* * Now save copies of the bitmaps in the relcache entry. We intentionally @@ -5460,6 +5503,7 @@ restart: relation->rd_idattr = bms_copy(idindexattrs); relation->rd_hotblockingattr = bms_copy(hotblockingattrs); relation->rd_summarizedattr = bms_copy(summarizedattrs); + relation->rd_expressionattr = bms_copy(hotblockingexprattrs); relation->rd_attrsvalid = true; MemoryContextSwitchTo(oldcxt); @@ -5476,6 +5520,8 @@ restart: return hotblockingattrs; case INDEX_ATTR_BITMAP_SUMMARIZED: return summarizedattrs; + case INDEX_ATTR_BITMAP_EXPRESSION: + return hotblockingexprattrs; default: elog(ERROR, "unknown attrKind %u", attrKind); return NULL; diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 8432be641ac..fc3250176eb 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -2992,7 +2992,7 @@ match_previous_words(int pattern_id, COMPLETE_WITH("("); /* ALTER TABLESPACE <foo> SET|RESET ( */ else if (Matches("ALTER", "TABLESPACE", MatchAny, "SET|RESET", "(")) - COMPLETE_WITH("seq_page_cost", "random_page_cost", + COMPLETE_WITH("seq_page_cost", "random_page_cost", "expression_checks", "effective_io_concurrency", "maintenance_io_concurrency"); /* ALTER TEXT SEARCH */ diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 1640d9c32f7..e12da1934e9 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -21,6 +21,7 @@ #include "access/skey.h" #include "access/table.h" /* for backward compatibility */ #include "access/tableam.h" +#include "executor/executor.h" #include "nodes/lockoptions.h" #include "nodes/primnodes.h" #include "storage/bufpage.h" @@ -339,7 +340,7 @@ extern TM_Result heap_update(Relation relation, ItemPointer otid, HeapTuple newtup, CommandId cid, Snapshot crosscheck, bool wait, struct TM_FailureData *tmfd, LockTupleMode *lockmode, - TU_UpdateIndexes *update_indexes); + TU_UpdateIndexes *update_indexes, struct EState *estate); extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple, CommandId cid, LockTupleMode mode, LockWaitPolicy wait_policy, bool follow_updates, diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h index 131c050c15f..62ed6592b85 100644 --- a/src/include/access/tableam.h +++ b/src/include/access/tableam.h @@ -38,6 +38,7 @@ struct IndexInfo; struct SampleScanState; struct VacuumParams; struct ValidateIndexState; +struct EState; /* * Bitmask values for the flags argument to the scan_begin callback. @@ -550,7 +551,8 @@ typedef struct TableAmRoutine bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode, - TU_UpdateIndexes *update_indexes); + TU_UpdateIndexes *update_indexes, + struct EState *estate); /* see table_tuple_lock() for reference about parameters */ TM_Result (*tuple_lock) (Relation rel, @@ -1541,12 +1543,12 @@ static inline TM_Result table_tuple_update(Relation rel, ItemPointer otid, TupleTableSlot *slot, CommandId cid, Snapshot snapshot, Snapshot crosscheck, bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode, - TU_UpdateIndexes *update_indexes) + TU_UpdateIndexes *update_indexes, struct EState *estate) { return rel->rd_tableam->tuple_update(rel, otid, slot, cid, snapshot, crosscheck, - wait, tmfd, - lockmode, update_indexes); + wait, tmfd, lockmode, + update_indexes, estate); } /* diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index d12e3f451d2..b5fe74f176b 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -689,6 +689,11 @@ extern void check_exclusion_constraint(Relation heap, Relation index, ItemPointer tupleid, const Datum *values, const bool *isnull, EState *estate, bool newIndex); +extern bool ExecExpressionIndexesUpdated(Relation relation, + Bitmapset *modified_attrs, + EState *estate, + HeapTuple old_tuple, + HeapTuple new_tuple); /* * prototypes from functions in execReplication.c diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index a323fa98bbb..6e5f60cad76 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -160,12 +160,15 @@ typedef struct ExprState * * NumIndexAttrs total number of columns in this index * NumIndexKeyAttrs number of key columns in index + * IndexAttrs bitmap of index attributes * IndexAttrNumbers underlying-rel attribute numbers used as keys * (zeroes indicate expressions). It also contains * info about included columns. * Expressions expr trees for expression entries, or NIL if none + * ExpressionAttrs bitmap of attributes used within the expression * ExpressionsState exec state for expressions, or NIL if none * Predicate partial-index predicate, or NIL if none + * PredicateAttrs bitmap of attributes used within the predicate * PredicateState exec state for predicate, or NIL if none * ExclusionOps Per-column exclusion operators, or NULL if none * ExclusionProcs Underlying function OIDs for ExclusionOps @@ -184,6 +187,7 @@ typedef struct ExprState * ParallelWorkers # of workers requested (excludes leader) * Am Oid of index AM * AmCache private cache area for index AM + * OpClassDataTypes operator class data types * Context memory context holding this IndexInfo * * ii_Concurrent, ii_BrokenHotChain, and ii_ParallelWorkers are used only @@ -195,10 +199,15 @@ typedef struct IndexInfo NodeTag type; int ii_NumIndexAttrs; /* total number of columns in index */ int ii_NumIndexKeyAttrs; /* number of key columns in index */ + Bitmapset *ii_IndexAttrs; AttrNumber ii_IndexAttrNumbers[INDEX_MAX_KEYS]; + uint16 ii_IndexAttrLen[INDEX_MAX_KEYS]; + Bitmapset *ii_IndexAttrByVal; List *ii_Expressions; /* list of Expr */ + Bitmapset *ii_ExpressionAttrs; List *ii_ExpressionsState; /* list of ExprState */ List *ii_Predicate; /* list of Expr */ + Bitmapset *ii_PredicateAttrs; ExprState *ii_PredicateState; Oid *ii_ExclusionOps; /* array with one entry per column */ Oid *ii_ExclusionProcs; /* array with one entry per column */ diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index db3e504c3d2..39975672da1 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -164,6 +164,7 @@ typedef struct RelationData Bitmapset *rd_idattr; /* included in replica identity index */ Bitmapset *rd_hotblockingattr; /* cols blocking HOT update */ Bitmapset *rd_summarizedattr; /* cols indexed by summarizing indexes */ + Bitmapset *rd_expressionattr; /* indexed cols referenced by expressions */ PublicationDesc *rd_pubdesc; /* publication descriptor, or NULL */ @@ -344,6 +345,7 @@ typedef struct StdRdOptions int parallel_workers; /* max number of parallel workers */ StdRdOptIndexCleanup vacuum_index_cleanup; /* controls index vacuuming */ bool vacuum_truncate; /* enables vacuum to truncate a relation */ + bool expression_checks; /* use expression to checks for changes */ /* * Fraction of pages in a relation that vacuum can eagerly scan and fail @@ -405,6 +407,14 @@ typedef struct StdRdOptions ((relation)->rd_options ? \ ((StdRdOptions *) (relation)->rd_options)->parallel_workers : (defaultpw)) +/* + * RelationGetExpressionChecks + * Returns the relation's expression_checks reloption setting. + */ +#define RelationGetExpressionChecks(relation) \ + ((relation)->rd_options ? \ + ((StdRdOptions *) (relation)->rd_options)->expression_checks : true) + /* ViewOptions->check_option values */ typedef enum ViewOptCheckOption { diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.h index a7c55db339e..0cc28cb97e2 100644 --- a/src/include/utils/relcache.h +++ b/src/include/utils/relcache.h @@ -63,6 +63,7 @@ typedef enum IndexAttrBitmapKind INDEX_ATTR_BITMAP_IDENTITY_KEY, INDEX_ATTR_BITMAP_HOT_BLOCKING, INDEX_ATTR_BITMAP_SUMMARIZED, + INDEX_ATTR_BITMAP_EXPRESSION, } IndexAttrBitmapKind; extern Bitmapset *RelationGetIndexAttrBitmap(Relation relation, diff --git a/src/test/regress/expected/heap_hot_updates.out b/src/test/regress/expected/heap_hot_updates.out new file mode 100644 index 00000000000..027c8760d69 --- /dev/null +++ b/src/test/regress/expected/heap_hot_updates.out @@ -0,0 +1,665 @@ +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table will have two columns and two indexes, one on the primary key +-- id and one on the expression (info->>'name'). That means that the indexed +-- attributes are 'id' and 'info'. +create table keyvalue(id integer primary key, info jsonb); +create index nameindex on keyvalue((info->>'name')); +insert into keyvalue values (1, '{"name": "john", "data": "some data"}'); +-- Disable expression checks. +ALTER TABLE keyvalue SET (expression_checks = false); +SELECT reloptions FROM pg_class WHERE relname = 'keyvalue'; + reloptions +--------------------------- + {expression_checks=false} +(1 row) + +-- While the indexed attribute "name" is unchanged we've disabled expression +-- checks so this update should not go HOT as the system can't determine if +-- the indexed attribute has changed without evaluating the expression. +update keyvalue set info='{"name": "john", "data": "something else"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 0 row + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Re-enable expression checks. +ALTER TABLE keyvalue SET (expression_checks = true); +SELECT reloptions FROM pg_class WHERE relname = 'keyvalue'; + reloptions +-------------------------- + {expression_checks=true} +(1 row) + +-- The indexed attribute "name" with value "john" is unchanged, expect a HOT update. +update keyvalue set info='{"name": "john", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 1 row + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- The following update changes the indexed attribute "name", this should not be a HOT update. +update keyvalue set info='{"name": "smith", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Now, this update does not change the indexed attribute "name" from "smith", this should be HOT. +update keyvalue set info='{"name": "smith", "data": "some more data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 2 rows now + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 2 +(1 row) + +drop table keyvalue; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table is the same as the previous one but it has a third index. The +-- index 'colindex' isn't an expression index, it indexes the entire value +-- in the info column. There are still only two indexed attributes for this +-- relation, the same two as before. The presence of an index on the entire +-- value of the info column should prevent HOT updates for any updates to any +-- portion of JSONB content in that column. +create table keyvalue(id integer primary key, info jsonb); +create index nameindex on keyvalue((info->>'name')); +create index colindex on keyvalue(info); +insert into keyvalue values (1, '{"name": "john", "data": "some data"}'); +-- This update doesn't change the value of the expression index, but it does +-- change the content of the info column and so should not be HOT because the +-- indexed value changed as a result of the update. +update keyvalue set info='{"name": "john", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 0 rows + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +drop table keyvalue; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- The table has one column docs and two indexes. They are both expression +-- indexes referencing the same column attribute (docs) but one is a partial +-- index. +CREATE TABLE ex (docs JSONB) WITH (fillfactor = 60); +INSERT INTO ex (docs) VALUES ('{"a": 0, "b": 0}'); +INSERT INTO ex (docs) SELECT jsonb_build_object('b', n) FROM generate_series(100, 10000) as n; +CREATE INDEX idx_ex_a ON ex ((docs->>'a')); +CREATE INDEX idx_ex_b ON ex ((docs->>'b')) WHERE (docs->>'b')::numeric > 9; +-- We're using BTREE indexes and for this test we want to make sure that they remain +-- in sync with changes to our relation. Force the choice of index scans below so +-- that we know we're checking the index's understanding of what values should be +-- in the index or not. +SET SESSION enable_seqscan = OFF; +SET SESSION enable_bitmapscan = OFF; +-- Leave 'a' unchanged but modify 'b' to a value outside of the index predicate. +-- This should be a HOT update because neither index is changed. +UPDATE ex SET docs = jsonb_build_object('a', 0, 'b', 1) WHERE (docs->>'a')::numeric = 0; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 row + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Let's check to make sure that the index does not contain a value for 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + QUERY PLAN +-------------------------------------------------------------- + Index Scan using idx_ex_b on ex + Filter: (((docs ->> 'b'::text))::numeric < '100'::numeric) +(2 rows) + +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + docs +------ +(0 rows) + +-- Leave 'a' unchanged but modify 'b' to a value within the index predicate. +-- This represents a change for field 'b' from unindexed to indexed and so +-- this should not take the HOT path. +UPDATE ex SET docs = jsonb_build_object('a', 0, 'b', 10) WHERE (docs->>'a')::numeric = 0; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Let's check to make sure that the index contains the new value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + QUERY PLAN +-------------------------------------------------------------- + Index Scan using idx_ex_b on ex + Filter: (((docs ->> 'b'::text))::numeric < '100'::numeric) +(2 rows) + +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + docs +------------------- + {"a": 0, "b": 10} +(1 row) + +-- This update modifies the value of 'a', an indexed field, so it also cannot +-- be a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 1, 'b', 10) WHERE (docs->>'b')::numeric = 10; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- This update changes both 'a' and 'b' to new values that require index updates, +-- this cannot use the HOT path. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 12) WHERE (docs->>'b')::numeric = 10; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Let's check to make sure that the index contains the new value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + QUERY PLAN +-------------------------------------------------------------- + Index Scan using idx_ex_b on ex + Filter: (((docs ->> 'b'::text))::numeric < '100'::numeric) +(2 rows) + +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + docs +------------------- + {"a": 2, "b": 12} +(1 row) + +-- This update changes 'b' to a value outside its predicate requiring that +-- we remove it from the index. That's a transition that can't be done +-- during a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 1) WHERE (docs->>'b')::numeric = 12; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Let's check to make sure that the index no longer contains the value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + QUERY PLAN +-------------------------------------------------------------- + Index Scan using idx_ex_b on ex + Filter: (((docs ->> 'b'::text))::numeric < '100'::numeric) +(2 rows) + +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + docs +------ +(0 rows) + +-- Disable expression checks. +ALTER TABLE ex SET (expression_checks = false); +SELECT reloptions FROM pg_class WHERE relname = 'ex'; + reloptions +----------------------------------------- + {fillfactor=60,expression_checks=false} +(1 row) + +-- This update changes 'b' to a value within its predicate just like it +-- previous value, which would allow for a HOT update but with expression +-- checks disabled we can't determine that so this should not be a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 2) WHERE (docs->>'b')::numeric = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Let's make sure we're recording HOT updates for our 'ex' relation properly in the system +-- table pg_stat_user_tables. Note that statistics are stored within a transaction context +-- first (xact) and then later into the global statistics for a relation, so first we need +-- to ensure pending stats are flushed. +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT + c.relname AS table_name, + -- Transaction statistics + pg_stat_get_xact_tuples_updated(c.oid) AS xact_updates, + pg_stat_get_xact_tuples_hot_updated(c.oid) AS xact_hot_updates, + ROUND(( + pg_stat_get_xact_tuples_hot_updated(c.oid)::float / + NULLIF(pg_stat_get_xact_tuples_updated(c.oid), 0) * 100 + )::numeric, 2) AS xact_hot_update_percentage, + -- Cumulative statistics + s.n_tup_upd AS total_updates, + s.n_tup_hot_upd AS hot_updates, + ROUND(( + s.n_tup_hot_upd::float / + NULLIF(s.n_tup_upd, 0) * 100 + )::numeric, 2) AS total_hot_update_percentage +FROM pg_class c +LEFT JOIN pg_stat_user_tables s ON c.relname = s.relname +WHERE c.relname = 'ex' +AND c.relnamespace = 'public'::regnamespace; + table_name | xact_updates | xact_hot_updates | xact_hot_update_percentage | total_updates | hot_updates | total_hot_update_percentage +------------+--------------+------------------+----------------------------+---------------+-------------+----------------------------- + ex | 0 | 0 | | 6 | 1 | 16.67 +(1 row) + +-- expect: 5 xact updates with 1 xact hot update and no cumulative updates as yet +SET SESSION enable_seqscan = ON; +SET SESSION enable_bitmapscan = ON; +DROP TABLE ex; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table has a single column 'email' and a unique constraint on it that +-- should preclude HOT updates. +CREATE TABLE users ( + user_id serial primary key, + name VARCHAR(255) NOT NULL, + email VARCHAR(255) NOT NULL, + EXCLUDE USING btree (lower(email) WITH =) +); +-- Add some data to the table and then update it in ways that should and should +-- not be HOT updates. +INSERT INTO users (name, email) VALUES +('user1', '[email protected]'), +('user2', '[email protected]'), +('taken', '[email protected]'), +('you', '[email protected]'), +('taken', '[email protected]'); +-- Should fail because of the unique constraint on the email column. +UPDATE users SET email = '[email protected]' WHERE email = '[email protected]'; +ERROR: conflicting key value violates exclusion constraint "users_lower_excl" +DETAIL: Key (lower(email::text))=([email protected]) conflicts with existing key (lower(email::text))=([email protected]). +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 0 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Should succeed because the email column is not being updated and should go HOT. +UPDATE users SET name = 'foo' WHERE email = '[email protected]'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 a single new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Create a partial index on the email column, updates +CREATE INDEX idx_users_email_no_example ON users (lower(email)) WHERE lower(email) LIKE '%@example.com%'; +-- An update that changes the email column but not the indexed portion of it and falls outside the constraint. +-- Shouldn't be a HOT update because of the exclusion constraint. +UPDATE users SET email = '[email protected]' WHERE name = 'you'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- An update that changes the email column but not the indexed portion of it and falls within the constraint. +-- Again, should fail constraint and fail to be a HOT update. +UPDATE users SET email = '[email protected]' WHERE name = 'you'; +ERROR: conflicting key value violates exclusion constraint "users_lower_excl" +DETAIL: Key (lower(email::text))=([email protected]) conflicts with existing key (lower(email::text))=([email protected]). +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +DROP TABLE users; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- Another test of constraints spoiling HOT updates, this time with a range. +CREATE TABLE events ( + id serial primary key, + name VARCHAR(255) NOT NULL, + event_time tstzrange, + constraint no_screening_time_overlap exclude using gist ( + event_time WITH && + ) +); +-- Add two non-overlapping events. +INSERT INTO events (id, event_time, name) +VALUES + (1, '["2023-01-01 19:00:00", "2023-01-01 20:45:00"]', 'event1'), + (2, '["2023-01-01 21:00:00", "2023-01-01 21:45:00"]', 'event2'); +-- Update the first event to overlap with the second, should fail the constraint and not be HOT. +UPDATE events SET event_time = '["2023-01-01 20:00:00", "2023-01-01 21:45:00"]' WHERE id = 1; +ERROR: conflicting key value violates exclusion constraint "no_screening_time_overlap" +DETAIL: Key (event_time)=(["Sun Jan 01 20:00:00 2023 PST","Sun Jan 01 21:45:00 2023 PST"]) conflicts with existing key (event_time)=(["Sun Jan 01 21:00:00 2023 PST","Sun Jan 01 21:45:00 2023 PST"]). +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 0 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Update the first event to not overlap with the second, again not HOT due to the constraint. +UPDATE events SET event_time = '["2023-01-01 22:00:00", "2023-01-01 22:45:00"]' WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 0 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Update the first event to not overlap with the second, this time we're HOT because we don't overlap with the constraint. +UPDATE events SET name = 'new name here' WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 1 one new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +DROP TABLE events; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- A test to ensure that only modified summarizing indexes are updated, not +-- all of them. +CREATE TABLE ex (id SERIAL primary key, att1 JSONB, att2 text, att3 text, att4 text) WITH (fillfactor = 60); +CREATE INDEX ex_expr1_idx ON ex USING btree((att1->'data')); +CREATE INDEX ex_sumr1_idx ON ex USING BRIN(att2); +CREATE INDEX ex_expr2_idx ON ex USING btree((att1->'a')); +CREATE INDEX ex_expr3_idx ON ex USING btree((att1->'b')); +CREATE INDEX ex_expr4_idx ON ex USING btree((att1->'c')); +CREATE INDEX ex_sumr2_idx ON ex USING BRIN(att3); +CREATE INDEX ex_sumr3_idx ON ex USING BRIN(att4); +CREATE INDEX ex_expr5_idx ON ex USING btree((att1->'d')); +INSERT INTO ex (att1, att2) VALUES ('{"data": []}'::json, 'nothing special'); +SELECT * FROM ex; + id | att1 | att2 | att3 | att4 +----+--------------+-----------------+------+------ + 1 | {"data": []} | nothing special | | +(1 row) + +-- Update att2 and att4 both are BRIN/summarizing indexes, this should be a HOT update and +-- only update two of the three summarizing indexes. +UPDATE ex SET att2 = 'special indeed', att4 = 'whatever'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 one new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +SELECT * FROM ex; + id | att1 | att2 | att3 | att4 +----+--------------+----------------+------+---------- + 1 | {"data": []} | special indeed | | whatever +(1 row) + +-- Update att1 and att2, only one is BRIN/summarizing, this should NOT be a HOT update. +UPDATE ex SET att1 = att1 || '{"data": "howdy"}', att2 = 'special, so special'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +SELECT * FROM ex; + id | att1 | att2 | att3 | att4 +----+-------------------+---------------------+------+---------- + 1 | {"data": "howdy"} | special, so special | | whatever +(1 row) + +-- Update att2, att3, and att4 all are BRIN/summarizing indexes, this should be a HOT update +-- and yet still update all three summarizing indexes. +UPDATE ex SET att2 = 'a', att3 = 'b', att4 = 'c'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 2 with one new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 2 +(1 row) + +SELECT * FROM ex; + id | att1 | att2 | att3 | att4 +----+-------------------+------+------+------ + 1 | {"data": "howdy"} | a | b | c +(1 row) + +-- Update att1, att2, and att3 all modified values are BRIN/summarizing indexes, this should be a HOT update +-- and yet still update all three summarizing indexes. +UPDATE ex SET att1 = '{"data": "howdy"}', att2 = 'd', att3 = 'e'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 3 with one new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 3 +(1 row) + +SELECT * FROM ex; + id | att1 | att2 | att3 | att4 +----+-------------------+------+------+------ + 1 | {"data": "howdy"} | d | e | c +(1 row) + +DROP TABLE ex; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- A test to ensure that summarizing indexes are not updated when they don't +-- change, but are updated when they do while not prefent HOT updates. +CREATE TABLE ex (att1 JSONB, att2 text) WITH (fillfactor = 60); +CREATE INDEX ex_expr1_idx ON ex USING btree((att1->'data')); +CREATE INDEX ex_sumr1_idx ON ex USING BRIN(att2); +INSERT INTO ex VALUES ('{"data": []}', 'nothing special'); +-- Update the unindexed value of att1, this should be a HOT update and and should +-- update the summarizing index. +UPDATE ex SET att1 = att1 || '{"status": "stalemate"}'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 one new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Update the indexed value of att2, a summarized value, this is a summarized +-- only update and should use the HOT path while still triggering an update to +-- the summarizing BRIN index. +UPDATE ex SET att2 = 'special indeed'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 2 one new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 2 +(1 row) + +-- Update to att1 doesn't change the indexed value while the update to att2 does, +-- this again is a summarized only update and should use the HOT path as well as +-- trigger an update to the BRIN index. +UPDATE ex SET att1 = att1 || '{"status": "checkmate"}', att2 = 'special, so special'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 3 one new HOT update + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 3 +(1 row) + +-- This updates both indexes, the expression index on att1 and the summarizing +-- index on att2. This should not be a HOT update because there are modified +-- indexes and only some are summarized, not all. This should force all +-- indexes to be updated. +UPDATE ex SET att1 = att1 || '{"data": [1,2,3]}', att2 = 'do you want to play a game?'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 4 both indexes updated + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 3 +(1 row) + +DROP TABLE ex; +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This test is for a table with a custom type and a custom operators on +-- the BTREE index. The question is, when comparing values for equality +-- to determine if there are changes on the index or not... shouldn't we +-- be using the custom operators? +-- Create a type +CREATE TYPE my_custom_type AS (val int); +-- Comparison functions (returns boolean) +CREATE FUNCTION my_custom_lt(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val < b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +CREATE FUNCTION my_custom_le(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val <= b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +CREATE FUNCTION my_custom_eq(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val = b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +CREATE FUNCTION my_custom_ge(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val >= b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +CREATE FUNCTION my_custom_gt(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val > b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +CREATE FUNCTION my_custom_ne(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val != b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +-- Comparison function (returns -1, 0, 1) +CREATE FUNCTION my_custom_cmp(a my_custom_type, b my_custom_type) RETURNS int AS $$ +BEGIN + IF a.val < b.val THEN + RETURN -1; + ELSIF a.val > b.val THEN + RETURN 1; + ELSE + RETURN 0; + END IF; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +-- Create the operators +CREATE OPERATOR < ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_lt, + COMMUTATOR = >, + NEGATOR = >= +); +CREATE OPERATOR <= ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_le, + COMMUTATOR = >=, + NEGATOR = > +); +CREATE OPERATOR = ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_eq, + COMMUTATOR = =, + NEGATOR = <> +); +CREATE OPERATOR >= ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_ge, + COMMUTATOR = <=, + NEGATOR = < +); +CREATE OPERATOR > ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_gt, + COMMUTATOR = <, + NEGATOR = <= +); +CREATE OPERATOR <> ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_ne, + COMMUTATOR = <>, + NEGATOR = = +); +-- Create the operator class (including the support function) +CREATE OPERATOR CLASS my_custom_ops + DEFAULT FOR TYPE my_custom_type USING btree AS + OPERATOR 1 <, + OPERATOR 2 <=, + OPERATOR 3 =, + OPERATOR 4 >=, + OPERATOR 5 >, + FUNCTION 1 my_custom_cmp(my_custom_type, my_custom_type); +-- Create the table +CREATE TABLE my_table ( + id int, + custom_val my_custom_type +); +-- Insert some data +INSERT INTO my_table (id, custom_val) VALUES +(1, ROW(3)::my_custom_type), +(2, ROW(1)::my_custom_type), +(3, ROW(4)::my_custom_type), +(4, ROW(2)::my_custom_type); +-- Create a function to use when indexing +CREATE OR REPLACE FUNCTION abs_val(val my_custom_type) RETURNS int AS $$ +BEGIN + RETURN abs(val.val); +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; +-- Create the index +CREATE INDEX idx_custom_val_abs ON my_table (abs_val(custom_val)); +-- Update 1 +UPDATE my_table SET custom_val = ROW(5)::my_custom_type WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Update 2 +UPDATE my_table SET custom_val = ROW(0)::my_custom_type WHERE custom_val < ROW(3)::my_custom_type; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Update 3 +UPDATE my_table SET custom_val = ROW(6)::my_custom_type WHERE id = 3; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 0 +(1 row) + +-- Update 4 +UPDATE my_table SET id = 5 WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + pg_stat_get_xact_tuples_hot_updated +------------------------------------- + 1 +(1 row) + +-- Query using the index +EXPLAIN (COSTS OFF) SELECT * FROM my_table WHERE abs_val(custom_val) = 6; + QUERY PLAN +------------------------------------- + Seq Scan on my_table + Filter: (abs_val(custom_val) = 6) +(2 rows) + +SELECT * FROM my_table WHERE abs_val(custom_val) = 6; + id | custom_val +----+------------ + 3 | (6) +(1 row) + +-- Clean up +DROP TABLE my_table CASCADE; +DROP OPERATOR CLASS my_custom_ops USING btree CASCADE; +DROP OPERATOR < (my_custom_type, my_custom_type); +DROP OPERATOR <= (my_custom_type, my_custom_type); +DROP OPERATOR = (my_custom_type, my_custom_type); +DROP OPERATOR >= (my_custom_type, my_custom_type); +DROP OPERATOR > (my_custom_type, my_custom_type); +DROP OPERATOR <> (my_custom_type, my_custom_type); +DROP FUNCTION my_custom_lt(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_le(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_eq(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_ge(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_gt(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_ne(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_cmp(my_custom_type, my_custom_type); +DROP FUNCTION abs_val(my_custom_type); +DROP TYPE my_custom_type CASCADE; diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 37b6d21e1f9..7ab9712e8a8 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -68,6 +68,11 @@ test: select_into select_distinct select_distinct_on select_implicit select_havi # ---------- test: brin gin gist spgist privileges init_privs security_label collate matview lock replica_identity rowsecurity object_address tablesample groupingsets drop_operator password identity generated_stored join_hash +# ---------- +# Another group of parallel tests +# ---------- +test: heap_hot_updates + # ---------- # Additional BRIN tests # ---------- diff --git a/src/test/regress/sql/heap_hot_updates.sql b/src/test/regress/sql/heap_hot_updates.sql new file mode 100644 index 00000000000..56ec4f2f96c --- /dev/null +++ b/src/test/regress/sql/heap_hot_updates.sql @@ -0,0 +1,481 @@ +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table will have two columns and two indexes, one on the primary key +-- id and one on the expression (info->>'name'). That means that the indexed +-- attributes are 'id' and 'info'. +create table keyvalue(id integer primary key, info jsonb); +create index nameindex on keyvalue((info->>'name')); +insert into keyvalue values (1, '{"name": "john", "data": "some data"}'); + +-- Disable expression checks. +ALTER TABLE keyvalue SET (expression_checks = false); +SELECT reloptions FROM pg_class WHERE relname = 'keyvalue'; + +-- While the indexed attribute "name" is unchanged we've disabled expression +-- checks so this update should not go HOT as the system can't determine if +-- the indexed attribute has changed without evaluating the expression. +update keyvalue set info='{"name": "john", "data": "something else"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 0 row + +-- Re-enable expression checks. +ALTER TABLE keyvalue SET (expression_checks = true); +SELECT reloptions FROM pg_class WHERE relname = 'keyvalue'; + +-- The indexed attribute "name" with value "john" is unchanged, expect a HOT update. +update keyvalue set info='{"name": "john", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 1 row + +-- The following update changes the indexed attribute "name", this should not be a HOT update. +update keyvalue set info='{"name": "smith", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 1 no new HOT updates + +-- Now, this update does not change the indexed attribute "name" from "smith", this should be HOT. +update keyvalue set info='{"name": "smith", "data": "some more data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 2 rows now +drop table keyvalue; + + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table is the same as the previous one but it has a third index. The +-- index 'colindex' isn't an expression index, it indexes the entire value +-- in the info column. There are still only two indexed attributes for this +-- relation, the same two as before. The presence of an index on the entire +-- value of the info column should prevent HOT updates for any updates to any +-- portion of JSONB content in that column. +create table keyvalue(id integer primary key, info jsonb); +create index nameindex on keyvalue((info->>'name')); +create index colindex on keyvalue(info); +insert into keyvalue values (1, '{"name": "john", "data": "some data"}'); + +-- This update doesn't change the value of the expression index, but it does +-- change the content of the info column and so should not be HOT because the +-- indexed value changed as a result of the update. +update keyvalue set info='{"name": "john", "data": "some other data"}' where id=1; +select pg_stat_get_xact_tuples_hot_updated('keyvalue'::regclass); -- expect: 0 rows +drop table keyvalue; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- The table has one column docs and two indexes. They are both expression +-- indexes referencing the same column attribute (docs) but one is a partial +-- index. +CREATE TABLE ex (docs JSONB) WITH (fillfactor = 60); +INSERT INTO ex (docs) VALUES ('{"a": 0, "b": 0}'); +INSERT INTO ex (docs) SELECT jsonb_build_object('b', n) FROM generate_series(100, 10000) as n; +CREATE INDEX idx_ex_a ON ex ((docs->>'a')); +CREATE INDEX idx_ex_b ON ex ((docs->>'b')) WHERE (docs->>'b')::numeric > 9; + +-- We're using BTREE indexes and for this test we want to make sure that they remain +-- in sync with changes to our relation. Force the choice of index scans below so +-- that we know we're checking the index's understanding of what values should be +-- in the index or not. +SET SESSION enable_seqscan = OFF; +SET SESSION enable_bitmapscan = OFF; + +-- Leave 'a' unchanged but modify 'b' to a value outside of the index predicate. +-- This should be a HOT update because neither index is changed. +UPDATE ex SET docs = jsonb_build_object('a', 0, 'b', 1) WHERE (docs->>'a')::numeric = 0; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 row +-- Let's check to make sure that the index does not contain a value for 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + +-- Leave 'a' unchanged but modify 'b' to a value within the index predicate. +-- This represents a change for field 'b' from unindexed to indexed and so +-- this should not take the HOT path. +UPDATE ex SET docs = jsonb_build_object('a', 0, 'b', 10) WHERE (docs->>'a')::numeric = 0; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates +-- Let's check to make sure that the index contains the new value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + +-- This update modifies the value of 'a', an indexed field, so it also cannot +-- be a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 1, 'b', 10) WHERE (docs->>'b')::numeric = 10; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + +-- This update changes both 'a' and 'b' to new values that require index updates, +-- this cannot use the HOT path. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 12) WHERE (docs->>'b')::numeric = 10; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates +-- Let's check to make sure that the index contains the new value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + +-- This update changes 'b' to a value outside its predicate requiring that +-- we remove it from the index. That's a transition that can't be done +-- during a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 1) WHERE (docs->>'b')::numeric = 12; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates +-- Let's check to make sure that the index no longer contains the value of 'b' +EXPLAIN (COSTS OFF) SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; +SELECT * FROM ex WHERE (docs->>'b')::numeric > 9 AND (docs->>'b')::numeric < 100; + +-- Disable expression checks. +ALTER TABLE ex SET (expression_checks = false); +SELECT reloptions FROM pg_class WHERE relname = 'ex'; + +-- This update changes 'b' to a value within its predicate just like it +-- previous value, which would allow for a HOT update but with expression +-- checks disabled we can't determine that so this should not be a HOT update. +UPDATE ex SET docs = jsonb_build_object('a', 2, 'b', 2) WHERE (docs->>'b')::numeric = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates + + +-- Let's make sure we're recording HOT updates for our 'ex' relation properly in the system +-- table pg_stat_user_tables. Note that statistics are stored within a transaction context +-- first (xact) and then later into the global statistics for a relation, so first we need +-- to ensure pending stats are flushed. +SELECT pg_stat_force_next_flush(); +SELECT + c.relname AS table_name, + -- Transaction statistics + pg_stat_get_xact_tuples_updated(c.oid) AS xact_updates, + pg_stat_get_xact_tuples_hot_updated(c.oid) AS xact_hot_updates, + ROUND(( + pg_stat_get_xact_tuples_hot_updated(c.oid)::float / + NULLIF(pg_stat_get_xact_tuples_updated(c.oid), 0) * 100 + )::numeric, 2) AS xact_hot_update_percentage, + -- Cumulative statistics + s.n_tup_upd AS total_updates, + s.n_tup_hot_upd AS hot_updates, + ROUND(( + s.n_tup_hot_upd::float / + NULLIF(s.n_tup_upd, 0) * 100 + )::numeric, 2) AS total_hot_update_percentage +FROM pg_class c +LEFT JOIN pg_stat_user_tables s ON c.relname = s.relname +WHERE c.relname = 'ex' +AND c.relnamespace = 'public'::regnamespace; +-- expect: 5 xact updates with 1 xact hot update and no cumulative updates as yet + +SET SESSION enable_seqscan = ON; +SET SESSION enable_bitmapscan = ON; + +DROP TABLE ex; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This table has a single column 'email' and a unique constraint on it that +-- should preclude HOT updates. +CREATE TABLE users ( + user_id serial primary key, + name VARCHAR(255) NOT NULL, + email VARCHAR(255) NOT NULL, + EXCLUDE USING btree (lower(email) WITH =) +); + +-- Add some data to the table and then update it in ways that should and should +-- not be HOT updates. +INSERT INTO users (name, email) VALUES +('user1', '[email protected]'), +('user2', '[email protected]'), +('taken', '[email protected]'), +('you', '[email protected]'), +('taken', '[email protected]'); + +-- Should fail because of the unique constraint on the email column. +UPDATE users SET email = '[email protected]' WHERE email = '[email protected]'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 0 no new HOT updates + +-- Should succeed because the email column is not being updated and should go HOT. +UPDATE users SET name = 'foo' WHERE email = '[email protected]'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 a single new HOT update + +-- Create a partial index on the email column, updates +CREATE INDEX idx_users_email_no_example ON users (lower(email)) WHERE lower(email) LIKE '%@example.com%'; + +-- An update that changes the email column but not the indexed portion of it and falls outside the constraint. +-- Shouldn't be a HOT update because of the exclusion constraint. +UPDATE users SET email = '[email protected]' WHERE name = 'you'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 no new HOT updates + +-- An update that changes the email column but not the indexed portion of it and falls within the constraint. +-- Again, should fail constraint and fail to be a HOT update. +UPDATE users SET email = '[email protected]' WHERE name = 'you'; +SELECT pg_stat_get_xact_tuples_hot_updated('users'::regclass); -- expect: 1 no new HOT updates + +DROP TABLE users; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- Another test of constraints spoiling HOT updates, this time with a range. +CREATE TABLE events ( + id serial primary key, + name VARCHAR(255) NOT NULL, + event_time tstzrange, + constraint no_screening_time_overlap exclude using gist ( + event_time WITH && + ) +); + +-- Add two non-overlapping events. +INSERT INTO events (id, event_time, name) +VALUES + (1, '["2023-01-01 19:00:00", "2023-01-01 20:45:00"]', 'event1'), + (2, '["2023-01-01 21:00:00", "2023-01-01 21:45:00"]', 'event2'); + +-- Update the first event to overlap with the second, should fail the constraint and not be HOT. +UPDATE events SET event_time = '["2023-01-01 20:00:00", "2023-01-01 21:45:00"]' WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 0 no new HOT updates + +-- Update the first event to not overlap with the second, again not HOT due to the constraint. +UPDATE events SET event_time = '["2023-01-01 22:00:00", "2023-01-01 22:45:00"]' WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 0 no new HOT updates + +-- Update the first event to not overlap with the second, this time we're HOT because we don't overlap with the constraint. +UPDATE events SET name = 'new name here' WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('events'::regclass); -- expect: 1 one new HOT update + +DROP TABLE events; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- A test to ensure that only modified summarizing indexes are updated, not +-- all of them. +CREATE TABLE ex (id SERIAL primary key, att1 JSONB, att2 text, att3 text, att4 text) WITH (fillfactor = 60); +CREATE INDEX ex_expr1_idx ON ex USING btree((att1->'data')); +CREATE INDEX ex_sumr1_idx ON ex USING BRIN(att2); +CREATE INDEX ex_expr2_idx ON ex USING btree((att1->'a')); +CREATE INDEX ex_expr3_idx ON ex USING btree((att1->'b')); +CREATE INDEX ex_expr4_idx ON ex USING btree((att1->'c')); +CREATE INDEX ex_sumr2_idx ON ex USING BRIN(att3); +CREATE INDEX ex_sumr3_idx ON ex USING BRIN(att4); +CREATE INDEX ex_expr5_idx ON ex USING btree((att1->'d')); +INSERT INTO ex (att1, att2) VALUES ('{"data": []}'::json, 'nothing special'); + +SELECT * FROM ex; + +-- Update att2 and att4 both are BRIN/summarizing indexes, this should be a HOT update and +-- only update two of the three summarizing indexes. +UPDATE ex SET att2 = 'special indeed', att4 = 'whatever'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 one new HOT update +SELECT * FROM ex; + +-- Update att1 and att2, only one is BRIN/summarizing, this should NOT be a HOT update. +UPDATE ex SET att1 = att1 || '{"data": "howdy"}', att2 = 'special, so special'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 no new HOT updates +SELECT * FROM ex; + +-- Update att2, att3, and att4 all are BRIN/summarizing indexes, this should be a HOT update +-- and yet still update all three summarizing indexes. +UPDATE ex SET att2 = 'a', att3 = 'b', att4 = 'c'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 2 with one new HOT update +SELECT * FROM ex; + +-- Update att1, att2, and att3 all modified values are BRIN/summarizing indexes, this should be a HOT update +-- and yet still update all three summarizing indexes. +UPDATE ex SET att1 = '{"data": "howdy"}', att2 = 'd', att3 = 'e'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 3 with one new HOT update +SELECT * FROM ex; + +DROP TABLE ex; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- A test to ensure that summarizing indexes are not updated when they don't +-- change, but are updated when they do while not prefent HOT updates. +CREATE TABLE ex (att1 JSONB, att2 text) WITH (fillfactor = 60); +CREATE INDEX ex_expr1_idx ON ex USING btree((att1->'data')); +CREATE INDEX ex_sumr1_idx ON ex USING BRIN(att2); +INSERT INTO ex VALUES ('{"data": []}', 'nothing special'); + +-- Update the unindexed value of att1, this should be a HOT update and and should +-- update the summarizing index. +UPDATE ex SET att1 = att1 || '{"status": "stalemate"}'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 1 one new HOT update + +-- Update the indexed value of att2, a summarized value, this is a summarized +-- only update and should use the HOT path while still triggering an update to +-- the summarizing BRIN index. +UPDATE ex SET att2 = 'special indeed'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 2 one new HOT update + +-- Update to att1 doesn't change the indexed value while the update to att2 does, +-- this again is a summarized only update and should use the HOT path as well as +-- trigger an update to the BRIN index. +UPDATE ex SET att1 = att1 || '{"status": "checkmate"}', att2 = 'special, so special'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 3 one new HOT update + +-- This updates both indexes, the expression index on att1 and the summarizing +-- index on att2. This should not be a HOT update because there are modified +-- indexes and only some are summarized, not all. This should force all +-- indexes to be updated. +UPDATE ex SET att1 = att1 || '{"data": [1,2,3]}', att2 = 'do you want to play a game?'; +SELECT pg_stat_get_xact_tuples_hot_updated('ex'::regclass); -- expect: 4 both indexes updated + +DROP TABLE ex; + +-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- +-- This test is for a table with a custom type and a custom operators on +-- the BTREE index. The question is, when comparing values for equality +-- to determine if there are changes on the index or not... shouldn't we +-- be using the custom operators? + +-- Create a type +CREATE TYPE my_custom_type AS (val int); + +-- Comparison functions (returns boolean) +CREATE FUNCTION my_custom_lt(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val < b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE FUNCTION my_custom_le(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val <= b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE FUNCTION my_custom_eq(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val = b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE FUNCTION my_custom_ge(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val >= b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE FUNCTION my_custom_gt(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val > b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +CREATE FUNCTION my_custom_ne(a my_custom_type, b my_custom_type) RETURNS boolean AS $$ +BEGIN + RETURN a.val != b.val; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +-- Comparison function (returns -1, 0, 1) +CREATE FUNCTION my_custom_cmp(a my_custom_type, b my_custom_type) RETURNS int AS $$ +BEGIN + IF a.val < b.val THEN + RETURN -1; + ELSIF a.val > b.val THEN + RETURN 1; + ELSE + RETURN 0; + END IF; +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +-- Create the operators +CREATE OPERATOR < ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_lt, + COMMUTATOR = >, + NEGATOR = >= +); + +CREATE OPERATOR <= ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_le, + COMMUTATOR = >=, + NEGATOR = > +); + +CREATE OPERATOR = ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_eq, + COMMUTATOR = =, + NEGATOR = <> +); + +CREATE OPERATOR >= ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_ge, + COMMUTATOR = <=, + NEGATOR = < +); + +CREATE OPERATOR > ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_gt, + COMMUTATOR = <, + NEGATOR = <= +); + +CREATE OPERATOR <> ( + LEFTARG = my_custom_type, + RIGHTARG = my_custom_type, + PROCEDURE = my_custom_ne, + COMMUTATOR = <>, + NEGATOR = = +); + +-- Create the operator class (including the support function) +CREATE OPERATOR CLASS my_custom_ops + DEFAULT FOR TYPE my_custom_type USING btree AS + OPERATOR 1 <, + OPERATOR 2 <=, + OPERATOR 3 =, + OPERATOR 4 >=, + OPERATOR 5 >, + FUNCTION 1 my_custom_cmp(my_custom_type, my_custom_type); + +-- Create the table +CREATE TABLE my_table ( + id int, + custom_val my_custom_type +); + +-- Insert some data +INSERT INTO my_table (id, custom_val) VALUES +(1, ROW(3)::my_custom_type), +(2, ROW(1)::my_custom_type), +(3, ROW(4)::my_custom_type), +(4, ROW(2)::my_custom_type); + +-- Create a function to use when indexing +CREATE OR REPLACE FUNCTION abs_val(val my_custom_type) RETURNS int AS $$ +BEGIN + RETURN abs(val.val); +END; +$$ LANGUAGE plpgsql IMMUTABLE STRICT; + +-- Create the index +CREATE INDEX idx_custom_val_abs ON my_table (abs_val(custom_val)); + +-- Update 1 +UPDATE my_table SET custom_val = ROW(5)::my_custom_type WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + +-- Update 2 +UPDATE my_table SET custom_val = ROW(0)::my_custom_type WHERE custom_val < ROW(3)::my_custom_type; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + +-- Update 3 +UPDATE my_table SET custom_val = ROW(6)::my_custom_type WHERE id = 3; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + +-- Update 4 +UPDATE my_table SET id = 5 WHERE id = 1; +SELECT pg_stat_get_xact_tuples_hot_updated('my_table'::regclass); + +-- Query using the index +EXPLAIN (COSTS OFF) SELECT * FROM my_table WHERE abs_val(custom_val) = 6; +SELECT * FROM my_table WHERE abs_val(custom_val) = 6; + +-- Clean up +DROP TABLE my_table CASCADE; +DROP OPERATOR CLASS my_custom_ops USING btree CASCADE; +DROP OPERATOR < (my_custom_type, my_custom_type); +DROP OPERATOR <= (my_custom_type, my_custom_type); +DROP OPERATOR = (my_custom_type, my_custom_type); +DROP OPERATOR >= (my_custom_type, my_custom_type); +DROP OPERATOR > (my_custom_type, my_custom_type); +DROP OPERATOR <> (my_custom_type, my_custom_type); +DROP FUNCTION my_custom_lt(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_le(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_eq(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_ge(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_gt(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_ne(my_custom_type, my_custom_type); +DROP FUNCTION my_custom_cmp(my_custom_type, my_custom_type); +DROP FUNCTION abs_val(my_custom_type); +DROP TYPE my_custom_type CASCADE; -- 2.42.0 ^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: Expanding HOT updates for expression and partial indexes @ 2025-03-05 22:56 Matthias van de Meent <[email protected]> parent: Burd, Greg <[email protected]> 1 sibling, 0 replies; 6+ messages in thread From: Matthias van de Meent @ 2025-03-05 22:56 UTC (permalink / raw) To: Burd, Greg <[email protected]>; +Cc: pgsql-hackers Hi, Sorry for the delay. This is a reply for the mail thread up to 17 Feb, so it might be very out-of-date by now, in which case sorry for the noise. On Mon, 17 Feb 2025 at 20:54, Burd, Greg <[email protected]> wrote: > On Feb 15, 2025, at 5:49 AM, Matthias van de Meent <[email protected]> wrote: > > > > In HEAD, we have a clear indication of which classes of indexes to > > update, with TU_UpdateIndexes. With this patch, we have to derive that > > from the (lack of) bits in the bitmap that might be output by the > > table_update procedure. > > Yes, but... that "clear indication" is lacking the ability to convey more detailed information. It doesn't tell you which summarizing indexes really need updating just that as a result of being on the HOT path all summarizing indexes require updates. Agreed that it's not great if you want to know about which indexes were meaningfully updated. I think that barring significant advances in performance of update checks, we can devise a way of transfering this info to the table_tuple_update caller once we get a need for more detailed information (e.g. this could be transfered through the IndexInfo* that's currently also used by index_unchanged_by_update). > > I think we can do with an additional parameter for which indexes would > > be updated (or store that info in the parameter which also will hold > > EState et al). I think it's cheaper that way, too - only when > > update_indexes could be TU_SUMMARIZING we might need the exact > > information for which indexes to insert new tuples into, and it only > > really needs to be sized to the number of summarizing indexes (usually > > small/nonexistent, but potentially huge). > > Okay, yes with this patch we need only concern ourselves with all, none, or some subset of summarizing as before. I'll work on the opaque parameter next iteration. Thanks! > > ----- > > > > I notice that ExecIndexesRequiringUpdates() does work on all indexes, > > rather than just indexes relevant to this exact phase of checking. I > > think that is a waste of time, so if we sort the indexes in order of > > [hotblocking without expressions, hotblocking with expressions, > > summarizing], then (with stored start/end indexes) we can save time in > > cases where there are comparatively few of the types we're not going > > to look at. > > If I plan on just having ExecIndexesRequiringUpdates() return a bool rather than a bitmap then sorting, or even just filtering the list of IndexInfo to only include indexes with expressions, makes sense. That way the only indexes in question in that function's loop will be those that may spoil the HOT path. When that list is length 0, we can skip the tests entirely. Yes, exactly. Though I'm not sure it should hit that path if the length is 0, as would mean we had expression indexes that matched the updated columns, but somehow none are in the list? > > You're extracting type info from the opclass, to use in > > datum_image_eq(). Couldn't you instead use the index relation's > > TupleDesc and its stored attribute information instead? That saves us > > from having to do further catalog lookups during execution. I'm also > > fairly sure that that information is supposed to be a more accurate > > representation of attributes' expression output types than the > > opclass' type information (though, they probably should match). > > I hadn't thought of that, I think it's a valid idea and I'll update accordingly. I think I understand what you are suggesting. Thanks, that change was exactly what I meant. > > > > ----- > > > > The operations applied in ExecIndexesRequiringUpdates partially > > duplicate those done in index_unchanged_by_update. Can we (partially) > > unify this, and pass which indexes were updated through the IndexInfo, > > rather than the current bitmap? > > I think I do that now, feel free to say otherwise. When the expression is checked in ExecIndexesExpressionsWereNotUpdated() I set: > > /* Shortcut index_unchanged_by_update(), we know the answer. */ indexInfo->ii_CheckedUnchanged = true; indexInfo->ii_IndexUnchanged = !changed; > > That prevents duplicate effort in index_unchanged_by_update(). Exactly, yes. > > ----- > > > > I don't see a good reason to add IndexInfo to Relation, by way of > > rd_indexInfoList. It seems like an ad-hoc way of passing data around, > > and I don't think that's the right way. > > At one point I'd created a way to get this set via relcache, I will resurrect that approach but I'm not sure it is what you were hinting at. AFAIK, we don't have IndexInfo in the relcaches currently. I'm very hesitant to add an executor node (!) subtype to catalog caches, as IndexInfos are also used to store temporary information about e.g. index tuple insertion state, which (if IndexInfo is stored in relcaches) would imply modifying relcache entries without any further locks, and I'm not sure that's at all an OK thing to do. > The current method avoids pulling a the lock on the index to build the list, but doing that once in relcache isn't horrible. Maybe you were suggesting using that opaque struct to pass around the list of IndexInfo? Let me know on this one if you had a specific idea. The swap I've made in v6 really just moves the IndexInfo list to a filtered list with a new name created in relcache. My main concern is the storage of executor nodes(!) directly in the relcache. I don't think we need that: We have relatively direct access to the right IndexInfo** in ResultRelInfo->ri_IndexRelationInfo, which I think should be sufficient for this purpose. (The relevant RRI is available in table_tuple_update caller ExecUpdateAct; and could be passed down by ExecSimpleRelationUpdate to simple_table_tuple_update, covering both (current, core) callers of table_tuple_update). That would then be passed down to the TableAM using an opaque pointer type; for example (names, file locations, exact layout all bikesheddable): /* tableam.h */ /* exact definition somewhere else, in e.g. an executor_internal.h */ typedef struct TU_UpdateIndexData TU_UpdateIndexData; table_tuple_update(..., TU_UpdateIndexData *idxupdate, ...) /* executor.h */ TU_UpdateIndexes UpdateDetermineChangedIndexes(TU_UpdateIndexData *idxupdate, TableTupleSlot *old, TableTupleSlot *new, bitmap *changed_atts, ...); /* executor_internal.h */ struct TU_UpdateIndexData { EState estate; IndexInfo **idxinfos; ... } ----- Looking at your later comments about RRI in patch v8, I think that would solve and clean up the way that you currently get access to the RRI and thus index set. Kind regards, Matthias van de Meent Neon (https://neon.tech) ^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2025-03-05 22:56 UTC | newest] Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2025-02-13 18:46 Re: Expanding HOT updates for expression and partial indexes Burd, Greg <[email protected]> 2025-02-15 10:49 ` Matthias van de Meent <[email protected]> 2025-02-17 19:53 ` Burd, Greg <[email protected]> 2025-02-18 18:09 ` Burd, Greg <[email protected]> 2025-03-05 17:20 ` Burd, Greg <[email protected]> 2025-03-05 22:56 ` Matthias van de Meent <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox