From: Greg Burd <greg@burd.me>
To: pgsql-hackers@postgresql.org
Cc: greg@burd.me
Cc: boekewurm+postgres@gmail.com
Cc: pgsql@j-davis.com
Cc: nathandbossart@gmail.com
Subject: Re: Expanding HOT updates for expression and partial indexes
Date: Thu, 22 Oct 2026 18:52:50 +0000
Message-ID: <atpXQaH1UBS3LY-6@floki> (raw)
In-Reply-To: <XtuZX8xWYFYMwfJ0BFIwteRjtst6uj1MVjGxiXNb8PuUJwkAUhvdQqeE1wbZPiYvmZ9G92o6CORDsp1vb3S8ci-lublEhHrFTil4GIYQBvU=@burd.me>
References: <7b8681a578b5fe77103948eeb7b5cdd80fabad5d.camel@j-davis.com>
<db765e0d-4674-4547-9a9d-6d4d9a0a123c@app.fastmail.com>
<acLppkl4hutHhiuH@nathan>
<4040d3a4-01c2-4f24-9025-57cfc19aea57@app.fastmail.com>
<acQYVfzHoL4lVTUE@nathan>
<6609db3402d0d2994144abc2081d1b790e644c0d.camel@j-davis.com>
<-ugooulGn7M5X4DFIzWHSgYRngUmXChD_a-JvcMaklnDNcHMGnQhcM82SpYY76Je5ss4TOrsQB5sv2USLRSlDcpwDX-6CgCCbGKlJuA0SOs=@protonmail.com>
<9XW-2xmCYWws1lnV5ELwZ67eaqyJNb9gJ6R2aUxntUuHetRia8NhzpSEl4gqtpKLtLwfqnBwnj5xOIs8N4Zd2x6x7gCyd5-HXocpMF4Fkj4=@burd.me>
<CAEze2WiCn74vqoqfeuK_9mhwFABRf7tTNdMUuMzoU=VRcQKkEQ@mail.gmail.com>
<XtuZX8xWYFYMwfJ0BFIwteRjtst6uj1MVjGxiXNb8PuUJwkAUhvdQqeE1wbZPiYvmZ9G92o6CORDsp1vb3S8ci-lublEhHrFTil4GIYQBvU=@burd.me>
Hello,
Attached is v43, eight patches on master (871dee0ccbe). It does what I
said I would in my last mail. The executor treats an indexed value
that is the same but stored differently as unchanged, for every access
method; the nbtree callback is gone; amcomparedatums is never asked
about a column the index can return; and the detoasting this costs is
measured below. 0009 and 0010 from v41 are still out of the set.
0005 amcomparedatums is not asked about a column the index can return
(index_can_return()), neither when the relcache decides which
columns are COMPARABLE nor when the executor compares an index's
datums. So the callback only answers for entries no index-only
scan can see, and an AM that later learns to return a column
loses the shortcut rather than giving a wrong answer.
dummy_index_am gets an amcanreturn driven by a reloption, and the
update its test makes HOT is not HOT with that turned on.
0006 Replaces "nbtree: implement amcomparedatums". When two values of
an indexed column differ as stored and one of them is compressed
or out of line, they are compared with datum_image_eq(), and if
they are the same the column counts as unchanged. That is done
once for each changed column, before any narrowing. The
narrowing then takes a column an index stores as changed exactly
when it is still in the changed set, so its values aren't
compared or fetched a second time; only the datums of expressions
are compared again, the same way. Columns of candidate keys and
of the replica identity are still compared as stored, because
heap_update() takes its lock strength and what it logs for
decoding from them; the cross-check in heap_update() in
assert-enabled builds allows for exactly that. hot_updates gets
four write-back cases: under SP-GiST, under a partial btree index
and under an expression that returns the value as is, all now
HOT, and under a unique index, which stays non-HOT.
Smaller things: heap_attr_equals()'s comment still named the function
0002 renamed, and a test had a redundant pg_stat_force_next_flush().
The cost of detoasting. Release builds on a c7i.8xlarge,
shared_buffers 8GB, fillfactor 50, with the data directory on tmpfs,
fsync and full_page_writes off, and a CHECKPOINT before each run. On
disk, checkpoints decided the result: the versions that make the
write-back cases HOT write less WAL, so their checkpoints come at
different points of the later workloads, which moved one of them by
40%. One UPDATE over every row of a 100k-row table (50k for hash,
500k for the controls), with 3000-byte values that compress, or 2500-
and 3200-byte values stored uncompressed out of line (STORAGE
EXTERNAL). Median of 12 runs (3 interleaved rounds of 5, the first
run of each dropped), in ms:
master v43 v43/master
value changes, same length
compressed, btree 1423 1498 +5.3%
out of line, btree 3428 3461 +0.9%
out of line, hash 1318 1491 +13.1%
value changes, different length
compressed, btree 6213 6250 +0.6%
same value written back
compressed, btree 3023 231 -92.4%
out of line, btree 2184 1245 -43.0%
compressed, SP-GiST 681 236 -65.3%
out of line, hash 730 660 -9.5%
controls
unindexed column changes 370 352 -4.9%
short indexed value changes 2113 2117 +0.2%
So your concern is right, and now we know what it costs. Nothing is
fetched when the lengths differ, or when an unchanged value keeps its
toast pointer. When a toasted value changes but keeps its length, the
old value is detoasted to find that it changed; the new one is still
in memory at that point. That costs 5% when the value is compressed,
and about 1% out of line under btree, where inserting a 2.5 kB key
costs much more. Out of line under hash is the worst case, at +13%,
and most of that isn't detoasting. hashcomparedatums() hashes both
3 kB values, and the insertion hashes the new one again, so perf shows
three times master's time in hash_bytes(); that is about two thirds of
the difference, and fetching the old value is the rest. v42 hashed the
same way. Without the changed set passed down, the narrowing compared
and fetched once more, and this case was +18%.
If the hashing matters, the executor could skip the callback for a
column whose type's equality implies image equality (btree's
equalimage): two values with different images then have the same hash
code only by collision. Numeric and nondeterministic collations,
where the callback earns its keep, would still be asked.
The write-back side is what ORMs and logical replication do with every
UPDATE: no version before v43 made the SP-GiST case HOT, and under hash
v42 made the write-back HOT but slower than master, because hashing
both values cost more than the index insertion saved; comparing images
first makes it 9.5% faster. The unindexed-column control came out
faster than master in every round, here and in an earlier run on disk.
I can't explain that and wouldn't count on it.
I haven't touched simple_heap_update(); I might try to deform both
tuples once, maybe it will be in the next version.
best.
-greg
Attachments:download all (8 files, .tar.gz)v43-0005-Add-amcomparedatums-index-AM-callback-for-HOT-up.patch (text/plain, 43.8K) download
From 348aeadbff5e2f570050276a7103d228277ff38d Mon Sep 17 00:00:00 2001
From: Greg Burd <greg@burd.me>
Date: Mon, 19 Oct 2026 19:10:37 -0400
Subject: [PATCH v43 5/8] Add amcomparedatums index AM callback for HOT updates
An UPDATE that changes a column an index stores directly, as a key or
included column, always gets new index entries, even when the index would
store exactly what it already does. Only the index's access method can
tell: a hash index stores a hash code, a GIN index the keys its operator
class extracts. Index access methods
for documents such as RUM store keys for a few paths of a document chosen
by operator class options, so to core every change to the document looks
like a change to the index.
Add an optional callback,
bool amcomparedatums(Relation index, IndexInfo *indexInfo, int attnum,
Oid typid, int16 typlen, bool typbyval,
Datum oldValue, Datum newValue);
which says whether the new, non-null value of an index column gives
exactly the same index entries as the old one. ExecUpdateModifiedIdxAttrs
already narrows the set of changed indexed attributes for those used only
through expressions or partial-index predicates, by forming each such
index's datums for the old and new row and comparing them byte for byte.
Treat a column that indexes also store directly the same way, if all of
those indexes have the callback, and ask the callback whenever an index
datum differs byte for byte. The new relcache bitmap
INDEX_ATTR_BITMAP_COMPARABLE names those columns.
The callback is not asked about a column the index can return in
index-only scans (amcanreturn): the index stores the value itself there,
so only the same value gives the same entries, and an access method that
learns to return a column later cannot be left giving a wrong answer.
The columns of candidate keys and of the replica identity are left out:
the table AM takes its tuple lock strength and what it logs for logical
decoding from the changed set, and those stay decided by value. So are the
indexes of exclusion constraints: the constraint is checked on the values
when an entry is inserted, and two values that give the same entries need
not have the same conflicts.
No core access method implements the callback yet, so nothing changes
until one does. dummy_index_am gets one, configured by its options, to
test the executor side.
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 51 ++++++
doc/src/sgml/storage.sgml | 8 +-
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/heap/README.HOT | 16 +-
src/backend/access/heap/heapam.c | 16 +-
src/backend/access/nbtree/nbtree.c | 1 +
src/backend/access/spgist/spgutils.c | 1 +
src/backend/executor/nodeModifyTable.c | 88 +++++++---
src/backend/utils/cache/relcache.c | 49 ++++++
src/include/access/amapi.h | 10 ++
src/include/nodes/execnodes.h | 3 +-
src/include/utils/rel.h | 3 +
src/include/utils/relcache.h | 1 +
src/test/modules/dummy_index_am/Makefile | 2 +-
.../modules/dummy_index_am/dummy_index_am.c | 38 ++++-
.../dummy_index_am/expected/comparedatums.out | 152 ++++++++++++++++++
src/test/modules/dummy_index_am/meson.build | 1 +
.../dummy_index_am/sql/comparedatums.sql | 82 ++++++++++
src/tools/pgindent/typedefs.list | 1 +
23 files changed, 492 insertions(+), 36 deletions(-)
create mode 100644 src/test/modules/dummy_index_am/expected/comparedatums.out
create mode 100644 src/test/modules/dummy_index_am/sql/comparedatums.sql
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.cindex 5111cdc6dd6..9fa7710739b 100644--- a/contrib/bloom/blutils.c+++ b/contrib/bloom/blutils.c@@ -155,6 +155,7 @@ blhandler(PG_FUNCTION_ARGS)
.amparallelrescan = NULL,
.amtranslatestrategy = NULL,
.amtranslatecmptype = NULL,
+ .amcomparedatums = NULL,
};
PG_RETURN_POINTER(&amroutine);
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgmlindex f48da318530..d91192bc011 100644--- a/doc/src/sgml/indexam.sgml+++ b/doc/src/sgml/indexam.sgml@@ -180,6 +180,9 @@ typedef struct IndexAmRoutine
/* interface functions to support planning */
amtranslate_strategy_function amtranslatestrategy; /* can be NULL */
amtranslate_cmptype_function amtranslatecmptype; /* can be NULL */
++ /* interface function to support HOT updates */+ amcomparedatums_function amcomparedatums; /* can be NULL */
} IndexAmRoutine;
</programlisting>
</para>
@@ -915,6 +918,54 @@ amtranslatecmptype (CompareType cmptype, Oid opfamily, Oid opcintype);
fully functional.
</para>
+ <para>+<programlisting>+bool+amcomparedatums (Relation indexRelation,+ IndexInfo *indexInfo,+ int attnum,+ Oid typid,+ int16 typlen,+ bool typbyval,+ Datum oldValue,+ Datum newValue);+</programlisting>+ Would inserting a row whose index column <parameter>attnum</parameter>+ (counting from 1) has the value <parameter>newValue</parameter> make+ exactly the same index entries as <parameter>oldValue</parameter> made?+ The values are not null, are of type <parameter>typid</parameter> (the+ type of the table column or index expression, which need not be the type+ the index stores), whose length and by-value property are+ <parameter>typlen</parameter> and <parameter>typbyval</parameter>, and+ are not equal byte for byte. The function is+ called during <command>UPDATE</command>, before the table access method+ is, for every index that uses a column whose value changed; if every such+ index answers <literal>true</literal> for every changed column, the update+ can be a heap-only tuple (<acronym>HOT</acronym>) update, which makes no new+ index entries at all (see <xref linkend="storage-hot"/>). The index then+ keeps the entries it made for the old value, and reaches the new row+ through them. <parameter>indexInfo</parameter> is the one+ <function>aminsert</function> would get for the new row, so the function+ may use <literal>ii_AmCache</literal> in the same way.+ </para>++ <para>+ So <literal>true</literal> is only correct if no scan can tell the old+ entries from the ones the new value would make: they must have the same+ keys, compared as the index compares them, and anything a scan returns+ from them must be the same. The function is not called for a column the+ index can return in index-only scans (see <function>amcanreturn</function>):+ the index stores the value itself there, so only the same value gives the+ same entries. If the function cannot be sure, it must return <literal>false</literal>, which+ is always correct. It must not modify the index, and it may be called for+ an index that is not yet valid. Columns of unique indexes that can be+ referenced by a foreign key, and of the replica identity index, are not+ compared this way, and the function is not called for the index of an+ exclusion constraint, whose check needs the values themselves. If <structfield>amcomparedatums</structfield> is+ <literal>NULL</literal>, a change to a column that the index stores+ always needs new index entries.+ </para>+
</sect1>
<sect1 id="index-scanning">
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgmlindex 4523d073a19..15b5dae1752 100644--- a/doc/src/sgml/storage.sgml+++ b/doc/src/sgml/storage.sgml@@ -1120,7 +1120,13 @@ data. Empty in ordinary tables.</entry>
<literal>SET</literal> clause changes only parts of the column that the
indexes do not read, for example through <function>jsonb_set</function>
or array subscripting, this is known without evaluating the indexes'
- expressions.+ expressions. A column that indexes store directly is treated the same+ way if the access method of each such index can tell whether the new+ value gives the same index entries as the old one; among the built-in+ methods, that depends on the method and on the change (see+ <xref linkend="indexam"/>). This does not apply to the columns of+ unique indexes that a foreign key can reference, of the replica+ identity index, or of exclusion constraints.
</para>
</listitem>
<listitem>
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.cindex 5059da8dce7..83da4850ddc 100644--- a/src/backend/access/brin/brin.c+++ b/src/backend/access/brin/brin.c@@ -307,6 +307,7 @@ brinhandler(PG_FUNCTION_ARGS)
.amparallelrescan = NULL,
.amtranslatestrategy = NULL,
.amtranslatecmptype = NULL,
+ .amcomparedatums = NULL,
};
PG_RETURN_POINTER(&amroutine);
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.cindex e7cba81d477..b57c016aea2 100644--- a/src/backend/access/gin/ginutil.c+++ b/src/backend/access/gin/ginutil.c@@ -90,6 +90,7 @@ ginhandler(PG_FUNCTION_ARGS)
.amestimateparallelscan = NULL,
.aminitparallelscan = NULL,
.amparallelrescan = NULL,
+ .amcomparedatums = NULL,
};
PG_RETURN_POINTER(&amroutine);
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.cindex 8565e225be7..0d562466821 100644--- a/src/backend/access/gist/gist.c+++ b/src/backend/access/gist/gist.c@@ -112,6 +112,7 @@ gisthandler(PG_FUNCTION_ARGS)
.amparallelrescan = NULL,
.amtranslatestrategy = NULL,
.amtranslatecmptype = gisttranslatecmptype,
+ .amcomparedatums = NULL,
};
PG_RETURN_POINTER(&amroutine);
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.cindex 63846c10840..0b05c4606e1 100644--- a/src/backend/access/hash/hash.c+++ b/src/backend/access/hash/hash.c@@ -123,6 +123,7 @@ hashhandler(PG_FUNCTION_ARGS)
.amparallelrescan = NULL,
.amtranslatestrategy = hashtranslatestrategy,
.amtranslatecmptype = hashtranslatecmptype,
+ .amcomparedatums = NULL,
};
PG_RETURN_POINTER(&amroutine);
diff --git a/src/backend/access/heap/README.HOT b/src/backend/access/heap/README.HOTindex 0377f508055..5c2dfc27f58 100644--- a/src/backend/access/heap/README.HOT+++ b/src/backend/access/heap/README.HOT@@ -41,10 +41,15 @@ A column that indexes reference only through expressions, or only as
partial indexes, may change if the update changes nothing those indexes
store: not the value of any of their expressions or columns, as compared
byte for byte, and not whether the tuple satisfies their predicates.
-The executor works that out before calling the table AM, and passes on-only the indexed columns whose change matters. An index that is not yet-ready for inserts, or whose build failed, counts as changed whenever a-column it references does; see "CREATE INDEX CONCURRENTLY" below.+The same holds for a column that indexes also store directly, if every+index that does so has an access method that can tell, from the old and+new values, that it would store the same entries (amcomparedatums), and+the column is in no candidate key, replica identity or exclusion+constraint. The executor works that out before calling the table AM,+and passes on only the indexed columns whose change matters. An index+that is not yet ready for inserts, or whose build failed, counts as+changed whenever a column it references does; see "CREATE INDEX+CONCURRENTLY" below.
Second, where the modified columns are only used in indexes that do not
contain tuple IDs, but maintain summaries of the indexed data by block.
@@ -506,7 +511,8 @@ Indexed column
actually be stored in the index --- it could be used in a
functional index's expression, or used in a partial index
predicate. A change to a column that indexes use only in these
- ways blocks HOT only if it changes what those indexes store.+ ways, or whose indexes can compare entries, blocks HOT only if it+ changes what those indexes store.
Redirecting line pointer
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.cindex e0cd1ab7612..a0e2d904beb 100644--- a/src/backend/access/heap/heapam.c+++ b/src/backend/access/heap/heapam.c@@ -3362,6 +3362,13 @@ heap_update(Relation relation, const ItemPointerData *otid, HeapTuple newtup,
check_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_INDEXED);
narrowable_attrs = RelationGetIndexAttrBitmap(relation,
INDEX_ATTR_BITMAP_EXPRESSION);
+ {+ Bitmapset *cmp_attrs = RelationGetIndexAttrBitmap(relation,+ INDEX_ATTR_BITMAP_COMPARABLE);++ narrowable_attrs = bms_add_members(narrowable_attrs, cmp_attrs);+ bms_free(cmp_attrs);+ }
#endif
block = ItemPointerGetBlockNumber(otid);
@@ -3801,10 +3808,11 @@ l2:
* attribute comparing the two tuples here finds changed is in the
* caller's set, so that a caller that misses a change, or compares the
* wrong tuples, is caught. The caller may leave out an attribute that
- * indexes use only through expressions or as partial indexes, having- * found that no index's entry changes (see ExecUpdateModifiedIdxAttrs()),- * and may report more: comparing slots, it cannot always tell that two- * values will be stored alike.+ * indexes use only through expressions or as partial indexes, or only in+ * indexes that can compare entries, having found that no index's entry+ * changes (see ExecUpdateModifiedIdxAttrs()), and may report more:+ * comparing slots, it cannot always tell that two values will be stored+ * alike.
*/
{
Bitmapset *changed = HeapUpdateModifiedIdxAttrs(relation, check_attrs,
diff --git a/src/backend/access/nbtree/nbtree.c b/src/backend/access/nbtree/nbtree.cindex 0abdd7b49f5..754ddaee71d 100644--- a/src/backend/access/nbtree/nbtree.c+++ b/src/backend/access/nbtree/nbtree.c@@ -171,6 +171,7 @@ bthandler(PG_FUNCTION_ARGS)
.amparallelrescan = btparallelrescan,
.amtranslatestrategy = bttranslatestrategy,
.amtranslatecmptype = bttranslatecmptype,
+ .amcomparedatums = NULL,
};
PG_RETURN_POINTER(&amroutine);
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.cindex f2ee333f60d..985a9474c9c 100644--- a/src/backend/access/spgist/spgutils.c+++ b/src/backend/access/spgist/spgutils.c@@ -97,6 +97,7 @@ spghandler(PG_FUNCTION_ARGS)
.amparallelrescan = NULL,
.amtranslatestrategy = NULL,
.amtranslatecmptype = NULL,
+ .amcomparedatums = NULL,
};
PG_RETURN_POINTER(&amroutine);
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.cindex 25bfb9bce55..4054f6da0df 100644--- a/src/backend/executor/nodeModifyTable.c+++ b/src/backend/executor/nodeModifyTable.c@@ -54,6 +54,8 @@
#include "postgres.h"
+#include "access/amapi.h"+#include "access/genam.h"
#include "access/htup_details.h"
#include "access/tableam.h"
#include "access/tupconvert.h"
@@ -198,41 +200,58 @@ static TupleTableSlot *ExecMergeNotMatched(ModifyTableContext *context,
bool canSetTag);
/*
- * Do the old and new index datums differ, byte for byte? Compare each- * column as the type FormIndexDatum() produced it in: the table column's- * type or the expression's, which need not be the type the index stores.+ * Do the old and new index datums differ? Compare each column as the type+ * FormIndexDatum() produced it in: the table column's type or the+ * expression's, which need not be the type the index stores. Datums that+ * differ byte for byte may still give the index the same entries; if its+ * access method can tell (amcomparedatums), ask it, but not for a column the+ * index can return in index-only scans, where only the same value will do.+ *+ * Not for an exclusion constraint's index, though: the constraint is checked+ * when an entry is inserted, with its operators, against the values, and two+ * values that give the same entries need not have the same conflicts. (A+ * value that is the same byte for byte has, so the check is not needed.)
*/
static bool
-IndexDatumsDiffer(Relation heapRel, IndexInfo *indexInfo,+IndexDatumsDiffer(Relation heapRel, Relation indexRel, IndexInfo *indexInfo,
const Datum *old_values, const bool *old_isnull,
const Datum *new_values, const bool *new_isnull)
{
ListCell *expr = list_head(indexInfo->ii_Expressions);
+ amcomparedatums_function cmp = indexRel->rd_index->indisexclusion ? NULL :+ indexRel->rd_indam->amcomparedatums;
for (int i = 0; i < indexInfo->ii_NumIndexAttrs; i++)
{
AttrNumber attnum = indexInfo->ii_IndexAttrNumbers[i];
+ Oid typid;
int16 typlen;
bool typbyval;
if (attnum != 0)
{
- CompactAttribute *att = TupleDescCompactAttr(RelationGetDescr(heapRel),- attnum - 1);+ Form_pg_attribute att = TupleDescAttr(RelationGetDescr(heapRel),+ attnum - 1);+ typid = att->atttypid;
typlen = att->attlen;
typbyval = att->attbyval;
}
else
{
- get_typlenbyval(exprType(lfirst(expr)), &typlen, &typbyval);+ typid = exprType(lfirst(expr));+ get_typlenbyval(typid, &typlen, &typbyval);
expr = lnext(indexInfo->ii_Expressions, expr);
}
if (old_isnull[i] != new_isnull[i])
return true;
- if (!old_isnull[i] &&- !datumIsEqual(old_values[i], new_values[i], typbyval, typlen))+ if (old_isnull[i] ||+ datumIsEqual(old_values[i], new_values[i], typbyval, typlen))+ continue;+ if (cmp == NULL || index_can_return(indexRel, i + 1) ||+ !cmp(indexRel, indexInfo, i + 1, typid, typlen, typbyval,+ old_values[i], new_values[i]))
return true;
}
return false;
@@ -248,9 +267,9 @@ IndexDatumsDiffer(Relation heapRel, IndexInfo *indexInfo,
* it in the query context.
*/
static bool
-IndexEntryChanges(ResultRelInfo *resultRelInfo, IndexInfo *indexInfo,- TupleTableSlot *old_tts, TupleTableSlot *new_tts,- EState *estate)+IndexEntryChanges(ResultRelInfo *resultRelInfo, Relation indexRel,+ IndexInfo *indexInfo, TupleTableSlot *old_tts,+ TupleTableSlot *new_tts, EState *estate)
{
ExprContext *econtext = GetPerTupleExprContext(estate);
TupleTableSlot *save_scantuple = econtext->ecxt_scantuple;
@@ -294,16 +313,18 @@ IndexEntryChanges(ResultRelInfo *resultRelInfo, IndexInfo *indexInfo,
FormIndexDatum(indexInfo, new_tts, estate, new_values, new_isnull);
econtext->ecxt_scantuple = save_scantuple;
- result = IndexDatumsDiffer(resultRelInfo->ri_RelationDesc, indexInfo,- old_values, old_isnull, new_values, new_isnull);+ result = IndexDatumsDiffer(resultRelInfo->ri_RelationDesc, indexRel,+ indexInfo, old_values, old_isnull,+ new_values, new_isnull);
MemoryContextSwitchTo(oldcxt);
return result;
}
/*
* Return the attributes of the i'th index that INDEX_ATTR_BITMAP_EXPRESSION
- * can contain: those it references through its expressions, and for a- * partial index all those it references. Builds the per-index cache on+ * or INDEX_ATTR_BITMAP_COMPARABLE can contain: those it references through+ * its expressions, and all those it references if it is a partial index or+ * its access method can compare entries. Builds the per-index cache on
* first use.
*/
static Bitmapset *
@@ -321,9 +342,12 @@ IndexExprAttrs(ResultRelInfo *resultRelInfo, int i, EState *estate)
Bitmapset *attrs = NULL;
pull_varattnos((Node *) ii->ii_Expressions, 1, &attrs);
- if (ii->ii_Predicate != NIL)+ pull_varattnos((Node *) ii->ii_Predicate, 1, &attrs);+ if (ii->ii_Predicate != NIL ||+ (resultRelInfo->ri_IndexRelationDescs[j] != NULL &&+ resultRelInfo->ri_IndexRelationDescs[j]->rd_indam->amcomparedatums != NULL &&+ !resultRelInfo->ri_IndexRelationDescs[j]->rd_index->indisexclusion))
{
- pull_varattnos((Node *) ii->ii_Predicate, 1, &attrs);
for (int k = 0; k < ii->ii_NumIndexAttrs; k++)
{
if (ii->ii_IndexAttrNumbers[k] != 0)
@@ -376,7 +400,9 @@ IndexUnaffected(ResultRelInfo *resultRelInfo, int i, Bitmapset *mine,
#ifdef USE_ASSERT_CHECKING
if (proven)
{
- if (IndexEntryChanges(resultRelInfo, indexInfo, old_tts, new_tts,+ if (IndexEntryChanges(resultRelInfo,+ resultRelInfo->ri_IndexRelationDescs[i],+ indexInfo, old_tts, new_tts,
estate))
elog(ERROR, "index \"%s\" proven unaffected by UPDATE, but its entry changed",
RelationGetRelationName(resultRelInfo->ri_IndexRelationDescs[i]));
@@ -387,7 +413,9 @@ IndexUnaffected(ResultRelInfo *resultRelInfo, int i, Bitmapset *mine,
return true;
#endif
- return !IndexEntryChanges(resultRelInfo, indexInfo, old_tts, new_tts,+ return !IndexEntryChanges(resultRelInfo,+ resultRelInfo->ri_IndexRelationDescs[i],+ indexInfo, old_tts, new_tts,
estate);
}
@@ -423,6 +451,12 @@ IndexUnaffected(ResultRelInfo *resultRelInfo, int i, Bitmapset *mine,
* update as if it had not changed the attribute: no index needs a new
* entry because of it.
*
+ * An attribute that non-summarizing indexes also reference directly, as key+ * or included columns, is a candidate too if all of those indexes' access+ * methods can compare entries (INDEX_ATTR_BITMAP_COMPARABLE): such an index+ * is unaffected when, column by column, its datums are equal or its access+ * method's amcomparedatums says they give the same entries.+ *
* An index that is not ready for inserts or not valid counts as affected by
* any change to an attribute it references, without evaluating anything.
* Such an index is being built, or its build failed, so the old row may be
@@ -430,9 +464,10 @@ IndexUnaffected(ResultRelInfo *resultRelInfo, int i, Bitmapset *mine,
* would make the UPDATE fail. It still constrains HOT, as README.HOT
* requires.
*
- * Narrowing never removes an attribute of a key or replica identity index,- * as those have neither expressions nor predicates, so the table AM may- * also take its lock mode and replica identity decisions from the result.+ * Narrowing never removes an attribute of a key or replica identity index:+ * those have neither expressions nor predicates, and their columns are left+ * out of INDEX_ATTR_BITMAP_COMPARABLE. So the table AM may also take its+ * lock mode and replica identity decisions from the result.
*/
Bitmapset *
ExecUpdateModifiedIdxAttrs(ResultRelInfo *resultRelInfo,
@@ -461,6 +496,13 @@ ExecUpdateModifiedIdxAttrs(ResultRelInfo *resultRelInfo,
*/
exprattrs = RelationGetIndexAttrBitmap(relation,
INDEX_ATTR_BITMAP_EXPRESSION);
+ {+ Bitmapset *cmpattrs = RelationGetIndexAttrBitmap(relation,+ INDEX_ATTR_BITMAP_COMPARABLE);++ exprattrs = bms_add_members(exprattrs, cmpattrs);+ bms_free(cmpattrs);+ }
candidates = bms_intersect(attrs, exprattrs);
if (bms_is_empty(candidates))
{
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.cindex c0941182b91..f93c456c86c 100644--- a/src/backend/utils/cache/relcache.c+++ b/src/backend/utils/cache/relcache.c@@ -30,6 +30,7 @@
#include <fcntl.h>
#include <unistd.h>
+#include "access/genam.h"
#include "access/htup_details.h"
#include "access/multixact.h"
#include "access/parallel.h"
@@ -2479,6 +2480,7 @@ RelationDestroyRelation(Relation relation, bool remember_tupdesc)
bms_free(relation->rd_summarizedattr);
bms_free(relation->rd_indexedattr);
bms_free(relation->rd_exprattr);
+ bms_free(relation->rd_cmpattr);
if (relation->rd_pubdesc)
pfree(relation->rd_pubdesc);
if (relation->rd_options)
@@ -5331,6 +5333,11 @@ RelationGetIndexPredicate(Relation relation)
* INDEX_ATTR_BITMAP_EXPRESSION Columns that non-summarizing indexes
* reference only through expressions or
* as partial indexes
+ * INDEX_ATTR_BITMAP_COMPARABLE Columns that non-summarizing indexes+ * reference directly only if their+ * access method can compare entries+ * (amcomparedatums), less key and replica+ * identity columns
*
* Attribute numbers are offset by FirstLowInvalidHeapAttributeNumber so that
* we can include system attributes (e.g., OID) in the bitmap representation.
@@ -5358,6 +5365,9 @@ RelationGetIndexAttrBitmap(Relation relation, IndexAttrBitmapKind attrKind)
Bitmapset *indexedattrs; /* columns referenced by any index */
Bitmapset *exprattrs; /* columns of expression or partial
* non-summarizing indexes */
+ Bitmapset *cmpattrs; /* plain columns of indexes whose AM can+ * compare entries (amcomparedatums) */+ Bitmapset *nocmpattrs; /* plain columns of indexes whose AM cannot */
Bitmapset *plainattrs; /* columns of other non-summarizing indexes,
* and plain columns of expression indexes */
List *indexoidlist;
@@ -5386,6 +5396,8 @@ RelationGetIndexAttrBitmap(Relation relation, IndexAttrBitmapKind attrKind)
return bms_copy(relation->rd_indexedattr);
case INDEX_ATTR_BITMAP_EXPRESSION:
return bms_copy(relation->rd_exprattr);
+ case INDEX_ATTR_BITMAP_COMPARABLE:+ return bms_copy(relation->rd_cmpattr);
default:
elog(ERROR, "unknown attrKind %u", attrKind);
}
@@ -5431,6 +5443,8 @@ restart:
hotblockingattrs = NULL;
summarizedattrs = NULL;
exprattrs = NULL;
+ cmpattrs = NULL;+ nocmpattrs = NULL;
plainattrs = NULL;
foreach(l, indexoidlist)
{
@@ -5523,6 +5537,14 @@ restart:
else
plainattrs = bms_add_member(plainattrs,
attrnum - FirstLowInvalidHeapAttributeNumber);
+ if (indexDesc->rd_indam->amcomparedatums != NULL &&+ !indexDesc->rd_index->indisexclusion &&+ !index_can_return(indexDesc, i + 1))+ cmpattrs = bms_add_member(cmpattrs,+ attrnum - FirstLowInvalidHeapAttributeNumber);+ else+ nocmpattrs = bms_add_member(nocmpattrs,+ attrnum - FirstLowInvalidHeapAttributeNumber);
}
if (isKey && i < indexDesc->rd_index->indnkeyatts)
@@ -5580,6 +5602,8 @@ restart:
bms_free(hotblockingattrs);
bms_free(summarizedattrs);
bms_free(exprattrs);
+ bms_free(cmpattrs);+ bms_free(nocmpattrs);
bms_free(plainattrs);
goto restart;
@@ -5599,6 +5623,26 @@ restart:
exprattrs = bms_del_members(exprattrs, summarizedattrs);
bms_free(plainattrs);
+ /*+ * Record what attributes non-summarizing indexes also reference directly,+ * as key or included columns, only in indexes whose access method can+ * tell whether a new value gives the same entries (amcomparedatums). A+ * change to one of these needs new index entries only if, for some index,+ * it does not. Like those above, and for the same reason, these leave+ * out what summarizing indexes reference. They also leave out the+ * columns of candidate keys and the replica identity, whose changes are+ * decided by value as before: the table AM takes its tuple lock strength+ * and what it logs for logical decoding from them. Indexes of exclusion+ * constraints count as unable to compare: the constraint is checked on+ * the values when an entry is inserted, and values that give the same+ * entries need not have the same conflicts.+ */+ cmpattrs = bms_del_members(cmpattrs, nocmpattrs);+ cmpattrs = bms_del_members(cmpattrs, summarizedattrs);+ cmpattrs = bms_del_members(cmpattrs, uindexattrs);+ cmpattrs = bms_del_members(cmpattrs, idindexattrs);+ bms_free(nocmpattrs);+
/* Don't leak the old values of these bitmaps, if any */
relation->rd_attrsvalid = false;
bms_free(relation->rd_keyattr);
@@ -5615,6 +5659,8 @@ restart:
relation->rd_indexedattr = NULL;
bms_free(relation->rd_exprattr);
relation->rd_exprattr = NULL;
+ bms_free(relation->rd_cmpattr);+ relation->rd_cmpattr = NULL;
/*
* Now save copies of the bitmaps in the relcache entry. We intentionally
@@ -5631,6 +5677,7 @@ restart:
relation->rd_summarizedattr = bms_copy(summarizedattrs);
relation->rd_indexedattr = bms_copy(indexedattrs);
relation->rd_exprattr = bms_copy(exprattrs);
+ relation->rd_cmpattr = bms_copy(cmpattrs);
relation->rd_attrsvalid = true;
MemoryContextSwitchTo(oldcxt);
@@ -5651,6 +5698,8 @@ restart:
return indexedattrs;
case INDEX_ATTR_BITMAP_EXPRESSION:
return exprattrs;
+ case INDEX_ATTR_BITMAP_COMPARABLE:+ return cmpattrs;
default:
elog(ERROR, "unknown attrKind %u", attrKind);
return NULL;
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.hindex 79240333530..f8e1b8197ae 100644--- a/src/include/access/amapi.h+++ b/src/include/access/amapi.h@@ -109,6 +109,13 @@ typedef CompareType (*amtranslate_strategy_function) (StrategyNumber strategy, O
/* translate general operator types to AM-specific strategies */
typedef StrategyNumber (*amtranslate_cmptype_function) (CompareType cmptype, Oid opfamily);
+/* would two non-null values of an index column give the same entries? */+typedef bool (*amcomparedatums_function) (Relation indexRelation,+ IndexInfo *indexInfo,+ int attnum, Oid typid,+ int16 typlen, bool typbyval,+ Datum oldValue, Datum newValue);+
/* build new index */
typedef IndexBuildResult *(*ambuild_function) (Relation heapRelation,
Relation indexRelation,
@@ -323,6 +330,9 @@ typedef struct IndexAmRoutine
/* interface functions to support planning */
amtranslate_strategy_function amtranslatestrategy; /* can be NULL */
amtranslate_cmptype_function amtranslatecmptype; /* can be NULL */
++ /* interface function to support HOT updates */+ amcomparedatums_function amcomparedatums; /* can be NULL */
} IndexAmRoutine;
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.hindex 716668c5aad..7c914ee001d 100644--- a/src/include/nodes/execnodes.h+++ b/src/include/nodes/execnodes.h@@ -504,7 +504,8 @@ typedef struct ResultRelInfo
/*
* For UPDATE, the table attributes each index references through
- * expressions or its predicate, offset by+ * expressions or its predicate, and directly if the index is partial or+ * its access method has amcomparedatums, offset by
* FirstLowInvalidHeapAttributeNumber. Built on first use; NULL until
* then.
*/
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.hindex fdd2a1e3213..b882c0e50f8 100644--- a/src/include/utils/rel.h+++ b/src/include/utils/rel.h@@ -167,6 +167,9 @@ typedef struct RelationData
Bitmapset *rd_indexedattr; /* cols referenced by any index */
Bitmapset *rd_exprattr; /* cols non-summarizing indexes reference only
* via expressions or as partial indexes */
+ Bitmapset *rd_cmpattr; /* cols indexed directly only by indexes with+ * amcomparedatums, less key and replica+ * identity cols */
PublicationDesc *rd_pubdesc; /* publication descriptor, or NULL */
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.hindex 955d0186c2d..26fd0af54dd 100644--- a/src/include/utils/relcache.h+++ b/src/include/utils/relcache.h@@ -74,6 +74,7 @@ typedef enum IndexAttrBitmapKind
INDEX_ATTR_BITMAP_SUMMARIZED,
INDEX_ATTR_BITMAP_INDEXED,
INDEX_ATTR_BITMAP_EXPRESSION,
+ INDEX_ATTR_BITMAP_COMPARABLE,
} IndexAttrBitmapKind;
extern Bitmapset *RelationGetIndexAttrBitmap(Relation relation,
diff --git a/src/test/modules/dummy_index_am/Makefile b/src/test/modules/dummy_index_am/Makefileindex aaf544a3854..6ca1f96d655 100644--- a/src/test/modules/dummy_index_am/Makefile+++ b/src/test/modules/dummy_index_am/Makefile@@ -6,7 +6,7 @@ EXTENSION = dummy_index_am
DATA = dummy_index_am--1.0.sql
PGFILEDESC = "dummy_index_am - index access method template"
-REGRESS = reloptions+REGRESS = comparedatums reloptions
ifdef USE_PGXS
PG_CONFIG = pg_config
diff --git a/src/test/modules/dummy_index_am/dummy_index_am.c b/src/test/modules/dummy_index_am/dummy_index_am.cindex 31f8d2b8161..18fe386f2cc 100644--- a/src/test/modules/dummy_index_am/dummy_index_am.c+++ b/src/test/modules/dummy_index_am/dummy_index_am.c@@ -16,8 +16,10 @@
#include "access/amapi.h"
#include "access/reloptions.h"
#include "catalog/index.h"
+#include "catalog/pg_type.h"
#include "commands/vacuum.h"
#include "nodes/pathnodes.h"
+#include "utils/rel.h"
PG_MODULE_MAGIC;
@@ -244,6 +246,39 @@ dioptions(Datum reloptions, bool validate)
di_relopt_tab, lengthof(di_relopt_tab));
}
+/*+ * amcanreturn: with option_ternary_1 set to true, claim that every column+ * can be returned, so that amcomparedatums is not to be asked about it.+ * (The index cannot actually be scanned.)+ */+static bool+dicanreturn(Relation index, int attno)+{+ DummyIndexOptions *opts = (DummyIndexOptions *) index->rd_options;++ return opts != NULL && opts->option_ternary_1 == PG_TERNARY_TRUE;+}++/*+ * Would newValue give the same entries as oldValue? The dummy index stores+ * nothing, so any answer is correct. For testing the executor's use of+ * amcomparedatums: with option_real set to 1, int4 values that are equal+ * modulo option_int count as giving the same entries.+ */+static bool+dicomparedatums(Relation index, IndexInfo *indexInfo, int attnum,+ Oid typid, int16 typlen, bool typbyval,+ Datum oldValue, Datum newValue)+{+ DummyIndexOptions *opts = (DummyIndexOptions *) index->rd_options;++ if (opts == NULL || opts->option_real != 1 || opts->option_int < 2 ||+ typid != INT4OID)+ return false;+ return DatumGetInt32(oldValue) % opts->option_int ==+ DatumGetInt32(newValue) % opts->option_int;+}+
/*
* Validator for index AM.
*/
@@ -324,7 +359,7 @@ dihandler(PG_FUNCTION_ARGS)
.aminsert = diinsert,
.ambulkdelete = dibulkdelete,
.amvacuumcleanup = divacuumcleanup,
- .amcanreturn = NULL,+ .amcanreturn = dicanreturn,
.amcostestimate = dicostestimate,
.amgettreeheight = NULL,
.amoptions = dioptions,
@@ -341,6 +376,7 @@ dihandler(PG_FUNCTION_ARGS)
.amestimateparallelscan = NULL,
.aminitparallelscan = NULL,
.amparallelrescan = NULL,
+ .amcomparedatums = dicomparedatums,
};
PG_RETURN_POINTER(&amroutine);
diff --git a/src/test/modules/dummy_index_am/expected/comparedatums.out b/src/test/modules/dummy_index_am/expected/comparedatums.outnew file mode 100644index 00000000000..754ba1fe1d0--- /dev/null+++ b/src/test/modules/dummy_index_am/expected/comparedatums.out@@ -0,0 +1,152 @@+-- Test the executor's use of amcomparedatums on UPDATE: a column that an+-- index stores directly does not block HOT if the index's access method+-- says the new value gives the same entries.+CREATE EXTENSION dummy_index_am;+CREATE TABLE dummy_cmp (id int, a int, b int) WITH (fillfactor = 50);+INSERT INTO dummy_cmp SELECT g, g, g FROM generate_series(1, 10) g;+-- With option_real = 1, dummy_index_am says int4 values equal modulo+-- option_int give the same entries. A btree index on another column, and+-- no index on b.+CREATE INDEX dummy_cmp_a ON dummy_cmp USING dummy_index_am (a)+ WITH (option_real = 1, option_int = 10);+CREATE INDEX dummy_cmp_id ON dummy_cmp (id);+CREATE FUNCTION dummy_cmp_hot(OUT updated bigint, OUT hot bigint)+LANGUAGE sql AS $$+ SELECT pg_stat_get_xact_tuples_updated('dummy_cmp'::regclass),+ pg_stat_get_xact_tuples_hot_updated('dummy_cmp'::regclass) $$;+-- count from zero: flush what earlier transactions counted+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++BEGIN;+-- a + 10 is equal modulo 10: HOT+UPDATE dummy_cmp SET a = a + 10;+SELECT * FROM dummy_cmp_hot();+ updated | hot +---------+-----+ 10 | 10+(1 row)++-- a + 1 is not: not HOT+UPDATE dummy_cmp SET a = a + 1;+SELECT * FROM dummy_cmp_hot();+ updated | hot +---------+-----+ 20 | 10+(1 row)++-- b is not indexed: HOT+UPDATE dummy_cmp SET b = b + 1;+SELECT * FROM dummy_cmp_hot();+ updated | hot +---------+-----+ 30 | 20+(1 row)++-- id + 100 needs a new entry in the btree index on id: not HOT, even though+-- a changes in a way dummy_index_am accepts+UPDATE dummy_cmp SET a = a + 10, id = id + 100;+SELECT * FROM dummy_cmp_hot();+ updated | hot +---------+-----+ 40 | 20+(1 row)++-- to and from null always needs new entries+UPDATE dummy_cmp SET a = NULL WHERE id = 101;+UPDATE dummy_cmp SET a = 2 WHERE id = 101;+SELECT * FROM dummy_cmp_hot();+ updated | hot +---------+-----+ 42 | 20+(1 row)++COMMIT;+-- A change to a column that another index also stores needs a new entry+-- if that index needs one: a + 10 does in a btree index on a, so not HOT.+CREATE INDEX dummy_cmp_a2 ON dummy_cmp (a);+-- count from zero: flush what earlier transactions counted+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++BEGIN;+UPDATE dummy_cmp SET a = a + 10;+SELECT * FROM dummy_cmp_hot();+ updated | hot +---------+-----+ 10 | 0+(1 row)++COMMIT;+DROP INDEX dummy_cmp_a2;+-- Nor is the callback asked about a column the index says it can return in+-- index-only scans (option_ternary_1 makes dummy_index_am say so): a + 10+-- is not HOT then.+ALTER INDEX dummy_cmp_a SET (option_ternary_1 = true);+-- count from zero: flush what earlier transactions counted+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++BEGIN;+UPDATE dummy_cmp SET a = a + 10;+SELECT * FROM dummy_cmp_hot();+ updated | hot +---------+-----+ 10 | 0+(1 row)++COMMIT;+ALTER INDEX dummy_cmp_a RESET (option_ternary_1);+-- The columns of a unique index that a foreign key could reference are+-- never compared this way. (Nor are those of the replica identity, but an+-- index that can be the replica identity is always such a key.) Without+-- the unique index, id + 100 would be HOT: dummy_index_am says it gives+-- the same entries, and nothing else indexes id.+DROP INDEX dummy_cmp_id;+CREATE INDEX dummy_cmp_id_d ON dummy_cmp USING dummy_index_am (id)+ WITH (option_real = 1, option_int = 100);+-- count from zero: flush what earlier transactions counted+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++BEGIN;+UPDATE dummy_cmp SET id = id + 100;+SELECT * FROM dummy_cmp_hot();+ updated | hot +---------+-----+ 10 | 10+(1 row)++COMMIT;+CREATE UNIQUE INDEX dummy_cmp_id_u ON dummy_cmp (id);+-- count from zero: flush what earlier transactions counted+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++BEGIN;+UPDATE dummy_cmp SET id = id + 100;+SELECT * FROM dummy_cmp_hot();+ updated | hot +---------+-----+ 10 | 0+(1 row)++COMMIT;+DROP TABLE dummy_cmp;+DROP FUNCTION dummy_cmp_hot;+DROP EXTENSION dummy_index_am;diff --git a/src/test/modules/dummy_index_am/meson.build b/src/test/modules/dummy_index_am/meson.buildindex 9cedfd2cf8b..a4174c2632c 100644--- a/src/test/modules/dummy_index_am/meson.build+++ b/src/test/modules/dummy_index_am/meson.build@@ -27,6 +27,7 @@ tests += {
'bd': meson.current_build_dir(),
'regress': {
'sql': [
+ 'comparedatums',
'reloptions',
],
},
diff --git a/src/test/modules/dummy_index_am/sql/comparedatums.sql b/src/test/modules/dummy_index_am/sql/comparedatums.sqlnew file mode 100644index 00000000000..cd8016a679b--- /dev/null+++ b/src/test/modules/dummy_index_am/sql/comparedatums.sql@@ -0,0 +1,82 @@+-- Test the executor's use of amcomparedatums on UPDATE: a column that an+-- index stores directly does not block HOT if the index's access method+-- says the new value gives the same entries.+CREATE EXTENSION dummy_index_am;+CREATE TABLE dummy_cmp (id int, a int, b int) WITH (fillfactor = 50);+INSERT INTO dummy_cmp SELECT g, g, g FROM generate_series(1, 10) g;+-- With option_real = 1, dummy_index_am says int4 values equal modulo+-- option_int give the same entries. A btree index on another column, and+-- no index on b.+CREATE INDEX dummy_cmp_a ON dummy_cmp USING dummy_index_am (a)+ WITH (option_real = 1, option_int = 10);+CREATE INDEX dummy_cmp_id ON dummy_cmp (id);+CREATE FUNCTION dummy_cmp_hot(OUT updated bigint, OUT hot bigint)+LANGUAGE sql AS $$+ SELECT pg_stat_get_xact_tuples_updated('dummy_cmp'::regclass),+ pg_stat_get_xact_tuples_hot_updated('dummy_cmp'::regclass) $$;+-- count from zero: flush what earlier transactions counted+SELECT pg_stat_force_next_flush();+BEGIN;+-- a + 10 is equal modulo 10: HOT+UPDATE dummy_cmp SET a = a + 10;+SELECT * FROM dummy_cmp_hot();+-- a + 1 is not: not HOT+UPDATE dummy_cmp SET a = a + 1;+SELECT * FROM dummy_cmp_hot();+-- b is not indexed: HOT+UPDATE dummy_cmp SET b = b + 1;+SELECT * FROM dummy_cmp_hot();+-- id + 100 needs a new entry in the btree index on id: not HOT, even though+-- a changes in a way dummy_index_am accepts+UPDATE dummy_cmp SET a = a + 10, id = id + 100;+SELECT * FROM dummy_cmp_hot();+-- to and from null always needs new entries+UPDATE dummy_cmp SET a = NULL WHERE id = 101;+UPDATE dummy_cmp SET a = 2 WHERE id = 101;+SELECT * FROM dummy_cmp_hot();+COMMIT;+-- A change to a column that another index also stores needs a new entry+-- if that index needs one: a + 10 does in a btree index on a, so not HOT.+CREATE INDEX dummy_cmp_a2 ON dummy_cmp (a);+-- count from zero: flush what earlier transactions counted+SELECT pg_stat_force_next_flush();+BEGIN;+UPDATE dummy_cmp SET a = a + 10;+SELECT * FROM dummy_cmp_hot();+COMMIT;+DROP INDEX dummy_cmp_a2;+-- Nor is the callback asked about a column the index says it can return in+-- index-only scans (option_ternary_1 makes dummy_index_am say so): a + 10+-- is not HOT then.+ALTER INDEX dummy_cmp_a SET (option_ternary_1 = true);+-- count from zero: flush what earlier transactions counted+SELECT pg_stat_force_next_flush();+BEGIN;+UPDATE dummy_cmp SET a = a + 10;+SELECT * FROM dummy_cmp_hot();+COMMIT;+ALTER INDEX dummy_cmp_a RESET (option_ternary_1);+-- The columns of a unique index that a foreign key could reference are+-- never compared this way. (Nor are those of the replica identity, but an+-- index that can be the replica identity is always such a key.) Without+-- the unique index, id + 100 would be HOT: dummy_index_am says it gives+-- the same entries, and nothing else indexes id.+DROP INDEX dummy_cmp_id;+CREATE INDEX dummy_cmp_id_d ON dummy_cmp USING dummy_index_am (id)+ WITH (option_real = 1, option_int = 100);+-- count from zero: flush what earlier transactions counted+SELECT pg_stat_force_next_flush();+BEGIN;+UPDATE dummy_cmp SET id = id + 100;+SELECT * FROM dummy_cmp_hot();+COMMIT;+CREATE UNIQUE INDEX dummy_cmp_id_u ON dummy_cmp (id);+-- count from zero: flush what earlier transactions counted+SELECT pg_stat_force_next_flush();+BEGIN;+UPDATE dummy_cmp SET id = id + 100;+SELECT * FROM dummy_cmp_hot();+COMMIT;+DROP TABLE dummy_cmp;+DROP FUNCTION dummy_cmp_hot;+DROP EXTENSION dummy_index_am;diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.listindex bccd3cac465..16e12f0854f 100644--- a/src/tools/pgindent/typedefs.list+++ b/src/tools/pgindent/typedefs.list@@ -3572,6 +3572,7 @@ ambuildempty_function
ambuildphasename_function
ambulkdelete_function
amcanreturn_function
+amcomparedatums_function
amcostestimate_function
amendscan_function
amestimateparallelscan_function
--
2.50.1
v43-0002-Identify-modified-indexed-attributes-in-the-exec.patch (text/plain, 53.5K) download
From b4ce0908f0f3c0afa1b6ad69f8903018dbeecaab Mon Sep 17 00:00:00 2001
From: Greg Burd <greg@burd.me>
Date: Tue, 10 Mar 2026 08:17:31 -0400
Subject: [PATCH v43 2/8] Identify modified indexed attributes in the executor
on UPDATE
heap_update() decides which indexed attributes an UPDATE changed by
comparing the old and new tuples inside HeapDetermineColumnsInfo(), with
the buffer locked. That knowledge is not heap-specific, and the
executor needs it before it calls the table AM to do anything smarter:
an index whose key did not change does not need a new entry, whatever
the table AM does. Determine it in the executor instead.
ExecUpdateModifiedIdxAttrs() compares the old and new slots over the
relation's indexed attributes and passes the result to
table_tuple_update() as a new modified_idx_attrs argument. It compares
every indexed attribute, not just those the UPDATE targets:
ExecGetAllUpdatedCols() names the targeted columns, and misses a column
a BEFORE ROW trigger changes with heap_modify_tuple() (as
tsvector_update_trigger() does), while logical replication apply and
REPACK (CONCURRENTLY) apply hand over whole tuples. The comparison is
the binary comparison heap_update() has always made, so an unchanged
toasted value still compares equal without being fetched.
heapam_tuple_update() derives the HOT decision and the lock mode from
the set, through the new HeapUpdateHotAllowable() and
HeapUpdateDetermineLockmode(), and passes both to heap_update().
simple_heap_update() has no slots: it fetches the old tuple and computes
the same set with HeapUpdateModifiedIdxAttrs(), the remainder of
HeapDetermineColumnsInfo(), before taking the buffer lock. heap_update()
keeps the replica identity decision, which needs the old tuple on the
page. The decisions about
HOT, lock mode and replica identity do not change; hot_updates' expected
output is unchanged.
The set to compare is the new INDEX_ATTR_BITMAP_INDEXED, the attributes
referenced by any index: the union of INDEX_ATTR_BITMAP_HOT_BLOCKING and
INDEX_ATTR_BITMAP_SUMMARIZED, whose meanings do not change.
simple_table_tuple_update() keeps its signature: it fetches the old
version and compares it with table_slot_modified_idx_attrs(), the
executor's comparison, now in tableam.c. Logical replication apply,
which has both slots, calls the new simple_table_tuple_update_ext() and
passes the set.
In builds with assertions, heap_update() compares the old tuple it
locked with the new one and checks that the caller's set is the same,
so that a caller passing a wrong set is caught.
The syscache-update-pruned isolation test waits at an injection point
before simple_heap_update() pins the old tuple's buffer, which now
happens before heap_update(); the point moves with it.
Discussion: https://postgr.es/m/78574B24-BE0A-42C5-8075-3FA9FA63B8FC@amazon.com
---
src/backend/access/heap/heapam.c | 448 ++++++++++++------
src/backend/access/heap/heapam_handler.c | 30 +-
src/backend/access/table/tableam.c | 110 ++++-
src/backend/commands/repack.c | 9 +-
src/backend/executor/execReplication.c | 9 +-
src/backend/executor/nodeModifyTable.c | 57 ++-
src/backend/utils/cache/relcache.c | 12 +
src/include/access/heapam.h | 9 +-
src/include/access/tableam.h | 19 +-
src/include/executor/executor.h | 3 +
src/include/utils/rel.h | 1 +
src/include/utils/relcache.h | 1 +
.../expected/syscache-update-pruned.out | 12 +-
.../specs/syscache-update-pruned.spec | 6 +-
14 files changed, 534 insertions(+), 192 deletions(-)
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.cindex 6a09a1a1517..729f9c5fb57 100644--- a/src/backend/access/heap/heapam.c+++ b/src/backend/access/heap/heapam.c@@ -37,6 +37,8 @@
#include "access/multixact.h"
#include "access/subtrans.h"
#include "access/syncscan.h"
+#include "access/sysattr.h"+#include "access/tableam.h"
#include "access/valid.h"
#include "access/visibilitymap.h"
#include "access/xloginsert.h"
@@ -44,8 +46,11 @@
#include "catalog/pg_database_d.h"
#include "commands/vacuum.h"
#include "executor/instrument_node.h"
+#include "executor/tuptable.h"+#include "nodes/lockoptions.h"
#include "pgstat.h"
#include "port/pg_bitutils.h"
+#include "storage/buf.h"
#include "storage/lmgr.h"
#include "storage/predicate.h"
#include "storage/proc.h"
@@ -53,6 +58,7 @@
#include "utils/datum.h"
#include "utils/injection_point.h"
#include "utils/inval.h"
+#include "utils/relcache.h"
#include "utils/spccache.h"
#include "utils/syscache.h"
@@ -71,11 +77,9 @@ static void check_lock_if_inplace_updateable_rel(Relation relation,
HeapTuple newtup);
static void check_inplace_rel_lock(HeapTuple oldtup);
#endif
-static Bitmapset *HeapDetermineColumnsInfo(Relation relation,- Bitmapset *interesting_cols,- Bitmapset *external_cols,- HeapTuple oldtup, HeapTuple newtup,- bool *has_external);+static Bitmapset *HeapUpdateModifiedIdxAttrs(Relation relation,+ const Bitmapset *attrs,+ HeapTuple oldtup, HeapTuple newtup);
static bool heap_acquire_tuplock(Relation relation, const ItemPointerData *tid,
LockTupleMode mode, LockWaitPolicy wait_policy,
bool *have_tuple_lock);
@@ -3258,7 +3262,13 @@ simple_heap_delete(Relation relation, const ItemPointerData *tid)
* heap_update - replace a tuple
*
* See table_tuple_update() for an explanation of the parameters, except that
- * this routine directly takes a tuple rather than a slot.+ * this routine directly takes a heap tuple rather than a slot, and that the+ * caller decides what table_tuple_update() leaves to the table AM:+ * modified_idx_attrs are the indexed attributes the update changes, lockmode+ * the tuple lock to take (HeapUpdateDetermineLockmode()), and hot_allowed+ * whether the new version may be heap-only (HeapUpdateHotAllowable()). The+ * caller then learns from HeapTupleIsHeapOnly(newtup) which indexes need new+ * entries.
*
* In the failure cases, the routine fills *tmfd with the tuple's t_ctid,
* t_xmax (resolving a possible MultiXact, if necessary), and t_cmax (the last
@@ -3268,17 +3278,12 @@ simple_heap_delete(Relation relation, const ItemPointerData *tid)
TM_Result
heap_update(Relation relation, const ItemPointerData *otid, HeapTuple newtup,
CommandId cid, uint32 options, Snapshot crosscheck, bool wait,
- TM_FailureData *tmfd, LockTupleMode *lockmode,- TU_UpdateIndexes *update_indexes)+ TM_FailureData *tmfd, LockTupleMode lockmode,+ const Bitmapset *modified_idx_attrs, bool hot_allowed)
{
TM_Result result;
TransactionId xid = GetCurrentTransactionId();
- Bitmapset *hot_attrs;- Bitmapset *sum_attrs;- Bitmapset *key_attrs;
Bitmapset *id_attrs;
- Bitmapset *interesting_attrs;- Bitmapset *modified_attrs;
ItemId lp;
HeapTupleData oldtup;
HeapTuple heaptup;
@@ -3301,7 +3306,6 @@ heap_update(Relation relation, const ItemPointerData *otid, HeapTuple newtup,
bool have_tuple_lock = false;
bool iscombo;
bool use_hot_update = false;
- bool summarized_update = false;
bool key_intact;
bool clear_all_visible = false;
bool clear_all_visible_new = false;
@@ -3309,7 +3313,10 @@ heap_update(Relation relation, const ItemPointerData *otid, HeapTuple newtup,
bool vmbuffer_new_modified = false;
bool checked_lockers;
bool locker_remains;
- bool id_has_external = false;+ bool rep_id_key_required = false;+#ifdef USE_ASSERT_CHECKING+ Bitmapset *check_attrs;+#endif
TransactionId xmax_new_tuple,
xmax_old_tuple;
uint16 infomask_old_tuple,
@@ -3340,33 +3347,19 @@ heap_update(Relation relation, const ItemPointerData *otid, HeapTuple newtup,
#endif
/*
- * Fetch the list of attributes to be checked for various operations.+ * Fetch the replica identity columns.
*
- * For HOT considerations, this is wasted effort if we fail to update or- * have to put the new tuple on a different page. But we must compute the- * list before obtaining buffer lock --- in the worst case, if we are- * doing an update on one of the relevant system catalogs, we could- * deadlock if we try to fetch the list later. In any case, the relcache- * caches the data so this is usually pretty cheap.- *- * We also need columns used by the replica identity and columns that are- * considered the "key" of rows in the table.- *- * Note that we get copies of each bitmap, so we need not worry about- * relcache flush happening midway through.- */- hot_attrs = RelationGetIndexAttrBitmap(relation,- INDEX_ATTR_BITMAP_HOT_BLOCKING);- sum_attrs = RelationGetIndexAttrBitmap(relation,- INDEX_ATTR_BITMAP_SUMMARIZED);- key_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_KEY);- id_attrs = RelationGetIndexAttrBitmap(relation,- INDEX_ATTR_BITMAP_IDENTITY_KEY);- interesting_attrs = NULL;- interesting_attrs = bms_add_members(interesting_attrs, hot_attrs);- interesting_attrs = bms_add_members(interesting_attrs, sum_attrs);- interesting_attrs = bms_add_members(interesting_attrs, key_attrs);- interesting_attrs = bms_add_members(interesting_attrs, id_attrs);+ * Note: We must compute the list before obtaining buffer lock. In the+ * worst case, if we are doing an update on one of the relevant system+ * catalogs, we could deadlock if we try to fetch the list later. Keep in+ * mind that relcache returns a copy of the bitmap, so we need not worry+ * about relcache flush happening midway through, but we do need to free+ * it.+ */+ id_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_IDENTITY_KEY);+#ifdef USE_ASSERT_CHECKING+ check_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_INDEXED);+#endif
block = ItemPointerGetBlockNumber(otid);
INJECTION_POINT("heap_update-before-pin", NULL);
@@ -3420,20 +3413,19 @@ heap_update(Relation relation, const ItemPointerData *otid, HeapTuple newtup,
tmfd->ctid = *otid;
tmfd->xmax = InvalidTransactionId;
tmfd->cmax = InvalidCommandId;
- *update_indexes = TU_None;- bms_free(hot_attrs);- bms_free(sum_attrs);- bms_free(key_attrs);
bms_free(id_attrs);
- /* modified_attrs not yet initialized */- bms_free(interesting_attrs);+#ifdef USE_ASSERT_CHECKING+ bms_free(check_attrs);+#endif+ /* modified_idx_attrs is owned by the caller, don't free it */+
return TM_Deleted;
}
/*
- * Fill in enough data in oldtup for HeapDetermineColumnsInfo to work- * properly.+ * Fill in enough data in oldtup to determine replica identity attribute+ * requirements.
*/
oldtup.t_tableOid = RelationGetRelid(relation);
oldtup.t_data = (HeapTupleHeader) PageGetItem(page, lp);
@@ -3444,16 +3436,54 @@ heap_update(Relation relation, const ItemPointerData *otid, HeapTuple newtup,
newtup->t_tableOid = RelationGetRelid(relation);
/*
- * Determine columns modified by the update. Additionally, identify- * whether any of the unmodified replica identity key attributes in the- * old tuple is externally stored or not. This is required because for- * such attributes the flattened value won't be WAL logged as part of the- * new tuple so we must include it as part of the old_key_tuple. See- * ExtractReplicaIdentity.+ * ExtractReplicaIdentity() needs to know if a modified indexed attribute+ * is used as a replica identity or if any of the replica identity+ * attributes are referenced in an index, unmodified, and are stored+ * externally in the old tuple being replaced. In those cases it may be+ * necessary to WAL log them so they are available to replicas.
*/
- modified_attrs = HeapDetermineColumnsInfo(relation, interesting_attrs,- id_attrs, &oldtup,- newtup, &id_has_external);+ rep_id_key_required = bms_overlap(modified_idx_attrs, id_attrs);+ if (!rep_id_key_required)+ {+ Bitmapset *attrs;+ TupleDesc tupdesc = RelationGetDescr(relation);+ int attidx = -1;++ /* the replica identity key attributes the update leaves unchanged */+ attrs = bms_difference(id_attrs, modified_idx_attrs);++ while ((attidx = bms_next_member(attrs, attidx)) >= 0)+ {+ /*+ * attidx is zero-based, attrnum is the normal attribute number+ */+ AttrNumber attrnum = attidx + FirstLowInvalidHeapAttributeNumber;+ Datum value;+ bool isnull;++ /*+ * These are replica identity columns: an index's plain key+ * columns, never system attributes.+ */+ Assert(attrnum > 0);++ value = heap_getattr(&oldtup, attrnum, tupdesc, &isnull);++ /* No need to check attributes that can't be stored externally */+ if (isnull ||+ TupleDescCompactAttr(tupdesc, attrnum - 1)->attlen != -1)+ continue;++ /* Check if the old tuple's attribute is stored externally */+ if (VARATT_IS_EXTERNAL((varlena *) DatumGetPointer(value)))+ {+ rep_id_key_required = true;+ break;+ }+ }++ bms_free(attrs);+ }
/*
* If we're not updating any "key" column, we can grab a weaker lock type.
@@ -3466,9 +3496,8 @@ heap_update(Relation relation, const ItemPointerData *otid, HeapTuple newtup,
* is updates that don't manipulate key columns, not those that
* serendipitously arrive at the same key values.
*/
- if (!bms_overlap(modified_attrs, key_attrs))+ if (lockmode == LockTupleNoKeyExclusive)
{
- *lockmode = LockTupleNoKeyExclusive;
mxact_status = MultiXactStatusNoKeyUpdate;
key_intact = true;
@@ -3485,7 +3514,7 @@ heap_update(Relation relation, const ItemPointerData *otid, HeapTuple newtup,
}
else
{
- *lockmode = LockTupleExclusive;+ Assert(lockmode == LockTupleExclusive);
mxact_status = MultiXactStatusUpdate;
key_intact = false;
}
@@ -3564,7 +3593,7 @@ l2:
bool current_is_member = false;
if (DoesMultiXactIdConflict((MultiXactId) xwait, infomask,
- *lockmode, ¤t_is_member))+ lockmode, ¤t_is_member))
{
LockBuffer(buffer, BUFFER_LOCK_UNLOCK);
@@ -3573,7 +3602,7 @@ l2:
* requesting a lock and already have one; avoids deadlock).
*/
if (!current_is_member)
- heap_acquire_tuplock(relation, &(oldtup.t_self), *lockmode,+ heap_acquire_tuplock(relation, &(oldtup.t_self), lockmode,
LockWaitBlock, &have_tuple_lock);
/* wait for multixact */
@@ -3669,7 +3698,7 @@ l2:
* lock.
*/
LockBuffer(buffer, BUFFER_LOCK_UNLOCK);
- heap_acquire_tuplock(relation, &(oldtup.t_self), *lockmode,+ heap_acquire_tuplock(relation, &(oldtup.t_self), lockmode,
LockWaitBlock, &have_tuple_lock);
XactLockTableWait(xwait, relation, &oldtup.t_self,
XLTW_Update);
@@ -3729,17 +3758,16 @@ l2:
tmfd->cmax = InvalidCommandId;
UnlockReleaseBuffer(buffer);
if (have_tuple_lock)
- UnlockTupleTuplock(relation, &(oldtup.t_self), *lockmode);+ UnlockTupleTuplock(relation, &(oldtup.t_self), lockmode);
if (vmbuffer != InvalidBuffer)
ReleaseBuffer(vmbuffer);
- *update_indexes = TU_None;- bms_free(hot_attrs);- bms_free(sum_attrs);- bms_free(key_attrs);
bms_free(id_attrs);
- bms_free(modified_attrs);- bms_free(interesting_attrs);+#ifdef USE_ASSERT_CHECKING+ bms_free(check_attrs);+#endif+ /* modified_idx_attrs is owned by the caller, don't free it */+
return result;
}
@@ -3760,6 +3788,26 @@ l2:
goto l2;
}
+#ifdef USE_ASSERT_CHECKING++ /*+ * The caller compared the version it fetched; this is the version being+ * updated, and no one else can have changed it. Check that every+ * attribute comparing the two tuples here finds changed is in the+ * caller's set, so that a caller that misses a change, or compares the+ * wrong tuples, is caught. The caller may report more: comparing slots,+ * it cannot always tell that two values will be stored alike.+ */+ {+ Bitmapset *changed = HeapUpdateModifiedIdxAttrs(relation, check_attrs,+ &oldtup, newtup);++ Assert(bms_is_subset(changed, modified_idx_attrs));+ bms_free(changed);+ bms_free(check_attrs);+ }+#endif+
/* Fill in transaction status data */
/*
@@ -3769,7 +3817,7 @@ l2:
compute_new_xmax_infomask(HeapTupleHeaderGetRawXmax(oldtup.t_data),
oldtup.t_data->t_infomask,
oldtup.t_data->t_infomask2,
- xid, *lockmode, true,+ xid, lockmode, true,
&xmax_old_tuple, &infomask_old_tuple,
&infomask2_old_tuple);
@@ -3886,7 +3934,7 @@ l2:
compute_new_xmax_infomask(HeapTupleHeaderGetRawXmax(oldtup.t_data),
oldtup.t_data->t_infomask,
oldtup.t_data->t_infomask2,
- xid, *lockmode, false,+ xid, lockmode, false,
&xmax_lock_old_tuple, &infomask_lock_old_tuple,
&infomask2_lock_old_tuple);
@@ -4081,23 +4129,11 @@ l2:
{
/*
* Since the new tuple is going into the same page, we might be able
- * to do a HOT update. Check if any of the index columns have been- * changed.+ * to do a HOT update, if the caller found that no index needs a new+ * entry (hot_allowed).
*/
- if (!bms_overlap(modified_attrs, hot_attrs))- {+ if (hot_allowed)
use_hot_update = true;
-- /*- * If none of the columns that are used in hot-blocking indexes- * were updated, we can apply HOT, but we do still need to check- * if we need to update the summarizing indexes, and update those- * indexes if the columns were updated, or we may fail to detect- * e.g. value bound changes in BRIN minmax indexes.- */- if (bms_overlap(modified_attrs, sum_attrs))- summarized_update = true;- }
}
else
{
@@ -4113,8 +4149,7 @@ l2:
* columns are modified or it has external data.
*/
old_key_tuple = ExtractReplicaIdentity(relation, &oldtup,
- bms_overlap(modified_attrs, id_attrs) ||- id_has_external,+ rep_id_key_required,
&old_key_copied);
clear_all_visible = PageIsAllVisible(page);
@@ -4342,7 +4377,7 @@ l2:
* Release the lmgr tuple lock, if we had it.
*/
if (have_tuple_lock)
- UnlockTupleTuplock(relation, &(oldtup.t_self), *lockmode);+ UnlockTupleTuplock(relation, &(oldtup.t_self), lockmode);
pgstat_count_heap_update(relation, use_hot_update, newbuf != buffer);
@@ -4356,31 +4391,11 @@ l2:
heap_freetuple(heaptup);
}
- /*- * If it is a HOT update, the update may still need to update summarized- * indexes, lest we fail to update those summaries and get incorrect- * results (for example, minmax bounds of the block may change with this- * update).- */- if (use_hot_update)- {- if (summarized_update)- *update_indexes = TU_Summarizing;- else- *update_indexes = TU_None;- }- else- *update_indexes = TU_All;-
if (old_key_tuple != NULL && old_key_copied)
heap_freetuple(old_key_tuple);
- bms_free(hot_attrs);- bms_free(sum_attrs);- bms_free(key_attrs);
bms_free(id_attrs);
- bms_free(modified_attrs);- bms_free(interesting_attrs);+ /* modified_idx_attrs is owned by the caller, don't free it */
return TM_Ok;
}
@@ -4509,7 +4524,7 @@ check_inplace_rel_lock(HeapTuple oldtup)
/*
* Check if the specified attribute's values are the same. Subroutine for
- * HeapDetermineColumnsInfo.+ * HeapUpdateModifiedIdxAttrs.
*/
static bool
heap_attr_equals(TupleDesc tupdesc, int attrnum, Datum value1, Datum value2,
@@ -4553,28 +4568,86 @@ heap_attr_equals(TupleDesc tupdesc, int attrnum, Datum value1, Datum value2,
}
/*
- * Check which columns are being updated.- *- * Given an updated tuple, determine (and return into the output bitmapset),- * from those listed as interesting, the set of columns that changed.- *- * has_external indicates if any of the unmodified attributes (from those- * listed as interesting) of the old tuple is a member of external_cols and is- * stored externally.+ * HOT updates are possible when either: a) there are no modified indexed+ * attributes, or b) the modified attributes are all on summarizing indexes.+ * Later, in heap_update(), we can choose to perform a HOT update if there is+ * space on the page for the new tuple and the following code has determined+ * that HOT is allowed.+ */+bool+HeapUpdateHotAllowable(Relation relation, const Bitmapset *modified_idx_attrs,+ bool *summarized_only)+{+ Bitmapset *hot_attrs;+ Bitmapset *sum_attrs;+ bool hot_allowed;++ *summarized_only = false;+ if (bms_is_empty(modified_idx_attrs))+ return true;++ hot_attrs = RelationGetIndexAttrBitmap(relation,+ INDEX_ATTR_BITMAP_HOT_BLOCKING);+ sum_attrs = RelationGetIndexAttrBitmap(relation,+ INDEX_ATTR_BITMAP_SUMMARIZED);++ /*+ * Unless a modified attribute is referenced by a non-summarizing index,+ * HOT is allowed; if a summarizing index references one, that index still+ * needs a new entry.+ */+ hot_allowed = !bms_overlap(modified_idx_attrs, hot_attrs);+ if (hot_allowed)+ *summarized_only = bms_overlap(modified_idx_attrs, sum_attrs);++ bms_free(hot_attrs);+ bms_free(sum_attrs);++ return hot_allowed;+}++/*+ * If we're not updating any attributes used when forming the index keys we can+ * grab a weaker lock type. This allows for more concurrency when we are+ * running simultaneously with foreign key checks.+ */+LockTupleMode+HeapUpdateDetermineLockmode(Relation relation, const Bitmapset *modified_idx_attrs)+{+ LockTupleMode lockmode = LockTupleExclusive;++ Bitmapset *key_attrs = RelationGetIndexAttrBitmap(relation,+ INDEX_ATTR_BITMAP_KEY);++ if (!bms_overlap(modified_idx_attrs, key_attrs))+ lockmode = LockTupleNoKeyExclusive;++ bms_free(key_attrs);++ return lockmode;+}++/*+ * Return a Bitmapset that contains the members of attrs, a set of indexed+ * attributes, whose values differ between oldtup and newtup. Reads no+ * catalogs, so it may be called with the buffer locked.
*/
static Bitmapset *
-HeapDetermineColumnsInfo(Relation relation,- Bitmapset *interesting_cols,- Bitmapset *external_cols,- HeapTuple oldtup, HeapTuple newtup,- bool *has_external)+HeapUpdateModifiedIdxAttrs(Relation relation, const Bitmapset *attrs,+ HeapTuple oldtup, HeapTuple newtup)
{
int attidx;
- Bitmapset *modified = NULL;+ Bitmapset *modified_idx_attrs = NULL;
TupleDesc tupdesc = RelationGetDescr(relation);
+ /*+ * As ExecUpdateModifiedIdxAttrs() does for the executor's slots, compare+ * every indexed attribute of the old tuple and the new to find the subset+ * that changed value, the "modified indexed attributes" or+ * "modified_idx_attrs".+ */
attidx = -1;
- while ((attidx = bms_next_member(interesting_cols, attidx)) >= 0)+ while ((attidx = bms_next_member(attrs, attidx)) >= 0)
{
/* attidx is zero-based, attrnum is the normal attribute number */
AttrNumber attrnum = attidx + FirstLowInvalidHeapAttributeNumber;
@@ -4590,7 +4663,7 @@ HeapDetermineColumnsInfo(Relation relation,
*/
if (attrnum == 0)
{
- modified = bms_add_member(modified, attidx);+ modified_idx_attrs = bms_add_member(modified_idx_attrs, attidx);
continue;
}
@@ -4603,7 +4676,7 @@ HeapDetermineColumnsInfo(Relation relation,
{
if (attrnum != TableOidAttributeNumber)
{
- modified = bms_add_member(modified, attidx);+ modified_idx_attrs = bms_add_member(modified_idx_attrs, attidx);
continue;
}
}
@@ -4619,29 +4692,10 @@ HeapDetermineColumnsInfo(Relation relation,
if (!heap_attr_equals(tupdesc, attrnum, value1,
value2, isnull1, isnull2))
- {- modified = bms_add_member(modified, attidx);- continue;- }-- /*- * No need to check attributes that can't be stored externally. Note- * that system attributes can't be stored externally.- */- if (attrnum < 0 || isnull1 ||- TupleDescCompactAttr(tupdesc, attrnum - 1)->attlen != -1)- continue;-- /*- * Check if the old tuple's attribute is stored externally and is a- * member of external_cols.- */- if (VARATT_IS_EXTERNAL((varlena *) DatumGetPointer(value1)) &&- bms_is_member(attidx, external_cols))- *has_external = true;+ modified_idx_attrs = bms_add_member(modified_idx_attrs, attidx);
}
- return modified;+ return modified_idx_attrs;
}
/*
@@ -4659,12 +4713,90 @@ simple_heap_update(Relation relation, const ItemPointerData *otid, HeapTuple tup
TM_Result result;
TM_FailureData tmfd;
LockTupleMode lockmode;
+ TupleTableSlot *slot;+ BufferHeapTupleTableSlot *bslot;+ HeapTuple oldtup;+ Bitmapset *indexed_attrs;+ Bitmapset *modified_idx_attrs;+ bool hot_allowed,+ summarized_only;+ Buffer buffer;- result = heap_update(relation, otid, tup,- GetCurrentCommandId(true), 0,- InvalidSnapshot,- true /* wait for commit */ ,- &tmfd, &lockmode, update_indexes);+ Assert(ItemPointerIsValid(otid));++ INJECTION_POINT("simple_heap_update-before-pin", NULL);++ /*+ * To update a heap tuple we need to find the set of modified indexed+ * attributes ("modified_idx_attrs") and use that to determine if a HOT+ * update is allowable or not. When updating heap tuples via execution of+ * UPDATE statements this set is constructed before calling into the table+ * AM's update function by ExecUpdateModifiedIdxAttrs() which compares the+ * old/new TupleTableSlots.+ *+ * Here things are a bit different, we have the old TID and the new tuple,+ * not two TupleTableSlots, but we still need to construct a similar+ * bitmap so as to be able to know if HOT updates are allowed or not.+ *+ * To do that we first have to fetch the old tuple itself, but because+ * heapam_fetch_row_version() is static, we replicate in part that code+ * here.+ *+ * heap_update() reads the old tuple again, under the buffer lock. The+ * comparison used to run there; running it here, before the lock, keeps+ * the lock short. Callers do not expect concurrent updates of the tuple+ * (see the header comment), so both reads see the same version; if one+ * happens anyway, heap_update() reports it.+ */+ slot = MakeTupleTableSlot(RelationGetDescr(relation), &TTSOpsBufferHeapTuple, 0);+ bslot = (BufferHeapTupleTableSlot *) slot;++ /*+ * Set the TID in the slot and then fetch the old tuple so we can examine+ * it+ */+ bslot->base.tupdata.t_self = *otid;+ if (!heap_fetch(relation, SnapshotAny, &bslot->base.tupdata, &buffer, false))+ {+ /*+ * heap_update() checks for !ItemIdIsNormal(lp) too, and returns+ * TM_Deleted, which is reported the same way below.+ */+ Assert(RelationSupportsSysCache(RelationGetRelid(relation)));++ ExecDropSingleTupleTableSlot(slot);+ elog(ERROR, "tuple concurrently deleted");+ }++ Assert(buffer != InvalidBuffer);++ /* Store in slot, transferring existing pin */+ ExecStorePinnedBufferHeapTuple(&bslot->base.tupdata, slot, buffer);+ oldtup = ExecFetchSlotHeapTuple(slot, false, NULL);++ indexed_attrs = RelationGetIndexAttrBitmap(relation,+ INDEX_ATTR_BITMAP_INDEXED);+ modified_idx_attrs = HeapUpdateModifiedIdxAttrs(relation, indexed_attrs,+ oldtup, tup);+ bms_free(indexed_attrs);+ lockmode = HeapUpdateDetermineLockmode(relation, modified_idx_attrs);+ hot_allowed = HeapUpdateHotAllowable(relation, modified_idx_attrs, &summarized_only);++ result = heap_update(relation, otid, tup, GetCurrentCommandId(true), 0,+ InvalidSnapshot, true /* wait for commit */ ,+ &tmfd, lockmode, modified_idx_attrs, hot_allowed);++ ExecDropSingleTupleTableSlot(slot);+ bms_free(modified_idx_attrs);++ /*+ * Decide whether new index entries are needed for the tuple+ *+ * If the update is not HOT, we must update all indexes. If the update is+ * HOT, it could be that we updated summarized columns, so we either+ * update only summarized indexes, or none at all.+ */+ *update_indexes = TU_None;
switch (result)
{
case TM_SelfModified:
@@ -4674,6 +4806,10 @@ simple_heap_update(Relation relation, const ItemPointerData *otid, HeapTuple tup
case TM_Ok:
/* done successfully */
+ if (!HeapTupleIsHeapOnly(tup))+ *update_indexes = TU_All;+ else if (summarized_only)+ *update_indexes = TU_Summarizing;
break;
case TM_Updated:
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.cindex 7d2065dec37..a2a57eac823 100644--- a/src/backend/access/heap/heapam_handler.c+++ b/src/backend/access/heap/heapam_handler.c@@ -218,20 +218,27 @@ static TM_Result
heapam_tuple_update(Relation relation, ItemPointer otid, TupleTableSlot *slot,
CommandId cid, uint32 options,
Snapshot snapshot, Snapshot crosscheck,
- bool wait, TM_FailureData *tmfd,- LockTupleMode *lockmode, TU_UpdateIndexes *update_indexes)+ bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode,+ const Bitmapset *modified_idx_attrs, TU_UpdateIndexes *update_indexes)
{
bool shouldFree = true;
HeapTuple tuple = ExecFetchSlotHeapTuple(slot, true, &shouldFree);
+ bool hot_allowed;+ bool summarized_only;
TM_Result result;
+ Assert(ItemPointerIsValid(otid));++ hot_allowed = HeapUpdateHotAllowable(relation, modified_idx_attrs, &summarized_only);+ *lockmode = HeapUpdateDetermineLockmode(relation, modified_idx_attrs);+
/* Update the tuple with table oid */
slot->tts_tableOid = RelationGetRelid(relation);
tuple->t_tableOid = slot->tts_tableOid;
result = heap_update(relation, otid, tuple, cid, options,
crosscheck, wait,
- tmfd, lockmode, update_indexes);+ tmfd, *lockmode, modified_idx_attrs, hot_allowed);
ItemPointerCopy(&tuple->t_self, &slot->tts_tid);
/*
@@ -244,16 +251,17 @@ heapam_tuple_update(Relation relation, ItemPointer otid, TupleTableSlot *slot,
* HOT, it could be that we updated summarized columns, so we either
* update only summarized indexes, or none at all.
*/
- if (result != TM_Ok)+ *update_indexes = TU_None;+ if (result == TM_Ok)
{
- Assert(*update_indexes == TU_None);- *update_indexes = TU_None;+ if (HeapTupleIsHeapOnly(tuple))+ {+ if (summarized_only)+ *update_indexes = TU_Summarizing;+ }+ else+ *update_indexes = TU_All;
}
- else if (!HeapTupleIsHeapOnly(tuple))- Assert(*update_indexes == TU_All);- else- Assert((*update_indexes == TU_Summarizing) ||- (*update_indexes == TU_None));
if (shouldFree)
pfree(tuple);
diff --git a/src/backend/access/table/tableam.c b/src/backend/access/table/tableam.cindex 3a1ff9a1d46..1ba18f0a215 100644--- a/src/backend/access/table/tableam.c+++ b/src/backend/access/table/tableam.c@@ -30,6 +30,9 @@
#include "storage/bufmgr.h"
#include "storage/shmem.h"
#include "storage/smgr.h"
+#include "utils/datum.h"+#include "utils/relcache.h"+#include "varatt.h"
/*
* Constants to control the behavior of block allocation to parallel workers
@@ -311,6 +314,83 @@ simple_table_tuple_delete(Relation rel, ItemPointer tid, Snapshot snapshot)
}
}
+/* Would two varlena values be stored the same, byte for byte? */+static bool+varlena_image_eq(Datum value1, Datum value2)+{+ varlena *v1 = (varlena *) DatumGetPointer(value1);+ varlena *v2 = (varlena *) DatumGetPointer(value2);++ if (VARATT_IS_EXTERNAL(v1) || VARATT_IS_EXTERNAL(v2) ||+ VARATT_IS_COMPRESSED(v1) || VARATT_IS_COMPRESSED(v2))+ return datumIsEqual(value1, value2, false, -1);++ return VARSIZE_ANY_EXHDR(v1) == VARSIZE_ANY_EXHDR(v2) &&+ memcmp(VARDATA_ANY(v1), VARDATA_ANY(v2), VARSIZE_ANY_EXHDR(v1)) == 0;+}++/*+ * table_slot_modified_idx_attrs - which indexed attributes differ?+ *+ * Return the set of attributes referenced by indexes of rel whose values+ * differ between old_slot and new_slot, offset by+ * FirstLowInvalidHeapAttributeNumber, in memory the caller frees. Values+ * are compared byte for byte as the table would store them, so an unchanged+ * toasted value compares equal without being fetched, and two values that+ * are equal but stored differently count as changed.+ *+ * A varlena value in a slot need not have the header it will be stored with:+ * a short one may have a 4-byte header until heap_form_tuple() packs it, for+ * instance when tuple routing has converted the slot. So an inline,+ * uncompressed varlena is compared without its header.+ */+Bitmapset *+table_slot_modified_idx_attrs(Relation rel, TupleTableSlot *old_slot,+ TupleTableSlot *new_slot)+{+ TupleDesc tupdesc = RelationGetDescr(rel);+ Bitmapset *attrs;+ int attidx = -1;++ attrs = RelationGetIndexAttrBitmap(rel, INDEX_ATTR_BITMAP_INDEXED);++ while ((attidx = bms_next_member(attrs, attidx)) >= 0)+ {+ AttrNumber attrnum = attidx + FirstLowInvalidHeapAttributeNumber;+ CompactAttribute *att;+ Datum value1,+ value2;+ bool null1,+ null2;++ /*+ * A whole-row reference is "changed" (it could only be unchanged+ * after a no-op update), and so is any system column other than+ * tableoid, which need not be set yet in the new tuple.+ */+ if (attrnum <= 0)+ {+ if (attrnum == TableOidAttributeNumber)+ attrs = bms_del_member(attrs, attidx);+ continue;+ }++ att = TupleDescCompactAttr(tupdesc, attrnum - 1);+ value1 = slot_getattr(old_slot, attrnum, &null1);+ value2 = slot_getattr(new_slot, attrnum, &null2);++ if (null1 != null2)+ continue;+ if (null1 ||+ (att->attlen == -1 ?+ varlena_image_eq(value1, value2) :+ datumIsEqual(value1, value2, att->attbyval, att->attlen)))+ attrs = bms_del_member(attrs, attidx);+ }++ return attrs;+}+
/*
* simple_table_tuple_update - replace a tuple
*
@@ -324,6 +404,32 @@ simple_table_tuple_update(Relation rel, ItemPointer otid,
TupleTableSlot *slot,
Snapshot snapshot,
TU_UpdateIndexes *update_indexes)
+{+ TupleTableSlot *oldslot = table_slot_create(rel, NULL);+ Bitmapset *modified_idx_attrs;++ if (!table_tuple_fetch_row_version(rel, otid, SnapshotAny, oldslot))+ elog(ERROR, "tuple concurrently deleted");+ modified_idx_attrs = table_slot_modified_idx_attrs(rel, oldslot, slot);+ ExecDropSingleTupleTableSlot(oldslot);++ simple_table_tuple_update_ext(rel, otid, slot, snapshot,+ modified_idx_attrs, update_indexes);+ bms_free(modified_idx_attrs);+}++/*+ * simple_table_tuple_update_ext - replace a tuple, given what changed+ *+ * As simple_table_tuple_update(), but the caller passes the indexed+ * attributes the update changes, as table_tuple_update() takes them.+ */+void+simple_table_tuple_update_ext(Relation rel, ItemPointer otid,+ TupleTableSlot *slot,+ Snapshot snapshot,+ const Bitmapset *modified_idx_attrs,+ TU_UpdateIndexes *update_indexes)
{
TM_Result result;
TM_FailureData tmfd;
@@ -333,7 +439,9 @@ simple_table_tuple_update(Relation rel, ItemPointer otid,
GetCurrentCommandId(true),
0, snapshot, InvalidSnapshot,
true /* wait for commit */ ,
- &tmfd, &lockmode, update_indexes);+ &tmfd, &lockmode,+ modified_idx_attrs,+ update_indexes);
switch (result)
{
diff --git a/src/backend/commands/repack.c b/src/backend/commands/repack.cindex 1b1a8213358..5fc5808e079 100644--- a/src/backend/commands/repack.c+++ b/src/backend/commands/repack.c@@ -2915,6 +2915,11 @@ apply_concurrent_update(Relation rel, TupleTableSlot *spilled_tuple,
TM_FailureData tmfd;
TU_UpdateIndexes update_indexes;
TM_Result res;
+ Bitmapset *modified_idx_attrs;++ modified_idx_attrs = ExecUpdateModifiedIdxAttrs(chgcxt->cc_rri,+ ondisk_tuple,+ spilled_tuple);
/*
* Carry out the update, skipping logical decoding for it.
@@ -2925,7 +2930,9 @@ apply_concurrent_update(Relation rel, TupleTableSlot *spilled_tuple,
InvalidSnapshot,
InvalidSnapshot,
false,
- &tmfd, &lockmode, &update_indexes);+ &tmfd, &lockmode,+ modified_idx_attrs, &update_indexes);+ bms_free(modified_idx_attrs);
if (res != TM_Ok)
ereport(ERROR,
errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.cindex 13644328164..5f894651d5e 100644--- a/src/backend/executor/execReplication.c+++ b/src/backend/executor/execReplication.c@@ -989,6 +989,7 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
bool skip_tuple = false;
Relation rel = resultRelInfo->ri_RelationDesc;
ItemPointer tid = &(searchslot->tts_tid);
+ Bitmapset *modified_idx_attrs;
/*
* We support only non-system tables, with
@@ -1027,8 +1028,12 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
if (rel->rd_rel->relispartition)
ExecPartitionCheck(resultRelInfo, slot, estate, true);
- simple_table_tuple_update(rel, tid, slot, estate->es_snapshot,- &update_indexes);+ modified_idx_attrs = ExecUpdateModifiedIdxAttrs(resultRelInfo,+ searchslot, slot);++ simple_table_tuple_update_ext(rel, tid, slot, estate->es_snapshot,+ modified_idx_attrs, &update_indexes);+ bms_free(modified_idx_attrs);
conflictindexes = resultRelInfo->ri_onConflictArbiterIndexes;
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.cindex 558ceece9bb..d31eae94fb2 100644--- a/src/backend/executor/nodeModifyTable.c+++ b/src/backend/executor/nodeModifyTable.c@@ -18,6 +18,7 @@
* ExecModifyTable - retrieve the next tuple from the node
* ExecEndModifyTable - shut down the ModifyTable node
* ExecReScanModifyTable - rescan the ModifyTable node
+ * ExecUpdateModifiedIdxAttrs - find set of updated indexed columns
*
* NOTES
* The ModifyTable node receives input from its outerPlan, which is
@@ -56,6 +57,7 @@
#include "access/htup_details.h"
#include "access/tableam.h"
#include "access/tupconvert.h"
+#include "access/tupdesc.h"
#include "access/xact.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
@@ -192,6 +194,40 @@ static TupleTableSlot *ExecMergeNotMatched(ModifyTableContext *context,
ResultRelInfo *resultRelInfo,
bool canSetTag);
+/*+ * ExecUpdateModifiedIdxAttrs+ *+ * Return the set of indexed attributes whose values differ between old_tts+ * and new_tts, using the FirstLowInvalidHeapAttributeNumber convention. The+ * result describes this one pair of tuples; the caller frees it.+ *+ * All indexed attributes are compared, not just those the UPDATE targets.+ * ExecGetAllUpdatedCols() names the columns an UPDATE targets, not the values+ * that changed: a BEFORE ROW trigger can change an indexed column with+ * heap_modify_tuple() (see tsvector_update_trigger()), and logical+ * replication and REPACK apply hand us whole tuples.+ *+ * Values are compared byte for byte as stored, as heap_update() always has+ * (see table_slot_modified_idx_attrs()). An unchanged toasted value keeps+ * its toast pointer, so it compares equal without being fetched. Two values+ * that are equal but stored differently count as changed; that costs an+ * index entry, never a missed one.+ *+ * simple_heap_update() makes the same determination from a TID and a+ * HeapTuple in HeapUpdateModifiedIdxAttrs(), and simple_table_tuple_update()+ * with table_slot_modified_idx_attrs().+ */+Bitmapset *+ExecUpdateModifiedIdxAttrs(ResultRelInfo *resultRelInfo,+ TupleTableSlot *old_tts,+ TupleTableSlot *new_tts)+{+ if (resultRelInfo->ri_NumIndices == 0)+ return NULL;++ return table_slot_modified_idx_attrs(resultRelInfo->ri_RelationDesc,+ old_tts, new_tts);+}
/*
* Verify that the tuples to be produced by INSERT match the
@@ -2203,14 +2239,17 @@ ExecUpdatePrepareSlot(ResultRelInfo *resultRelInfo,
*/
static TM_Result
ExecUpdateAct(ModifyTableContext *context, ResultRelInfo *resultRelInfo,
- ItemPointer tupleid, HeapTuple oldtuple, TupleTableSlot *slot,- bool canSetTag, UpdateContext *updateCxt)+ ItemPointer tupleid, HeapTuple oldtuple, TupleTableSlot *oldSlot,+ TupleTableSlot *slot, bool canSetTag, UpdateContext *updateCxt)
{
EState *estate = context->estate;
Relation resultRelationDesc = resultRelInfo->ri_RelationDesc;
bool partition_constraint_failed;
TM_Result result;
+ /* The set of modified indexed attributes that trigger new index entries */+ Bitmapset *modified_idx_attrs = NULL;+
updateCxt->crossPartUpdate = false;
/*
@@ -2326,8 +2365,10 @@ lreplace:
if (resultRelationDesc->rd_att->constr)
ExecConstraints(resultRelInfo, slot, estate);
+ modified_idx_attrs = ExecUpdateModifiedIdxAttrs(resultRelInfo, oldSlot, slot);+
/*
- * replace the heap tuple+ * Call into the table AM to update the heap tuple.
*
* Note: if es_crosscheck_snapshot isn't InvalidSnapshot, we check that
* the row to be updated is visible to that snapshot, and throw a
@@ -2342,7 +2383,9 @@ lreplace:
estate->es_crosscheck_snapshot,
true /* wait for commit */ ,
&context->tmfd, &updateCxt->lockmode,
+ modified_idx_attrs,
&updateCxt->updateIndexes);
+ bms_free(modified_idx_attrs);
return result;
}
@@ -2583,8 +2626,8 @@ ExecUpdate(ModifyTableContext *context, ResultRelInfo *resultRelInfo,
*/
redo_act:
lockedtid = *tupleid;
- result = ExecUpdateAct(context, resultRelInfo, tupleid, oldtuple, slot,- canSetTag, &updateCxt);+ result = ExecUpdateAct(context, resultRelInfo, tupleid, oldtuple, oldSlot,+ slot, canSetTag, &updateCxt);
/*
* If ExecUpdateAct reports that a cross-partition update was done,
@@ -3435,8 +3478,8 @@ lmerge_matched:
Assert(oldtuple == NULL);
result = ExecUpdateAct(context, resultRelInfo, tupleid,
- NULL, newslot, canSetTag,- &updateCxt);+ NULL, resultRelInfo->ri_oldTupleSlot,+ newslot, canSetTag, &updateCxt);
/*
* As in ExecUpdate(), if ExecUpdateAct() reports that a
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.cindex d8f04a05309..7b83882bf28 100644--- a/src/backend/utils/cache/relcache.c+++ b/src/backend/utils/cache/relcache.c@@ -2477,6 +2477,7 @@ RelationDestroyRelation(Relation relation, bool remember_tupdesc)
bms_free(relation->rd_idattr);
bms_free(relation->rd_hotblockingattr);
bms_free(relation->rd_summarizedattr);
+ bms_free(relation->rd_indexedattr);
if (relation->rd_pubdesc)
pfree(relation->rd_pubdesc);
if (relation->rd_options)
@@ -5325,6 +5326,7 @@ RelationGetIndexPredicate(Relation relation)
* index (empty if FULL)
* INDEX_ATTR_BITMAP_HOT_BLOCKING Columns that block updates from being HOT
* INDEX_ATTR_BITMAP_SUMMARIZED Columns included in summarizing indexes
+ * INDEX_ATTR_BITMAP_INDEXED Columns referenced by any index
*
* Attribute numbers are offset by FirstLowInvalidHeapAttributeNumber so that
* we can include system attributes (e.g., OID) in the bitmap representation.
@@ -5349,6 +5351,7 @@ RelationGetIndexAttrBitmap(Relation relation, IndexAttrBitmapKind attrKind)
Bitmapset *idindexattrs; /* columns in the replica identity */
Bitmapset *hotblockingattrs; /* columns with HOT blocking indexes */
Bitmapset *summarizedattrs; /* columns with summarizing indexes */
+ Bitmapset *indexedattrs; /* columns referenced by any index */
List *indexoidlist;
List *newindexoidlist;
Oid relpkindex;
@@ -5371,6 +5374,8 @@ RelationGetIndexAttrBitmap(Relation relation, IndexAttrBitmapKind attrKind)
return bms_copy(relation->rd_hotblockingattr);
case INDEX_ATTR_BITMAP_SUMMARIZED:
return bms_copy(relation->rd_summarizedattr);
+ case INDEX_ATTR_BITMAP_INDEXED:+ return bms_copy(relation->rd_indexedattr);
default:
elog(ERROR, "unknown attrKind %u", attrKind);
}
@@ -5551,6 +5556,8 @@ restart:
goto restart;
}
+ indexedattrs = bms_union(hotblockingattrs, summarizedattrs);+
/* Don't leak the old values of these bitmaps, if any */
relation->rd_attrsvalid = false;
bms_free(relation->rd_keyattr);
@@ -5563,6 +5570,8 @@ restart:
relation->rd_hotblockingattr = NULL;
bms_free(relation->rd_summarizedattr);
relation->rd_summarizedattr = NULL;
+ bms_free(relation->rd_indexedattr);+ relation->rd_indexedattr = NULL;
/*
* Now save copies of the bitmaps in the relcache entry. We intentionally
@@ -5577,6 +5586,7 @@ restart:
relation->rd_idattr = bms_copy(idindexattrs);
relation->rd_hotblockingattr = bms_copy(hotblockingattrs);
relation->rd_summarizedattr = bms_copy(summarizedattrs);
+ relation->rd_indexedattr = bms_copy(indexedattrs);
relation->rd_attrsvalid = true;
MemoryContextSwitchTo(oldcxt);
@@ -5593,6 +5603,8 @@ restart:
return hotblockingattrs;
case INDEX_ATTR_BITMAP_SUMMARIZED:
return summarizedattrs;
+ case INDEX_ATTR_BITMAP_INDEXED:+ return indexedattrs;
default:
elog(ERROR, "unknown attrKind %u", attrKind);
return NULL;
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.hindex 9e35961fb9e..fa7c540425c 100644--- a/src/include/access/heapam.h+++ b/src/include/access/heapam.h@@ -389,8 +389,13 @@ extern TM_Result heap_update(Relation relation, const ItemPointerData *otid,
HeapTuple newtup,
CommandId cid, uint32 options,
Snapshot crosscheck, bool wait,
- TM_FailureData *tmfd, LockTupleMode *lockmode,- TU_UpdateIndexes *update_indexes);+ TM_FailureData *tmfd, LockTupleMode lockmode,+ const Bitmapset *modified_idx_attrs, bool hot_allowed);+extern bool HeapUpdateHotAllowable(Relation relation,+ const Bitmapset *modified_idx_attrs,+ bool *summarized_only);+extern LockTupleMode HeapUpdateDetermineLockmode(Relation relation,+ const Bitmapset *modified_idx_attrs);
extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
CommandId cid, LockTupleMode mode, LockWaitPolicy wait_policy,
bool follow_updates,
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.hindex ea3f2a6be99..a3174fb20f1 100644--- a/src/include/access/tableam.h+++ b/src/include/access/tableam.h@@ -586,6 +586,7 @@ typedef struct TableAmRoutine
bool wait,
TM_FailureData *tmfd,
LockTupleMode *lockmode,
+ const Bitmapset *modified_idx_attrs,
TU_UpdateIndexes *update_indexes);
/* see table_tuple_lock() for reference about parameters */
@@ -1594,6 +1595,10 @@ table_tuple_delete(Relation rel, ItemPointer tid, CommandId cid,
* during table rewrites where RelationIsLogicallyLogged(rel) is not
* yet accurate for the new relation.
* crosscheck - if not InvalidSnapshot, also check old tuple against this
+ * modified_idx_attrs - the indexed attributes whose values differ between+ * the old tuple and the new one, offset by+ * FirstLowInvalidHeapAttributeNumber; see ExecUpdateModifiedIdxAttrs().+ * The caller owns it.
*
* Output parameters:
* slot - newly constructed tuple data to store
@@ -1621,12 +1626,12 @@ table_tuple_update(Relation rel, ItemPointer otid, TupleTableSlot *slot,
CommandId cid, uint32 options,
Snapshot snapshot, Snapshot crosscheck,
bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode,
- TU_UpdateIndexes *update_indexes)+ const Bitmapset *modified_idx_attrs, TU_UpdateIndexes *update_indexes)
{
return rel->rd_tableam->tuple_update(rel, otid, slot,
cid, options, snapshot, crosscheck,
- wait, tmfd,- lockmode, update_indexes);+ wait, tmfd, lockmode,+ modified_idx_attrs, update_indexes);
}
/*
@@ -2112,6 +2117,14 @@ extern void simple_table_tuple_delete(Relation rel, ItemPointer tid,
extern void simple_table_tuple_update(Relation rel, ItemPointer otid,
TupleTableSlot *slot, Snapshot snapshot,
TU_UpdateIndexes *update_indexes);
+extern void simple_table_tuple_update_ext(Relation rel, ItemPointer otid,+ TupleTableSlot *slot,+ Snapshot snapshot,+ const Bitmapset *modified_idx_attrs,+ TU_UpdateIndexes *update_indexes);+extern Bitmapset *table_slot_modified_idx_attrs(Relation rel,+ TupleTableSlot *old_slot,+ TupleTableSlot *new_slot);
/* ----------------------------------------------------------------------------
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.hindex 152a1dfa568..5cbdb092564 100644--- a/src/include/executor/executor.h+++ b/src/include/executor/executor.h@@ -817,5 +817,8 @@ extern ResultRelInfo *ExecLookupResultRelByOid(ModifyTableState *node,
Oid resultoid,
bool missing_ok,
bool update_cache);
+extern Bitmapset *ExecUpdateModifiedIdxAttrs(ResultRelInfo *relinfo,+ TupleTableSlot *old_tts,+ TupleTableSlot *new_tts);
#endif /* EXECUTOR_H */
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.hindex 7b8c2b1e362..55b459adba8 100644--- a/src/include/utils/rel.h+++ b/src/include/utils/rel.h@@ -164,6 +164,7 @@ typedef struct RelationData
Bitmapset *rd_idattr; /* included in replica identity index */
Bitmapset *rd_hotblockingattr; /* cols blocking HOT update */
Bitmapset *rd_summarizedattr; /* cols indexed by summarizing indexes */
+ Bitmapset *rd_indexedattr; /* cols referenced by any index */
PublicationDesc *rd_pubdesc; /* publication descriptor, or NULL */
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.hindex 3e97da6a538..417990e8022 100644--- a/src/include/utils/relcache.h+++ b/src/include/utils/relcache.h@@ -72,6 +72,7 @@ typedef enum IndexAttrBitmapKind
INDEX_ATTR_BITMAP_IDENTITY_KEY,
INDEX_ATTR_BITMAP_HOT_BLOCKING,
INDEX_ATTR_BITMAP_SUMMARIZED,
+ INDEX_ATTR_BITMAP_INDEXED,
} IndexAttrBitmapKind;
extern Bitmapset *RelationGetIndexAttrBitmap(Relation relation,
diff --git a/src/test/modules/injection_points/expected/syscache-update-pruned.out b/src/test/modules/injection_points/expected/syscache-update-pruned.outindex a6a4e8db996..07ef67a1eb4 100644--- a/src/test/modules/injection_points/expected/syscache-update-pruned.out+++ b/src/test/modules/injection_points/expected/syscache-update-pruned.out@@ -16,8 +16,8 @@ step wakeinval4:
step at2: <... completed>
step wakeinval4: <... completed>
step wakegrant4:
- SELECT FROM injection_points_detach('heap_update-before-pin');- SELECT FROM injection_points_wakeup('heap_update-before-pin');+ SELECT FROM injection_points_detach('simple_heap_update-before-pin');+ SELECT FROM injection_points_wakeup('simple_heap_update-before-pin');
<waiting ...>
step grant1: <... completed>
ERROR: tuple concurrently deleted
@@ -42,8 +42,8 @@ step mkrels4:
SELECT FROM vactest.mkrels('intruder', 1, 100); -- repopulate LP_UNUSED
step wakegrant4:
- SELECT FROM injection_points_detach('heap_update-before-pin');- SELECT FROM injection_points_wakeup('heap_update-before-pin');+ SELECT FROM injection_points_detach('simple_heap_update-before-pin');+ SELECT FROM injection_points_wakeup('simple_heap_update-before-pin');
<waiting ...>
step grant1: <... completed>
ERROR: duplicate key value violates unique constraint "pg_class_oid_index"
@@ -71,8 +71,8 @@ step at2: <... completed>
step wakeinval4: <... completed>
step at4: ALTER TABLE vactest.child50 INHERIT vactest.orig50;
step wakegrant4:
- SELECT FROM injection_points_detach('heap_update-before-pin');- SELECT FROM injection_points_wakeup('heap_update-before-pin');+ SELECT FROM injection_points_detach('simple_heap_update-before-pin');+ SELECT FROM injection_points_wakeup('simple_heap_update-before-pin');
<waiting ...>
step grant1: <... completed>
step wakegrant4: <... completed>
diff --git a/src/test/modules/injection_points/specs/syscache-update-pruned.spec b/src/test/modules/injection_points/specs/syscache-update-pruned.specindex e3a4295bd12..fef9ac895a1 100644--- a/src/test/modules/injection_points/specs/syscache-update-pruned.spec+++ b/src/test/modules/injection_points/specs/syscache-update-pruned.spec@@ -103,7 +103,7 @@ session s1
setup {
SET debug_discard_caches = 0;
SELECT FROM injection_points_set_local();
- SELECT FROM injection_points_attach('heap_update-before-pin', 'wait');+ SELECT FROM injection_points_attach('simple_heap_update-before-pin', 'wait');
}
step cachefill1 { SELECT FROM vactest.reloid_catcache_set('vactest.orig50'); }
step grant1 { GRANT SELECT ON vactest.orig50 TO PUBLIC; }
@@ -140,8 +140,8 @@ step mkrels4 {
SELECT FROM vactest.mkrels('intruder', 1, 100); -- repopulate LP_UNUSED
}
step wakegrant4 {
- SELECT FROM injection_points_detach('heap_update-before-pin');- SELECT FROM injection_points_wakeup('heap_update-before-pin');+ SELECT FROM injection_points_detach('simple_heap_update-before-pin');+ SELECT FROM injection_points_wakeup('simple_heap_update-before-pin');
}
step at4 { ALTER TABLE vactest.child50 INHERIT vactest.orig50; }
step wakeinval4 {
--
2.50.1
v43-0008-hash-implement-amcomparedatums.patch (text/plain, 9.8K) download
From 9ce22b7d50392d494f4da4d574f9d13a3ecb3023 Mon Sep 17 00:00:00 2001
From: Greg Burd <greg@burd.me>
Date: Fri, 16 Oct 2026 12:50:30 -0400
Subject: [PATCH v43 8/8] hash: implement amcomparedatums
A hash index stores only the hash code of the value, and every scan, plain
or bitmap, rechecks the heap tuple. So an updated row gives the same entry
exactly when the new value has the same hash code as the old one.
Updates that become HOT include changing the display scale of a numeric,
which hash_numeric() ignores, and changing a string to one equal under a
nondeterministic collation.
A test checks that an exclusion constraint using a hash index still sees a
change to a value with the same hash code (hashint8(1) = hashint8(2^32)):
the executor does not consult amcomparedatums for exclusion constraints.
---
src/backend/access/hash/hash.c | 20 +++-
src/include/access/hash.h | 3 +
src/test/regress/expected/hash_index.out | 124 +++++++++++++++++++++++
src/test/regress/sql/hash_index.sql | 59 +++++++++++
4 files changed, 205 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.cindex 0b05c4606e1..8677b6b3f96 100644--- a/src/backend/access/hash/hash.c+++ b/src/backend/access/hash/hash.c@@ -123,7 +123,7 @@ hashhandler(PG_FUNCTION_ARGS)
.amparallelrescan = NULL,
.amtranslatestrategy = hashtranslatestrategy,
.amtranslatecmptype = hashtranslatecmptype,
- .amcomparedatums = NULL,+ .amcomparedatums = hashcomparedatums,
};
PG_RETURN_POINTER(&amroutine);
@@ -1029,3 +1029,21 @@ hashtranslatecmptype(CompareType cmptype, Oid opfamily)
return HTEqualStrategyNumber;
return InvalidStrategy;
}
++/*+ * hashcomparedatums() -- would newValue give the same index entry as oldValue?+ *+ * A hash index stores only the hash code of the value, and every scan+ * rechecks the heap tuple, so the entries are the same if the hash codes+ * are. Then, for example, numeric 1.0 and 1.00, or strings equal under a+ * case-insensitive collation, make no new entries.+ */+bool+hashcomparedatums(Relation index, IndexInfo *indexInfo, int attnum,+ Oid typid, int16 typlen, bool typbyval,+ Datum oldValue, Datum newValue)+{+ Assert(attnum == 1);+ return _hash_datum2hashkey(index, oldValue) ==+ _hash_datum2hashkey(index, newValue);+}diff --git a/src/include/access/hash.h b/src/include/access/hash.hindex a8702f0e5ea..2ff2aac9822 100644--- a/src/include/access/hash.h+++ b/src/include/access/hash.h@@ -389,6 +389,9 @@ extern void hashadjustmembers(Oid opfamilyoid,
extern CompareType hashtranslatestrategy(StrategyNumber strategy, Oid opfamily);
extern StrategyNumber hashtranslatecmptype(CompareType cmptype, Oid opfamily);
+extern bool hashcomparedatums(Relation index, struct IndexInfo *indexInfo,+ int attnum, Oid typid, int16 typlen,+ bool typbyval, Datum oldValue, Datum newValue);
/* private routines */
diff --git a/src/test/regress/expected/hash_index.out b/src/test/regress/expected/hash_index.outindex 0b099b9e771..0192ccd8ae3 100644--- a/src/test/regress/expected/hash_index.out+++ b/src/test/regress/expected/hash_index.out@@ -375,3 +375,127 @@ CREATE INDEX hash_f8_index2 ON hash_f8_heap USING hash (random float8_ops)
WITH (fillfactor=101);
ERROR: value 101 out of bounds for option "fillfactor"
DETAIL: Valid values are between "10" and "100".
+--+-- An UPDATE to a value with the same hash code is HOT: the index stores+-- only the code, and scans recheck the heap tuple.+--+CREATE TABLE hash_hot (id int, n numeric) WITH (fillfactor = 10);+CREATE INDEX hash_hot_n ON hash_hot USING hash (n);+CREATE INDEX hash_hot_n2 ON hash_hot USING hash ((n * 2));+INSERT INTO hash_hot SELECT g, g FROM generate_series(1, 50) g;+-- (pg_stat_get_xact_* count this transaction's updates, and earlier+-- transactions' not yet reported)+CREATE FUNCTION hash_hot_count(OUT updated bigint, OUT hot bigint)+LANGUAGE sql AS $$+ SELECT pg_stat_get_xact_tuples_updated('hash_hot'::regclass),+ pg_stat_get_xact_tuples_hot_updated('hash_hot'::regclass) $$;+BEGIN;+-- numeric hashes ignore the display scale, in the column and in the+-- expression: HOT+UPDATE hash_hot SET n = n + 0.00;+SELECT * FROM hash_hot_count();+ updated | hot +---------+-----+ 50 | 50+(1 row)++-- a different value: not HOT+UPDATE hash_hot SET n = n + 100 WHERE id <= 5;+SELECT * FROM hash_hot_count();+ updated | hot +---------+-----+ 55 | 50+(1 row)++COMMIT;+SET enable_seqscan = off;+SET enable_bitmapscan = off;+EXPLAIN (COSTS OFF) SELECT id, n FROM hash_hot WHERE n = 7;+ QUERY PLAN +-----------------------------------------+ Index Scan using hash_hot_n on hash_hot+ Index Cond: (n = '7'::numeric)+(2 rows)++SELECT id, n FROM hash_hot WHERE n = 7;+ id | n +----+------+ 7 | 7.00+(1 row)++SELECT id, n FROM hash_hot WHERE n = 7.000;+ id | n +----+------+ 7 | 7.00+(1 row)++SELECT id, n FROM hash_hot WHERE n = 3;+ id | n +----+---+(0 rows)++SELECT id, n FROM hash_hot WHERE n = 103;+ id | n +----+--------+ 3 | 103.00+(1 row)++SELECT id, n FROM hash_hot WHERE n * 2 = 14;+ id | n +----+------+ 7 | 7.00+(1 row)++RESET enable_bitmapscan;+SET enable_indexscan = off;+SELECT count(*) FROM hash_hot h WHERE n = (SELECT n FROM hash_hot WHERE id = h.id);+ count +-------+ 50+(1 row)++RESET enable_indexscan;+RESET enable_seqscan;+DROP TABLE hash_hot;+-- An exclusion constraint is checked on the values, so a change to a value+-- with the same hash code must still make a new entry and be checked.+-- hashint8 folds the high half into the low, so 1 and 2^32 collide.+SELECT hashint8(1) = hashint8(4294967296) AS same_hash;+ same_hash +-----------+ t+(1 row)++CREATE TABLE hash_hot (id int, k int8, EXCLUDE USING hash (k WITH =))+ WITH (fillfactor = 10);+INSERT INTO hash_hot VALUES (1, 1), (2, 4294967296);+UPDATE hash_hot SET k = 4294967296 WHERE id = 1;+ERROR: conflicting key value violates exclusion constraint "hash_hot_k_excl"+DETAIL: Key (k)=(4294967296) conflicts with existing key (k)=(4294967296).+-- pg_stat_get_xact_* also count earlier transactions' updates not yet+-- reported; report them, so the counts below start from zero+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++BEGIN;+DELETE FROM hash_hot WHERE id = 2;+-- without a conflict, the update is checked and is not HOT+UPDATE hash_hot SET k = 4294967296 WHERE id = 1;+SELECT * FROM hash_hot_count();+ updated | hot +---------+-----+ 1 | 0+(1 row)++COMMIT;+SELECT * FROM hash_hot;+ id | k +----+------------+ 1 | 4294967296+(1 row)++DROP TABLE hash_hot;+DROP FUNCTION hash_hot_count;diff --git a/src/test/regress/sql/hash_index.sql b/src/test/regress/sql/hash_index.sqlindex 3ebdaeaf172..051e7427041 100644--- a/src/test/regress/sql/hash_index.sql+++ b/src/test/regress/sql/hash_index.sql@@ -351,3 +351,62 @@ CREATE INDEX hash_f8_index2 ON hash_f8_heap USING hash (random float8_ops)
WITH (fillfactor=9);
CREATE INDEX hash_f8_index2 ON hash_f8_heap USING hash (random float8_ops)
WITH (fillfactor=101);
++--+-- An UPDATE to a value with the same hash code is HOT: the index stores+-- only the code, and scans recheck the heap tuple.+--+CREATE TABLE hash_hot (id int, n numeric) WITH (fillfactor = 10);+CREATE INDEX hash_hot_n ON hash_hot USING hash (n);+CREATE INDEX hash_hot_n2 ON hash_hot USING hash ((n * 2));+INSERT INTO hash_hot SELECT g, g FROM generate_series(1, 50) g;+-- (pg_stat_get_xact_* count this transaction's updates, and earlier+-- transactions' not yet reported)+CREATE FUNCTION hash_hot_count(OUT updated bigint, OUT hot bigint)+LANGUAGE sql AS $$+ SELECT pg_stat_get_xact_tuples_updated('hash_hot'::regclass),+ pg_stat_get_xact_tuples_hot_updated('hash_hot'::regclass) $$;+BEGIN;+-- numeric hashes ignore the display scale, in the column and in the+-- expression: HOT+UPDATE hash_hot SET n = n + 0.00;+SELECT * FROM hash_hot_count();+-- a different value: not HOT+UPDATE hash_hot SET n = n + 100 WHERE id <= 5;+SELECT * FROM hash_hot_count();+COMMIT;+SET enable_seqscan = off;+SET enable_bitmapscan = off;+EXPLAIN (COSTS OFF) SELECT id, n FROM hash_hot WHERE n = 7;+SELECT id, n FROM hash_hot WHERE n = 7;+SELECT id, n FROM hash_hot WHERE n = 7.000;+SELECT id, n FROM hash_hot WHERE n = 3;+SELECT id, n FROM hash_hot WHERE n = 103;+SELECT id, n FROM hash_hot WHERE n * 2 = 14;+RESET enable_bitmapscan;+SET enable_indexscan = off;+SELECT count(*) FROM hash_hot h WHERE n = (SELECT n FROM hash_hot WHERE id = h.id);+RESET enable_indexscan;+RESET enable_seqscan;+DROP TABLE hash_hot;++-- An exclusion constraint is checked on the values, so a change to a value+-- with the same hash code must still make a new entry and be checked.+-- hashint8 folds the high half into the low, so 1 and 2^32 collide.+SELECT hashint8(1) = hashint8(4294967296) AS same_hash;+CREATE TABLE hash_hot (id int, k int8, EXCLUDE USING hash (k WITH =))+ WITH (fillfactor = 10);+INSERT INTO hash_hot VALUES (1, 1), (2, 4294967296);+UPDATE hash_hot SET k = 4294967296 WHERE id = 1;+-- pg_stat_get_xact_* also count earlier transactions' updates not yet+-- reported; report them, so the counts below start from zero+SELECT pg_stat_force_next_flush();+BEGIN;+DELETE FROM hash_hot WHERE id = 2;+-- without a conflict, the update is checked and is not HOT+UPDATE hash_hot SET k = 4294967296 WHERE id = 1;+SELECT * FROM hash_hot_count();+COMMIT;+SELECT * FROM hash_hot;+DROP TABLE hash_hot;+DROP FUNCTION hash_hot_count;--
2.50.1
v43-0004-Skip-evaluating-indexes-that-an-UPDATE-s-SET-cla.patch (text/plain, 99.6K) download
From 41e36dacc2f94115fe8362f48056f797c3865f84 Mon Sep 17 00:00:00 2001
From: Greg Burd <greg@burd.me>
Date: Thu, 15 Oct 2026 12:51:22 -0400
Subject: [PATCH v43 4/8] Skip evaluating indexes that an UPDATE's SET clause
cannot affect
The previous commit lets an UPDATE be HOT when it changes a column that
indexes use only through expressions or partial-index predicates, if
evaluating those expressions and predicates on the old and new rows gives
the same results. For a large structured column, such as a jsonb
document, that means reading the document twice per index on every row,
even when the SET clause plainly changes a part no index reads:
CREATE INDEX ON t ((doc->>'name'));
UPDATE t SET doc = jsonb_set(doc, '{visits}', ...);
Work this out once per statement, when the update projection is set up,
from the plan tree alone. For each SET target column that expression or
partial indexes use, find which parts of the column the SET expression may
change, and which parts each index reads; an index whose parts cannot
overlap the changed ones needs no evaluation for that column. Nothing at
run time is involved, so the result is the same with or without JIT
compilation, and there is no per-row state to reset.
Parts are named by paths of object keys and array positions. A function
describes its calls through its support function, with the new
SupportRequestSubattr request: which parts of which argument a call reads,
or which parts of its source argument it may change, given its constant
arguments. Array and jsonb subscripting, composite field selection and
assignment, COALESCE and domain coercion are understood directly. The
jsonb accessors (->, ->>, #>, #>>, jsonb_extract_path[_text]) and
modifiers (jsonb_set, jsonb_set_lax, jsonb_insert, -, #-, and || with a
constant object) get a support function, jsonb_subattr_support().
Extensions can describe their own document types and functions the same
way. External Params of a generic plan are read as constants, so
prepared statements benefit as well.
The analysis gives up for a column unless the subplan is a plain scan of
the target relation (directly, or under an Append for inheritance and
partitioning), the statement is an UPDATE rather than a MERGE, there are
no BEFORE ROW UPDATE triggers, and every reference to the column in the
SET expression and the index is one it understands. At run time, the
conclusion is not used for a row whose old or new value of the column is
null. Two paths are taken to overlap unless they differ in a key that
cannot be read as an array position, so changes that move array elements,
negative positions and keys such as "01" or " 1" are all covered.
Assert-enabled builds evaluate the indexes anyway and raise an error if
the conclusion was wrong.
This only saves work: an update is HOT exactly when it was before.
Each conclusion is logged at DEBUG1. amcheck's new TAP test
007_subattr_proof reads them from the server log; check_subattr checks
the HOT counts and the indexes for many SET clauses, and misc_functions
checks HOT counts and an index scan in builds without assertions.
---
contrib/amcheck/Makefile | 2 +-
contrib/amcheck/expected/check_subattr.out | 517 ++++++++++++++
contrib/amcheck/meson.build | 2 +
contrib/amcheck/sql/check_subattr.sql | 162 +++++
contrib/amcheck/t/007_subattr_proof.pl | 123 ++++
doc/src/sgml/storage.sgml | 6 +-
doc/src/sgml/xfunc.sgml | 15 +
src/backend/executor/Makefile | 1 +
src/backend/executor/execSubattr.c | 630 ++++++++++++++++++
src/backend/executor/meson.build | 1 +
src/backend/executor/nodeModifyTable.c | 197 +++++-
src/backend/utils/adt/jsonfuncs.c | 275 ++++++++
src/include/catalog/pg_proc.dat | 63 +-
src/include/executor/execSubattr.h | 22 +
src/include/nodes/execnodes.h | 12 +
src/include/nodes/supportnodes.h | 74 ++
.../isolation/expected/update-subattr-epq.out | 65 ++
src/test/isolation/isolation_schedule | 1 +
.../isolation/specs/update-subattr-epq.spec | 44 ++
src/test/regress/expected/misc_functions.out | 68 ++
src/test/regress/regress.c | 38 ++
src/test/regress/sql/misc_functions.sql | 42 ++
src/tools/pgindent/typedefs.list | 5 +
23 files changed, 2336 insertions(+), 29 deletions(-)
create mode 100644 contrib/amcheck/expected/check_subattr.out
create mode 100644 contrib/amcheck/sql/check_subattr.sql
create mode 100644 contrib/amcheck/t/007_subattr_proof.pl
create mode 100644 src/backend/executor/execSubattr.c
create mode 100644 src/include/executor/execSubattr.h
create mode 100644 src/test/isolation/expected/update-subattr-epq.out
create mode 100644 src/test/isolation/specs/update-subattr-epq.spec
diff --git a/contrib/amcheck/Makefile b/contrib/amcheck/Makefileindex 1b7a63cbaa4..1ca619e1777 100644--- a/contrib/amcheck/Makefile+++ b/contrib/amcheck/Makefile@@ -13,7 +13,7 @@ DATA = amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck
amcheck--1.3--1.4.sql amcheck--1.4--1.5.sql
PGFILEDESC = "amcheck - function for verifying relation integrity"
-REGRESS = check check_btree check_gin check_heap+REGRESS = check check_btree check_gin check_heap check_subattr
EXTRA_INSTALL = contrib/pg_walinspect
TAP_TESTS = 1
diff --git a/contrib/amcheck/expected/check_subattr.out b/contrib/amcheck/expected/check_subattr.outnew file mode 100644index 00000000000..266da431504--- /dev/null+++ b/contrib/amcheck/expected/check_subattr.out@@ -0,0 +1,517 @@+--+-- Updates of parts of jsonb, array and composite columns+--+-- The executor works out from an UPDATE's SET expressions which parts of a+-- column it can change, and skips evaluating the expression and partial+-- indexes that read only other parts. A wrong conclusion would leave an+-- index without an entry for a row, so each case checks every index against+-- the heap; assert-enabled builds also evaluate the indexes anyway and fail+-- if the conclusion was wrong. HOT counts show which updates avoided new+-- index entries, whether or not anything was skipped.+--+SET client_min_messages = warning;+CREATE TABLE subattr_results (name text, hot bigint, upd bigint, amcheck text);+CREATE FUNCTION subattr_probe(name text, upd text, arg jsonb DEFAULT NULL)+RETURNS void LANGUAGE plpgsql AS $f$+DECLARE h0 bigint; u0 bigint; ac text := 'ok'; i regclass;+BEGIN+ CREATE TABLE subattr (id int PRIMARY KEY, d jsonb, a int[], x int)+ WITH (fillfactor = 10);+ CREATE INDEX subattr_name ON subattr ((d->>'name'));+ CREATE INDEX subattr_nested ON subattr ((d->'o'->>'k'));+ CREATE INDEX subattr_path ON subattr ((d #>> '{p,q}'));+ CREATE INDEX subattr_arr0 ON subattr ((d->'tags'->>0));+ CREATE INDEX subattr_sub ON subattr ((d['s']));+ CREATE INDEX subattr_part ON subattr (id) WHERE (d->>'flag') = 'on';+ CREATE INDEX subattr_a2 ON subattr ((a[2]));+ INSERT INTO subattr SELECT g,+ jsonb_build_object('name', 'n' || g, 'age', g, 'tags', '[1,2,3]'::jsonb,+ 'o', jsonb_build_object('k', 'k' || g, 'z', 1),+ 'p', jsonb_build_object('q', 'q' || g), 's', 's' || g,+ 'flag', CASE WHEN g % 2 = 0 THEN 'on' ELSE 'off' END,+ '1', 'one', 'arr', '[10,20,30]'::jsonb),+ ARRAY[g, g + 1, g + 2], 0+ FROM generate_series(1, 50) g;+ IF name LIKE 'trigger%' THEN+ CREATE FUNCTION subattr_trg() RETURNS trigger LANGUAGE plpgsql AS+ $t$BEGIN NEW.d := jsonb_set(NEW.d, '{name}', '"renamed"'); RETURN NEW; END$t$;+ CREATE TRIGGER subattr_trg BEFORE UPDATE ON subattr+ FOR EACH ROW EXECUTE FUNCTION subattr_trg();+ END IF;+ h0 := pg_stat_get_xact_tuples_hot_updated('subattr'::regclass);+ u0 := pg_stat_get_xact_tuples_updated('subattr'::regclass);+ IF arg IS NULL THEN EXECUTE upd; ELSE EXECUTE upd USING arg; END IF;+ FOR i IN SELECT indexrelid::regclass FROM pg_index+ WHERE indrelid = 'subattr'::regclass ORDER BY 1 LOOP+ BEGIN PERFORM bt_index_check(i, true);+ EXCEPTION WHEN OTHERS THEN+ ac := CASE WHEN ac = 'ok' THEN '' ELSE ac || '; ' END ||+ i::text || ': ' || SQLERRM;+ END;+ END LOOP;+ INSERT INTO subattr_results VALUES (name,+ pg_stat_get_xact_tuples_hot_updated('subattr'::regclass) - h0,+ pg_stat_get_xact_tuples_updated('subattr'::regclass) - u0, ac);+ DROP TABLE subattr;+ DROP FUNCTION IF EXISTS subattr_trg();+END $f$;+-- Nothing indexed changes.+SELECT subattr_probe('set-age', $$UPDATE subattr SET d = jsonb_set(d, '{age}', '1')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('set-o-z', $$UPDATE subattr SET d = jsonb_set(d, '{o,z}', '2')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('set-p-other', $$UPDATE subattr SET d = jsonb_set(d, '{p,r}', '2')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('delete-age', $$UPDATE subattr SET d = d - 'age'$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('delete-keys', $$UPDATE subattr SET d = d - '{age,zz}'::text[]$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('delete-path-age', $$UPDATE subattr SET d = d #- '{age}'$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('delete-arr-elem', $$UPDATE subattr SET d = jsonb_set(d, '{tags}', (d->'tags') - 0)$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('concat-other', $$UPDATE subattr SET d = d || '{"age": 5, "new": 1}'$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('subscript-age', $$UPDATE subattr SET d['age'] = '7'$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('nested-mutators', $$UPDATE subattr SET d = jsonb_set(jsonb_set(d, '{age}', '1'), '{new}', '2')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('set-lax-age', $$UPDATE subattr SET d = jsonb_set_lax(d, '{age}', NULL, true, 'delete_key')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('insert-o-new', $$UPDATE subattr SET d = jsonb_insert(d, '{o,new}', '1')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('array-a3', $$UPDATE subattr SET a[3] = 99$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('array-a9-extend', $$UPDATE subattr SET a[9] = 99$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('set-arr-elem', $$UPDATE subattr SET d = jsonb_set(d, '{arr,1}', '5')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('param-path', $$UPDATE subattr SET d = jsonb_set(d, ARRAY['age'], $1)$$, '3');+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('coalesce-wrap', $$UPDATE subattr SET d = COALESCE(jsonb_set(d, '{age}', '1'), d)$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('flag-unchanged', $$UPDATE subattr SET d = jsonb_set(d, '{age}', '1') WHERE id % 2 = 0$$);+ subattr_probe +---------------+ +(1 row)++-- An indexed part changes, or might.+SELECT subattr_probe('set-name', $$UPDATE subattr SET d = jsonb_set(d, '{name}', '"x"')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('set-o', $$UPDATE subattr SET d = jsonb_set(d, '{o}', '{}')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('set-o-k', $$UPDATE subattr SET d = jsonb_set(d, '{o,k}', '"x"')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('set-p', $$UPDATE subattr SET d = jsonb_set(d, '{p}', '1')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('delete-name', $$UPDATE subattr SET d = d - 'name'$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('delete-keys-name', $$UPDATE subattr SET d = d - '{age,name}'::text[]$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('delete-path-name', $$UPDATE subattr SET d = d #- '{name}'$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('delete-path-o-k', $$UPDATE subattr SET d = d #- '{o,k}'$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('insert-tags-0', $$UPDATE subattr SET d = jsonb_insert(d, '{tags,0}', '"x"')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('insert-tags-1', $$UPDATE subattr SET d = jsonb_insert(d, '{tags,1}', '"x"')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('insert-tags-neg', $$UPDATE subattr SET d = jsonb_insert(d, '{tags,-1}', '"x"')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('set-tags-0', $$UPDATE subattr SET d = jsonb_set(d, '{tags,0}', '9')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('set-tags-neg3', $$UPDATE subattr SET d = jsonb_set(d, '{tags,-3}', '9')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('set-tags-sp0', $$UPDATE subattr SET d = jsonb_set(d, ARRAY['tags',' 0'], '9')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('set-tags-plus0', $$UPDATE subattr SET d = jsonb_set(d, ARRAY['tags','+0'], '9')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('set-tags-00', $$UPDATE subattr SET d = jsonb_set(d, ARRAY['tags','00'], '9')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('delete-path-tags-0', $$UPDATE subattr SET d = d #- '{tags,0}'$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('set-lax-tags-0', $$UPDATE subattr SET d = jsonb_set_lax(d, '{tags,0}', NULL, true, 'delete_key')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('concat-name', $$UPDATE subattr SET d = d || '{"name": "x"}'$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('concat-array', $$UPDATE subattr SET d = d || '[1]'$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('concat-left', $$UPDATE subattr SET d = '{"name": "x"}' || d$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('whole-replace', $$UPDATE subattr SET d = '{"name": "x"}'$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('mutator-other-col', $$UPDATE subattr SET d = jsonb_set(jsonb_build_object('name', 'z' || x), '{age}', '1')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('case-wrap', $$UPDATE subattr SET d = CASE WHEN id > 0 THEN jsonb_set(d, '{name}', '"x"') ELSE d END$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('set-lax-name', $$UPDATE subattr SET d = jsonb_set_lax(d, '{name}', NULL, true, 'delete_key')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('subscript-name', $$UPDATE subattr SET d['name'] = '"x"'$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('subscript-s', $$UPDATE subattr SET d['s'] = '"x"'$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('subscript-int', $$UPDATE subattr SET d['tags'][0] = '9'$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('set-flag', $$UPDATE subattr SET d = jsonb_set(d, '{flag}', '"on"')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('set-1-key', $$UPDATE subattr SET d = jsonb_set(d, '{1}', '"two"')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('trigger-renames', $$UPDATE subattr SET d = jsonb_set(d, '{age}', '1')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('array-a2', $$UPDATE subattr SET a[2] = 99$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('array-a-neg', $$UPDATE subattr SET a[-5] = 99$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('array-whole', $$UPDATE subattr SET a = a || 99$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('param-path-name', $$UPDATE subattr SET d = jsonb_set(d, ARRAY['name'], $1)$$, '"x"');+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('param-doc', $$UPDATE subattr SET d = $1$$, '{"name":"x"}');+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('self-ref-value', $$UPDATE subattr SET d = jsonb_set(d, '{age}', d->'name')$$);+ subattr_probe +---------------+ +(1 row)++SELECT subattr_probe('null-doc', $$UPDATE subattr SET d = jsonb_set(NULL::jsonb, '{age}', '1')$$);+ subattr_probe +---------------+ +(1 row)++SELECT name, upd, hot, amcheck FROM subattr_results ORDER BY name COLLATE "C";+ name | upd | hot | amcheck +--------------------+-----+-----+---------+ array-a-neg | 50 | 50 | ok+ array-a2 | 50 | 0 | ok+ array-a3 | 50 | 50 | ok+ array-a9-extend | 50 | 50 | ok+ array-whole | 50 | 50 | ok+ case-wrap | 50 | 0 | ok+ coalesce-wrap | 50 | 50 | ok+ concat-array | 50 | 0 | ok+ concat-left | 50 | 50 | ok+ concat-name | 50 | 0 | ok+ concat-other | 50 | 50 | ok+ delete-age | 50 | 50 | ok+ delete-arr-elem | 50 | 0 | ok+ delete-keys | 50 | 50 | ok+ delete-keys-name | 50 | 0 | ok+ delete-name | 50 | 0 | ok+ delete-path-age | 50 | 50 | ok+ delete-path-name | 50 | 0 | ok+ delete-path-o-k | 50 | 0 | ok+ delete-path-tags-0 | 50 | 0 | ok+ flag-unchanged | 25 | 25 | ok+ insert-o-new | 50 | 50 | ok+ insert-tags-0 | 50 | 0 | ok+ insert-tags-1 | 50 | 50 | ok+ insert-tags-neg | 50 | 50 | ok+ mutator-other-col | 50 | 0 | ok+ nested-mutators | 50 | 50 | ok+ null-doc | 50 | 0 | ok+ param-doc | 50 | 0 | ok+ param-path | 50 | 50 | ok+ param-path-name | 50 | 0 | ok+ self-ref-value | 50 | 50 | ok+ set-1-key | 50 | 50 | ok+ set-age | 50 | 50 | ok+ set-arr-elem | 50 | 50 | ok+ set-flag | 50 | 25 | ok+ set-lax-age | 50 | 50 | ok+ set-lax-name | 50 | 0 | ok+ set-lax-tags-0 | 50 | 0 | ok+ set-name | 50 | 0 | ok+ set-o | 50 | 0 | ok+ set-o-k | 50 | 0 | ok+ set-o-z | 50 | 50 | ok+ set-p | 50 | 0 | ok+ set-p-other | 50 | 50 | ok+ set-tags-0 | 50 | 0 | ok+ set-tags-00 | 50 | 0 | ok+ set-tags-neg3 | 50 | 0 | ok+ set-tags-plus0 | 50 | 0 | ok+ set-tags-sp0 | 50 | 0 | ok+ subscript-age | 50 | 50 | ok+ subscript-int | 50 | 0 | ok+ subscript-name | 50 | 0 | ok+ subscript-s | 50 | 0 | ok+ trigger-renames | 50 | 0 | ok+ whole-replace | 50 | 0 | ok+(56 rows)++DROP TABLE subattr_results;+DROP FUNCTION subattr_probe;+-- Shapes that need their own tables: composite fields, domains, generic+-- plans, a strict modifier that returns null, and references the analysis+-- does not follow. t/007_subattr_proof.pl checks what is concluded.+CREATE TYPE subattr_pair AS (k int, v int);+CREATE DOMAIN subattr_doc AS jsonb CHECK (VALUE ? 'name');+CREATE TABLE subattr (id int PRIMARY KEY, d subattr_doc, p subattr_pair,+ r jsonb, n jsonb) WITH (fillfactor = 50);+CREATE INDEX subattr_name ON subattr ((d->>'name'));+CREATE INDEX subattr_pk ON subattr (((p).k));+CREATE INDEX subattr_whole ON subattr ((r->>'name')) WHERE r IS NOT NULL;+CREATE INDEX subattr_row ON subattr (id) WHERE subattr IS NOT NULL;+CREATE INDEX subattr_text ON subattr ((n::text));+CREATE INDEX subattr_ep ON subattr ((jsonb_extract_path(n, 'a', 'b')));+INSERT INTO subattr SELECT g, jsonb_build_object('name', 'n' || g),+ ROW(g, g)::subattr_pair, jsonb_build_object('name', 'r' || g),+ '{"a": {"b": 1}}'+ FROM generate_series(1, 4) g;+UPDATE subattr SET d = jsonb_set(d, '{age}', '1');+UPDATE subattr SET p.v = 0;+UPDATE subattr SET p.k = (p).k + 10;+UPDATE subattr SET r = jsonb_set(r, '{age}', '1');+UPDATE subattr SET n = jsonb_set(n, '{c}', '1');+UPDATE subattr SET n = jsonb_set(n, '{a,c}', '1');+UPDATE subattr SET d = jsonb_set(d, ARRAY[NULL, 'x'], '1') WHERE false;+UPDATE subattr SET d = jsonb_set(d, '{age}', (SELECT d->'name'));+PREPARE subattr_p(text[], jsonb) AS+ UPDATE subattr SET d = jsonb_set(d, $1, $2);+SET plan_cache_mode = force_generic_plan;+EXECUTE subattr_p('{age}', '2');+EXECUTE subattr_p('{name}', '"x"');+RESET plan_cache_mode;+DEALLOCATE subattr_p;+-- jsonb_set returns null for a null new value; such rows are evaluated+UPDATE subattr SET r = jsonb_set(r, '{age}', NULL) WHERE id <= 2;+SELECT id, d, p, r FROM subattr ORDER BY id;+ id | d | p | r +----+-------------------------+--------+--------------------------+ 1 | {"age": 2, "name": "x"} | (11,0) | + 2 | {"age": 2, "name": "x"} | (12,0) | + 3 | {"age": 2, "name": "x"} | (13,0) | {"age": 1, "name": "r3"}+ 4 | {"age": 2, "name": "x"} | (14,0) | {"age": 1, "name": "r4"}+(4 rows)++SELECT bt_index_check(indexrelid, true) FROM pg_index+ WHERE indrelid = 'subattr'::regclass ORDER BY indexrelid::regclass::text;+ bt_index_check +----------------+ + + + + + + +(7 rows)++DROP TABLE subattr;+DROP DOMAIN subattr_doc;+DROP TYPE subattr_pair;diff --git a/contrib/amcheck/meson.build b/contrib/amcheck/meson.buildindex d5137ef691d..e39fa220057 100644--- a/contrib/amcheck/meson.build+++ b/contrib/amcheck/meson.build@@ -40,6 +40,7 @@ tests += {
'check_btree',
'check_gin',
'check_heap',
+ 'check_subattr',
],
},
'tap': {
@@ -50,6 +51,7 @@ tests += {
't/004_verify_nbtree_unique.pl',
't/005_pitr.pl',
't/006_verify_gin.pl',
+ 't/007_subattr_proof.pl',
],
},
}
diff --git a/contrib/amcheck/sql/check_subattr.sql b/contrib/amcheck/sql/check_subattr.sqlnew file mode 100644index 00000000000..bd98be3af55--- /dev/null+++ b/contrib/amcheck/sql/check_subattr.sql@@ -0,0 +1,162 @@+--+-- Updates of parts of jsonb, array and composite columns+--+-- The executor works out from an UPDATE's SET expressions which parts of a+-- column it can change, and skips evaluating the expression and partial+-- indexes that read only other parts. A wrong conclusion would leave an+-- index without an entry for a row, so each case checks every index against+-- the heap; assert-enabled builds also evaluate the indexes anyway and fail+-- if the conclusion was wrong. HOT counts show which updates avoided new+-- index entries, whether or not anything was skipped.+--+SET client_min_messages = warning;+CREATE TABLE subattr_results (name text, hot bigint, upd bigint, amcheck text);+CREATE FUNCTION subattr_probe(name text, upd text, arg jsonb DEFAULT NULL)+RETURNS void LANGUAGE plpgsql AS $f$+DECLARE h0 bigint; u0 bigint; ac text := 'ok'; i regclass;+BEGIN+ CREATE TABLE subattr (id int PRIMARY KEY, d jsonb, a int[], x int)+ WITH (fillfactor = 10);+ CREATE INDEX subattr_name ON subattr ((d->>'name'));+ CREATE INDEX subattr_nested ON subattr ((d->'o'->>'k'));+ CREATE INDEX subattr_path ON subattr ((d #>> '{p,q}'));+ CREATE INDEX subattr_arr0 ON subattr ((d->'tags'->>0));+ CREATE INDEX subattr_sub ON subattr ((d['s']));+ CREATE INDEX subattr_part ON subattr (id) WHERE (d->>'flag') = 'on';+ CREATE INDEX subattr_a2 ON subattr ((a[2]));+ INSERT INTO subattr SELECT g,+ jsonb_build_object('name', 'n' || g, 'age', g, 'tags', '[1,2,3]'::jsonb,+ 'o', jsonb_build_object('k', 'k' || g, 'z', 1),+ 'p', jsonb_build_object('q', 'q' || g), 's', 's' || g,+ 'flag', CASE WHEN g % 2 = 0 THEN 'on' ELSE 'off' END,+ '1', 'one', 'arr', '[10,20,30]'::jsonb),+ ARRAY[g, g + 1, g + 2], 0+ FROM generate_series(1, 50) g;+ IF name LIKE 'trigger%' THEN+ CREATE FUNCTION subattr_trg() RETURNS trigger LANGUAGE plpgsql AS+ $t$BEGIN NEW.d := jsonb_set(NEW.d, '{name}', '"renamed"'); RETURN NEW; END$t$;+ CREATE TRIGGER subattr_trg BEFORE UPDATE ON subattr+ FOR EACH ROW EXECUTE FUNCTION subattr_trg();+ END IF;+ h0 := pg_stat_get_xact_tuples_hot_updated('subattr'::regclass);+ u0 := pg_stat_get_xact_tuples_updated('subattr'::regclass);+ IF arg IS NULL THEN EXECUTE upd; ELSE EXECUTE upd USING arg; END IF;+ FOR i IN SELECT indexrelid::regclass FROM pg_index+ WHERE indrelid = 'subattr'::regclass ORDER BY 1 LOOP+ BEGIN PERFORM bt_index_check(i, true);+ EXCEPTION WHEN OTHERS THEN+ ac := CASE WHEN ac = 'ok' THEN '' ELSE ac || '; ' END ||+ i::text || ': ' || SQLERRM;+ END;+ END LOOP;+ INSERT INTO subattr_results VALUES (name,+ pg_stat_get_xact_tuples_hot_updated('subattr'::regclass) - h0,+ pg_stat_get_xact_tuples_updated('subattr'::regclass) - u0, ac);+ DROP TABLE subattr;+ DROP FUNCTION IF EXISTS subattr_trg();+END $f$;++-- Nothing indexed changes.+SELECT subattr_probe('set-age', $$UPDATE subattr SET d = jsonb_set(d, '{age}', '1')$$);+SELECT subattr_probe('set-o-z', $$UPDATE subattr SET d = jsonb_set(d, '{o,z}', '2')$$);+SELECT subattr_probe('set-p-other', $$UPDATE subattr SET d = jsonb_set(d, '{p,r}', '2')$$);+SELECT subattr_probe('delete-age', $$UPDATE subattr SET d = d - 'age'$$);+SELECT subattr_probe('delete-keys', $$UPDATE subattr SET d = d - '{age,zz}'::text[]$$);+SELECT subattr_probe('delete-path-age', $$UPDATE subattr SET d = d #- '{age}'$$);+SELECT subattr_probe('delete-arr-elem', $$UPDATE subattr SET d = jsonb_set(d, '{tags}', (d->'tags') - 0)$$);+SELECT subattr_probe('concat-other', $$UPDATE subattr SET d = d || '{"age": 5, "new": 1}'$$);+SELECT subattr_probe('subscript-age', $$UPDATE subattr SET d['age'] = '7'$$);+SELECT subattr_probe('nested-mutators', $$UPDATE subattr SET d = jsonb_set(jsonb_set(d, '{age}', '1'), '{new}', '2')$$);+SELECT subattr_probe('set-lax-age', $$UPDATE subattr SET d = jsonb_set_lax(d, '{age}', NULL, true, 'delete_key')$$);+SELECT subattr_probe('insert-o-new', $$UPDATE subattr SET d = jsonb_insert(d, '{o,new}', '1')$$);+SELECT subattr_probe('array-a3', $$UPDATE subattr SET a[3] = 99$$);+SELECT subattr_probe('array-a9-extend', $$UPDATE subattr SET a[9] = 99$$);+SELECT subattr_probe('set-arr-elem', $$UPDATE subattr SET d = jsonb_set(d, '{arr,1}', '5')$$);+SELECT subattr_probe('param-path', $$UPDATE subattr SET d = jsonb_set(d, ARRAY['age'], $1)$$, '3');+SELECT subattr_probe('coalesce-wrap', $$UPDATE subattr SET d = COALESCE(jsonb_set(d, '{age}', '1'), d)$$);+SELECT subattr_probe('flag-unchanged', $$UPDATE subattr SET d = jsonb_set(d, '{age}', '1') WHERE id % 2 = 0$$);++-- An indexed part changes, or might.+SELECT subattr_probe('set-name', $$UPDATE subattr SET d = jsonb_set(d, '{name}', '"x"')$$);+SELECT subattr_probe('set-o', $$UPDATE subattr SET d = jsonb_set(d, '{o}', '{}')$$);+SELECT subattr_probe('set-o-k', $$UPDATE subattr SET d = jsonb_set(d, '{o,k}', '"x"')$$);+SELECT subattr_probe('set-p', $$UPDATE subattr SET d = jsonb_set(d, '{p}', '1')$$);+SELECT subattr_probe('delete-name', $$UPDATE subattr SET d = d - 'name'$$);+SELECT subattr_probe('delete-keys-name', $$UPDATE subattr SET d = d - '{age,name}'::text[]$$);+SELECT subattr_probe('delete-path-name', $$UPDATE subattr SET d = d #- '{name}'$$);+SELECT subattr_probe('delete-path-o-k', $$UPDATE subattr SET d = d #- '{o,k}'$$);+SELECT subattr_probe('insert-tags-0', $$UPDATE subattr SET d = jsonb_insert(d, '{tags,0}', '"x"')$$);+SELECT subattr_probe('insert-tags-1', $$UPDATE subattr SET d = jsonb_insert(d, '{tags,1}', '"x"')$$);+SELECT subattr_probe('insert-tags-neg', $$UPDATE subattr SET d = jsonb_insert(d, '{tags,-1}', '"x"')$$);+SELECT subattr_probe('set-tags-0', $$UPDATE subattr SET d = jsonb_set(d, '{tags,0}', '9')$$);+SELECT subattr_probe('set-tags-neg3', $$UPDATE subattr SET d = jsonb_set(d, '{tags,-3}', '9')$$);+SELECT subattr_probe('set-tags-sp0', $$UPDATE subattr SET d = jsonb_set(d, ARRAY['tags',' 0'], '9')$$);+SELECT subattr_probe('set-tags-plus0', $$UPDATE subattr SET d = jsonb_set(d, ARRAY['tags','+0'], '9')$$);+SELECT subattr_probe('set-tags-00', $$UPDATE subattr SET d = jsonb_set(d, ARRAY['tags','00'], '9')$$);+SELECT subattr_probe('delete-path-tags-0', $$UPDATE subattr SET d = d #- '{tags,0}'$$);+SELECT subattr_probe('set-lax-tags-0', $$UPDATE subattr SET d = jsonb_set_lax(d, '{tags,0}', NULL, true, 'delete_key')$$);+SELECT subattr_probe('concat-name', $$UPDATE subattr SET d = d || '{"name": "x"}'$$);+SELECT subattr_probe('concat-array', $$UPDATE subattr SET d = d || '[1]'$$);+SELECT subattr_probe('concat-left', $$UPDATE subattr SET d = '{"name": "x"}' || d$$);+SELECT subattr_probe('whole-replace', $$UPDATE subattr SET d = '{"name": "x"}'$$);+SELECT subattr_probe('mutator-other-col', $$UPDATE subattr SET d = jsonb_set(jsonb_build_object('name', 'z' || x), '{age}', '1')$$);+SELECT subattr_probe('case-wrap', $$UPDATE subattr SET d = CASE WHEN id > 0 THEN jsonb_set(d, '{name}', '"x"') ELSE d END$$);+SELECT subattr_probe('set-lax-name', $$UPDATE subattr SET d = jsonb_set_lax(d, '{name}', NULL, true, 'delete_key')$$);+SELECT subattr_probe('subscript-name', $$UPDATE subattr SET d['name'] = '"x"'$$);+SELECT subattr_probe('subscript-s', $$UPDATE subattr SET d['s'] = '"x"'$$);+SELECT subattr_probe('subscript-int', $$UPDATE subattr SET d['tags'][0] = '9'$$);+SELECT subattr_probe('set-flag', $$UPDATE subattr SET d = jsonb_set(d, '{flag}', '"on"')$$);+SELECT subattr_probe('set-1-key', $$UPDATE subattr SET d = jsonb_set(d, '{1}', '"two"')$$);+SELECT subattr_probe('trigger-renames', $$UPDATE subattr SET d = jsonb_set(d, '{age}', '1')$$);+SELECT subattr_probe('array-a2', $$UPDATE subattr SET a[2] = 99$$);+SELECT subattr_probe('array-a-neg', $$UPDATE subattr SET a[-5] = 99$$);+SELECT subattr_probe('array-whole', $$UPDATE subattr SET a = a || 99$$);+SELECT subattr_probe('param-path-name', $$UPDATE subattr SET d = jsonb_set(d, ARRAY['name'], $1)$$, '"x"');+SELECT subattr_probe('param-doc', $$UPDATE subattr SET d = $1$$, '{"name":"x"}');+SELECT subattr_probe('self-ref-value', $$UPDATE subattr SET d = jsonb_set(d, '{age}', d->'name')$$);+SELECT subattr_probe('null-doc', $$UPDATE subattr SET d = jsonb_set(NULL::jsonb, '{age}', '1')$$);++SELECT name, upd, hot, amcheck FROM subattr_results ORDER BY name COLLATE "C";+DROP TABLE subattr_results;+DROP FUNCTION subattr_probe;++-- Shapes that need their own tables: composite fields, domains, generic+-- plans, a strict modifier that returns null, and references the analysis+-- does not follow. t/007_subattr_proof.pl checks what is concluded.+CREATE TYPE subattr_pair AS (k int, v int);+CREATE DOMAIN subattr_doc AS jsonb CHECK (VALUE ? 'name');+CREATE TABLE subattr (id int PRIMARY KEY, d subattr_doc, p subattr_pair,+ r jsonb, n jsonb) WITH (fillfactor = 50);+CREATE INDEX subattr_name ON subattr ((d->>'name'));+CREATE INDEX subattr_pk ON subattr (((p).k));+CREATE INDEX subattr_whole ON subattr ((r->>'name')) WHERE r IS NOT NULL;+CREATE INDEX subattr_row ON subattr (id) WHERE subattr IS NOT NULL;+CREATE INDEX subattr_text ON subattr ((n::text));+CREATE INDEX subattr_ep ON subattr ((jsonb_extract_path(n, 'a', 'b')));+INSERT INTO subattr SELECT g, jsonb_build_object('name', 'n' || g),+ ROW(g, g)::subattr_pair, jsonb_build_object('name', 'r' || g),+ '{"a": {"b": 1}}'+ FROM generate_series(1, 4) g;+UPDATE subattr SET d = jsonb_set(d, '{age}', '1');+UPDATE subattr SET p.v = 0;+UPDATE subattr SET p.k = (p).k + 10;+UPDATE subattr SET r = jsonb_set(r, '{age}', '1');+UPDATE subattr SET n = jsonb_set(n, '{c}', '1');+UPDATE subattr SET n = jsonb_set(n, '{a,c}', '1');+UPDATE subattr SET d = jsonb_set(d, ARRAY[NULL, 'x'], '1') WHERE false;+UPDATE subattr SET d = jsonb_set(d, '{age}', (SELECT d->'name'));+PREPARE subattr_p(text[], jsonb) AS+ UPDATE subattr SET d = jsonb_set(d, $1, $2);+SET plan_cache_mode = force_generic_plan;+EXECUTE subattr_p('{age}', '2');+EXECUTE subattr_p('{name}', '"x"');+RESET plan_cache_mode;+DEALLOCATE subattr_p;+-- jsonb_set returns null for a null new value; such rows are evaluated+UPDATE subattr SET r = jsonb_set(r, '{age}', NULL) WHERE id <= 2;+SELECT id, d, p, r FROM subattr ORDER BY id;+SELECT bt_index_check(indexrelid, true) FROM pg_index+ WHERE indrelid = 'subattr'::regclass ORDER BY indexrelid::regclass::text;+DROP TABLE subattr;+DROP DOMAIN subattr_doc;+DROP TYPE subattr_pair;diff --git a/contrib/amcheck/t/007_subattr_proof.pl b/contrib/amcheck/t/007_subattr_proof.plnew file mode 100644index 00000000000..44c41abe446--- /dev/null+++ b/contrib/amcheck/t/007_subattr_proof.pl@@ -0,0 +1,123 @@++# Copyright (c) 2026, PostgreSQL Global Development Group++# What an UPDATE's SET clause proves about the indexes it cannot change.+# The executor logs each conclusion at DEBUG1; this reads the server log,+# so that other messages cannot disturb the test. check_subattr checks+# that the conclusions are right.++use strict;+use warnings FATAL => 'all';++use PostgreSQL::Test::Cluster;+use PostgreSQL::Test::Utils;++use Test::More;++my $node = PostgreSQL::Test::Cluster->new('subattr');+$node->init;+$node->append_conf('postgresql.conf', 'autovacuum = off');+$node->start;++# Run one statement at log_min_messages = debug1, and return the+# conclusions it logged, as sorted "column/index" pairs.+sub concluded+{+ my ($sql) = @_;+ my $offset = -s $node->logfile;+ my @found;++ $node->safe_psql('postgres', "SET log_min_messages = debug1; $sql");+ my $log = slurp_file($node->logfile, $offset);+ while ($log =~ /UPDATE of column "(\w+)" cannot change index "(\w+)"/g)+ {+ push @found, "$1/$2";+ }+ return join(' ', sort @found);+}++$node->safe_psql(+ 'postgres', q{+ CREATE TABLE subattr (id int PRIMARY KEY, d jsonb, a int[]);+ CREATE INDEX subattr_name ON subattr ((d->>'name'));+ CREATE INDEX subattr_tags ON subattr ((d->'tags'->>0));+ CREATE INDEX subattr_a2 ON subattr ((a[2]));+ INSERT INTO subattr VALUES (1, '{"name": "a", "tags": [1, 2]}', '{1,2,3}');+});++# nothing is concluded for an UPDATE of a single row+is(concluded(q{UPDATE subattr SET d = jsonb_set(d, '{age}', '1')}),+ '', 'single row: nothing concluded');++$node->safe_psql('postgres',+ q{INSERT INTO subattr VALUES (2, '{"name": "b", "tags": [1, 2]}', '{1,2,3}')}+);++is( concluded(q{UPDATE subattr SET d = jsonb_set(d, '{age}', '1')}),+ 'd/subattr_name d/subattr_tags',+ 'an unindexed key changes neither index');+is( concluded(q{UPDATE subattr SET d = jsonb_set(d, '{tags,1}', '9')}),+ 'd/subattr_name', 'another array element: the name index only');+is( concluded(q{UPDATE subattr SET d = d - 'name', a[1] = 0}),+ 'a/subattr_a2 d/subattr_tags',+ 'deleting a key, and another array element');+is( concluded(q{UPDATE subattr SET d = d || '{"age": 2}'}),+ 'd/subattr_name d/subattr_tags',+ 'concatenating an object with other keys');++$node->safe_psql(+ 'postgres', q{+ DROP TABLE subattr;+ CREATE TYPE subattr_pair AS (k int, v int);+ CREATE DOMAIN subattr_doc AS jsonb CHECK (VALUE ? 'name');+ CREATE TABLE subattr (id int PRIMARY KEY, d subattr_doc, p subattr_pair,+ r jsonb, n jsonb) WITH (fillfactor = 50);+ CREATE INDEX subattr_name ON subattr ((d->>'name'));+ CREATE INDEX subattr_pk ON subattr (((p).k));+ CREATE INDEX subattr_whole ON subattr ((r->>'name')) WHERE r IS NOT NULL;+ CREATE INDEX subattr_row ON subattr (id) WHERE subattr IS NOT NULL;+ CREATE INDEX subattr_text ON subattr ((n::text));+ CREATE INDEX subattr_ep ON subattr ((jsonb_extract_path(n, 'a', 'b')));+ INSERT INTO subattr SELECT g, jsonb_build_object('name', 'n' || g),+ ROW(g, g)::subattr_pair, jsonb_build_object('name', 'r' || g),+ '{"a": {"b": 1}}'+ FROM generate_series(1, 4) g;+});++is(concluded(q{UPDATE subattr SET d = jsonb_set(d, '{age}', '1')}),+ 'd/subattr_name', 'a domain over jsonb');+is(concluded(q{UPDATE subattr SET p.v = 0}),+ 'p/subattr_pk', 'another field of a composite');+is(concluded(q{UPDATE subattr SET p.k = (p).k + 10}),+ '', 'the indexed field of a composite');+is(concluded(q{UPDATE subattr SET r = jsonb_set(r, '{age}', '1')}),+ '', 'a column a predicate tests as a whole');+is(concluded(q{UPDATE subattr SET n = jsonb_set(n, '{c}', '1')}),+ 'n/subattr_ep', 'jsonb_extract_path, another key');+is(concluded(q{UPDATE subattr SET n = jsonb_set(n, '{a,c}', '1')}),+ 'n/subattr_ep', 'jsonb_extract_path, a sibling key');+is( concluded(+ q{UPDATE subattr SET d = jsonb_set(d, ARRAY[NULL, 'x'], '1') WHERE false}+ ),+ '',+ 'a null path element');+is( concluded(+ q{UPDATE subattr SET d = jsonb_set(d, '{age}', (SELECT d->'name'))}),+ 'd/subattr_name', 'a sublink in the new value');+is( concluded(+ q{SET plan_cache_mode = force_generic_plan;+ PREPARE subattr_p(text[], jsonb) AS+ UPDATE subattr SET d = jsonb_set(d, $1, $2);+ EXECUTE subattr_p('{age}', '2');}),+ 'd/subattr_name',+ 'a generic plan, a parameter path');+is( concluded(+ q{SET plan_cache_mode = force_generic_plan;+ PREPARE subattr_p(text[], jsonb) AS+ UPDATE subattr SET d = jsonb_set(d, $1, $2);+ EXECUTE subattr_p('{name}', '"x"');}),+ '',+ 'a generic plan, a parameter path to the indexed key');++$node->stop;+done_testing();diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgmlindex 4d65523e7b9..4523d073a19 100644--- a/doc/src/sgml/storage.sgml+++ b/doc/src/sgml/storage.sgml@@ -1116,7 +1116,11 @@ data. Empty in ordinary tables.</entry>
expressions nor whether the row satisfies its predicate. The values are
compared as stored, so equal values with different representations, such
as <literal>1.0</literal> and <literal>1.00</literal> of type
- <type>numeric</type>, count as different.+ <type>numeric</type>, count as different. When the update's+ <literal>SET</literal> clause changes only parts of the column that the+ indexes do not read, for example through <function>jsonb_set</function>+ or array subscripting, this is known without evaluating the indexes'+ expressions.
</para>
</listitem>
<listitem>
diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgmlindex bc04d9dfc13..366bf9abd1c 100644--- a/doc/src/sgml/xfunc.sgml+++ b/doc/src/sgml/xfunc.sgml@@ -4304,4 +4304,19 @@ supportfn(internal) returns internal
To create such conditions, the support function must implement
the <literal>SupportRequestIndexCondition</literal> request type.
</para>
++ <para>+ For functions that read part of a structured value, such as a member of a+ document, or that return a copy of a structured value with some parts+ changed, the executor can use a description of which parts are read or+ changed. When an <command>UPDATE</command> changes a column only through+ such functions, and every expression index and partial-index predicate+ that uses the column reads only other parts of it, the executor knows+ without evaluating them that the update leaves those indexes unaffected+ (see <xref linkend="storage-hot"/>). To describe a call, the support+ function must implement the <literal>SupportRequestSubattr</literal>+ request type. Its answer must be exact for every value the call could be+ given: an answer that omits a part the call reads or changes can cause+ index entries to be missing.+ </para>
</sect1>
diff --git a/src/backend/executor/Makefile b/src/backend/executor/Makefileindex 11118d0ce02..77f0448ce79 100644--- a/src/backend/executor/Makefile+++ b/src/backend/executor/Makefile@@ -28,6 +28,7 @@ OBJS = \
execReplication.o \
execSRF.o \
execScan.o \
+ execSubattr.o \
execTuples.o \
execUtils.o \
functions.o \
diff --git a/src/backend/executor/execSubattr.c b/src/backend/executor/execSubattr.cnew file mode 100644index 00000000000..35a4986bc88--- /dev/null+++ b/src/backend/executor/execSubattr.c@@ -0,0 +1,630 @@+/*-------------------------------------------------------------------------+ *+ * execSubattr.c+ * Prove that an UPDATE cannot change what an expression or partial index+ * stores, from the shape of the SET expressions and the index+ * definitions alone.+ *+ * ExecUpdateModifiedIdxAttrs() finds the indexed columns an UPDATE changed+ * by comparing the old and new rows. For a column used only through index+ * expressions or partial-index predicates it then evaluates those+ * expressions and predicates on both rows. That is always right, but for a+ * large structured value (a jsonb document, an array, an extension's+ * document type) it reads the value twice per index, and often the SET+ * expression already tells us that the parts an index reads cannot have+ * changed:+ *+ * CREATE INDEX ON t ((doc->>'name'));+ * UPDATE t SET doc = jsonb_set(doc, '{visits}', ...);+ *+ * This module works that out once per statement, from the plan tree, so+ * that nothing at run time (the expression interpreter, JIT-compiled code,+ * the functions themselves) has to cooperate. Functions+ * take part through their support function by answering a+ * SupportRequestSubattr: which parts of which argument a call reads, or+ * which parts of its source argument it may change. Array and jsonb+ * subscripting are understood directly. Anything this module does not+ * understand makes it give up for that index, and the executor then+ * evaluates it as before.+ *+ * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group+ *+ * IDENTIFICATION+ * src/backend/executor/execSubattr.c+ *+ *-------------------------------------------------------------------------+ */+#include "postgres.h"++#include <ctype.h>++#include "catalog/pg_type.h"+#include "executor/execSubattr.h"+#include "miscadmin.h"+#include "nodes/makefuncs.h"+#include "nodes/nodeFuncs.h"+#include "nodes/params.h"+#include "nodes/supportnodes.h"+#include "utils/builtins.h"+#include "utils/datum.h"+#include "utils/fmgroids.h"+#include "utils/lsyscache.h"+#include "utils/rel.h"++/* The column the walkers look for, and what they have found. */+typedef struct SubattrWalk+{+ int varno;+ AttrNumber attnum;+ bool known; /* false: could not tell */+ List *paths; /* List of List of SubattrPathElem */+} SubattrWalk;++static bool read_walker(Node *node, SubattrWalk *walk);++/*+ * Only RelabelType is looked through: any other wrapper keeps an expression+ * from matching the target Var, which just loses the proof.+ */+static Node *+strip_relabel(Node *node)+{+ while (node && IsA(node, RelabelType))+ node = (Node *) ((RelabelType *) node)->arg;+ return node;+}++static bool+is_target_var(Node *node, SubattrWalk *walk)+{+ node = strip_relabel(node);+ return node && IsA(node, Var) &&+ ((Var *) node)->varno == walk->varno &&+ ((Var *) node)->varattno == walk->attnum &&+ ((Var *) node)->varlevelsup == 0;+}++static List *+make_path_elem(SubattrPathElemKind kind, const char *key, int position)+{+ SubattrPathElem *e = palloc0_object(SubattrPathElem);++ e->kind = kind;+ e->key = key ? pstrdup(key) : NULL;+ e->position = position;+ return list_make1(e);+}++/* The function and arguments of a FuncExpr or OpExpr, else InvalidOid. */+static Oid+call_of(Node *node, List **args)+{+ if (IsA(node, FuncExpr))+ {+ *args = ((FuncExpr *) node)->args;+ return ((FuncExpr *) node)->funcid;+ }+ if (IsA(node, OpExpr))+ {+ OpExpr *op = (OpExpr *) node;++ *args = op->args;+ return OidIsValid(op->opfuncid) ? op->opfuncid : get_opcode(op->opno);+ }+ return InvalidOid;+}++/*+ * Ask the support function of funcid about a call with these arguments.+ * Returns false if there is none or it declines.+ */+static bool+ask_support(Oid funcid, List *args, SubattrRequestKind kind,+ SupportRequestSubattr *req)+{+ RegProcedure prosupport = get_func_support(funcid);++ if (!OidIsValid(prosupport))+ return false;++ memset(req, 0, sizeof(*req));+ req->type = T_SupportRequestSubattr;+ req->kind = kind;+ req->funcid = funcid;+ req->args = args;+ req->source = -1;++ if (DatumGetPointer(OidFunctionCall1(prosupport, PointerGetDatum(req))) !=+ (Pointer) req)+ return false;+ if (req->source < 0 || req->source >= list_length(args))+ return false;++ /* an answer we can't read counts as no answer */+ foreach_ptr(List, path, req->paths)+ {+ foreach_ptr(SubattrPathElem, e, path)+ {+ if (e == NULL ||+ (e->kind != SUBATTR_ELEM_KEY &&+ e->kind != SUBATTR_ELEM_POSITION &&+ e->kind != SUBATTR_ELEM_ANY_POSITION) ||+ (e->kind == SUBATTR_ELEM_KEY && e->key == NULL))+ return false;+ }+ }+ return true;+}++/*+ * The path named by the subscripts of an array or jsonb subscripting+ * expression, if they are all constants.+ *+ * jsonb subscripting turns an integer subscript into its text form and then+ * reads each subscript as an object key or, in an array, as a position, so+ * jsonb subscripts are always keys here; see paths_may_overlap() for how a+ * key that reads as an integer is treated.+ */+static bool+subscript_path(SubscriptingRef *sbsref, List **path)+{+ ListCell *lc;+ List *result = NIL;+ RegProcedure handler = get_typsubscript(sbsref->refcontainertype, NULL);++ if (handler != F_ARRAY_SUBSCRIPT_HANDLER &&+ handler != F_JSONB_SUBSCRIPT_HANDLER)+ return false;+ if (sbsref->reflowerindexpr != NIL)+ return false; /* a slice */++ foreach(lc, sbsref->refupperindexpr)+ {+ Node *sub = strip_relabel(lfirst(lc));+ Const *c = (Const *) sub;++ if (!sub || !IsA(sub, Const) || c->constisnull)+ return false;+ if (handler == F_ARRAY_SUBSCRIPT_HANDLER && c->consttype == INT4OID)+ result = list_concat(result,+ make_path_elem(SUBATTR_ELEM_POSITION, NULL,+ DatumGetInt32(c->constvalue)));+ else if (handler == F_JSONB_SUBSCRIPT_HANDLER &&+ c->consttype == INT4OID)+ result = list_concat(result,+ make_path_elem(SUBATTR_ELEM_KEY,+ psprintf("%d", DatumGetInt32(c->constvalue)),+ 0));+ else if (handler == F_JSONB_SUBSCRIPT_HANDLER &&+ c->consttype == TEXTOID)+ result = list_concat(result,+ make_path_elem(SUBATTR_ELEM_KEY,+ TextDatumGetCString(c->constvalue),+ 0));+ else+ return false;+ }+ *path = result;+ return true;+}++/*+ * If node is a chain of part-reading accessors ending at the target column,+ * return the paths it reads in *paths. *subvalue says whether the result of+ * the chain is itself the part at the (single) path, so that an enclosing+ * accessor can extend the path.+ */+static bool+accessor_path(Node *node, SubattrWalk *walk, List **paths, bool *subvalue)+{+ List *args;+ Oid funcid;++ check_stack_depth();++ node = strip_relabel(node);+ if (is_target_var(node, walk))+ {+ *paths = list_make1(NIL);+ *subvalue = true;+ return true;+ }++ if (IsA(node, SubscriptingRef) &&+ ((SubscriptingRef *) node)->refassgnexpr == NULL)+ {+ SubscriptingRef *sbsref = (SubscriptingRef *) node;+ List *inner;+ List *mine;+ bool innersub;++ if (!accessor_path((Node *) sbsref->refexpr, walk, &inner, &innersub) ||+ !innersub || !subscript_path(sbsref, &mine))+ return false;+ *paths = list_make1(list_concat(linitial(inner), mine));+ *subvalue = true; /* fetching an element yields the element */+ return true;+ }++ if (IsA(node, FieldSelect))+ {+ FieldSelect *fselect = (FieldSelect *) node;+ List *inner;+ bool innersub;++ if (!accessor_path((Node *) fselect->arg, walk, &inner, &innersub) ||+ !innersub)+ return false;+ *paths = list_make1(list_concat(linitial(inner),+ make_path_elem(SUBATTR_ELEM_POSITION,+ NULL,+ fselect->fieldnum)));+ *subvalue = true;+ return true;+ }++ funcid = call_of(node, &args);+ if (OidIsValid(funcid))+ {+ SupportRequestSubattr req;+ List *inner;+ bool innersub;+ ListCell *lc;++ /*+ * A call that reads several paths cannot claim to return the part at+ * a path, so nothing can extend them. An extended path is only+ * understood if the inner chain names one part.+ */+ if (!ask_support(funcid, args, SUBATTR_READ, &req) ||+ req.paths == NIL ||+ (req.subvalue && list_length(req.paths) != 1) ||+ !accessor_path(list_nth(args, req.source), walk, &inner, &innersub) ||+ !innersub)+ return false;++ /* The other arguments must not read the column at all. */+ for (int i = 0; i < list_length(args); i++)+ {+ SubattrWalk other = {walk->varno, walk->attnum, true, NIL};++ if (i == req.source)+ continue;+ read_walker(list_nth(args, i), &other);+ if (other.paths != NIL || !other.known)+ return false;+ }++ *paths = NIL;+ foreach(lc, req.paths)+ {+ List *path = list_concat_copy(linitial(inner), lfirst(lc));++ *paths = lappend(*paths, path);+ }+ *subvalue = req.subvalue;+ return true;+ }++ return false;+}++/*+ * Collect, into walk->paths, every part of the target column that "node"+ * reads. A reference to the column that is not the start of an accessor+ * chain we understand reads the whole value.+ */+static bool+read_walker(Node *node, SubattrWalk *walk)+{+ List *paths;+ bool subvalue;++ if (node == NULL)+ return false;++ if (accessor_path(node, walk, &paths, &subvalue))+ {+ walk->paths = list_concat(walk->paths, paths);+ return false;+ }++ if (IsA(node, Var))+ {+ Var *var = (Var *) node;++ /* a whole-row reference reads every column */+ if (var->varno == walk->varno && var->varattno == InvalidAttrNumber &&+ var->varlevelsup == 0)+ walk->known = false;+ return false;+ }++ if (IsA(node, SubLink) || IsA(node, SubPlan))+ {+ walk->known = false; /* not attempting to see inside */+ return true;+ }++ return expression_tree_walker(node, read_walker, walk);+}++/*+ * Record the parts of the target column that the SET expression "expr" may+ * change. It must be the column itself, passed through modifications we+ * understand; anything else is unknown.+ */+static void+collect_modifies(Node *expr, SubattrWalk *walk)+{+ Oid funcid;+ List *args;++ check_stack_depth();+ expr = strip_relabel(expr);+ if (!walk->known)+ return;+ if (expr == NULL)+ {+ walk->known = false;+ return;+ }++ if (is_target_var(expr, walk))+ return; /* this part is unchanged */++ /* a domain check passes the value through or fails */+ if (IsA(expr, CoerceToDomain))+ {+ collect_modifies((Node *) ((CoerceToDomain *) expr)->arg, walk);+ return;+ }++ if (IsA(expr, SubscriptingRef) &&+ ((SubscriptingRef *) expr)->refassgnexpr != NULL)+ {+ SubscriptingRef *sbsref = (SubscriptingRef *) expr;+ List *path;++ /*+ * col[subscripts] := value. Assigning beyond the bounds of an array+ * fills the gap with nulls, which read the same as the missing+ * elements did; jsonb assignment may create containers on the way to+ * the path, but those are at a prefix of it, which overlaps.+ */+ if (!subscript_path(sbsref, &path))+ {+ walk->known = false;+ return;+ }+ walk->paths = lappend(walk->paths, path);+ collect_modifies((Node *) sbsref->refexpr, walk);+ return;+ }++ if (IsA(expr, FieldStore))+ {+ FieldStore *fstore = (FieldStore *) expr;+ ListCell *lc;++ /* col.field := value, for one or more fields */+ foreach(lc, fstore->fieldnums)+ walk->paths = lappend(walk->paths,+ make_path_elem(SUBATTR_ELEM_POSITION, NULL,+ lfirst_int(lc)));+ collect_modifies((Node *) fstore->arg, walk);+ return;+ }++ funcid = call_of(expr, &args);+ if (OidIsValid(funcid))+ {+ SupportRequestSubattr req;++ if (ask_support(funcid, args, SUBATTR_MODIFY, &req))+ {+ walk->paths = list_concat(walk->paths, req.paths);+ collect_modifies(list_nth(args, req.source), walk);+ return;+ }+ }++ /* COALESCE(f(col, ...), col): each branch is the column, modified */+ if (IsA(expr, CoalesceExpr))+ {+ ListCell *lc;++ foreach(lc, ((CoalesceExpr *) expr)->args)+ collect_modifies(lfirst(lc), walk);+ return;+ }++ walk->known = false;+}++/*+ * Could the key be read as an array position? jsonb reads a path element+ * as a position with strtoint(), which allows leading white space and a+ * sign. Trailing white space is allowed too: answering yes too often only+ * makes two paths overlap that need not. White space is what strtol()+ * skips, isspace() in the current locale; digits are only '0' to '9'+ * there, so they are tested by hand.+ */+static bool+key_may_be_position(const char *key)+{+ const char *p = key;++ while (isspace((unsigned char) *p))+ p++;+ if (*p == '-' || *p == '+')+ p++;+ if (*p < '0' || *p > '9')+ return false;+ while (*p >= '0' && *p <= '9')+ p++;+ while (isspace((unsigned char) *p))+ p++;+ return *p == '\0';+}++/*+ * Can paths a and b name overlapping parts? See SupportRequestSubattr.+ *+ * Two keys that both read as integers are taken to overlap even if they+ * differ: "01" and "1" select the same array element, and inserting into or+ * deleting from an array, or appending past its end, moves or creates+ * elements at positions other than the one named.+ */+static bool+paths_may_overlap(List *a, List *b)+{+ ListCell *la,+ *lb;++ forboth(la, a, lb, b)+ {+ SubattrPathElem *ea = lfirst(la);+ SubattrPathElem *eb = lfirst(lb);++ if (ea->kind == SUBATTR_ELEM_KEY && eb->kind == SUBATTR_ELEM_KEY)+ {+ if (strcmp(ea->key, eb->key) != 0 &&+ !(key_may_be_position(ea->key) &&+ key_may_be_position(eb->key)))+ return false;+ }+ else if (ea->kind == SUBATTR_ELEM_KEY || eb->kind == SUBATTR_ELEM_KEY)+ {+ /* a key against a position or any position */+ SubattrPathElem *key = ea->kind == SUBATTR_ELEM_KEY ? ea : eb;++ if (!key_may_be_position(key->key))+ return false;+ }+ else if (ea->kind == SUBATTR_ELEM_POSITION &&+ eb->kind == SUBATTR_ELEM_POSITION)+ {+ if (ea->position != eb->position)+ return false;+ }+ else+ {+ /* any position against a position or any position: overlap */+ Assert(ea->kind == SUBATTR_ELEM_ANY_POSITION ||+ eb->kind == SUBATTR_ELEM_ANY_POSITION);+ }+ }+ return true;+}++/*+ * Replace external Params whose values are fixed for the execution by+ * Consts, as eval_const_expressions() does, so that support functions see+ * the values of the arguments of a generic plan. Nothing is evaluated.+ */+static Node *+substitute_params(Node *node, ParamListInfo params)+{+ if (node == NULL)+ return NULL;+ if (IsA(node, Param) && ((Param *) node)->paramkind == PARAM_EXTERN)+ {+ Param *param = (Param *) node;+ ParamExternData prmdata;+ ParamExternData *prm;++ if (param->paramid <= 0 || param->paramid > params->numParams)+ return node;+ if (params->paramFetch != NULL)+ prm = params->paramFetch(params, param->paramid, true, &prmdata);+ else+ prm = ¶ms->params[param->paramid - 1];++ if (prm != NULL && prm->ptype == param->paramtype &&+ OidIsValid(prm->ptype) &&+ (prm->pflags & PARAM_FLAG_CONST))+ {+ int16 typlen;+ bool typbyval;++ get_typlenbyval(param->paramtype, &typlen, &typbyval);+ return (Node *) makeConst(param->paramtype, param->paramtypmod,+ param->paramcollid, (int) typlen,+ prm->isnull ? (Datum) 0 :+ datumCopy(prm->value, typbyval, typlen),+ prm->isnull, typbyval);+ }+ return node;+ }+ return expression_tree_mutator(node, substitute_params, params);+}++/*+ * ExecSubattrProveUntouched+ *+ * setexpr computes column attnum of the result relation in an UPDATE, with+ * Vars of varno referring to the row being updated, and Params whose values+ * are in params. Return the indexes,+ * as positions in resultRelInfo's index arrays, that provably read nothing+ * of the column that setexpr may change, provided the column is not null in+ * the old or the new row. An index that uses the column in any way we do+ * not understand, or as a plain key or INCLUDE column, is never included.+ */+Bitmapset *+ExecSubattrProveUntouched(ResultRelInfo *resultRelInfo, Node *setexpr,+ int varno, AttrNumber attnum, ParamListInfo params)+{+ SubattrWalk mods = {varno, attnum, true, NIL};+ Bitmapset *result = NULL;++ if (params != NULL && params->numParams > 0)+ setexpr = substitute_params(setexpr, params);+ collect_modifies(setexpr, &mods);+ if (!mods.known || mods.paths == NIL)+ return NULL;++ for (int i = 0; i < resultRelInfo->ri_NumIndices; i++)+ {+ IndexInfo *ii = resultRelInfo->ri_IndexRelationInfo[i];+ SubattrWalk reads = {1, attnum, true, NIL};+ bool disjoint = true;+ ListCell *lr;++ if (resultRelInfo->ri_IndexRelationDescs[i] == NULL)+ continue;+ for (int k = 0; k < ii->ii_NumIndexAttrs; k++)+ {+ if (ii->ii_IndexAttrNumbers[k] == attnum)+ reads.known = false;+ }+ if (!reads.known)+ continue;++ /* index expressions and predicates refer to the table as varno 1 */+ read_walker((Node *) ii->ii_Expressions, &reads);+ read_walker((Node *) ii->ii_Predicate, &reads);+ if (!reads.known || reads.paths == NIL)+ continue;++ foreach(lr, reads.paths)+ {+ ListCell *lm;++ foreach(lm, mods.paths)+ {+ if (paths_may_overlap(lfirst(lr), lfirst(lm)))+ {+ disjoint = false;+ break;+ }+ }+ if (!disjoint)+ break;+ }+ if (disjoint)+ result = bms_add_member(result, i);+ }++ return result;+}diff --git a/src/backend/executor/meson.build b/src/backend/executor/meson.buildindex dc45be0b2ce..0d758e234a2 100644--- a/src/backend/executor/meson.build+++ b/src/backend/executor/meson.build@@ -16,6 +16,7 @@ backend_sources += files(
'execReplication.c',
'execSRF.c',
'execScan.c',
+ 'execSubattr.c',
'execTuples.c',
'execUtils.c',
'functions.c',
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.cindex 5a042eb83a3..25bfb9bce55 100644--- a/src/backend/executor/nodeModifyTable.c+++ b/src/backend/executor/nodeModifyTable.c@@ -63,6 +63,7 @@
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
+#include "executor/execSubattr.h"
#include "executor/instrument.h"
#include "executor/nodeModifyTable.h"
#include "foreign/fdwapi.h"
@@ -338,6 +339,58 @@ IndexExprAttrs(ResultRelInfo *resultRelInfo, int i, EState *estate)
return resultRelInfo->ri_IndexExprAttrs[i];
}
+/*+ * Is the i'th index unaffected by the update from old_tts to new_tts, where+ * "mine" are the changed attributes it uses? Use the statement-level proof+ * (ri_IndexUntouchedAttrs) when it covers them all and none is null in+ * either row; otherwise evaluate. Assert builds evaluate anyway and check+ * that the proof agrees.+ */+static bool+IndexUnaffected(ResultRelInfo *resultRelInfo, int i, Bitmapset *mine,+ TupleTableSlot *old_tts, TupleTableSlot *new_tts,+ EState *estate)+{+ IndexInfo *indexInfo = resultRelInfo->ri_IndexRelationInfo[i];+ bool proven = false;++ if (resultRelInfo->ri_IndexUntouchedAttrs != NULL &&+ bms_is_subset(mine, resultRelInfo->ri_IndexUntouchedAttrs[i]))+ {+ int attidx = -1;++ proven = true;+ while ((attidx = bms_next_member(mine, attidx)) >= 0)+ {+ AttrNumber attnum = attidx + FirstLowInvalidHeapAttributeNumber;++ if (slot_attisnull(old_tts, attnum) ||+ slot_attisnull(new_tts, attnum))+ {+ proven = false;+ break;+ }+ }+ }++#ifdef USE_ASSERT_CHECKING+ if (proven)+ {+ if (IndexEntryChanges(resultRelInfo, indexInfo, old_tts, new_tts,+ estate))+ elog(ERROR, "index \"%s\" proven unaffected by UPDATE, but its entry changed",+ RelationGetRelationName(resultRelInfo->ri_IndexRelationDescs[i]));+ return true;+ }+#else+ if (proven)+ return true;+#endif++ return !IndexEntryChanges(resultRelInfo, indexInfo, old_tts, new_tts,+ estate);+}+
/*
* ExecUpdateModifiedIdxAttrs
*
@@ -456,8 +509,8 @@ ExecUpdateModifiedIdxAttrs(ResultRelInfo *resultRelInfo,
if (!indexInfo->ii_ReadyForInserts ||
!resultRelInfo->ri_IndexRelationDescs[i]->rd_index->indisvalid ||
- IndexEntryChanges(resultRelInfo, indexInfo,- old_tts, new_tts, estate))+ !IndexUnaffected(resultRelInfo, i, mine, old_tts, new_tts,+ estate))
{
keep = bms_add_members(keep, mine);
if (bms_is_subset(candidates, keep))
@@ -1003,6 +1056,136 @@ ExecInitInsertProjection(ModifyTableState *mtstate,
resultRelInfo->ri_projectNewInfoValid = true;
}
+/*+ * ExecInitUpdateUntouched+ * Work out, per index, which SET target columns the statement cannot+ * change in any part the index reads. See execSubattr.c.+ *+ * Only for a plain UPDATE whose subplan is a scan of the result relation+ * itself, so that its target list computes each new value from Vars of the+ * row being updated. (MERGE keeps its SET expressions elsewhere.) Joins,+ * subqueries and the like bring other rows' values, and are left to+ * ExecUpdateModifiedIdxAttrs() to evaluate.+ */+static void+ExecInitUpdateUntouched(ModifyTableState *mtstate,+ ResultRelInfo *resultRelInfo)+{+ EState *estate = mtstate->ps.state;+ Plan *subplan = outerPlan(mtstate->ps.plan);+ int whichrel = resultRelInfo - mtstate->resultRelInfo;+ List *updateColnos;+ Relation rel = resultRelInfo->ri_RelationDesc;+ Index rti = resultRelInfo->ri_RangeTableIndex;+ Bitmapset *exprattrs;+ ListCell *lt,+ *lc;+ MemoryContext tmpcxt;+ MemoryContext oldcxt;++ Assert(whichrel >= 0 && whichrel < mtstate->mt_nrels);+ updateColnos = (List *) list_nth(mtstate->mt_updateColnosLists, whichrel);++ resultRelInfo->ri_IndexUntouchedAttrs = NULL;+ if (resultRelInfo->ri_NumIndices == 0 || rti == 0)+ return;++ /*+ * For an inheritance tree or partitioned table, each child's rows come+ * from its own scan under an Append, laid out the same way.+ */+ if (IsA(subplan, Append))+ {+ Plan *found = NULL;++ foreach(lc, ((Append *) subplan)->appendplans)+ {+ Plan *child = lfirst(lc);++ if (IsA(child, SeqScan) || IsA(child, IndexScan) ||+ IsA(child, BitmapHeapScan) || IsA(child, TidScan) ||+ IsA(child, TidRangeScan))+ {+ if (((Scan *) child)->scanrelid == rti)+ {+ /* a relation is scanned once per Append */+ if (found != NULL)+ return;+ found = child;+ }+ }+ }+ if (found == NULL)+ return;+ subplan = found;+ }++ if (!(IsA(subplan, SeqScan) || IsA(subplan, IndexScan) ||+ IsA(subplan, BitmapHeapScan) || IsA(subplan, TidScan) ||+ IsA(subplan, TidRangeScan)) ||+ ((Scan *) subplan)->scanrelid != rti)+ return;++ /*+ * A BEFORE ROW trigger may change any column of the new row, which the+ * SET expressions do not show.+ */+ if (resultRelInfo->ri_TrigDesc &&+ resultRelInfo->ri_TrigDesc->trig_update_before_row)+ return;++ exprattrs = RelationGetIndexAttrBitmap(rel, INDEX_ATTR_BITMAP_EXPRESSION);+ if (bms_is_empty(exprattrs))+ return;++ /* the analysis is thrown away; only the bitmaps are kept */+ tmpcxt = AllocSetContextCreate(CurrentMemoryContext, "subattr proof",+ ALLOCSET_SMALL_SIZES);+ oldcxt = MemoryContextSwitchTo(tmpcxt);++ /*+ * ExecBuildUpdateProjection() has checked that updateColnos pairs with+ * the target list's non-junk entries, which come first.+ */+ forboth(lt, subplan->targetlist, lc, updateColnos)+ {+ TargetEntry *tle = lfirst_node(TargetEntry, lt);+ AttrNumber attnum = lfirst_int(lc);+ Bitmapset *untouched;+ int i;++ Assert(!tle->resjunk);+ if (!bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber,+ exprattrs))+ continue;++ untouched = ExecSubattrProveUntouched(resultRelInfo,+ (Node *) tle->expr,+ rti, attnum,+ estate->es_param_list_info);+ i = -1;+ while ((i = bms_next_member(untouched, i)) >= 0)+ {+ ereport(DEBUG1,+ errmsg_internal("UPDATE of column \"%s\" cannot change index \"%s\"",+ NameStr(TupleDescAttr(RelationGetDescr(rel),+ attnum - 1)->attname),+ RelationGetRelationName(resultRelInfo->ri_IndexRelationDescs[i])));+ MemoryContextSwitchTo(estate->es_query_cxt);+ if (resultRelInfo->ri_IndexUntouchedAttrs == NULL)+ resultRelInfo->ri_IndexUntouchedAttrs =+ palloc0_array(Bitmapset *, resultRelInfo->ri_NumIndices);+ resultRelInfo->ri_IndexUntouchedAttrs[i] =+ bms_add_member(resultRelInfo->ri_IndexUntouchedAttrs[i],+ attnum - FirstLowInvalidHeapAttributeNumber);+ MemoryContextSwitchTo(tmpcxt);+ }+ }+ MemoryContextSwitchTo(oldcxt);+ MemoryContextDelete(tmpcxt);+ bms_free(exprattrs);+}+
/*
* ExecInitUpdateProjection
* Do one-time initialization of projection data for UPDATE tuples.
@@ -2629,6 +2812,16 @@ lreplace:
if (resultRelationDesc->rd_att->constr)
ExecConstraints(resultRelInfo, slot, estate);
+ /*+ * From the second row on, use what the SET clause shows about which+ * indexes it cannot affect. For a single row, working that out costs+ * more than evaluating the indexes it would save.+ */+ if (unlikely(resultRelInfo->ri_UpdateCount < 2) &&+ ++resultRelInfo->ri_UpdateCount == 2 &&+ context->mtstate->operation == CMD_UPDATE)+ ExecInitUpdateUntouched(context->mtstate, resultRelInfo);+
modified_idx_attrs = ExecUpdateModifiedIdxAttrs(resultRelInfo, oldSlot, slot,
estate);
diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.cindex 274ed9d3184..9b3936bb60a 100644--- a/src/backend/utils/adt/jsonfuncs.c+++ b/src/backend/utils/adt/jsonfuncs.c@@ -29,6 +29,7 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "nodes/miscnodes.h"
+#include "nodes/supportnodes.h"
#include "parser/parse_coerce.h"
#include "utils/array.h"
#include "utils/builtins.h"
@@ -6169,3 +6170,277 @@ json_check_mutability(Oid typoid, bool is_jsonb, bool *has_mutable)
break;
}
}
++/*+ * Turn a text[] Const into a SubattrPathElem path, or return false.+ */+static bool+jsonb_subattr_text_path(Node *arg, List **path)+{+ Const *c = (Const *) arg;+ Datum *elems;+ bool *nulls;+ int nelems;+ List *result = NIL;++ if (!arg || !IsA(arg, Const) || c->constisnull ||+ c->consttype != TEXTARRAYOID)+ return false;++ deconstruct_array_builtin(DatumGetArrayTypeP(c->constvalue), TEXTOID,+ &elems, &nulls, &nelems);+ for (int i = 0; i < nelems; i++)+ {+ SubattrPathElem *e;++ if (nulls[i])+ return false; /* an error at run time; claim nothing */+ e = palloc0_object(SubattrPathElem);+ e->kind = SUBATTR_ELEM_KEY;+ e->key = TextDatumGetCString(elems[i]);+ result = lappend(result, e);+ }+ *path = result;+ return true;+}++/* A one-key path from a text Const, or an int4 Const in text form. */+static bool+jsonb_subattr_one_elem(Node *arg, List **path)+{+ Const *c = (Const *) arg;+ SubattrPathElem *e;++ if (!arg || !IsA(arg, Const) || c->constisnull)+ return false;+ e = palloc0_object(SubattrPathElem);+ e->kind = SUBATTR_ELEM_KEY;+ if (c->consttype == TEXTOID)+ e->key = TextDatumGetCString(c->constvalue);+ else if (c->consttype == INT4OID)+ e->key = psprintf("%d", DatumGetInt32(c->constvalue));+ else+ return false;+ *path = list_make1(e);+ return true;+}++/* path || [any position] */+static List *+jsonb_subattr_any_position(List *prefix)+{+ SubattrPathElem *e = palloc0_object(SubattrPathElem);++ e->kind = SUBATTR_ELEM_ANY_POSITION;+ return lappend(prefix, e);+}++/*+ * jsonb_subattr_support+ * Support function for the jsonb accessors and modifiers: tells the+ * executor which parts of the document a call reads or may change.+ * See SupportRequestSubattr.+ *+ * Paths are reported as keys throughout, also where a function takes an+ * integer: jsonb reads every path element as an object key or, inside an+ * array, as a position, and the executor treats any key that reads as an+ * integer as possibly every position.+ *+ * Every path an accessor reads includes the containers on the way to it:+ * "a"->"b" reads "a" too, and so overlaps a change to "a", as a prefix.+ */+Datum+jsonb_subattr_support(PG_FUNCTION_ARGS)+{+ Node *rawreq = (Node *) PG_GETARG_POINTER(0);+ SupportRequestSubattr *req;+ List *args;+ List *path;++ if (!IsA(rawreq, SupportRequestSubattr))+ PG_RETURN_POINTER(NULL);+ req = (SupportRequestSubattr *) rawreq;+ args = req->args;++ if (req->kind == SUBATTR_READ)+ {+ switch (req->funcid)+ {+ case F_JSONB_OBJECT_FIELD:+ case F_JSONB_ARRAY_ELEMENT:+ if (!jsonb_subattr_one_elem(lsecond(args), &path))+ PG_RETURN_POINTER(NULL);+ req->subvalue = true;+ break;+ case F_JSONB_OBJECT_FIELD_TEXT:+ case F_JSONB_ARRAY_ELEMENT_TEXT:+ if (!jsonb_subattr_one_elem(lsecond(args), &path))+ PG_RETURN_POINTER(NULL);+ req->subvalue = false;+ break;+ case F_JSONB_EXTRACT_PATH:+ if (!jsonb_subattr_text_path(lsecond(args), &path))+ PG_RETURN_POINTER(NULL);+ req->subvalue = true;+ break;+ case F_JSONB_EXTRACT_PATH_TEXT:+ if (!jsonb_subattr_text_path(lsecond(args), &path))+ PG_RETURN_POINTER(NULL);+ req->subvalue = false;+ break;+ default:+ PG_RETURN_POINTER(NULL);+ }+ req->source = 0;+ req->paths = list_make1(path);+ PG_RETURN_POINTER(req);+ }++ Assert(req->kind == SUBATTR_MODIFY);+ req->source = 0;+ switch (req->funcid)+ {+ case F_JSONB_SET:+ case F_JSONB_SET_LAX:++ /*+ * jsonb_set(target, path, new, create_if_missing) replaces or+ * creates the part at path. In an array it may instead add an+ * element at either end, and jsonb_set_lax() with a null new+ * value may delete the part, moving later elements; but an array+ * position is named by a key that reads as an integer, which the+ * executor takes to overlap every position.+ */+ if (!jsonb_subattr_text_path(lsecond(args), &path))+ PG_RETURN_POINTER(NULL);+ req->paths = list_make1(path);+ break;++ case F_JSONB_DELETE_PATH:+ /* jsonb #- path deletes the part at path; as above */+ if (!jsonb_subattr_text_path(lsecond(args), &path))+ PG_RETURN_POINTER(NULL);+ req->paths = list_make1(path);+ break;++ case F_JSONB_INSERT:++ /*+ * jsonb_insert(target, path, new, after) adds a member at path to+ * an object, or an element next to the one at path in an array,+ * moving the later elements: the container may change in a key at+ * path or in any of its positions.+ */+ if (!jsonb_subattr_text_path(lsecond(args), &path) ||+ path == NIL)+ PG_RETURN_POINTER(NULL);+ req->paths = list_make2(path,+ jsonb_subattr_any_position(list_copy_head(path,+ list_length(path) - 1)));+ break;++ case F_JSONB_DELETE_JSONB_TEXT:+ case F_JSONB_DELETE_JSONB__TEXT:+ {+ /*+ * jsonb - key and jsonb - keys[] delete top-level members of+ * an object, or matching string elements of an array, moving+ * the others.+ */+ Const *c = (Const *) lsecond(args);+ Datum *elems;+ bool *nulls;+ int nelems;++ if (!c || !IsA(c, Const) || c->constisnull)+ PG_RETURN_POINTER(NULL);+ if (req->funcid == F_JSONB_DELETE_JSONB_TEXT)+ {+ /* a one-element array of keys */+ elems = palloc_object(Datum);+ nulls = palloc0_object(bool);+ elems[0] = c->constvalue;+ nelems = 1;+ }+ else+ deconstruct_array_builtin(DatumGetArrayTypeP(c->constvalue),+ TEXTOID, &elems, &nulls,+ &nelems);+ req->paths = list_make1(jsonb_subattr_any_position(NIL));+ for (int i = 0; i < nelems; i++)+ {+ SubattrPathElem *e;++ if (nulls[i])+ continue; /* deletes nothing */+ e = palloc0_object(SubattrPathElem);+ e->kind = SUBATTR_ELEM_KEY;+ e->key = TextDatumGetCString(elems[i]);+ req->paths = lappend(req->paths, list_make1(e));+ }+ break;+ }++ case F_JSONB_DELETE_JSONB_INT4:+ /* jsonb - int deletes an array element, moving later ones */+ req->paths = list_make1(jsonb_subattr_any_position(NIL));+ break;++ case F_JSONB_CONCAT:+ {+ /*+ * doc || '{"k": v, ...}' with a constant object, or the other+ * way around. For an object doc, only the constant's+ * top-level keys change. For an array or scalar doc, the+ * result is an array of its elements (or the scalar) and the+ * object, in some order: positions change, and no key that+ * does not read as an integer reads anything before or after.+ * Any other constant (an array, a scalar) turns an object doc+ * into an array, which changes every key; give up.+ */+ Const *c;+ Jsonb *jb;+ JsonbIterator *it;+ JsonbValue v;+ JsonbIteratorToken r;++ if (IsA(lsecond(args), Const))+ {+ c = (Const *) lsecond(args);+ req->source = 0;+ }+ else if (IsA(linitial(args), Const))+ {+ c = (Const *) linitial(args);+ req->source = 1;+ }+ else+ PG_RETURN_POINTER(NULL);+ if (c->constisnull)+ PG_RETURN_POINTER(NULL);+ jb = DatumGetJsonbP(c->constvalue);+ if (!JB_ROOT_IS_OBJECT(jb))+ PG_RETURN_POINTER(NULL);++ req->paths = list_make1(jsonb_subattr_any_position(NIL));+ it = JsonbIteratorInit(&jb->root);+ while ((r = JsonbIteratorNext(&it, &v, true)) != WJB_DONE)+ {+ if (r == WJB_KEY)+ {+ SubattrPathElem *e = palloc0_object(SubattrPathElem);++ e->kind = SUBATTR_ELEM_KEY;+ e->key = pnstrdup(v.val.string.val, v.val.string.len);+ req->paths = lappend(req->paths, list_make1(e));+ }+ }+ break;+ }++ default:+ PG_RETURN_POINTER(NULL);+ }++ PG_RETURN_POINTER(req);+}diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.datindex 1f21a8eb6d4..06e14df9d87 100644--- a/src/include/catalog/pg_proc.dat+++ b/src/include/catalog/pg_proc.dat@@ -10541,31 +10541,35 @@
proargdefaults => '{false}', prosrc => 'jsonb_strip_nulls' },
{ oid => '3478',
- proname => 'jsonb_object_field', prorettype => 'jsonb',- proargtypes => 'jsonb text', proargnames => '{from_json, field_name}',- prosrc => 'jsonb_object_field' },+ proname => 'jsonb_object_field', prosupport => 'jsonb_subattr_support',+ prorettype => 'jsonb', proargtypes => 'jsonb text',+ proargnames => '{from_json, field_name}', prosrc => 'jsonb_object_field' },
{ oid => '3214',
- proname => 'jsonb_object_field_text', prorettype => 'text',- proargtypes => 'jsonb text', proargnames => '{from_json, field_name}',+ proname => 'jsonb_object_field_text', prosupport => 'jsonb_subattr_support',+ prorettype => 'text', proargtypes => 'jsonb text',+ proargnames => '{from_json, field_name}',
prosrc => 'jsonb_object_field_text' },
{ oid => '3215',
- proname => 'jsonb_array_element', prorettype => 'jsonb',- proargtypes => 'jsonb int4', proargnames => '{from_json, element_index}',+ proname => 'jsonb_array_element', prosupport => 'jsonb_subattr_support',+ prorettype => 'jsonb', proargtypes => 'jsonb int4',+ proargnames => '{from_json, element_index}',
prosrc => 'jsonb_array_element' },
{ oid => '3216',
- proname => 'jsonb_array_element_text', prorettype => 'text',- proargtypes => 'jsonb int4', proargnames => '{from_json, element_index}',+ proname => 'jsonb_array_element_text', prosupport => 'jsonb_subattr_support',+ prorettype => 'text', proargtypes => 'jsonb int4',+ proargnames => '{from_json, element_index}',
prosrc => 'jsonb_array_element_text' },
{ oid => '3217', descr => 'get value from jsonb with path elements',
- proname => 'jsonb_extract_path', provariadic => 'text', prorettype => 'jsonb',+ proname => 'jsonb_extract_path', provariadic => 'text',+ prosupport => 'jsonb_subattr_support', prorettype => 'jsonb',
proargtypes => 'jsonb _text', proallargtypes => '{jsonb,_text}',
proargmodes => '{i,v}', proargnames => '{from_json,path_elems}',
prosrc => 'jsonb_extract_path' },
{ oid => '3940', descr => 'get value from jsonb as text with path elements',
proname => 'jsonb_extract_path_text', provariadic => 'text',
- prorettype => 'text', proargtypes => 'jsonb _text',- proallargtypes => '{jsonb,_text}', proargmodes => '{i,v}',- proargnames => '{from_json,path_elems}',+ prosupport => 'jsonb_subattr_support', prorettype => 'text',+ proargtypes => 'jsonb _text', proallargtypes => '{jsonb,_text}',+ proargmodes => '{i,v}', proargnames => '{from_json,path_elems}',
prosrc => 'jsonb_extract_path_text' },
{ oid => '3219', descr => 'elements of a jsonb array',
proname => 'jsonb_array_elements', prorows => '100', proretset => 't',
@@ -10694,38 +10698,44 @@
proargtypes => 'internal int2 jsonb int4 internal internal internal',
prosrc => 'gin_triconsistent_jsonb_path' },
{ oid => '3301',
- proname => 'jsonb_concat', prorettype => 'jsonb',- proargtypes => 'jsonb jsonb', prosrc => 'jsonb_concat' },+ proname => 'jsonb_concat', prosupport => 'jsonb_subattr_support',+ prorettype => 'jsonb', proargtypes => 'jsonb jsonb',+ prosrc => 'jsonb_concat' },
{ oid => '3302',
- proname => 'jsonb_delete', prorettype => 'jsonb', proargtypes => 'jsonb text',+ proname => 'jsonb_delete', prosupport => 'jsonb_subattr_support',+ prorettype => 'jsonb', proargtypes => 'jsonb text',
prosrc => 'jsonb_delete' },
{ oid => '3303',
- proname => 'jsonb_delete', prorettype => 'jsonb', proargtypes => 'jsonb int4',+ proname => 'jsonb_delete', prosupport => 'jsonb_subattr_support',+ prorettype => 'jsonb', proargtypes => 'jsonb int4',
prosrc => 'jsonb_delete_idx' },
{ oid => '3343',
- proname => 'jsonb_delete', provariadic => 'text', prorettype => 'jsonb',+ proname => 'jsonb_delete', provariadic => 'text',+ prosupport => 'jsonb_subattr_support', prorettype => 'jsonb',
proargtypes => 'jsonb _text', proallargtypes => '{jsonb,_text}',
proargmodes => '{i,v}', proargnames => '{from_json,path_elems}',
prosrc => 'jsonb_delete_array' },
{ oid => '3304',
- proname => 'jsonb_delete_path', prorettype => 'jsonb',- proargtypes => 'jsonb _text', prosrc => 'jsonb_delete_path' },+ proname => 'jsonb_delete_path', prosupport => 'jsonb_subattr_support',+ prorettype => 'jsonb', proargtypes => 'jsonb _text',+ prosrc => 'jsonb_delete_path' },
{ oid => '5054', descr => 'Set part of a jsonb, handle NULL value',
- proname => 'jsonb_set_lax', proisstrict => 'f', prorettype => 'jsonb',+ proname => 'jsonb_set_lax', prosupport => 'jsonb_subattr_support',+ proisstrict => 'f', prorettype => 'jsonb',
proargtypes => 'jsonb _text jsonb bool text',
proargnames => '{jsonb_in,path,replacement,create_if_missing,null_value_treatment}',
proargdefaults => '{true,use_json_null}', prosrc => 'jsonb_set_lax' },
{ oid => '3305', descr => 'Set part of a jsonb',
- proname => 'jsonb_set', prorettype => 'jsonb',- proargtypes => 'jsonb _text jsonb bool',+ proname => 'jsonb_set', prosupport => 'jsonb_subattr_support',+ prorettype => 'jsonb', proargtypes => 'jsonb _text jsonb bool',
proargnames => '{jsonb_in,path,replacement,create_if_missing}',
proargdefaults => '{true}', prosrc => 'jsonb_set' },
{ oid => '3306', descr => 'Indented text from jsonb',
proname => 'jsonb_pretty', prorettype => 'text', proargtypes => 'jsonb',
prosrc => 'jsonb_pretty' },
{ oid => '3579', descr => 'Insert value into a jsonb',
- proname => 'jsonb_insert', prorettype => 'jsonb',- proargtypes => 'jsonb _text jsonb bool',+ proname => 'jsonb_insert', prosupport => 'jsonb_subattr_support',+ prorettype => 'jsonb', proargtypes => 'jsonb _text jsonb bool',
proargnames => '{jsonb_in,path,replacement,insert_after}',
proargdefaults => '{false}', prosrc => 'jsonb_insert' },
@@ -12469,6 +12479,9 @@
proname => 'raw_array_subscript_handler', prorettype => 'internal',
proargtypes => 'internal', prosrc => 'raw_array_subscript_handler' },
# type subscripting support
+{ oid => '8008', descr => 'support for jsonb accessors and modifiers',+ proname => 'jsonb_subattr_support', prorettype => 'internal',+ proargtypes => 'internal', prosrc => 'jsonb_subattr_support' },
{ oid => '6098', descr => 'jsonb subscripting logic',
proname => 'jsonb_subscript_handler', prorettype => 'internal',
proargtypes => 'internal', prosrc => 'jsonb_subscript_handler' },
diff --git a/src/include/executor/execSubattr.h b/src/include/executor/execSubattr.hnew file mode 100644index 00000000000..61c0fb58404--- /dev/null+++ b/src/include/executor/execSubattr.h@@ -0,0 +1,22 @@+/*-------------------------------------------------------------------------+ *+ * execSubattr.h+ * Prove that an UPDATE cannot change what an index reads of a column.+ *+ * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group+ *+ * src/include/executor/execSubattr.h+ *+ *-------------------------------------------------------------------------+ */+#ifndef EXECSUBATTR_H+#define EXECSUBATTR_H++#include "nodes/execnodes.h"++extern Bitmapset *ExecSubattrProveUntouched(ResultRelInfo *resultRelInfo,+ Node *setexpr, int varno,+ AttrNumber attnum,+ ParamListInfo params);++#endif /* EXECSUBATTR_H */diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.hindex 220aaa8b965..716668c5aad 100644--- a/src/include/nodes/execnodes.h+++ b/src/include/nodes/execnodes.h@@ -510,6 +510,18 @@ typedef struct ResultRelInfo
*/
Bitmapset **ri_IndexExprAttrs;
+ /*+ * For UPDATE, per index, the table attributes (offset as above) that the+ * statement's SET expressions provably cannot change any part of that the+ * index reads, as long as the attribute is not null before or after.+ * Worked out when ri_UpdateCount, which counts updated rows up to then,+ * reaches 2; NULL if nothing was proven. It holds for the whole+ * statement, across rescans: it depends only on the plan and on Params+ * whose values are fixed for the execution.+ */+ Bitmapset **ri_IndexUntouchedAttrs;+ int ri_UpdateCount;+
/*
* For UPDATE/DELETE/MERGE result relations, the attribute number of the
* row identity junk attribute in the source plan's output tuples
diff --git a/src/include/nodes/supportnodes.h b/src/include/nodes/supportnodes.hindex 132a2bcd8af..779d65c8bce 100644--- a/src/include/nodes/supportnodes.h+++ b/src/include/nodes/supportnodes.h@@ -445,4 +445,78 @@ typedef struct SupportRequestModifyInPlace
int paramid; /* ID of Param(s) representing variable */
} SupportRequestModifyInPlace;
+/*+ * The SupportRequestSubattr request lets the executor learn, without running+ * anything, which parts of a structured value (a jsonb document, an array,+ * an extension's own document type) a function call reads or may change.+ * The executor uses the answers to skip work when an UPDATE cannot change+ * what an index stores. A wrong answer can leave an index without an entry+ * for a row, so a support function must answer only what it can guarantee,+ * and otherwise decline by returning NULL.+ *+ * A part is named by a path: a List of SubattrPathElem, from the outermost+ * level in. An empty path (NIL) names the whole value.+ *+ * The support function sets "source" to the index in "args" of the argument+ * the call reads (SUBATTR_READ) or copies from (SUBATTR_MODIFY). It may+ * rely on the values of other arguments only if they are Consts; the+ * executor replaces external Params by Consts holding their values before+ * asking. If the answer depends on anything else, decline.+ *+ * SUBATTR_READ: the call's result is computed only from the parts of+ * args[source] named in "paths", and from its other arguments. A call may+ * name several paths. Set "subvalue" if the call has a single path and+ * returns exactly the part that path names, as a value of a type whose parts+ * are named the same way (jsonb's -> does; ->> returns text and does not),+ * so that reading a part of the result reads the part at the combined path.+ *+ * SUBATTR_MODIFY: the call returns args[source] changed at most at the+ * parts named in "paths": every other part reads the same, byte for byte,+ * through any SUBATTR_READ call. A change that moves other parts, as+ * inserting into or deleting from an array moves later elements, must name+ * them: SUBATTR_ELEM_ANY_POSITION names every element of an array. If the+ * source argument is null, or the result is null, the executor does not+ * rely on the answer.+ *+ * Two paths are taken to name overlapping parts unless, at some depth, both+ * have an element and those elements cannot name the same part: two+ * different keys that do not both read as integers ("1" and "01" may name+ * the same array element), two different positions, or a position or+ * SUBATTR_ELEM_ANY_POSITION against a key that does not read as an integer.+ * A path overlaps every path it is a prefix of.+ */+typedef enum SubattrPathElemKind+{+ SUBATTR_ELEM_KEY, /* object member by name */+ SUBATTR_ELEM_POSITION, /* array element by position */+ SUBATTR_ELEM_ANY_POSITION, /* any array element */+} SubattrPathElemKind;++typedef struct SubattrPathElem+{+ SubattrPathElemKind kind;+ char *key; /* SUBATTR_ELEM_KEY: null-terminated name */+ int position; /* SUBATTR_ELEM_POSITION */+} SubattrPathElem;++typedef enum SubattrRequestKind+{+ SUBATTR_READ,+ SUBATTR_MODIFY,+} SubattrRequestKind;++typedef struct SupportRequestSubattr+{+ NodeTag type;++ SubattrRequestKind kind; /* what the executor asks */+ Oid funcid; /* PG_PROC OID of the target function */+ List *args; /* arguments to the function */++ /* Set by the support function before returning the request: */+ int source; /* index into args of the value read/copied */+ List *paths; /* List of List of SubattrPathElem */+ bool subvalue; /* SUBATTR_READ: returns the part at paths */+} SupportRequestSubattr;+
#endif /* SUPPORTNODES_H */
diff --git a/src/test/isolation/expected/update-subattr-epq.out b/src/test/isolation/expected/update-subattr-epq.outnew file mode 100644index 00000000000..d54e266e40e--- /dev/null+++ b/src/test/isolation/expected/update-subattr-epq.out@@ -0,0 +1,65 @@+Parsed test spec with 2 sessions++starting permutation: s2_begin s2_status s1_n s2_commit s1_check+step s2_begin: BEGIN;+step s2_status: UPDATE subattr_epq SET doc = jsonb_set(doc, ARRAY['status'], to_jsonb('b'::text)) WHERE id = 2;+step s1_n: UPDATE subattr_epq SET doc = jsonb_set(doc, ARRAY['n'], '1'); <waiting ...>+step s2_commit: COMMIT;+step s1_n: <... completed>+step s1_check: + SET enable_seqscan = off;+ SET enable_bitmapscan = off;+ SELECT id FROM subattr_epq WHERE doc->>'status' = 'b';+ SELECT id FROM subattr_epq WHERE (doc->>'vip') = 'yes' AND id IN (1, 2);+ RESET enable_seqscan;+ RESET enable_bitmapscan;+ SELECT id, doc FROM subattr_epq ORDER BY id;++id+--+ 2+(1 row)++id+--+ 1+ 2+(2 rows)++id|doc +--+-------------------------------------+ 1|{"n": 1, "vip": "yes", "status": "a"}+ 2|{"n": 1, "vip": "yes", "status": "b"}+(2 rows)+++starting permutation: s2_begin s2_vip s1_n s2_commit s1_check+step s2_begin: BEGIN;+step s2_vip: UPDATE subattr_epq SET doc = jsonb_set(doc, ARRAY['vip'], to_jsonb('no'::text)) WHERE id = 2;+step s1_n: UPDATE subattr_epq SET doc = jsonb_set(doc, ARRAY['n'], '1'); <waiting ...>+step s2_commit: COMMIT;+step s1_n: <... completed>+step s1_check: + SET enable_seqscan = off;+ SET enable_bitmapscan = off;+ SELECT id FROM subattr_epq WHERE doc->>'status' = 'b';+ SELECT id FROM subattr_epq WHERE (doc->>'vip') = 'yes' AND id IN (1, 2);+ RESET enable_seqscan;+ RESET enable_bitmapscan;+ SELECT id, doc FROM subattr_epq ORDER BY id;++id+--+(0 rows)++id+--+ 1+(1 row)++id|doc +--+-------------------------------------+ 1|{"n": 1, "vip": "yes", "status": "a"}+ 2|{"n": 1, "vip": "no", "status": "a"} +(2 rows)+diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_scheduleindex 2e99ca04305..b76a884d646 100644--- a/src/test/isolation/isolation_schedule+++ b/src/test/isolation/isolation_schedule@@ -44,6 +44,7 @@ test: subxid-overflow
test: eval-plan-qual
test: eval-plan-qual-trigger
test: update-epq-indexes
+test: update-subattr-epq
test: eval-plan-qual-partition-prune
test: inplace-inval
test: intra-grant-inplace
diff --git a/src/test/isolation/specs/update-subattr-epq.spec b/src/test/isolation/specs/update-subattr-epq.specnew file mode 100644index 00000000000..c611c2cad45--- /dev/null+++ b/src/test/isolation/specs/update-subattr-epq.spec@@ -0,0 +1,44 @@+# An UPDATE whose SET clause cannot change what an index reads, waiting on+# a concurrent UPDATE that does change it. The waiting UPDATE recomputes+# its new row from the version the other one wrote, and must leave the+# index entry for that version in place. The waiting row is the second+# row of the UPDATE, so that the conclusion drawn from the SET clause is+# used for it.++setup+{+ CREATE TABLE subattr_epq (id int PRIMARY KEY, doc jsonb)+ WITH (fillfactor = 50);+ CREATE INDEX subattr_epq_status ON subattr_epq ((doc->>'status'));+ CREATE INDEX subattr_epq_vip ON subattr_epq (id) WHERE (doc->>'vip') = 'yes';+ INSERT INTO subattr_epq+ SELECT g, jsonb_build_object('status', 'a', 'vip', 'yes', 'n', 0)+ FROM generate_series(1, 2) g;+}++teardown+{+ DROP TABLE subattr_epq;+}++session s1+step s1_n { UPDATE subattr_epq SET doc = jsonb_set(doc, ARRAY['n'], '1'); }+step s1_check+{+ SET enable_seqscan = off;+ SET enable_bitmapscan = off;+ SELECT id FROM subattr_epq WHERE doc->>'status' = 'b';+ SELECT id FROM subattr_epq WHERE (doc->>'vip') = 'yes' AND id IN (1, 2);+ RESET enable_seqscan;+ RESET enable_bitmapscan;+ SELECT id, doc FROM subattr_epq ORDER BY id;+}++session s2+step s2_begin { BEGIN; }+step s2_status { UPDATE subattr_epq SET doc = jsonb_set(doc, ARRAY['status'], to_jsonb('b'::text)) WHERE id = 2; }+step s2_vip { UPDATE subattr_epq SET doc = jsonb_set(doc, ARRAY['vip'], to_jsonb('no'::text)) WHERE id = 2; }+step s2_commit { COMMIT; }++permutation s2_begin s2_status s1_n s2_commit s1_check+permutation s2_begin s2_vip s1_n s2_commit s1_checkdiff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.outindex 35d6ee2ce9e..291b317ccf0 100644--- a/src/test/regress/expected/misc_functions.out+++ b/src/test/regress/expected/misc_functions.out@@ -631,6 +631,74 @@ SELECT * FROM tenk1 a JOIN my_gen_series(1,10) g ON a.unique1 = g;
Index Cond: (unique1 = g.g)
(4 rows)
+--+-- Test SupportRequestSubattr with a function that reads two parts of a+-- document. An UPDATE that changes neither need not evaluate an index on+-- it; one that changes either must, and must give the index a new entry.+-- Only assert-enabled builds check that no index skipped this way changes;+-- amcheck's check_subattr test shows the skips, at DEBUG1.+--+CREATE FUNCTION test_subattr_support(internal)+ RETURNS internal+ AS :'regresslib', 'test_subattr_support'+ LANGUAGE C STRICT;+-- (plpgsql, so that it is not inlined and the support function is asked)+CREATE FUNCTION two_keys(jsonb, text, text) RETURNS text+ LANGUAGE plpgsql IMMUTABLE STRICT+ AS $$ BEGIN RETURN ($1->>$2) || '/' || ($1->>$3); END $$+ SUPPORT test_subattr_support;+CREATE TABLE subattr_two (id int, d jsonb) WITH (fillfactor = 50);+CREATE INDEX subattr_two_idx ON subattr_two (two_keys(d, 'a', 'b'));+INSERT INTO subattr_two SELECT g, jsonb_build_object('a', g, 'b', g, 'c', g)+ FROM generate_series(1, 2) g;+BEGIN;+UPDATE subattr_two SET d = jsonb_set(d, '{c}', '0');+-- both rows HOT: the index's value didn't change+SELECT pg_stat_get_xact_tuples_hot_updated('subattr_two'::regclass) AS hot;+ hot +-----+ 2+(1 row)++UPDATE subattr_two SET d = jsonb_set(d, '{b}', '0');+-- still 2: neither row HOT, the index needs new entries+SELECT pg_stat_get_xact_tuples_hot_updated('subattr_two'::regclass) AS hot;+ hot +-----+ 2+(1 row)++COMMIT;+-- through the index, which has no entry for '2/0' unless the second+-- update made one+SET enable_seqscan = off;+SET enable_bitmapscan = off;+EXPLAIN (COSTS OFF)+SELECT id FROM subattr_two WHERE two_keys(d, 'a', 'b') = '2/0';+ QUERY PLAN +-----------------------------------------------------------------+ Index Scan using subattr_two_idx on subattr_two+ Index Cond: (two_keys(d, 'a'::text, 'b'::text) = '2/0'::text)+(2 rows)++SELECT id FROM subattr_two WHERE two_keys(d, 'a', 'b') = '2/0';+ id +----+ 2+(1 row)++RESET enable_bitmapscan;+RESET enable_seqscan;+SELECT id, two_keys(d, 'a', 'b') FROM subattr_two ORDER BY id;+ id | two_keys +----+----------+ 1 | 1/0+ 2 | 2/0+(2 rows)++DROP TABLE subattr_two;+DROP FUNCTION two_keys;+DROP FUNCTION test_subattr_support;
--
-- Test SupportRequestInlineInFrom request
--
diff --git a/src/test/regress/regress.c b/src/test/regress/regress.cindex 3deb4ed7203..060ac9544a5 100644--- a/src/test/regress/regress.c+++ b/src/test/regress/regress.c@@ -750,6 +750,44 @@ is_catalog_text_unique_index_oid(PG_FUNCTION_ARGS)
return BoolGetDatum(IsCatalogTextUniqueIndexOid(PG_GETARG_OID(0)));
}
+/*+ * test_subattr_support: SupportRequestSubattr for a function of+ * (jsonb, text, text) that reads the two top-level keys its constant+ * arguments name, and returns something that is not a part of the document+ * (so subvalue is false).+ */+PG_FUNCTION_INFO_V1(test_subattr_support);+Datum+test_subattr_support(PG_FUNCTION_ARGS)+{+ Node *rawreq = (Node *) PG_GETARG_POINTER(0);+ SupportRequestSubattr *req;+ List *paths = NIL;++ if (!IsA(rawreq, SupportRequestSubattr))+ PG_RETURN_POINTER(NULL);+ req = (SupportRequestSubattr *) rawreq;+ if (req->kind != SUBATTR_READ || list_length(req->args) != 3)+ PG_RETURN_POINTER(NULL);++ for (int i = 1; i <= 2; i++)+ {+ Const *c = (Const *) list_nth(req->args, i);+ SubattrPathElem *e;++ if (!IsA(c, Const) || c->constisnull || c->consttype != TEXTOID)+ PG_RETURN_POINTER(NULL);+ e = palloc0_object(SubattrPathElem);+ e->kind = SUBATTR_ELEM_KEY;+ e->key = TextDatumGetCString(c->constvalue);+ paths = lappend(paths, list_make1(e));+ }+ req->source = 0;+ req->paths = paths;+ req->subvalue = false;+ PG_RETURN_POINTER(req);+}+
PG_FUNCTION_INFO_V1(test_support_func);
Datum
test_support_func(PG_FUNCTION_ARGS)
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sqlindex 94b91964c04..8b787e66768 100644--- a/src/test/regress/sql/misc_functions.sql+++ b/src/test/regress/sql/misc_functions.sql@@ -236,6 +236,48 @@ SELECT * FROM tenk1 a JOIN my_gen_series(1,1000) g ON a.unique1 = g;
EXPLAIN (COSTS OFF)
SELECT * FROM tenk1 a JOIN my_gen_series(1,10) g ON a.unique1 = g;
+--+-- Test SupportRequestSubattr with a function that reads two parts of a+-- document. An UPDATE that changes neither need not evaluate an index on+-- it; one that changes either must, and must give the index a new entry.+-- Only assert-enabled builds check that no index skipped this way changes;+-- amcheck's check_subattr test shows the skips, at DEBUG1.+--+CREATE FUNCTION test_subattr_support(internal)+ RETURNS internal+ AS :'regresslib', 'test_subattr_support'+ LANGUAGE C STRICT;+-- (plpgsql, so that it is not inlined and the support function is asked)+CREATE FUNCTION two_keys(jsonb, text, text) RETURNS text+ LANGUAGE plpgsql IMMUTABLE STRICT+ AS $$ BEGIN RETURN ($1->>$2) || '/' || ($1->>$3); END $$+ SUPPORT test_subattr_support;+CREATE TABLE subattr_two (id int, d jsonb) WITH (fillfactor = 50);+CREATE INDEX subattr_two_idx ON subattr_two (two_keys(d, 'a', 'b'));+INSERT INTO subattr_two SELECT g, jsonb_build_object('a', g, 'b', g, 'c', g)+ FROM generate_series(1, 2) g;+BEGIN;+UPDATE subattr_two SET d = jsonb_set(d, '{c}', '0');+-- both rows HOT: the index's value didn't change+SELECT pg_stat_get_xact_tuples_hot_updated('subattr_two'::regclass) AS hot;+UPDATE subattr_two SET d = jsonb_set(d, '{b}', '0');+-- still 2: neither row HOT, the index needs new entries+SELECT pg_stat_get_xact_tuples_hot_updated('subattr_two'::regclass) AS hot;+COMMIT;+-- through the index, which has no entry for '2/0' unless the second+-- update made one+SET enable_seqscan = off;+SET enable_bitmapscan = off;+EXPLAIN (COSTS OFF)+SELECT id FROM subattr_two WHERE two_keys(d, 'a', 'b') = '2/0';+SELECT id FROM subattr_two WHERE two_keys(d, 'a', 'b') = '2/0';+RESET enable_bitmapscan;+RESET enable_seqscan;+SELECT id, two_keys(d, 'a', 'b') FROM subattr_two ORDER BY id;+DROP TABLE subattr_two;+DROP FUNCTION two_keys;+DROP FUNCTION test_subattr_support;+
--
-- Test SupportRequestInlineInFrom request
--
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.listindex a7a8dbdd23f..bccd3cac465 100644--- a/src/tools/pgindent/typedefs.list+++ b/src/tools/pgindent/typedefs.list@@ -3021,6 +3021,10 @@ SubXactCallback
SubXactCallbackItem
SubXactEvent
SubXactInfo
+SubattrPathElem+SubattrPathElemKind+SubattrRequestKind+SubattrWalk
SubqueryScan
SubqueryScanPath
SubqueryScanState
@@ -3044,6 +3048,7 @@ SupportRequestRows
SupportRequestSelectivity
SupportRequestSimplify
SupportRequestSimplifyAggref
+SupportRequestSubattr
SupportRequestWFuncMonotonic
Syn
SyncOps
--
2.50.1
v43-0007-GIN-implement-amcomparedatums.patch (text/plain, 13.0K) download
From 1b1050b30777bc454c7e6a70c250bbfecb5ec8da Mon Sep 17 00:00:00 2001
From: Greg Burd <greg@burd.me>
Date: Fri, 16 Oct 2026 12:35:16 -0400
Subject: [PATCH v43 7/8] GIN: implement amcomparedatums
A GIN index stores, for each row, the keys its operator class extracts from
the indexed value, and nothing else about the value. Scans see a row only
as its set of keys: that is all the consistent function is given, and what
it cannot decide from them it leaves to a recheck against the heap tuple.
So an updated row gives the same entries exactly when extractValue gives
the same keys for the new value, compared as GIN compares keys. Keys equal
under the compare function share one entry whichever came first, so the
compare function is the right test.
Run ginExtractEntries() on both values, which sorts and deduplicates them
and represents empty items and null keys as GIN does, and compare. The
GinState is cached in ii_AmCache, as gininsert() caches it, so it is built
once per statement.
Updates that become HOT include reordering or repeating the elements of an
array column under array_ops, and changing a tsvector, or a text column
under a to_tsvector() expression index, in ways that keep its lexemes, such
as moving words around.
---
contrib/amcheck/expected/check_gin.out | 125 +++++++++++++++++++++++++
contrib/amcheck/sql/check_gin.sql | 71 ++++++++++++++
src/backend/access/gin/ginutil.c | 61 +++++++++++-
src/include/access/gin_private.h | 3 +
4 files changed, 259 insertions(+), 1 deletion(-)
diff --git a/contrib/amcheck/expected/check_gin.out b/contrib/amcheck/expected/check_gin.outindex 8dd01ced8d1..cb5cb777219 100644--- a/contrib/amcheck/expected/check_gin.out+++ b/contrib/amcheck/expected/check_gin.out@@ -88,3 +88,128 @@ SELECT gin_index_check('gin_check_multicolumn_idx');
-- cleanup
DROP TABLE gin_check_multicolumn;
+-- An UPDATE that leaves a GIN index's keys for the row as they are is HOT:+-- reordering or repeating array elements, or changing the positions in a+-- tsvector but not its lexemes. Scans through the index must still find+-- exactly the rows a sequential scan finds.+CREATE TABLE gin_hot (id int, tags int[], tsv tsvector, body text)+ WITH (fillfactor = 10);+CREATE INDEX gin_hot_tags ON gin_hot USING gin (tags);+CREATE INDEX gin_hot_tsv ON gin_hot USING gin (tsv);+CREATE INDEX gin_hot_body ON gin_hot USING gin (to_tsvector('simple', body));+INSERT INTO gin_hot+ SELECT g, ARRAY[g % 7, g % 11, g % 13], to_tsvector('simple', 'a b c ' || g),+ 'one two ' || g+ FROM generate_series(1, 100) g;+CREATE FUNCTION gin_hot_count(OUT updated bigint, OUT hot bigint)+LANGUAGE sql AS $$+ SELECT pg_stat_get_xact_tuples_updated('gin_hot'::regclass),+ pg_stat_get_xact_tuples_hot_updated('gin_hot'::regclass) $$;+-- How many rows each tag, lexeme and word matches through the index (a+-- bitmap scan, as GIN has no other) and not (a sequential scan), and the+-- number of mismatches.+CREATE FUNCTION gin_hot_count_via(via_index bool, q text) RETURNS bigint+LANGUAGE plpgsql AS $$+DECLARE c bigint;+BEGIN+ PERFORM set_config('enable_seqscan', CASE WHEN via_index THEN 'off' ELSE 'on' END, true);+ PERFORM set_config('enable_bitmapscan', CASE WHEN via_index THEN 'on' ELSE 'off' END, true);+ PERFORM set_config('enable_indexscan', 'off', true);+ EXECUTE 'SELECT count(*) FROM gin_hot WHERE ' || q INTO c;+ RETURN c;+END $$;+CREATE FUNCTION gin_hot_mismatches() RETURNS bigint LANGUAGE sql AS $$+ SELECT count(*) FROM+ (SELECT format('tags @> ARRAY[%s]', k) FROM generate_series(0, 13) k+ UNION ALL+ SELECT format('tsv @@ to_tsquery(''simple'', %L)', w)+ FROM unnest(ARRAY['a', 'b', 'c', 'd', 'one', 'two', 'three', '7']) w+ UNION ALL+ SELECT format('to_tsvector(''simple'', body) @@ to_tsquery(''simple'', %L)', w)+ FROM unnest(ARRAY['a', 'b', 'c', 'd', 'one', 'two', 'three', '7']) w) q(q)+ WHERE gin_hot_count_via(true, q) <> gin_hot_count_via(false, q)+$$;+BEGIN;+-- same keys: HOT+UPDATE gin_hot SET tags = ARRAY[tags[3], tags[1], tags[2], tags[1]];+SELECT * FROM gin_hot_count();+ updated | hot +---------+-----+ 100 | 100+(1 row)++UPDATE gin_hot SET tsv = to_tsvector('simple', 'c b a a ' || id);+SELECT * FROM gin_hot_count();+ updated | hot +---------+-----+ 200 | 200+(1 row)++UPDATE gin_hot SET body = id || ' two one one';+SELECT * FROM gin_hot_count();+ updated | hot +---------+-----+ 300 | 300+(1 row)++-- different keys: not HOT+UPDATE gin_hot SET tags = tags || 0 WHERE id <= 10;+SELECT * FROM gin_hot_count();+ updated | hot +---------+-----+ 310 | 301+(1 row)++UPDATE gin_hot SET tsv = to_tsvector('simple', 'a b d ' || id) WHERE id <= 10;+SELECT * FROM gin_hot_count();+ updated | hot +---------+-----+ 320 | 301+(1 row)++UPDATE gin_hot SET body = 'three ' || body WHERE id <= 10;+SELECT * FROM gin_hot_count();+ updated | hot +---------+-----+ 330 | 301+(1 row)++-- to and from no keys, and null+UPDATE gin_hot SET tags = '{}' WHERE id = 11;+UPDATE gin_hot SET tags = NULL WHERE id = 11;+UPDATE gin_hot SET tags = '{1}' WHERE id = 11;+SELECT * FROM gin_hot_count();+ updated | hot +---------+-----+ 333 | 301+(1 row)++COMMIT;+SELECT gin_hot_mismatches();+ gin_hot_mismatches +--------------------+ 0+(1 row)++SELECT gin_index_check('gin_hot_tags');+ gin_index_check +-----------------+ +(1 row)++SELECT gin_index_check('gin_hot_tsv');+ gin_index_check +-----------------+ +(1 row)++SELECT gin_index_check('gin_hot_body');+ gin_index_check +-----------------+ +(1 row)++DROP TABLE gin_hot;+DROP FUNCTION gin_hot_count;+DROP FUNCTION gin_hot_mismatches;+DROP FUNCTION gin_hot_count_via;diff --git a/contrib/amcheck/sql/check_gin.sql b/contrib/amcheck/sql/check_gin.sqlindex 11caed3d6a8..a4b529ba200 100644--- a/contrib/amcheck/sql/check_gin.sql+++ b/contrib/amcheck/sql/check_gin.sql@@ -60,3 +60,74 @@ SELECT gin_index_check('gin_check_multicolumn_idx');
-- cleanup
DROP TABLE gin_check_multicolumn;
++-- An UPDATE that leaves a GIN index's keys for the row as they are is HOT:+-- reordering or repeating array elements, or changing the positions in a+-- tsvector but not its lexemes. Scans through the index must still find+-- exactly the rows a sequential scan finds.+CREATE TABLE gin_hot (id int, tags int[], tsv tsvector, body text)+ WITH (fillfactor = 10);+CREATE INDEX gin_hot_tags ON gin_hot USING gin (tags);+CREATE INDEX gin_hot_tsv ON gin_hot USING gin (tsv);+CREATE INDEX gin_hot_body ON gin_hot USING gin (to_tsvector('simple', body));+INSERT INTO gin_hot+ SELECT g, ARRAY[g % 7, g % 11, g % 13], to_tsvector('simple', 'a b c ' || g),+ 'one two ' || g+ FROM generate_series(1, 100) g;+CREATE FUNCTION gin_hot_count(OUT updated bigint, OUT hot bigint)+LANGUAGE sql AS $$+ SELECT pg_stat_get_xact_tuples_updated('gin_hot'::regclass),+ pg_stat_get_xact_tuples_hot_updated('gin_hot'::regclass) $$;+-- How many rows each tag, lexeme and word matches through the index (a+-- bitmap scan, as GIN has no other) and not (a sequential scan), and the+-- number of mismatches.+CREATE FUNCTION gin_hot_count_via(via_index bool, q text) RETURNS bigint+LANGUAGE plpgsql AS $$+DECLARE c bigint;+BEGIN+ PERFORM set_config('enable_seqscan', CASE WHEN via_index THEN 'off' ELSE 'on' END, true);+ PERFORM set_config('enable_bitmapscan', CASE WHEN via_index THEN 'on' ELSE 'off' END, true);+ PERFORM set_config('enable_indexscan', 'off', true);+ EXECUTE 'SELECT count(*) FROM gin_hot WHERE ' || q INTO c;+ RETURN c;+END $$;+CREATE FUNCTION gin_hot_mismatches() RETURNS bigint LANGUAGE sql AS $$+ SELECT count(*) FROM+ (SELECT format('tags @> ARRAY[%s]', k) FROM generate_series(0, 13) k+ UNION ALL+ SELECT format('tsv @@ to_tsquery(''simple'', %L)', w)+ FROM unnest(ARRAY['a', 'b', 'c', 'd', 'one', 'two', 'three', '7']) w+ UNION ALL+ SELECT format('to_tsvector(''simple'', body) @@ to_tsquery(''simple'', %L)', w)+ FROM unnest(ARRAY['a', 'b', 'c', 'd', 'one', 'two', 'three', '7']) w) q(q)+ WHERE gin_hot_count_via(true, q) <> gin_hot_count_via(false, q)+$$;+BEGIN;+-- same keys: HOT+UPDATE gin_hot SET tags = ARRAY[tags[3], tags[1], tags[2], tags[1]];+SELECT * FROM gin_hot_count();+UPDATE gin_hot SET tsv = to_tsvector('simple', 'c b a a ' || id);+SELECT * FROM gin_hot_count();+UPDATE gin_hot SET body = id || ' two one one';+SELECT * FROM gin_hot_count();+-- different keys: not HOT+UPDATE gin_hot SET tags = tags || 0 WHERE id <= 10;+SELECT * FROM gin_hot_count();+UPDATE gin_hot SET tsv = to_tsvector('simple', 'a b d ' || id) WHERE id <= 10;+SELECT * FROM gin_hot_count();+UPDATE gin_hot SET body = 'three ' || body WHERE id <= 10;+SELECT * FROM gin_hot_count();+-- to and from no keys, and null+UPDATE gin_hot SET tags = '{}' WHERE id = 11;+UPDATE gin_hot SET tags = NULL WHERE id = 11;+UPDATE gin_hot SET tags = '{1}' WHERE id = 11;+SELECT * FROM gin_hot_count();+COMMIT;+SELECT gin_hot_mismatches();+SELECT gin_index_check('gin_hot_tags');+SELECT gin_index_check('gin_hot_tsv');+SELECT gin_index_check('gin_hot_body');+DROP TABLE gin_hot;+DROP FUNCTION gin_hot_count;+DROP FUNCTION gin_hot_mismatches;+DROP FUNCTION gin_hot_count_via;diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.cindex b57c016aea2..8fbdb35e6af 100644--- a/src/backend/access/gin/ginutil.c+++ b/src/backend/access/gin/ginutil.c@@ -23,6 +23,7 @@
#include "commands/progress.h"
#include "commands/vacuum.h"
#include "miscadmin.h"
+#include "nodes/execnodes.h"
#include "storage/indexfsm.h"
#include "utils/builtins.h"
#include "utils/index_selfuncs.h"
@@ -90,7 +91,7 @@ ginhandler(PG_FUNCTION_ARGS)
.amestimateparallelscan = NULL,
.aminitparallelscan = NULL,
.amparallelrescan = NULL,
- .amcomparedatums = NULL,+ .amcomparedatums = gincomparedatums,
};
PG_RETURN_POINTER(&amroutine);
@@ -680,3 +681,61 @@ ginbuildphasename(int64 phasenum)
return NULL;
}
}
++/*+ * gincomparedatums() -- would newValue give the same entries as oldValue?+ *+ * A GIN index stores, for each item, the keys its operator class extracts,+ * and nothing else about the item: scans see an item only as the set of its+ * keys, which is all the consistent function is shown and all a lossy match+ * leaves to recheck. So two values give the same entries if they have the+ * same keys, as GIN compares them. Keys equal under the compare function+ * share one entry in the index whichever of them came first, so the compare+ * function is the right test. Then, for example, reordering or repeating+ * the elements of an array, or changing a tsvector's positions but not its+ * lexemes, makes no new entries.+ */+bool+gincomparedatums(Relation index, IndexInfo *indexInfo, int attnum,+ Oid typid, int16 typlen, bool typbyval,+ Datum oldValue, Datum newValue)+{+ GinState *ginstate = (GinState *) indexInfo->ii_AmCache;+ Datum *oldEntries;+ Datum *newEntries;+ GinNullCategory *oldCategories;+ GinNullCategory *newCategories;+ int32 noldEntries;+ int32 nnewEntries;++ /* the same cache as gininsert(), for the same statement */+ if (ginstate == NULL)+ {+ MemoryContext oldCtx = MemoryContextSwitchTo(indexInfo->ii_Context);++ ginstate = palloc_object(GinState);+ initGinState(ginstate, index);+ indexInfo->ii_AmCache = ginstate;+ MemoryContextSwitchTo(oldCtx);+ }++ oldEntries = ginExtractEntries(ginstate, (OffsetNumber) attnum,+ oldValue, false,+ &noldEntries, &oldCategories);+ newEntries = ginExtractEntries(ginstate, (OffsetNumber) attnum,+ newValue, false,+ &nnewEntries, &newCategories);++ if (noldEntries != nnewEntries)+ return false;+ for (int32 i = 0; i < noldEntries; i++)+ {+ /* there could be many entries, so be willing to abort here */+ CHECK_FOR_INTERRUPTS();+ if (ginCompareEntries(ginstate, (OffsetNumber) attnum,+ oldEntries[i], oldCategories[i],+ newEntries[i], newCategories[i]) != 0)+ return false;+ }+ return true;+}diff --git a/src/include/access/gin_private.h b/src/include/access/gin_private.hindex 3c5fd6ba817..115f29edb6f 100644--- a/src/include/access/gin_private.h+++ b/src/include/access/gin_private.h@@ -106,6 +106,9 @@ extern OffsetNumber gintuple_get_attrnum(GinState *ginstate, IndexTuple tuple);
extern Datum gintuple_get_key(GinState *ginstate, IndexTuple tuple,
GinNullCategory *category);
extern char *ginbuildphasename(int64 phasenum);
+extern bool gincomparedatums(Relation index, struct IndexInfo *indexInfo,+ int attnum, Oid typid, int16 typlen,+ bool typbyval, Datum oldValue, Datum newValue);
/* gininsert.c */
extern IndexBuildResult *ginbuild(Relation heap, Relation index,
--
2.50.1
v43-0001-Improve-test-coverage-for-heap-only-tuple-HOT-up.patch (text/plain, 65.6K) download
From 481ba1507e6cad3f04110c2ed06cc19382e6aa63 Mon Sep 17 00:00:00 2001
From: Greg Burd <greg@burd.me>
Date: Tue, 10 Mar 2026 09:28:15 -0400
Subject: [PATCH v43 1/8] Improve test coverage for heap-only tuple (HOT)
update behavior
Add regression coverage for the existing Heap-Only Tuple (HOT) update
rules, so that later changes to them show up as changes to the expected
output. The new hot_updates regression test covers:
- HOT and non-HOT updates- summarizing (BRIN) indexes, which do not block HOT- TOAST columns- partial, multi-column and unique indexes- partitioned tables- expression indexes whose input changes while the expression's value
does or does not, including expressions stored as another type
- partial indexes, with the row inside, outside, entering or leaving the
predicate
- BEFORE ROW triggers that change an indexed column
A hot_updates test in contrib/pageinspect checks the HOT chain an update
leaves on the page, and that VACUUM collapses it.
Discussion: https://postgr.es/m/78574B24-BE0A-42C5-8075-3FA9FA63B8FC@amazon.com
---
contrib/pageinspect/Makefile | 3 +-
contrib/pageinspect/expected/hot_updates.out | 130 ++++
contrib/pageinspect/meson.build | 1 +
contrib/pageinspect/sql/hot_updates.sql | 112 +++
.../isolation/expected/update-epq-indexes.out | 94 +++
src/test/isolation/isolation_schedule | 1 +
.../isolation/specs/update-epq-indexes.spec | 55 ++
src/test/regress/expected/hot_updates.out | 723 ++++++++++++++++++
src/test/regress/parallel_schedule | 2 +-
src/test/regress/sql/hot_updates.sql | 468 ++++++++++++
10 files changed, 1587 insertions(+), 2 deletions(-)
create mode 100644 contrib/pageinspect/expected/hot_updates.out
create mode 100644 contrib/pageinspect/sql/hot_updates.sql
create mode 100644 src/test/isolation/expected/update-epq-indexes.out
create mode 100644 src/test/isolation/specs/update-epq-indexes.spec
create mode 100644 src/test/regress/expected/hot_updates.out
create mode 100644 src/test/regress/sql/hot_updates.sql
diff --git a/contrib/pageinspect/Makefile b/contrib/pageinspect/Makefileindex eae989569d0..231bd82137e 100644--- a/contrib/pageinspect/Makefile+++ b/contrib/pageinspect/Makefile@@ -32,7 +32,8 @@ REGRESS = \
gin \
gist \
hash \
- oldextversions+ oldextversions \+ hot_updates
ifdef USE_PGXS
PG_CONFIG = pg_config
diff --git a/contrib/pageinspect/expected/hot_updates.out b/contrib/pageinspect/expected/hot_updates.outnew file mode 100644index 00000000000..9ce6a34f4e3--- /dev/null+++ b/contrib/pageinspect/expected/hot_updates.out@@ -0,0 +1,130 @@+--+-- HOT_UPDATES+-- The HOT chain an update leaves on the page. src/test/regress's+-- hot_updates test covers which updates are HOT.+--+-- oldextversions, which runs just before this, drops the extension; this+-- creates it again and drops it at the end, leaving things as they were+CREATE EXTENSION pageinspect;+-- Sum of committed and in-progress (non-HOT, HOT) update counters.+CREATE FUNCTION get_hot_count(rel_name text)+RETURNS TABLE (+ updates BIGINT,+ hot BIGINT+) AS $$+DECLARE+ rel_oid oid;+BEGIN+ rel_oid := rel_name::regclass::oid;+ updates := COALESCE(pg_stat_get_tuples_updated(rel_oid), 0) ++ COALESCE(pg_stat_get_xact_tuples_updated(rel_oid), 0);+ hot := COALESCE(pg_stat_get_tuples_hot_updated(rel_oid), 0) ++ COALESCE(pg_stat_get_xact_tuples_hot_updated(rel_oid), 0);+ RETURN NEXT;+END;+$$ LANGUAGE plpgsql;+-- True iff another live tuple on the same page has its t_ctid pointing at+-- target_ctid, i.e. target_ctid is a later member of a HOT chain.+CREATE FUNCTION has_hot_chain(rel_name text, target_ctid tid)+RETURNS boolean LANGUAGE sql AS $$+ SELECT EXISTS (+ SELECT FROM heap_page_items(get_raw_page(rel_name,+ tid_block(target_ctid)::int))+ WHERE lp_flags = 1 AND t_ctid = target_ctid+ AND lp <> tid_offset(target_ctid))+$$;+-- The HOT chain through start_ctid, from the live tuple pointing at it if+-- there is one, following t_ctid on the page until a tuple points at itself.+CREATE FUNCTION print_hot_chain(rel_name text, start_ctid tid)+RETURNS TABLE (chain_position int, ctid tid, lp_flags text, t_ctid tid)+LANGUAGE sql AS $$+ WITH RECURSIVE items AS (+ SELECT * FROM heap_page_items(get_raw_page(rel_name,+ tid_block(start_ctid)::int))+ ), chain (chain_position, lp) AS (+ SELECT 0, coalesce((SELECT lp FROM items+ WHERE lp_flags = 1 AND t_ctid = start_ctid+ AND lp <> tid_offset(start_ctid)),+ tid_offset(start_ctid))+ UNION ALL+ SELECT c.chain_position + 1, tid_offset(i.t_ctid)+ FROM chain c JOIN items i ON i.lp = c.lp+ WHERE tid_block(i.t_ctid) = tid_block(start_ctid)+ AND tid_offset(i.t_ctid) <> c.lp+ )+ SELECT c.chain_position,+ format('(%s,%s)', tid_block(start_ctid), c.lp)::tid,+ CASE i.lp_flags WHEN 0 THEN 'unused (0)' WHEN 1 THEN 'normal (1)'+ WHEN 2 THEN 'redirect (2)' WHEN 3 THEN 'dead (3)' END,+ i.t_ctid+ FROM chain c JOIN items i ON i.lp = c.lp+ ORDER BY c.chain_position+$$;+-- ---------------------------------------------------------------------------+-- Basic HOT: update of a non-indexed column+-- ---------------------------------------------------------------------------+-- (No autovacuum: the chain printed below must not be pruned first.)+CREATE TABLE hot_test (+ id int PRIMARY KEY,+ indexed_col int,+ non_indexed_col text+) WITH (fillfactor = 50, autovacuum_enabled = off);+CREATE INDEX hot_test_indexed_idx ON hot_test(indexed_col);+INSERT INTO hot_test VALUES (1, 100, 'initial');+INSERT INTO hot_test VALUES (2, 200, 'initial');+INSERT INTO hot_test VALUES (3, 300, 'initial');+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++SELECT * FROM get_hot_count('hot_test');+ updates | hot +---------+-----+ 0 | 0+(1 row)++-- Three classic HOT updates (non-indexed col).+UPDATE hot_test SET non_indexed_col = 'updated1' WHERE id = 1;+UPDATE hot_test SET non_indexed_col = 'updated2' WHERE id = 2;+UPDATE hot_test SET non_indexed_col = 'updated3' WHERE id = 3;+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++SELECT * FROM get_hot_count('hot_test');+ updates | hot +---------+-----+ 3 | 3+(1 row)++-- Chain-of-1 on id=1 still has a predecessor line pointer.+WITH current_tuple AS (SELECT ctid FROM hot_test WHERE id = 1)+SELECT has_hot_chain('hot_test', current_tuple.ctid) AS has_chain,+ chain_position, print_hot_chain.ctid, lp_flags, t_ctid+FROM current_tuple, LATERAL print_hot_chain('hot_test', current_tuple.ctid);+ has_chain | chain_position | ctid | lp_flags | t_ctid +-----------+----------------+-------+------------+--------+ t | 0 | (0,1) | normal (1) | (0,4)+ t | 1 | (0,4) | normal (1) | (0,4)+(2 rows)++-- VACUUM collapses the chain.+VACUUM hot_test;+WITH current_tuple AS (SELECT ctid FROM hot_test WHERE id = 1)+SELECT has_hot_chain('hot_test', current_tuple.ctid) AS has_chain,+ chain_position, print_hot_chain.ctid, lp_flags, t_ctid+FROM current_tuple, LATERAL print_hot_chain('hot_test', current_tuple.ctid);+ has_chain | chain_position | ctid | lp_flags | t_ctid +-----------+----------------+-------+------------+--------+ f | 0 | (0,4) | normal (1) | (0,4)+(1 row)++DROP TABLE hot_test;+DROP FUNCTION has_hot_chain(text, tid);+DROP FUNCTION print_hot_chain(text, tid);+DROP FUNCTION get_hot_count(text);+DROP EXTENSION pageinspect;diff --git a/contrib/pageinspect/meson.build b/contrib/pageinspect/meson.buildindex c43ea400a4d..105a5b7d68e 100644--- a/contrib/pageinspect/meson.build+++ b/contrib/pageinspect/meson.build@@ -56,6 +56,7 @@ tests += {
'hash',
'checksum',
'oldextversions',
+ 'hot_updates',
],
},
}
diff --git a/contrib/pageinspect/sql/hot_updates.sql b/contrib/pageinspect/sql/hot_updates.sqlnew file mode 100644index 00000000000..14df5bdb180--- /dev/null+++ b/contrib/pageinspect/sql/hot_updates.sql@@ -0,0 +1,112 @@+--+-- HOT_UPDATES+-- The HOT chain an update leaves on the page. src/test/regress's+-- hot_updates test covers which updates are HOT.+--++-- oldextversions, which runs just before this, drops the extension; this+-- creates it again and drops it at the end, leaving things as they were+CREATE EXTENSION pageinspect;++-- Sum of committed and in-progress (non-HOT, HOT) update counters.+CREATE FUNCTION get_hot_count(rel_name text)+RETURNS TABLE (+ updates BIGINT,+ hot BIGINT+) AS $$+DECLARE+ rel_oid oid;+BEGIN+ rel_oid := rel_name::regclass::oid;+ updates := COALESCE(pg_stat_get_tuples_updated(rel_oid), 0) ++ COALESCE(pg_stat_get_xact_tuples_updated(rel_oid), 0);+ hot := COALESCE(pg_stat_get_tuples_hot_updated(rel_oid), 0) ++ COALESCE(pg_stat_get_xact_tuples_hot_updated(rel_oid), 0);+ RETURN NEXT;+END;+$$ LANGUAGE plpgsql;++-- True iff another live tuple on the same page has its t_ctid pointing at+-- target_ctid, i.e. target_ctid is a later member of a HOT chain.+CREATE FUNCTION has_hot_chain(rel_name text, target_ctid tid)+RETURNS boolean LANGUAGE sql AS $$+ SELECT EXISTS (+ SELECT FROM heap_page_items(get_raw_page(rel_name,+ tid_block(target_ctid)::int))+ WHERE lp_flags = 1 AND t_ctid = target_ctid+ AND lp <> tid_offset(target_ctid))+$$;++-- The HOT chain through start_ctid, from the live tuple pointing at it if+-- there is one, following t_ctid on the page until a tuple points at itself.+CREATE FUNCTION print_hot_chain(rel_name text, start_ctid tid)+RETURNS TABLE (chain_position int, ctid tid, lp_flags text, t_ctid tid)+LANGUAGE sql AS $$+ WITH RECURSIVE items AS (+ SELECT * FROM heap_page_items(get_raw_page(rel_name,+ tid_block(start_ctid)::int))+ ), chain (chain_position, lp) AS (+ SELECT 0, coalesce((SELECT lp FROM items+ WHERE lp_flags = 1 AND t_ctid = start_ctid+ AND lp <> tid_offset(start_ctid)),+ tid_offset(start_ctid))+ UNION ALL+ SELECT c.chain_position + 1, tid_offset(i.t_ctid)+ FROM chain c JOIN items i ON i.lp = c.lp+ WHERE tid_block(i.t_ctid) = tid_block(start_ctid)+ AND tid_offset(i.t_ctid) <> c.lp+ )+ SELECT c.chain_position,+ format('(%s,%s)', tid_block(start_ctid), c.lp)::tid,+ CASE i.lp_flags WHEN 0 THEN 'unused (0)' WHEN 1 THEN 'normal (1)'+ WHEN 2 THEN 'redirect (2)' WHEN 3 THEN 'dead (3)' END,+ i.t_ctid+ FROM chain c JOIN items i ON i.lp = c.lp+ ORDER BY c.chain_position+$$;++-- ---------------------------------------------------------------------------+-- Basic HOT: update of a non-indexed column+-- ---------------------------------------------------------------------------+-- (No autovacuum: the chain printed below must not be pruned first.)+CREATE TABLE hot_test (+ id int PRIMARY KEY,+ indexed_col int,+ non_indexed_col text+) WITH (fillfactor = 50, autovacuum_enabled = off);+CREATE INDEX hot_test_indexed_idx ON hot_test(indexed_col);++INSERT INTO hot_test VALUES (1, 100, 'initial');+INSERT INTO hot_test VALUES (2, 200, 'initial');+INSERT INTO hot_test VALUES (3, 300, 'initial');++SELECT pg_stat_force_next_flush();+SELECT * FROM get_hot_count('hot_test');++-- Three classic HOT updates (non-indexed col).+UPDATE hot_test SET non_indexed_col = 'updated1' WHERE id = 1;+UPDATE hot_test SET non_indexed_col = 'updated2' WHERE id = 2;+UPDATE hot_test SET non_indexed_col = 'updated3' WHERE id = 3;+SELECT pg_stat_force_next_flush();+SELECT * FROM get_hot_count('hot_test');++-- Chain-of-1 on id=1 still has a predecessor line pointer.+WITH current_tuple AS (SELECT ctid FROM hot_test WHERE id = 1)+SELECT has_hot_chain('hot_test', current_tuple.ctid) AS has_chain,+ chain_position, print_hot_chain.ctid, lp_flags, t_ctid+FROM current_tuple, LATERAL print_hot_chain('hot_test', current_tuple.ctid);++-- VACUUM collapses the chain.+VACUUM hot_test;++WITH current_tuple AS (SELECT ctid FROM hot_test WHERE id = 1)+SELECT has_hot_chain('hot_test', current_tuple.ctid) AS has_chain,+ chain_position, print_hot_chain.ctid, lp_flags, t_ctid+FROM current_tuple, LATERAL print_hot_chain('hot_test', current_tuple.ctid);++DROP TABLE hot_test;++DROP FUNCTION has_hot_chain(text, tid);+DROP FUNCTION print_hot_chain(text, tid);+DROP FUNCTION get_hot_count(text);+DROP EXTENSION pageinspect;diff --git a/src/test/isolation/expected/update-epq-indexes.out b/src/test/isolation/expected/update-epq-indexes.outnew file mode 100644index 00000000000..3a28bffa596--- /dev/null+++ b/src/test/isolation/expected/update-epq-indexes.out@@ -0,0 +1,94 @@+Parsed test spec with 2 sessions++starting permutation: s2_begin s2_status s1_status s2_commit s1_check+step s2_begin: BEGIN;+step s2_status: UPDATE hot_epq SET status = 'b' WHERE id = 1;+step s1_status: UPDATE hot_epq SET status = 'a' WHERE id = 1; <waiting ...>+step s2_commit: COMMIT;+step s1_status: <... completed>+step s1_check: + SET enable_seqscan = off;+ SET enable_bitmapscan = off;+ SELECT count(*) AS status_a FROM hot_epq WHERE status = 'a';+ SELECT count(*) AS doc_a FROM hot_epq WHERE doc->>'status' = 'a';+ SELECT count(*) AS vip FROM hot_epq WHERE vip AND id = 1;+ RESET enable_seqscan;+ RESET enable_bitmapscan;++status_a+--------+ 1+(1 row)++doc_a+-----+ 1+(1 row)++vip+---+ 1+(1 row)+++starting permutation: s2_begin s2_doc s1_doc s2_commit s1_check+step s2_begin: BEGIN;+step s2_doc: UPDATE hot_epq SET doc = jsonb_set(doc, ARRAY['status'], to_jsonb('b'::text)) WHERE id = 1;+step s1_doc: UPDATE hot_epq SET doc = jsonb_set(doc, ARRAY['status'], to_jsonb('a'::text)) WHERE id = 1; <waiting ...>+step s2_commit: COMMIT;+step s1_doc: <... completed>+step s1_check: + SET enable_seqscan = off;+ SET enable_bitmapscan = off;+ SELECT count(*) AS status_a FROM hot_epq WHERE status = 'a';+ SELECT count(*) AS doc_a FROM hot_epq WHERE doc->>'status' = 'a';+ SELECT count(*) AS vip FROM hot_epq WHERE vip AND id = 1;+ RESET enable_seqscan;+ RESET enable_bitmapscan;++status_a+--------+ 1+(1 row)++doc_a+-----+ 1+(1 row)++vip+---+ 1+(1 row)+++starting permutation: s2_begin s2_vip s1_vip s2_commit s1_check+step s2_begin: BEGIN;+step s2_vip: UPDATE hot_epq SET vip = false WHERE id = 1;+step s1_vip: UPDATE hot_epq SET vip = true WHERE id = 1; <waiting ...>+step s2_commit: COMMIT;+step s1_vip: <... completed>+step s1_check: + SET enable_seqscan = off;+ SET enable_bitmapscan = off;+ SELECT count(*) AS status_a FROM hot_epq WHERE status = 'a';+ SELECT count(*) AS doc_a FROM hot_epq WHERE doc->>'status' = 'a';+ SELECT count(*) AS vip FROM hot_epq WHERE vip AND id = 1;+ RESET enable_seqscan;+ RESET enable_bitmapscan;++status_a+--------+ 1+(1 row)++doc_a+-----+ 1+(1 row)++vip+---+ 1+(1 row)+diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_scheduleindex bf9a2037c70..d7da899aa85 100644--- a/src/test/isolation/isolation_schedule+++ b/src/test/isolation/isolation_schedule@@ -43,6 +43,7 @@ test: fk-fastpath-null-key
test: subxid-overflow
test: eval-plan-qual
test: eval-plan-qual-trigger
+test: update-epq-indexes
test: eval-plan-qual-partition-prune
test: inplace-inval
test: intra-grant-inplace
diff --git a/src/test/isolation/specs/update-epq-indexes.spec b/src/test/isolation/specs/update-epq-indexes.specnew file mode 100644index 00000000000..09420c1a060--- /dev/null+++ b/src/test/isolation/specs/update-epq-indexes.spec@@ -0,0 +1,55 @@+# UPDATEs whose target row is updated concurrently, with a BEFORE ROW UPDATE+# trigger. The trigger code locks the latest version and rebuilds the new+# row from it, so deciding which indexes the UPDATE changes must compare the+# new row with that latest version, not with the one the UPDATE first read.+#+# The row starts with status a, inside the partial index. s2 changes it and+# holds the lock; s1 then sets it back and waits. Compared with the version+# s1 first read, s1 would change nothing, and every index would keep the+# entry for the version s2 wrote.++setup+{+ CREATE TABLE hot_epq (id int PRIMARY KEY, status text, doc jsonb, vip bool)+ WITH (fillfactor = 50);+ CREATE INDEX hot_epq_status ON hot_epq (status);+ CREATE INDEX hot_epq_doc ON hot_epq ((doc->>'status'));+ CREATE INDEX hot_epq_vip ON hot_epq (id) WHERE vip;+ INSERT INTO hot_epq VALUES (1, 'a', jsonb_build_object('status', 'a'), true);+ CREATE FUNCTION hot_epq_trig() RETURNS trigger LANGUAGE plpgsql AS+ $$BEGIN RETURN NEW; END$$;+ CREATE TRIGGER hot_epq_before BEFORE UPDATE ON hot_epq+ FOR EACH ROW EXECUTE FUNCTION hot_epq_trig();+}++teardown+{+ DROP TABLE hot_epq;+ DROP FUNCTION hot_epq_trig();+}++session s1+step s1_status { UPDATE hot_epq SET status = 'a' WHERE id = 1; }+step s1_doc { UPDATE hot_epq SET doc = jsonb_set(doc, ARRAY['status'], to_jsonb('a'::text)) WHERE id = 1; }+step s1_vip { UPDATE hot_epq SET vip = true WHERE id = 1; }+step s1_check+{+ SET enable_seqscan = off;+ SET enable_bitmapscan = off;+ SELECT count(*) AS status_a FROM hot_epq WHERE status = 'a';+ SELECT count(*) AS doc_a FROM hot_epq WHERE doc->>'status' = 'a';+ SELECT count(*) AS vip FROM hot_epq WHERE vip AND id = 1;+ RESET enable_seqscan;+ RESET enable_bitmapscan;+}++session s2+step s2_begin { BEGIN; }+step s2_status { UPDATE hot_epq SET status = 'b' WHERE id = 1; }+step s2_doc { UPDATE hot_epq SET doc = jsonb_set(doc, ARRAY['status'], to_jsonb('b'::text)) WHERE id = 1; }+step s2_vip { UPDATE hot_epq SET vip = false WHERE id = 1; }+step s2_commit { COMMIT; }++permutation s2_begin s2_status s1_status s2_commit s1_check+permutation s2_begin s2_doc s1_doc s2_commit s1_check+permutation s2_begin s2_vip s1_vip s2_commit s1_checkdiff --git a/src/test/regress/expected/hot_updates.out b/src/test/regress/expected/hot_updates.outnew file mode 100644index 00000000000..f83a686f3a9--- /dev/null+++ b/src/test/regress/expected/hot_updates.out@@ -0,0 +1,723 @@+--+-- HOT_UPDATES+-- Test Heap-Only Tuple (HOT) update decisions+--+-- Expected output records which UPDATEs are HOT, so that a change to the+-- rules shows up as a change to this file. contrib/pageinspect's+-- hot_updates test looks at the HOT chain itself.+--+-- Sum of committed and in-progress (non-HOT, HOT) update counters.+CREATE FUNCTION get_hot_count(rel_name text)+RETURNS TABLE (+ updates BIGINT,+ hot BIGINT+) AS $$+DECLARE+ rel_oid oid;+BEGIN+ rel_oid := rel_name::regclass::oid;+ updates := COALESCE(pg_stat_get_tuples_updated(rel_oid), 0) ++ COALESCE(pg_stat_get_xact_tuples_updated(rel_oid), 0);+ hot := COALESCE(pg_stat_get_tuples_hot_updated(rel_oid), 0) ++ COALESCE(pg_stat_get_xact_tuples_hot_updated(rel_oid), 0);+ RETURN NEXT;+END;+$$ LANGUAGE plpgsql;+-- ---------------------------------------------------------------------------+-- Summarizing indexes (BRIN) do not block HOT+-- ---------------------------------------------------------------------------+CREATE TABLE hot_test (+ id int PRIMARY KEY,+ ts timestamp,+ value int,+ brin_col int+) WITH (fillfactor = 50);+CREATE INDEX hot_test_ts_brin ON hot_test USING brin(ts);+CREATE INDEX hot_test_brin_col_brin ON hot_test USING brin(brin_col);+INSERT INTO hot_test VALUES (1, '2024-01-01', 100, 1000);+-- BRIN columns are summarizing; updating them stays classic HOT even+-- though their values change.+UPDATE hot_test SET ts = '2024-01-02', brin_col = 2000 WHERE id = 1;+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++SELECT * FROM get_hot_count('hot_test');+ updates | hot +---------+-----+ 1 | 1+(1 row)++-- Non-indexed column: also HOT.+UPDATE hot_test SET value = 200 WHERE id = 1;+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++SELECT * FROM get_hot_count('hot_test');+ updates | hot +---------+-----+ 2 | 2+(1 row)++DROP TABLE hot_test;+-- ---------------------------------------------------------------------------+-- TOAST participates in HOT (non-indexed column paths only)+-- ---------------------------------------------------------------------------+CREATE TABLE hot_test (+ id int PRIMARY KEY,+ indexed_col int,+ large_text text,+ small_text text+) WITH (fillfactor = 50);+CREATE INDEX hot_test_idx ON hot_test(indexed_col);+-- stored out of line at any block size+ALTER TABLE hot_test ALTER COLUMN large_text SET STORAGE EXTERNAL;+SELECT current_setting('block_size')::int / 2 AS large_len \gset+INSERT INTO hot_test VALUES (1, 100, repeat('x', :large_len), 'small');+SELECT pg_column_toast_chunk_id(large_text) IS NOT NULL AS toasted+ FROM hot_test;+ toasted +---------+ t+(1 row)++-- Non-indexed, non-TOAST column: HOT.+UPDATE hot_test SET small_text = 'updated';+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++SELECT * FROM get_hot_count('hot_test');+ updates | hot +---------+-----+ 1 | 1+(1 row)++-- TOAST column, indexed_col unchanged: HOT.+UPDATE hot_test SET large_text = repeat('y', :large_len);+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++SELECT * FROM get_hot_count('hot_test');+ updates | hot +---------+-----+ 2 | 2+(1 row)++DROP TABLE hot_test;+-- ---------------------------------------------------------------------------+-- Partial index where update leaves indexed attrs unchanged+-- ---------------------------------------------------------------------------+CREATE TABLE hot_test (+ id int PRIMARY KEY,+ status text,+ data text+) WITH (fillfactor = 50);+CREATE INDEX hot_test_active_idx ON hot_test(status) WHERE status = 'active';+INSERT INTO hot_test VALUES (1, 'active', 'data1');+INSERT INTO hot_test VALUES (2, 'inactive', 'data2');+INSERT INTO hot_test VALUES (3, 'deleted', 'data3');+-- Update data on a row whose status matches the partial predicate: HOT.+UPDATE hot_test SET data = 'updated1' WHERE id = 1;+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++SELECT * FROM get_hot_count('hot_test');+ updates | hot +---------+-----+ 1 | 1+(1 row)++-- Update data on a row outside the predicate: HOT.+UPDATE hot_test SET data = 'updated2' WHERE id = 2;+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++SELECT * FROM get_hot_count('hot_test');+ updates | hot +---------+-----+ 2 | 2+(1 row)++SELECT id, status FROM hot_test WHERE status = 'active';+ id | status +----+--------+ 1 | active+(1 row)++DROP TABLE hot_test;+-- ---------------------------------------------------------------------------+-- Multi-column btree: update of non-indexed column+-- ---------------------------------------------------------------------------+CREATE TABLE hot_test (+ id int PRIMARY KEY,+ col_a int,+ col_b int,+ col_c int,+ data text+) WITH (fillfactor = 50);+CREATE INDEX hot_test_ab_idx ON hot_test(col_a, col_b);+INSERT INTO hot_test VALUES (1, 10, 20, 30, 'data');+-- col_c not in any index: HOT.+UPDATE hot_test SET col_c = 35;+-- data not in any index: HOT.+UPDATE hot_test SET data = 'updated';+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++SELECT * FROM get_hot_count('hot_test');+ updates | hot +---------+-----+ 2 | 2+(1 row)++DROP TABLE hot_test;+-- ---------------------------------------------------------------------------+-- Unique index: update of non-indexed column + uniqueness enforcement+-- ---------------------------------------------------------------------------+CREATE TABLE hot_test (+ id int PRIMARY KEY,+ unique_col int UNIQUE,+ data text+) WITH (fillfactor = 50);+INSERT INTO hot_test VALUES (1, 100, 'data1');+INSERT INTO hot_test VALUES (2, 200, 'data2');+UPDATE hot_test SET data = 'updated';+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++SELECT * FROM get_hot_count('hot_test');+ updates | hot +---------+-----+ 2 | 2+(1 row)++SELECT id, unique_col, data FROM hot_test ORDER BY id;+ id | unique_col | data +----+------------+---------+ 1 | 100 | updated+ 2 | 200 | updated+(2 rows)++-- Unique constraint still enforced on any path.+UPDATE hot_test SET unique_col = 100 WHERE id = 2;+ERROR: duplicate key value violates unique constraint "hot_test_unique_col_key"+DETAIL: Key (unique_col)=(100) already exists.+DROP TABLE hot_test;+-- ---------------------------------------------------------------------------+-- Partitioned tables: HOT within a partition+-- ---------------------------------------------------------------------------+CREATE TABLE hot_test_partitioned (+ id int,+ partition_key int,+ indexed_col int,+ data text,+ PRIMARY KEY (id, partition_key)+) PARTITION BY RANGE (partition_key);+CREATE TABLE hot_test_part1 PARTITION OF hot_test_partitioned+ FOR VALUES FROM (1) TO (100) WITH (fillfactor = 50);+CREATE TABLE hot_test_part2 PARTITION OF hot_test_partitioned+ FOR VALUES FROM (100) TO (200) WITH (fillfactor = 50);+CREATE INDEX hot_test_part_idx ON hot_test_partitioned(indexed_col);+INSERT INTO hot_test_partitioned VALUES (1, 50, 100, 'initial1');+INSERT INTO hot_test_partitioned VALUES (2, 150, 200, 'initial2');+UPDATE hot_test_partitioned SET data = 'updated1' WHERE id = 1;+UPDATE hot_test_partitioned SET data = 'updated2' WHERE id = 2;+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++SELECT * FROM get_hot_count('hot_test_part1');+ updates | hot +---------+-----+ 1 | 1+(1 row)++SELECT * FROM get_hot_count('hot_test_part2');+ updates | hot +---------+-----+ 1 | 1+(1 row)++SELECT id FROM hot_test_partitioned WHERE indexed_col = 100;+ id +----+ 1+(1 row)++SELECT id FROM hot_test_partitioned WHERE indexed_col = 200;+ id +----+ 2+(1 row)++DROP TABLE hot_test_partitioned CASCADE;+-- An update through the parent that writes back an indexed column's value+-- is HOT. Routing hands the partition a new tuple whose short text value+-- has not been packed yet; it is still the same value.+CREATE TABLE hot_test_partitioned (id int UNIQUE, tag char, n int)+ PARTITION BY LIST (id);+CREATE TABLE hot_test_part1 PARTITION OF hot_test_partitioned+ (tag UNIQUE) FOR VALUES IN (1, 2) WITH (fillfactor = 50);+INSERT INTO hot_test_partitioned VALUES (1, 'a', 0), (2, 'b', 0);+INSERT INTO hot_test_partitioned VALUES (1, 'a', 1)+ ON CONFLICT (id) DO UPDATE SET tag = excluded.tag, n = excluded.n;+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++SELECT * FROM get_hot_count('hot_test_part1');+ updates | hot +---------+-----+ 1 | 1+(1 row)++DROP TABLE hot_test_partitioned CASCADE;+-- ---------------------------------------------------------------------------+-- Expression index on a JSONB field+-- ---------------------------------------------------------------------------+CREATE TABLE hot_jsonb_test (+ id int PRIMARY KEY,+ data jsonb+) WITH (fillfactor = 50);+CREATE INDEX hot_jsonb_name_idx ON hot_jsonb_test ((data->>'name'));+INSERT INTO hot_jsonb_test VALUES+ (1, '{"name":"Alice","age":30,"city":"NYC"}'),+ (2, '{"name":"Bob","age":25,"city":"LA"}');+-- Each UPDATE changes the indexed column but not the indexed expression.+UPDATE hot_jsonb_test SET data = jsonb_set(data, '{age}', '31') WHERE id = 1;+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++SELECT * FROM get_hot_count('hot_jsonb_test');+ updates | hot +---------+-----+ 1 | 0+(1 row)++UPDATE hot_jsonb_test SET data = data - 'city' WHERE id = 2;+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++SELECT * FROM get_hot_count('hot_jsonb_test');+ updates | hot +---------+-----+ 2 | 0+(1 row)++UPDATE hot_jsonb_test SET data = jsonb_insert(data, '{country}', '"USA"') WHERE id = 2;+SELECT pg_stat_force_next_flush();+ pg_stat_force_next_flush +--------------------------+ +(1 row)++SELECT * FROM get_hot_count('hot_jsonb_test');+ updates | hot +---------+-----+ 3 | 0+(1 row)++DROP TABLE hot_jsonb_test;+-- ---------------------------------------------------------------------------+-- Expression and partial indexes+--+-- update_case() runs one UPDATE on a fresh table and reports how many rows+-- it updated, and how many of those with HOT.+-- ---------------------------------------------------------------------------+CREATE FUNCTION update_case(setup text, upd text,+ OUT updated bigint, OUT hot bigint)+LANGUAGE plpgsql AS $$+BEGIN+ EXECUTE 'DROP TABLE IF EXISTS hot_expr_test';+ EXECUTE setup;+ updated := pg_stat_get_xact_tuples_updated('hot_expr_test'::regclass);+ hot := pg_stat_get_xact_tuples_hot_updated('hot_expr_test'::regclass);+ EXECUTE upd;+ updated := pg_stat_get_xact_tuples_updated('hot_expr_test'::regclass) - updated;+ hot := pg_stat_get_xact_tuples_hot_updated('hot_expr_test'::regclass) - hot;+END;+$$;+-- Expression value unchanged.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, s text) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (lower(s));+ INSERT INTO hot_expr_test SELECT g, 'abc' || g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET s = upper(s)$$);+NOTICE: table "hot_expr_test" does not exist, skipping+ updated | hot +---------+-----+ 10 | 0+(1 row)++SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, a int, b int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test ((a + b));+ INSERT INTO hot_expr_test SELECT g, g, 0 FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET a = a - 1, b = b + 1$$);+ updated | hot +---------+-----+ 10 | 0+(1 row)++-- Expression value changed.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, d jsonb) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test ((d->>'name'));+ INSERT INTO hot_expr_test SELECT g, jsonb_build_object('name', 'n' || g)+ FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET d = jsonb_set(d, '{name}', '"x"')$$);+ updated | hot +---------+-----+ 10 | 0+(1 row)++-- Equal values that are stored differently.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, n numeric) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test ((n + 0));+ INSERT INTO hot_expr_test SELECT g, 3 FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET n = 3.0$$);+ updated | hot +---------+-----+ 10 | 0+(1 row)++-- The index stores a type other than the expression's: name_ops stores+-- cstring, and gist point_ops stores box.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, s text) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test ((lower(s)::name));+ INSERT INTO hot_expr_test SELECT g, 'abc' || g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET s = upper(s)$$);+ updated | hot +---------+-----+ 10 | 0+(1 row)++SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, p point) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test USING gist ((point(p[0], 0)));+ INSERT INTO hot_expr_test SELECT g, point(g, g) FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET p = point(p[0], p[1] + 1)$$);+ updated | hot +---------+-----+ 10 | 0+(1 row)++-- An expression that is a point, stored by the gist index as a box.+-- Comparing it as the box the index stores rather than as the point it is+-- crashed the server in 2018 (2877.1541538838@sss.pgh.pa.us).+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, d smallint) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test USING gist ((point(d::integer % 7, 0)));+ INSERT INTO hot_expr_test SELECT g, 1 FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET d = d + 7$$);+ updated | hot +---------+-----+ 10 | 0+(1 row)++-- A plain index on the same column as an expression index.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, s text) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (lower(s));+ CREATE INDEX ON hot_expr_test (s);+ INSERT INTO hot_expr_test SELECT g, 'abc' || g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET s = upper(s)$$);+ updated | hot +---------+-----+ 10 | 0+(1 row)++-- A BRIN index whose column changed, and an expression index whose value+-- did not.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, s text, b int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (lower(s));+ CREATE INDEX ON hot_expr_test USING brin (b);+ INSERT INTO hot_expr_test SELECT g, 'abc' || g, g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET s = upper(s), b = b + 1$$);+ updated | hot +---------+-----+ 10 | 0+(1 row)++-- A BEFORE ROW trigger that changes the expression's value.+CREATE FUNCTION hot_expr_trig() RETURNS trigger LANGUAGE plpgsql AS+ $$BEGIN NEW.s := NEW.s || 'z'; RETURN NEW; END$$;+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, s text, x int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (lower(s));+ CREATE TRIGGER t BEFORE UPDATE ON hot_expr_test+ FOR EACH ROW EXECUTE FUNCTION hot_expr_trig();+ INSERT INTO hot_expr_test SELECT g, 'abc' || g, 0 FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET x = 1$$);+ updated | hot +---------+-----+ 10 | 0+(1 row)++DROP FUNCTION hot_expr_trig() CASCADE;+NOTICE: drop cascades to trigger t on table hot_expr_test+-- Partial index, both versions outside the predicate, changing the key or+-- the predicate's column.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, status text, v int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (v) WHERE status = 'active';+ INSERT INTO hot_expr_test SELECT g, 'done', g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET v = v + 1$$);+ updated | hot +---------+-----+ 10 | 0+(1 row)++SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, status text, v int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (v) WHERE status = 'active';+ INSERT INTO hot_expr_test SELECT g, 'done', g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET status = 'archived'$$);+ updated | hot +---------+-----+ 10 | 0+(1 row)++-- Both versions inside the predicate with the key unchanged.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, status text, v int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (v) WHERE status <> 'x';+ INSERT INTO hot_expr_test SELECT g, 'a', g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET status = 'b'$$);+ updated | hot +---------+-----+ 10 | 0+(1 row)++-- Entering or leaving the predicate, or changing the key inside it.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, status text, v int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (v) WHERE status = 'active';+ INSERT INTO hot_expr_test SELECT g, 'active', g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET status = 'done'$$);+ updated | hot +---------+-----+ 10 | 0+(1 row)++SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, status text, v int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (v) WHERE status = 'active';+ INSERT INTO hot_expr_test SELECT g, 'done', g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET status = 'active'$$);+ updated | hot +---------+-----+ 10 | 0+(1 row)++SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, status text, v int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (v) WHERE status = 'active';+ INSERT INTO hot_expr_test SELECT g, 'active', g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET v = v + 1000$$);+ updated | hot +---------+-----+ 10 | 0+(1 row)++-- Rows updated outside a partial index must not be found through it.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, status text, v int) WITH (fillfactor = 50);+ CREATE INDEX hot_expr_test_v ON hot_expr_test (v) WHERE status = 'active';+ INSERT INTO hot_expr_test SELECT g, CASE WHEN g <= 5 THEN 'active' ELSE 'done' END, g+ FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET v = v + 100 WHERE status = 'done'$$);+ updated | hot +---------+-----+ 5 | 0+(1 row)++SET enable_seqscan = off;+SET enable_bitmapscan = off;+EXPLAIN (COSTS OFF) SELECT id FROM hot_expr_test WHERE status = 'active' AND v > 0;+ QUERY PLAN +---------------------------------------------------+ Index Scan using hot_expr_test_v on hot_expr_test+ Index Cond: (v > 0)+(2 rows)++SELECT id FROM hot_expr_test WHERE status = 'active' AND v > 0 ORDER BY v;+ id +----+ 1+ 2+ 3+ 4+ 5+(5 rows)++RESET enable_seqscan;+RESET enable_bitmapscan;+SELECT id FROM hot_expr_test WHERE status = 'active' AND v > 0 ORDER BY v;+ id +----+ 1+ 2+ 3+ 4+ 5+(5 rows)++DROP TABLE hot_expr_test;+-- One UPDATE whose rows each need a different index to decide: some change+-- the expression index, some only enter or leave the partial index created+-- after it, some change neither. The partial index must hold exactly the+-- rows that satisfy its predicate.+CREATE TABLE hot_expr_multi (id int PRIMARY KEY, doc jsonb) WITH (fillfactor = 30);+CREATE INDEX hot_expr_multi_status ON hot_expr_multi ((doc->>'status'));+CREATE INDEX hot_expr_multi_vip ON hot_expr_multi ((doc->>'email'))+ WHERE (doc->>'vip')::bool;+INSERT INTO hot_expr_multi+ SELECT g, jsonb_build_object('status', 'a', 'email', 'e' || g,+ 'vip', g % 2 = 0, 'n', 0)+ FROM generate_series(1, 300) g;+UPDATE hot_expr_multi SET doc = CASE+ WHEN id % 4 = 0 THEN jsonb_set(doc, '{status}', '"b"')+ WHEN id % 4 = 1 THEN jsonb_set(doc, '{vip}', to_jsonb(NOT (doc->>'vip')::bool))+ WHEN id % 4 = 2 THEN jsonb_set(jsonb_set(doc, '{status}', '"c"'),+ '{vip}', to_jsonb(NOT (doc->>'vip')::bool))+ ELSE jsonb_set(doc, '{n}', '1') END;+SET enable_seqscan = off;+SET enable_bitmapscan = off;+SELECT count(*) FROM hot_expr_multi+ WHERE (doc->>'vip')::bool AND doc->>'email' >= 'e';+ count +-------+ 150+(1 row)++RESET enable_seqscan;+RESET enable_bitmapscan;+SELECT count(*) FROM hot_expr_multi WHERE (doc->>'vip')::bool;+ count +-------+ 150+(1 row)++DROP TABLE hot_expr_multi;+-- An index whose concurrent build failed is not ready for inserts, and its+-- expression need not be computable for every row. An UPDATE must not+-- evaluate it.+CREATE TABLE hot_expr_cic (id int PRIMARY KEY, x int) WITH (fillfactor = 50);+INSERT INTO hot_expr_cic VALUES (1, 1), (2, 0);+CREATE INDEX CONCURRENTLY hot_expr_cic_idx ON hot_expr_cic ((100 / x));+ERROR: division by zero+SELECT indisready, indisvalid FROM pg_index+ WHERE indexrelid = 'hot_expr_cic_idx'::regclass;+ indisready | indisvalid +------------+------------+ f | f+(1 row)++UPDATE hot_expr_cic SET x = 0 WHERE id = 1;+UPDATE hot_expr_cic SET x = 1 WHERE id = 2;+SELECT * FROM hot_expr_cic ORDER BY id;+ id | x +----+---+ 1 | 0+ 2 | 1+(2 rows)++DROP TABLE hot_expr_cic;+-- Writing back a value that is stored compressed in the old row, and not+-- yet in the new one. An SP-GiST index, whose access method cannot compare+-- entries, and which returns the value in index-only scans.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, t text) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test USING spgist (t);+ INSERT INTO hot_expr_test SELECT g, repeat('x', 3000) || g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET t = repeat('x', 3000) || id$$);+NOTICE: table "hot_expr_test" does not exist, skipping+ updated | hot +---------+-----+ 10 | 0+(1 row)++SET enable_seqscan = off;+SET enable_bitmapscan = off;+SELECT count(*) FROM hot_expr_test WHERE t = repeat('x', 3000) || 7;+ count +-------+ 1+(1 row)++RESET enable_seqscan;+RESET enable_bitmapscan;+-- The same in a partial index evaluated for another column's change.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, t text, x int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (t) WHERE x >= 0;+ INSERT INTO hot_expr_test SELECT g, repeat('x', 3000) || g, 0 FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET t = repeat('x', 3000) || id, x = x + 1$$);+ updated | hot +---------+-----+ 10 | 0+(1 row)++-- The same through an expression that returns the column's value as is.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, t text, x int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test ((coalesce(t, ''))) WHERE x >= 0;+ INSERT INTO hot_expr_test SELECT g, repeat('x', 3000) || g, 0 FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET t = repeat('x', 3000) || id, x = x + 1$$);+ updated | hot +---------+-----+ 10 | 0+(1 row)++-- ... but not for a column of a unique index, which the table's lock+-- strength is decided by as stored.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int, t text) WITH (fillfactor = 50);+ CREATE UNIQUE INDEX ON hot_expr_test (t);+ INSERT INTO hot_expr_test SELECT g, repeat('x', 3000) || g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET t = repeat('x', 3000) || id$$);+ updated | hot +---------+-----+ 10 | 0+(1 row)++DROP TABLE hot_expr_test;+DROP FUNCTION update_case(text, text);+DROP FUNCTION get_hot_count(text);diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_scheduleindex 75063f87a4a..bb8e5f958c1 100644--- a/src/test/regress/parallel_schedule+++ b/src/test/regress/parallel_schedule@@ -81,7 +81,7 @@ test: create_table_like alter_generic alter_operator misc async dbsize merge mis
# collate.linux.utf8 and collate.icu.utf8 tests cannot be run in parallel with each other
# psql depends on create_am
# amutils depends on geometry, create_index_spgist, hash_index, brin
-test: rules psql psql_crosstab psql_pipeline amutils stats_ext collate.linux.utf8 collate.windows.win1252+test: rules psql psql_crosstab psql_pipeline amutils stats_ext collate.linux.utf8 collate.windows.win1252 hot_updates
# ----------
# Run these alone so they don't run out of parallel workers
diff --git a/src/test/regress/sql/hot_updates.sql b/src/test/regress/sql/hot_updates.sqlnew file mode 100644index 00000000000..c523c908c6b--- /dev/null+++ b/src/test/regress/sql/hot_updates.sql@@ -0,0 +1,468 @@+--+-- HOT_UPDATES+-- Test Heap-Only Tuple (HOT) update decisions+--+-- Expected output records which UPDATEs are HOT, so that a change to the+-- rules shows up as a change to this file. contrib/pageinspect's+-- hot_updates test looks at the HOT chain itself.+--++-- Sum of committed and in-progress (non-HOT, HOT) update counters.+CREATE FUNCTION get_hot_count(rel_name text)+RETURNS TABLE (+ updates BIGINT,+ hot BIGINT+) AS $$+DECLARE+ rel_oid oid;+BEGIN+ rel_oid := rel_name::regclass::oid;+ updates := COALESCE(pg_stat_get_tuples_updated(rel_oid), 0) ++ COALESCE(pg_stat_get_xact_tuples_updated(rel_oid), 0);+ hot := COALESCE(pg_stat_get_tuples_hot_updated(rel_oid), 0) ++ COALESCE(pg_stat_get_xact_tuples_hot_updated(rel_oid), 0);+ RETURN NEXT;+END;+$$ LANGUAGE plpgsql;++-- ---------------------------------------------------------------------------+-- Summarizing indexes (BRIN) do not block HOT+-- ---------------------------------------------------------------------------+CREATE TABLE hot_test (+ id int PRIMARY KEY,+ ts timestamp,+ value int,+ brin_col int+) WITH (fillfactor = 50);+CREATE INDEX hot_test_ts_brin ON hot_test USING brin(ts);+CREATE INDEX hot_test_brin_col_brin ON hot_test USING brin(brin_col);++INSERT INTO hot_test VALUES (1, '2024-01-01', 100, 1000);++-- BRIN columns are summarizing; updating them stays classic HOT even+-- though their values change.+UPDATE hot_test SET ts = '2024-01-02', brin_col = 2000 WHERE id = 1;+SELECT pg_stat_force_next_flush();+SELECT * FROM get_hot_count('hot_test');++-- Non-indexed column: also HOT.+UPDATE hot_test SET value = 200 WHERE id = 1;+SELECT pg_stat_force_next_flush();+SELECT * FROM get_hot_count('hot_test');++DROP TABLE hot_test;++-- ---------------------------------------------------------------------------+-- TOAST participates in HOT (non-indexed column paths only)+-- ---------------------------------------------------------------------------+CREATE TABLE hot_test (+ id int PRIMARY KEY,+ indexed_col int,+ large_text text,+ small_text text+) WITH (fillfactor = 50);+CREATE INDEX hot_test_idx ON hot_test(indexed_col);+-- stored out of line at any block size+ALTER TABLE hot_test ALTER COLUMN large_text SET STORAGE EXTERNAL;+SELECT current_setting('block_size')::int / 2 AS large_len \gset++INSERT INTO hot_test VALUES (1, 100, repeat('x', :large_len), 'small');+SELECT pg_column_toast_chunk_id(large_text) IS NOT NULL AS toasted+ FROM hot_test;++-- Non-indexed, non-TOAST column: HOT.+UPDATE hot_test SET small_text = 'updated';+SELECT pg_stat_force_next_flush();+SELECT * FROM get_hot_count('hot_test');++-- TOAST column, indexed_col unchanged: HOT.+UPDATE hot_test SET large_text = repeat('y', :large_len);+SELECT pg_stat_force_next_flush();+SELECT * FROM get_hot_count('hot_test');++DROP TABLE hot_test;++-- ---------------------------------------------------------------------------+-- Partial index where update leaves indexed attrs unchanged+-- ---------------------------------------------------------------------------+CREATE TABLE hot_test (+ id int PRIMARY KEY,+ status text,+ data text+) WITH (fillfactor = 50);+CREATE INDEX hot_test_active_idx ON hot_test(status) WHERE status = 'active';++INSERT INTO hot_test VALUES (1, 'active', 'data1');+INSERT INTO hot_test VALUES (2, 'inactive', 'data2');+INSERT INTO hot_test VALUES (3, 'deleted', 'data3');++-- Update data on a row whose status matches the partial predicate: HOT.+UPDATE hot_test SET data = 'updated1' WHERE id = 1;+SELECT pg_stat_force_next_flush();+SELECT * FROM get_hot_count('hot_test');++-- Update data on a row outside the predicate: HOT.+UPDATE hot_test SET data = 'updated2' WHERE id = 2;+SELECT pg_stat_force_next_flush();+SELECT * FROM get_hot_count('hot_test');++SELECT id, status FROM hot_test WHERE status = 'active';++DROP TABLE hot_test;++-- ---------------------------------------------------------------------------+-- Multi-column btree: update of non-indexed column+-- ---------------------------------------------------------------------------+CREATE TABLE hot_test (+ id int PRIMARY KEY,+ col_a int,+ col_b int,+ col_c int,+ data text+) WITH (fillfactor = 50);+CREATE INDEX hot_test_ab_idx ON hot_test(col_a, col_b);++INSERT INTO hot_test VALUES (1, 10, 20, 30, 'data');++-- col_c not in any index: HOT.+UPDATE hot_test SET col_c = 35;+-- data not in any index: HOT.+UPDATE hot_test SET data = 'updated';+SELECT pg_stat_force_next_flush();+SELECT * FROM get_hot_count('hot_test');++DROP TABLE hot_test;++-- ---------------------------------------------------------------------------+-- Unique index: update of non-indexed column + uniqueness enforcement+-- ---------------------------------------------------------------------------+CREATE TABLE hot_test (+ id int PRIMARY KEY,+ unique_col int UNIQUE,+ data text+) WITH (fillfactor = 50);++INSERT INTO hot_test VALUES (1, 100, 'data1');+INSERT INTO hot_test VALUES (2, 200, 'data2');++UPDATE hot_test SET data = 'updated';+SELECT pg_stat_force_next_flush();+SELECT * FROM get_hot_count('hot_test');++SELECT id, unique_col, data FROM hot_test ORDER BY id;++-- Unique constraint still enforced on any path.+UPDATE hot_test SET unique_col = 100 WHERE id = 2;++DROP TABLE hot_test;++-- ---------------------------------------------------------------------------+-- Partitioned tables: HOT within a partition+-- ---------------------------------------------------------------------------+CREATE TABLE hot_test_partitioned (+ id int,+ partition_key int,+ indexed_col int,+ data text,+ PRIMARY KEY (id, partition_key)+) PARTITION BY RANGE (partition_key);++CREATE TABLE hot_test_part1 PARTITION OF hot_test_partitioned+ FOR VALUES FROM (1) TO (100) WITH (fillfactor = 50);+CREATE TABLE hot_test_part2 PARTITION OF hot_test_partitioned+ FOR VALUES FROM (100) TO (200) WITH (fillfactor = 50);++CREATE INDEX hot_test_part_idx ON hot_test_partitioned(indexed_col);++INSERT INTO hot_test_partitioned VALUES (1, 50, 100, 'initial1');+INSERT INTO hot_test_partitioned VALUES (2, 150, 200, 'initial2');++UPDATE hot_test_partitioned SET data = 'updated1' WHERE id = 1;+UPDATE hot_test_partitioned SET data = 'updated2' WHERE id = 2;++SELECT pg_stat_force_next_flush();+SELECT * FROM get_hot_count('hot_test_part1');+SELECT * FROM get_hot_count('hot_test_part2');++SELECT id FROM hot_test_partitioned WHERE indexed_col = 100;+SELECT id FROM hot_test_partitioned WHERE indexed_col = 200;++DROP TABLE hot_test_partitioned CASCADE;++-- An update through the parent that writes back an indexed column's value+-- is HOT. Routing hands the partition a new tuple whose short text value+-- has not been packed yet; it is still the same value.+CREATE TABLE hot_test_partitioned (id int UNIQUE, tag char, n int)+ PARTITION BY LIST (id);+CREATE TABLE hot_test_part1 PARTITION OF hot_test_partitioned+ (tag UNIQUE) FOR VALUES IN (1, 2) WITH (fillfactor = 50);+INSERT INTO hot_test_partitioned VALUES (1, 'a', 0), (2, 'b', 0);+INSERT INTO hot_test_partitioned VALUES (1, 'a', 1)+ ON CONFLICT (id) DO UPDATE SET tag = excluded.tag, n = excluded.n;+SELECT pg_stat_force_next_flush();+SELECT * FROM get_hot_count('hot_test_part1');+DROP TABLE hot_test_partitioned CASCADE;++-- ---------------------------------------------------------------------------+-- Expression index on a JSONB field+-- ---------------------------------------------------------------------------+CREATE TABLE hot_jsonb_test (+ id int PRIMARY KEY,+ data jsonb+) WITH (fillfactor = 50);+CREATE INDEX hot_jsonb_name_idx ON hot_jsonb_test ((data->>'name'));++INSERT INTO hot_jsonb_test VALUES+ (1, '{"name":"Alice","age":30,"city":"NYC"}'),+ (2, '{"name":"Bob","age":25,"city":"LA"}');++-- Each UPDATE changes the indexed column but not the indexed expression.+UPDATE hot_jsonb_test SET data = jsonb_set(data, '{age}', '31') WHERE id = 1;+SELECT pg_stat_force_next_flush();+SELECT * FROM get_hot_count('hot_jsonb_test');++UPDATE hot_jsonb_test SET data = data - 'city' WHERE id = 2;+SELECT pg_stat_force_next_flush();+SELECT * FROM get_hot_count('hot_jsonb_test');++UPDATE hot_jsonb_test SET data = jsonb_insert(data, '{country}', '"USA"') WHERE id = 2;+SELECT pg_stat_force_next_flush();+SELECT * FROM get_hot_count('hot_jsonb_test');++DROP TABLE hot_jsonb_test;+++-- ---------------------------------------------------------------------------+-- Expression and partial indexes+--+-- update_case() runs one UPDATE on a fresh table and reports how many rows+-- it updated, and how many of those with HOT.+-- ---------------------------------------------------------------------------+CREATE FUNCTION update_case(setup text, upd text,+ OUT updated bigint, OUT hot bigint)+LANGUAGE plpgsql AS $$+BEGIN+ EXECUTE 'DROP TABLE IF EXISTS hot_expr_test';+ EXECUTE setup;+ updated := pg_stat_get_xact_tuples_updated('hot_expr_test'::regclass);+ hot := pg_stat_get_xact_tuples_hot_updated('hot_expr_test'::regclass);+ EXECUTE upd;+ updated := pg_stat_get_xact_tuples_updated('hot_expr_test'::regclass) - updated;+ hot := pg_stat_get_xact_tuples_hot_updated('hot_expr_test'::regclass) - hot;+END;+$$;++-- Expression value unchanged.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, s text) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (lower(s));+ INSERT INTO hot_expr_test SELECT g, 'abc' || g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET s = upper(s)$$);++SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, a int, b int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test ((a + b));+ INSERT INTO hot_expr_test SELECT g, g, 0 FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET a = a - 1, b = b + 1$$);++-- Expression value changed.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, d jsonb) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test ((d->>'name'));+ INSERT INTO hot_expr_test SELECT g, jsonb_build_object('name', 'n' || g)+ FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET d = jsonb_set(d, '{name}', '"x"')$$);++-- Equal values that are stored differently.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, n numeric) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test ((n + 0));+ INSERT INTO hot_expr_test SELECT g, 3 FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET n = 3.0$$);++-- The index stores a type other than the expression's: name_ops stores+-- cstring, and gist point_ops stores box.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, s text) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test ((lower(s)::name));+ INSERT INTO hot_expr_test SELECT g, 'abc' || g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET s = upper(s)$$);++SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, p point) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test USING gist ((point(p[0], 0)));+ INSERT INTO hot_expr_test SELECT g, point(g, g) FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET p = point(p[0], p[1] + 1)$$);++-- An expression that is a point, stored by the gist index as a box.+-- Comparing it as the box the index stores rather than as the point it is+-- crashed the server in 2018 (2877.1541538838@sss.pgh.pa.us).+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, d smallint) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test USING gist ((point(d::integer % 7, 0)));+ INSERT INTO hot_expr_test SELECT g, 1 FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET d = d + 7$$);++-- A plain index on the same column as an expression index.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, s text) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (lower(s));+ CREATE INDEX ON hot_expr_test (s);+ INSERT INTO hot_expr_test SELECT g, 'abc' || g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET s = upper(s)$$);++-- A BRIN index whose column changed, and an expression index whose value+-- did not.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, s text, b int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (lower(s));+ CREATE INDEX ON hot_expr_test USING brin (b);+ INSERT INTO hot_expr_test SELECT g, 'abc' || g, g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET s = upper(s), b = b + 1$$);++-- A BEFORE ROW trigger that changes the expression's value.+CREATE FUNCTION hot_expr_trig() RETURNS trigger LANGUAGE plpgsql AS+ $$BEGIN NEW.s := NEW.s || 'z'; RETURN NEW; END$$;+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, s text, x int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (lower(s));+ CREATE TRIGGER t BEFORE UPDATE ON hot_expr_test+ FOR EACH ROW EXECUTE FUNCTION hot_expr_trig();+ INSERT INTO hot_expr_test SELECT g, 'abc' || g, 0 FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET x = 1$$);+DROP FUNCTION hot_expr_trig() CASCADE;++-- Partial index, both versions outside the predicate, changing the key or+-- the predicate's column.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, status text, v int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (v) WHERE status = 'active';+ INSERT INTO hot_expr_test SELECT g, 'done', g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET v = v + 1$$);++SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, status text, v int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (v) WHERE status = 'active';+ INSERT INTO hot_expr_test SELECT g, 'done', g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET status = 'archived'$$);++-- Both versions inside the predicate with the key unchanged.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, status text, v int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (v) WHERE status <> 'x';+ INSERT INTO hot_expr_test SELECT g, 'a', g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET status = 'b'$$);++-- Entering or leaving the predicate, or changing the key inside it.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, status text, v int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (v) WHERE status = 'active';+ INSERT INTO hot_expr_test SELECT g, 'active', g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET status = 'done'$$);++SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, status text, v int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (v) WHERE status = 'active';+ INSERT INTO hot_expr_test SELECT g, 'done', g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET status = 'active'$$);++SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, status text, v int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (v) WHERE status = 'active';+ INSERT INTO hot_expr_test SELECT g, 'active', g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET v = v + 1000$$);++-- Rows updated outside a partial index must not be found through it.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, status text, v int) WITH (fillfactor = 50);+ CREATE INDEX hot_expr_test_v ON hot_expr_test (v) WHERE status = 'active';+ INSERT INTO hot_expr_test SELECT g, CASE WHEN g <= 5 THEN 'active' ELSE 'done' END, g+ FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET v = v + 100 WHERE status = 'done'$$);+SET enable_seqscan = off;+SET enable_bitmapscan = off;+EXPLAIN (COSTS OFF) SELECT id FROM hot_expr_test WHERE status = 'active' AND v > 0;+SELECT id FROM hot_expr_test WHERE status = 'active' AND v > 0 ORDER BY v;+RESET enable_seqscan;+RESET enable_bitmapscan;+SELECT id FROM hot_expr_test WHERE status = 'active' AND v > 0 ORDER BY v;++DROP TABLE hot_expr_test;++-- One UPDATE whose rows each need a different index to decide: some change+-- the expression index, some only enter or leave the partial index created+-- after it, some change neither. The partial index must hold exactly the+-- rows that satisfy its predicate.+CREATE TABLE hot_expr_multi (id int PRIMARY KEY, doc jsonb) WITH (fillfactor = 30);+CREATE INDEX hot_expr_multi_status ON hot_expr_multi ((doc->>'status'));+CREATE INDEX hot_expr_multi_vip ON hot_expr_multi ((doc->>'email'))+ WHERE (doc->>'vip')::bool;+INSERT INTO hot_expr_multi+ SELECT g, jsonb_build_object('status', 'a', 'email', 'e' || g,+ 'vip', g % 2 = 0, 'n', 0)+ FROM generate_series(1, 300) g;+UPDATE hot_expr_multi SET doc = CASE+ WHEN id % 4 = 0 THEN jsonb_set(doc, '{status}', '"b"')+ WHEN id % 4 = 1 THEN jsonb_set(doc, '{vip}', to_jsonb(NOT (doc->>'vip')::bool))+ WHEN id % 4 = 2 THEN jsonb_set(jsonb_set(doc, '{status}', '"c"'),+ '{vip}', to_jsonb(NOT (doc->>'vip')::bool))+ ELSE jsonb_set(doc, '{n}', '1') END;+SET enable_seqscan = off;+SET enable_bitmapscan = off;+SELECT count(*) FROM hot_expr_multi+ WHERE (doc->>'vip')::bool AND doc->>'email' >= 'e';+RESET enable_seqscan;+RESET enable_bitmapscan;+SELECT count(*) FROM hot_expr_multi WHERE (doc->>'vip')::bool;+DROP TABLE hot_expr_multi;++-- An index whose concurrent build failed is not ready for inserts, and its+-- expression need not be computable for every row. An UPDATE must not+-- evaluate it.+CREATE TABLE hot_expr_cic (id int PRIMARY KEY, x int) WITH (fillfactor = 50);+INSERT INTO hot_expr_cic VALUES (1, 1), (2, 0);+CREATE INDEX CONCURRENTLY hot_expr_cic_idx ON hot_expr_cic ((100 / x));+SELECT indisready, indisvalid FROM pg_index+ WHERE indexrelid = 'hot_expr_cic_idx'::regclass;+UPDATE hot_expr_cic SET x = 0 WHERE id = 1;+UPDATE hot_expr_cic SET x = 1 WHERE id = 2;+SELECT * FROM hot_expr_cic ORDER BY id;+DROP TABLE hot_expr_cic;++-- Writing back a value that is stored compressed in the old row, and not+-- yet in the new one. An SP-GiST index, whose access method cannot compare+-- entries, and which returns the value in index-only scans.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, t text) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test USING spgist (t);+ INSERT INTO hot_expr_test SELECT g, repeat('x', 3000) || g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET t = repeat('x', 3000) || id$$);+SET enable_seqscan = off;+SET enable_bitmapscan = off;+SELECT count(*) FROM hot_expr_test WHERE t = repeat('x', 3000) || 7;+RESET enable_seqscan;+RESET enable_bitmapscan;+-- The same in a partial index evaluated for another column's change.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, t text, x int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test (t) WHERE x >= 0;+ INSERT INTO hot_expr_test SELECT g, repeat('x', 3000) || g, 0 FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET t = repeat('x', 3000) || id, x = x + 1$$);+-- The same through an expression that returns the column's value as is.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int PRIMARY KEY, t text, x int) WITH (fillfactor = 50);+ CREATE INDEX ON hot_expr_test ((coalesce(t, ''))) WHERE x >= 0;+ INSERT INTO hot_expr_test SELECT g, repeat('x', 3000) || g, 0 FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET t = repeat('x', 3000) || id, x = x + 1$$);+-- ... but not for a column of a unique index, which the table's lock+-- strength is decided by as stored.+SELECT * FROM update_case(+ $$CREATE TABLE hot_expr_test (id int, t text) WITH (fillfactor = 50);+ CREATE UNIQUE INDEX ON hot_expr_test (t);+ INSERT INTO hot_expr_test SELECT g, repeat('x', 3000) || g FROM generate_series(1, 10) g$$,+ $$UPDATE hot_expr_test SET t = repeat('x', 3000) || id$$);+DROP TABLE hot_expr_test;++DROP FUNCTION update_case(text, text);++DROP FUNCTION get_hot_count(text);
base-commit: 871dee0ccbe292cf4c2874aa7b460d242262d242
--
2.50.1
v43-0003-Allow-HOT-updates-when-expression-and-partial-in.patch (text/plain, 42.9K) download
From d74624c06baa831e12655a7b319b1fbd5dfb548b Mon Sep 17 00:00:00 2001
From: Greg Burd <greg@burd.me>
Date: Mon, 19 Oct 2026 18:55:41 -0400
Subject: [PATCH v43 3/8] Allow HOT updates when expression and partial indexes
are unaffected
An UPDATE that changes a column used by an expression index cannot be
HOT today, even when the expression's value does not change: an index
on (data->>'name') blocks HOT for every change to data. Partial
indexes are treated the same way, so an index WHERE status = 'active'
blocks HOT for a row that is outside the predicate before and after
the update. For JSONB documents with a few indexed fields that is
most UPDATEs.
Since the previous commit the executor computes which indexed
attributes an UPDATE changed before calling the table AM, outside any
buffer lock, so it can now look at what the indexes would store.
INDEX_ATTR_BITMAP_EXPRESSION, a new relcache bitmap, holds the
attributes that non-summarizing indexes reference only through
expressions or as partial indexes. When the changed attributes are all
in that set, or are referenced otherwise only by summarizing indexes,
ExecUpdateModifiedIdxAttrs() evaluates each affected index's predicate
and index datums for the old and new rows, and drops from the changed
set every attribute whose change no index sees. The table AM decides
the rest as before, so for heap the update can be HOT.
An index sees the change if:
- the row enters or leaves its predicate. A HOT chain would leave the
index pointing at a row outside the predicate, which the planner
relies on not happening;
- the row is inside the predicate and any index datum differs. Datums
are compared byte for byte as stored, using the type of the value
FormIndexDatum() produced (the expression's type, not the index
column's, which can differ through opckeytype), so equal values with
different representations still count as changed;
- the index is not yet ready for inserts or not valid. README.HOT
requires such an index to constrain HOT, and a failed build's
expression need not be computable for every row.
The comparison is made per row, after BEFORE ROW triggers and after
EvalPlanQual has replaced the old and new rows with those built from the
latest version, so neither a concurrent update nor an earlier row of the
same statement can supply the answer. update-epq-indexes, added with
the tests, covers the concurrent case with a BEFORE ROW trigger.
Nothing is evaluated when an update changes a column that a plain
index uses, so the cost of an update that cannot be HOT is the
attribute comparison it already paid. Key and replica identity indexes
have neither expressions nor predicates, so the lock mode and the
replica identity decision see the same attributes as before; an
assertion checks this.
This is the second attempt. Commit c203d6cf8 compared index datums
using the index's storage type and crashed for an index whose stored
type differs from the expression's; it was disabled in 5d28c9bd7 and
reverted in 1c53c4dec. The hot_updates test now covers such indexes
(name_ops, gist point_ops). Its expected output changes for the
expression and partial index cases this makes HOT, and for no others.
Discussion: https://postgr.es/m/78574B24-BE0A-42C5-8075-3FA9FA63B8FC@amazon.com
Discussion: https://postgr.es/m/20181106185255.776mstcyehnc63ty@alvherre.pgsql
# Conflicts:
# contrib/pageinspect/expected/hot_updates.out
# src/backend/executor/nodeModifyTable.c
# src/backend/utils/cache/relcache.c
# src/include/utils/rel.h
# src/include/utils/relcache.h
---
contrib/amcheck/expected/check_btree.out | 83 ++++++
contrib/amcheck/sql/check_btree.sql | 50 ++++
doc/src/sgml/storage.sgml | 9 +-
src/backend/access/heap/README.HOT | 18 +-
src/backend/access/heap/heapam.c | 14 +-
src/backend/commands/repack.c | 3 +-
src/backend/executor/execReplication.c | 3 +-
src/backend/executor/nodeModifyTable.c | 273 +++++++++++++++++-
src/backend/utils/cache/relcache.c | 46 +++
src/include/executor/executor.h | 3 +-
src/include/nodes/execnodes.h | 8 +
src/include/utils/rel.h | 2 +
src/include/utils/relcache.h | 1 +
.../isolation/expected/cic-expression-hot.out | 43 +++
src/test/isolation/isolation_schedule | 1 +
.../isolation/specs/cic-expression-hot.spec | 43 +++
src/test/regress/expected/hot_updates.out | 26 +-
17 files changed, 599 insertions(+), 27 deletions(-)
create mode 100644 src/test/isolation/expected/cic-expression-hot.out
create mode 100644 src/test/isolation/specs/cic-expression-hot.spec
diff --git a/contrib/amcheck/expected/check_btree.out b/contrib/amcheck/expected/check_btree.outindex 4fcee5da463..6fc1ee97902 100644--- a/contrib/amcheck/expected/check_btree.out+++ b/contrib/amcheck/expected/check_btree.out@@ -283,6 +283,89 @@ SELECT bt_index_check('varlena_bug_idx', true);
(1 row)
+-- HOT updates of rows indexed by expression and partial indexes must leave+-- every index entry consistent with the heap.+CREATE TABLE hot_expr (id int PRIMARY KEY, s text, d jsonb,+ status text, v int, n numeric) WITH (fillfactor = 50);+CREATE INDEX hot_expr_lower ON hot_expr (lower(s));+CREATE INDEX hot_expr_name ON hot_expr ((d->>'name'));+CREATE INDEX hot_expr_partial ON hot_expr (v) WHERE status = 'active';+CREATE INDEX hot_expr_numeric ON hot_expr ((n + 0));+INSERT INTO hot_expr+ SELECT g, 'abc' || g, jsonb_build_object('name', 'n' || g, 'age', g),+ CASE WHEN g % 2 = 0 THEN 'active' ELSE 'done' END, g, 3+ FROM generate_series(1, 200) g;+UPDATE hot_expr SET s = upper(s);+UPDATE hot_expr SET d = jsonb_set(d, '{age}', '0');+UPDATE hot_expr SET v = v + 1000 WHERE status = 'done';+UPDATE hot_expr SET status = 'done' WHERE id % 4 = 0;+UPDATE hot_expr SET status = 'active' WHERE id % 3 = 0;+UPDATE hot_expr SET n = 3.0 WHERE id % 5 = 0;+UPDATE hot_expr SET d = jsonb_set(d, '{name}', '"x"') WHERE id % 7 = 0;+SELECT bt_index_check('hot_expr_lower', heapallindexed => true);+ bt_index_check +----------------+ +(1 row)++SELECT bt_index_check('hot_expr_name', heapallindexed => true);+ bt_index_check +----------------+ +(1 row)++SELECT bt_index_check('hot_expr_partial', heapallindexed => true);+ bt_index_check +----------------+ +(1 row)++SELECT bt_index_check('hot_expr_numeric', heapallindexed => true);+ bt_index_check +----------------+ +(1 row)++SELECT bt_index_parent_check('hot_expr_partial', heapallindexed => true);+ bt_index_parent_check +-----------------------+ +(1 row)++DROP TABLE hot_expr;+-- One UPDATE in which different rows are decided by different indexes: an+-- expression index created first, a partial index created after it.+CREATE TABLE hot_expr_multi (id int PRIMARY KEY, doc jsonb) WITH (fillfactor = 30);+CREATE INDEX hot_expr_multi_status ON hot_expr_multi ((doc->>'status'));+CREATE INDEX hot_expr_multi_vip ON hot_expr_multi ((doc->>'email'))+ WHERE (doc->>'vip')::bool;+INSERT INTO hot_expr_multi+ SELECT g, jsonb_build_object('status', 'a', 'email', 'e' || g,+ 'vip', g % 2 = 0, 'n', 0)+ FROM generate_series(1, 3000) g;+UPDATE hot_expr_multi SET doc = CASE+ WHEN id % 4 = 0 THEN jsonb_set(doc, '{status}', '"b"')+ WHEN id % 4 = 1 THEN jsonb_set(doc, '{vip}', to_jsonb(NOT (doc->>'vip')::bool))+ WHEN id % 4 = 2 THEN jsonb_set(jsonb_set(doc, '{status}', '"c"'),+ '{vip}', to_jsonb(NOT (doc->>'vip')::bool))+ ELSE jsonb_set(doc, '{n}', '1') END;+UPDATE hot_expr_multi SET doc = CASE+ WHEN id % 3 = 0 THEN jsonb_set(doc, '{vip}', to_jsonb(NOT (doc->>'vip')::bool))+ WHEN id % 3 = 1 THEN jsonb_set(doc, '{status}', '"d"')+ ELSE jsonb_set(doc, '{n}', '2') END;+SELECT bt_index_check('hot_expr_multi_status', heapallindexed => true);+ bt_index_check +----------------+ +(1 row)++SELECT bt_index_check('hot_expr_multi_vip', heapallindexed => true);+ bt_index_check +----------------+ +(1 row)++DROP TABLE hot_expr_multi;
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
diff --git a/contrib/amcheck/sql/check_btree.sql b/contrib/amcheck/sql/check_btree.sqlindex a5585085fad..4c54aad5d9f 100644--- a/contrib/amcheck/sql/check_btree.sql+++ b/contrib/amcheck/sql/check_btree.sql@@ -183,6 +183,56 @@ INSERT INTO varlena_bug VALUES (repeat('Test', 250));
ALTER TABLE varlena_bug ALTER COLUMN v SET STORAGE extended;
SELECT bt_index_check('varlena_bug_idx', true);
+-- HOT updates of rows indexed by expression and partial indexes must leave+-- every index entry consistent with the heap.+CREATE TABLE hot_expr (id int PRIMARY KEY, s text, d jsonb,+ status text, v int, n numeric) WITH (fillfactor = 50);+CREATE INDEX hot_expr_lower ON hot_expr (lower(s));+CREATE INDEX hot_expr_name ON hot_expr ((d->>'name'));+CREATE INDEX hot_expr_partial ON hot_expr (v) WHERE status = 'active';+CREATE INDEX hot_expr_numeric ON hot_expr ((n + 0));+INSERT INTO hot_expr+ SELECT g, 'abc' || g, jsonb_build_object('name', 'n' || g, 'age', g),+ CASE WHEN g % 2 = 0 THEN 'active' ELSE 'done' END, g, 3+ FROM generate_series(1, 200) g;+UPDATE hot_expr SET s = upper(s);+UPDATE hot_expr SET d = jsonb_set(d, '{age}', '0');+UPDATE hot_expr SET v = v + 1000 WHERE status = 'done';+UPDATE hot_expr SET status = 'done' WHERE id % 4 = 0;+UPDATE hot_expr SET status = 'active' WHERE id % 3 = 0;+UPDATE hot_expr SET n = 3.0 WHERE id % 5 = 0;+UPDATE hot_expr SET d = jsonb_set(d, '{name}', '"x"') WHERE id % 7 = 0;+SELECT bt_index_check('hot_expr_lower', heapallindexed => true);+SELECT bt_index_check('hot_expr_name', heapallindexed => true);+SELECT bt_index_check('hot_expr_partial', heapallindexed => true);+SELECT bt_index_check('hot_expr_numeric', heapallindexed => true);+SELECT bt_index_parent_check('hot_expr_partial', heapallindexed => true);+DROP TABLE hot_expr;++-- One UPDATE in which different rows are decided by different indexes: an+-- expression index created first, a partial index created after it.+CREATE TABLE hot_expr_multi (id int PRIMARY KEY, doc jsonb) WITH (fillfactor = 30);+CREATE INDEX hot_expr_multi_status ON hot_expr_multi ((doc->>'status'));+CREATE INDEX hot_expr_multi_vip ON hot_expr_multi ((doc->>'email'))+ WHERE (doc->>'vip')::bool;+INSERT INTO hot_expr_multi+ SELECT g, jsonb_build_object('status', 'a', 'email', 'e' || g,+ 'vip', g % 2 = 0, 'n', 0)+ FROM generate_series(1, 3000) g;+UPDATE hot_expr_multi SET doc = CASE+ WHEN id % 4 = 0 THEN jsonb_set(doc, '{status}', '"b"')+ WHEN id % 4 = 1 THEN jsonb_set(doc, '{vip}', to_jsonb(NOT (doc->>'vip')::bool))+ WHEN id % 4 = 2 THEN jsonb_set(jsonb_set(doc, '{status}', '"c"'),+ '{vip}', to_jsonb(NOT (doc->>'vip')::bool))+ ELSE jsonb_set(doc, '{n}', '1') END;+UPDATE hot_expr_multi SET doc = CASE+ WHEN id % 3 = 0 THEN jsonb_set(doc, '{vip}', to_jsonb(NOT (doc->>'vip')::bool))+ WHEN id % 3 = 1 THEN jsonb_set(doc, '{status}', '"d"')+ ELSE jsonb_set(doc, '{n}', '2') END;+SELECT bt_index_check('hot_expr_multi_status', heapallindexed => true);+SELECT bt_index_check('hot_expr_multi_vip', heapallindexed => true);+DROP TABLE hot_expr_multi;+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgmlindex 83de016eaa5..4d65523e7b9 100644--- a/doc/src/sgml/storage.sgml+++ b/doc/src/sgml/storage.sgml@@ -1109,7 +1109,14 @@ data. Empty in ordinary tables.</entry>
The update does not modify any columns referenced by the table's indexes,
not including summarizing indexes. The only summarizing index method in
the core <productname>PostgreSQL</productname> distribution is <link
- linkend="brin">BRIN</link>.+ linkend="brin">BRIN</link>. A column that indexes reference only through+ <link linkend="indexes-expressional">expressions</link>, or only as+ <link linkend="indexes-partial">partial indexes</link>, does not count as+ modified if the update changes neither the value of any such index's+ expressions nor whether the row satisfies its predicate. The values are+ compared as stored, so equal values with different representations, such+ as <literal>1.0</literal> and <literal>1.00</literal> of type+ <type>numeric</type>, count as different.
</para>
</listitem>
<listitem>
diff --git a/src/backend/access/heap/README.HOT b/src/backend/access/heap/README.HOTindex 74e407f375a..0377f508055 100644--- a/src/backend/access/heap/README.HOT+++ b/src/backend/access/heap/README.HOT@@ -37,6 +37,14 @@ First, where a tuple is repeatedly updated in ways that do not change
its indexed columns. (Here, "indexed column" means any column referenced
at all in an index definition, including for example columns that are
tested in a partial-index predicate but are not stored in the index.)
+A column that indexes reference only through expressions, or only as+partial indexes, may change if the update changes nothing those indexes+store: not the value of any of their expressions or columns, as compared+byte for byte, and not whether the tuple satisfies their predicates.+The executor works that out before calling the table AM, and passes on+only the indexed columns whose change matters. An index that is not yet+ready for inserts, or whose build failed, counts as changed whenever a+column it references does; see "CREATE INDEX CONCURRENTLY" below.
Second, where the modified columns are only used in indexes that do not
contain tuple IDs, but maintain summaries of the indexed data by block.
@@ -477,9 +485,10 @@ Heap-only tuple
HOT-safe
A proposed tuple update is said to be HOT-safe if it changes
- none of the tuple's indexed columns. It will only become an- actual HOT update if we can find room on the same page for- the new tuple version.+ none of the tuple's indexed columns, other than in ways the+ indexes that use them cannot see (see above). It will only+ become an actual HOT update if we can find room on the same page+ for the new tuple version.
HOT update
@@ -496,7 +505,8 @@ Indexed column
A column used in an index definition. The column might not
actually be stored in the index --- it could be used in a
functional index's expression, or used in a partial index
- predicate. HOT treats all these cases alike.+ predicate. A change to a column that indexes use only in these+ ways blocks HOT only if it changes what those indexes store.
Redirecting line pointer
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.cindex 729f9c5fb57..e0cd1ab7612 100644--- a/src/backend/access/heap/heapam.c+++ b/src/backend/access/heap/heapam.c@@ -3316,6 +3316,7 @@ heap_update(Relation relation, const ItemPointerData *otid, HeapTuple newtup,
bool rep_id_key_required = false;
#ifdef USE_ASSERT_CHECKING
Bitmapset *check_attrs;
+ Bitmapset *narrowable_attrs;
#endif
TransactionId xmax_new_tuple,
xmax_old_tuple;
@@ -3359,6 +3360,8 @@ heap_update(Relation relation, const ItemPointerData *otid, HeapTuple newtup,
id_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_IDENTITY_KEY);
#ifdef USE_ASSERT_CHECKING
check_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_INDEXED);
+ narrowable_attrs = RelationGetIndexAttrBitmap(relation,+ INDEX_ATTR_BITMAP_EXPRESSION);
#endif
block = ItemPointerGetBlockNumber(otid);
@@ -3417,6 +3420,7 @@ heap_update(Relation relation, const ItemPointerData *otid, HeapTuple newtup,
bms_free(id_attrs);
#ifdef USE_ASSERT_CHECKING
bms_free(check_attrs);
+ bms_free(narrowable_attrs);
#endif
/* modified_idx_attrs is owned by the caller, don't free it */
@@ -3765,6 +3769,7 @@ l2:
bms_free(id_attrs);
#ifdef USE_ASSERT_CHECKING
bms_free(check_attrs);
+ bms_free(narrowable_attrs);
#endif
/* modified_idx_attrs is owned by the caller, don't free it */
@@ -3795,16 +3800,21 @@ l2:
* updated, and no one else can have changed it. Check that every
* attribute comparing the two tuples here finds changed is in the
* caller's set, so that a caller that misses a change, or compares the
- * wrong tuples, is caught. The caller may report more: comparing slots,- * it cannot always tell that two values will be stored alike.+ * wrong tuples, is caught. The caller may leave out an attribute that+ * indexes use only through expressions or as partial indexes, having+ * found that no index's entry changes (see ExecUpdateModifiedIdxAttrs()),+ * and may report more: comparing slots, it cannot always tell that two+ * values will be stored alike.
*/
{
Bitmapset *changed = HeapUpdateModifiedIdxAttrs(relation, check_attrs,
&oldtup, newtup);
+ changed = bms_del_members(changed, narrowable_attrs);
Assert(bms_is_subset(changed, modified_idx_attrs));
bms_free(changed);
bms_free(check_attrs);
+ bms_free(narrowable_attrs);
}
#endif
diff --git a/src/backend/commands/repack.c b/src/backend/commands/repack.cindex 5fc5808e079..4294c860e12 100644--- a/src/backend/commands/repack.c+++ b/src/backend/commands/repack.c@@ -2919,7 +2919,8 @@ apply_concurrent_update(Relation rel, TupleTableSlot *spilled_tuple,
modified_idx_attrs = ExecUpdateModifiedIdxAttrs(chgcxt->cc_rri,
ondisk_tuple,
- spilled_tuple);+ spilled_tuple,+ chgcxt->cc_estate);
/*
* Carry out the update, skipping logical decoding for it.
diff --git a/src/backend/executor/execReplication.c b/src/backend/executor/execReplication.cindex 5f894651d5e..64e2ca2297d 100644--- a/src/backend/executor/execReplication.c+++ b/src/backend/executor/execReplication.c@@ -1029,7 +1029,8 @@ ExecSimpleRelationUpdate(ResultRelInfo *resultRelInfo,
ExecPartitionCheck(resultRelInfo, slot, estate, true);
modified_idx_attrs = ExecUpdateModifiedIdxAttrs(resultRelInfo,
- searchslot, slot);+ searchslot, slot,+ estate);
simple_table_tuple_update_ext(rel, tid, slot, estate->es_snapshot,
modified_idx_attrs, &update_indexes);
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.cindex d31eae94fb2..5a042eb83a3 100644--- a/src/backend/executor/nodeModifyTable.c+++ b/src/backend/executor/nodeModifyTable.c@@ -59,6 +59,7 @@
#include "access/tupconvert.h"
#include "access/tupdesc.h"
#include "access/xact.h"
+#include "catalog/index.h"
#include "commands/trigger.h"
#include "executor/execPartition.h"
#include "executor/executor.h"
@@ -75,6 +76,7 @@
#include "utils/builtins.h"
#include "utils/datum.h"
#include "utils/injection_point.h"
+#include "utils/lsyscache.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
@@ -194,6 +196,148 @@ static TupleTableSlot *ExecMergeNotMatched(ModifyTableContext *context,
ResultRelInfo *resultRelInfo,
bool canSetTag);
+/*+ * Do the old and new index datums differ, byte for byte? Compare each+ * column as the type FormIndexDatum() produced it in: the table column's+ * type or the expression's, which need not be the type the index stores.+ */+static bool+IndexDatumsDiffer(Relation heapRel, IndexInfo *indexInfo,+ const Datum *old_values, const bool *old_isnull,+ const Datum *new_values, const bool *new_isnull)+{+ ListCell *expr = list_head(indexInfo->ii_Expressions);++ for (int i = 0; i < indexInfo->ii_NumIndexAttrs; i++)+ {+ AttrNumber attnum = indexInfo->ii_IndexAttrNumbers[i];+ int16 typlen;+ bool typbyval;++ if (attnum != 0)+ {+ CompactAttribute *att = TupleDescCompactAttr(RelationGetDescr(heapRel),+ attnum - 1);++ typlen = att->attlen;+ typbyval = att->attbyval;+ }+ else+ {+ get_typlenbyval(exprType(lfirst(expr)), &typlen, &typbyval);+ expr = lnext(indexInfo->ii_Expressions, expr);+ }++ if (old_isnull[i] != new_isnull[i])+ return true;+ if (!old_isnull[i] &&+ !datumIsEqual(old_values[i], new_values[i], typbyval, typlen))+ return true;+ }+ return false;+}++/*+ * Would the update from old_tts to new_tts give this index a new entry, or+ * take one away?+ *+ * The index datums are formed in the per-tuple memory context, which+ * callers reset for each row. The predicate's ExprState is kept in the+ * IndexInfo, as ExecInsertIndexTuples() keeps it; ExecPrepareQual() builds+ * it in the query context.+ */+static bool+IndexEntryChanges(ResultRelInfo *resultRelInfo, IndexInfo *indexInfo,+ TupleTableSlot *old_tts, TupleTableSlot *new_tts,+ EState *estate)+{+ ExprContext *econtext = GetPerTupleExprContext(estate);+ TupleTableSlot *save_scantuple = econtext->ecxt_scantuple;+ MemoryContext oldcxt;+ Datum old_values[INDEX_MAX_KEYS];+ bool old_isnull[INDEX_MAX_KEYS];+ Datum new_values[INDEX_MAX_KEYS];+ bool new_isnull[INDEX_MAX_KEYS];+ bool result;++ if (indexInfo->ii_Predicate != NIL)+ {+ bool old_member;+ bool new_member;++ if (indexInfo->ii_PredicateState == NULL)+ indexInfo->ii_PredicateState =+ ExecPrepareQual(indexInfo->ii_Predicate, estate);++ econtext->ecxt_scantuple = old_tts;+ old_member = ExecQual(indexInfo->ii_PredicateState, econtext);+ econtext->ecxt_scantuple = new_tts;+ new_member = ExecQual(indexInfo->ii_PredicateState, econtext);+ econtext->ecxt_scantuple = save_scantuple;++ /*+ * A row that leaves or enters the index's predicate changes which+ * rows the index holds. A row outside it on both sides is none of+ * the index's business.+ */+ if (old_member != new_member)+ return true;+ if (!old_member)+ return false;+ }++ oldcxt = MemoryContextSwitchTo(econtext->ecxt_per_tuple_memory);+ econtext->ecxt_scantuple = old_tts;+ FormIndexDatum(indexInfo, old_tts, estate, old_values, old_isnull);+ econtext->ecxt_scantuple = new_tts;+ FormIndexDatum(indexInfo, new_tts, estate, new_values, new_isnull);+ econtext->ecxt_scantuple = save_scantuple;++ result = IndexDatumsDiffer(resultRelInfo->ri_RelationDesc, indexInfo,+ old_values, old_isnull, new_values, new_isnull);+ MemoryContextSwitchTo(oldcxt);+ return result;+}++/*+ * Return the attributes of the i'th index that INDEX_ATTR_BITMAP_EXPRESSION+ * can contain: those it references through its expressions, and for a+ * partial index all those it references. Builds the per-index cache on+ * first use.+ */+static Bitmapset *+IndexExprAttrs(ResultRelInfo *resultRelInfo, int i, EState *estate)+{+ if (resultRelInfo->ri_IndexExprAttrs == NULL)+ {+ MemoryContext oldcxt = MemoryContextSwitchTo(estate->es_query_cxt);++ resultRelInfo->ri_IndexExprAttrs =+ palloc0_array(Bitmapset *, resultRelInfo->ri_NumIndices);+ for (int j = 0; j < resultRelInfo->ri_NumIndices; j++)+ {+ IndexInfo *ii = resultRelInfo->ri_IndexRelationInfo[j];+ Bitmapset *attrs = NULL;++ pull_varattnos((Node *) ii->ii_Expressions, 1, &attrs);+ if (ii->ii_Predicate != NIL)+ {+ pull_varattnos((Node *) ii->ii_Predicate, 1, &attrs);+ for (int k = 0; k < ii->ii_NumIndexAttrs; k++)+ {+ if (ii->ii_IndexAttrNumbers[k] != 0)+ attrs = bms_add_member(attrs,+ ii->ii_IndexAttrNumbers[k] -+ FirstLowInvalidHeapAttributeNumber);+ }+ }+ resultRelInfo->ri_IndexExprAttrs[j] = attrs;+ }+ MemoryContextSwitchTo(oldcxt);+ }+ return resultRelInfo->ri_IndexExprAttrs[i];+}+
/*
* ExecUpdateModifiedIdxAttrs
*
@@ -216,17 +360,137 @@ static TupleTableSlot *ExecMergeNotMatched(ModifyTableContext *context,
* simple_heap_update() makes the same determination from a TID and a
* HeapTuple in HeapUpdateModifiedIdxAttrs(), and simple_table_tuple_update()
* with table_slot_modified_idx_attrs().
+ *+ * The set is then narrowed to what the indexes see, using estate. An+ * attribute that non-summarizing indexes reference only through expressions+ * or as partial indexes (INDEX_ATTR_BITMAP_EXPRESSION) is dropped from the+ * set if, for every such index that references it, the update moves the row+ * neither into nor out of the index's predicate and, if the row is in it,+ * changes none of the index's datums. The table AM can then treat the+ * update as if it had not changed the attribute: no index needs a new+ * entry because of it.+ *+ * An index that is not ready for inserts or not valid counts as affected by+ * any change to an attribute it references, without evaluating anything.+ * Such an index is being built, or its build failed, so the old row may be+ * one its expressions or predicate cannot be evaluated for; evaluating them+ * would make the UPDATE fail. It still constrains HOT, as README.HOT+ * requires.+ *+ * Narrowing never removes an attribute of a key or replica identity index,+ * as those have neither expressions nor predicates, so the table AM may+ * also take its lock mode and replica identity decisions from the result.
*/
Bitmapset *
ExecUpdateModifiedIdxAttrs(ResultRelInfo *resultRelInfo,
TupleTableSlot *old_tts,
- TupleTableSlot *new_tts)+ TupleTableSlot *new_tts,+ EState *estate)
{
+ Relation relation = resultRelInfo->ri_RelationDesc;+ Bitmapset *attrs;+ Bitmapset *exprattrs;+ Bitmapset *candidates;+
if (resultRelInfo->ri_NumIndices == 0)
return NULL;
- return table_slot_modified_idx_attrs(resultRelInfo->ri_RelationDesc,- old_tts, new_tts);+ attrs = table_slot_modified_idx_attrs(relation, old_tts, new_tts);++ if (bms_is_empty(attrs))+ return attrs;++ /*+ * Narrowing can only make the update HOT if every changed attribute is+ * either a candidate or referenced only by summarizing indexes. Otherwise+ * some index needs a new entry anyway, and evaluating expressions would+ * be wasted work.+ */+ exprattrs = RelationGetIndexAttrBitmap(relation,+ INDEX_ATTR_BITMAP_EXPRESSION);+ candidates = bms_intersect(attrs, exprattrs);+ if (bms_is_empty(candidates))+ {+ bms_free(exprattrs);+ return attrs;+ }+ {+ Bitmapset *rest = bms_difference(attrs, exprattrs);+ Bitmapset *hotattrs = RelationGetIndexAttrBitmap(relation,+ INDEX_ATTR_BITMAP_HOT_BLOCKING);+ bool hopeless = bms_overlap(rest, hotattrs);++ bms_free(rest);+ bms_free(hotattrs);+ bms_free(exprattrs);+ if (hopeless)+ {+ bms_free(candidates);+ return attrs;+ }+ }++ /*+ * Keep each candidate that some index sees change. An index whose+ * candidates are all kept already need not be asked.+ */+ {+ Bitmapset *keep = NULL;++ for (int i = 0; i < resultRelInfo->ri_NumIndices; i++)+ {+ IndexInfo *indexInfo = resultRelInfo->ri_IndexRelationInfo[i];+ Bitmapset *mine;++ if (resultRelInfo->ri_IndexRelationDescs[i] == NULL ||+ indexInfo->ii_Summarizing)+ continue;++ mine = bms_intersect(IndexExprAttrs(resultRelInfo, i, estate),+ candidates);+ if (bms_is_empty(mine) || bms_is_subset(mine, keep))+ {+ bms_free(mine);+ continue;+ }++ if (!indexInfo->ii_ReadyForInserts ||+ !resultRelInfo->ri_IndexRelationDescs[i]->rd_index->indisvalid ||+ IndexEntryChanges(resultRelInfo, indexInfo,+ old_tts, new_tts, estate))+ {+ keep = bms_add_members(keep, mine);+ if (bms_is_subset(candidates, keep))+ {+ bms_free(mine);+ break;+ }+ }+ bms_free(mine);+ }++ candidates = bms_del_members(candidates, keep);+ bms_free(keep);+ }++#ifdef USE_ASSERT_CHECKING+ {+ Bitmapset *keyattrs = RelationGetIndexAttrBitmap(relation,+ INDEX_ATTR_BITMAP_KEY);+ Bitmapset *idattrs = RelationGetIndexAttrBitmap(relation,+ INDEX_ATTR_BITMAP_IDENTITY_KEY);++ Assert(!bms_overlap(candidates, keyattrs));+ Assert(!bms_overlap(candidates, idattrs));+ bms_free(keyattrs);+ bms_free(idattrs);+ }+#endif++ attrs = bms_del_members(attrs, candidates);+ bms_free(candidates);++ return attrs;
}
/*
@@ -2365,7 +2629,8 @@ lreplace:
if (resultRelationDesc->rd_att->constr)
ExecConstraints(resultRelInfo, slot, estate);
- modified_idx_attrs = ExecUpdateModifiedIdxAttrs(resultRelInfo, oldSlot, slot);+ modified_idx_attrs = ExecUpdateModifiedIdxAttrs(resultRelInfo, oldSlot, slot,+ estate);
/*
* Call into the table AM to update the heap tuple.
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.cindex 7b83882bf28..c0941182b91 100644--- a/src/backend/utils/cache/relcache.c+++ b/src/backend/utils/cache/relcache.c@@ -2478,6 +2478,7 @@ RelationDestroyRelation(Relation relation, bool remember_tupdesc)
bms_free(relation->rd_hotblockingattr);
bms_free(relation->rd_summarizedattr);
bms_free(relation->rd_indexedattr);
+ bms_free(relation->rd_exprattr);
if (relation->rd_pubdesc)
pfree(relation->rd_pubdesc);
if (relation->rd_options)
@@ -5327,6 +5328,9 @@ RelationGetIndexPredicate(Relation relation)
* INDEX_ATTR_BITMAP_HOT_BLOCKING Columns that block updates from being HOT
* INDEX_ATTR_BITMAP_SUMMARIZED Columns included in summarizing indexes
* INDEX_ATTR_BITMAP_INDEXED Columns referenced by any index
+ * INDEX_ATTR_BITMAP_EXPRESSION Columns that non-summarizing indexes+ * reference only through expressions or+ * as partial indexes
*
* Attribute numbers are offset by FirstLowInvalidHeapAttributeNumber so that
* we can include system attributes (e.g., OID) in the bitmap representation.
@@ -5352,6 +5356,10 @@ RelationGetIndexAttrBitmap(Relation relation, IndexAttrBitmapKind attrKind)
Bitmapset *hotblockingattrs; /* columns with HOT blocking indexes */
Bitmapset *summarizedattrs; /* columns with summarizing indexes */
Bitmapset *indexedattrs; /* columns referenced by any index */
+ Bitmapset *exprattrs; /* columns of expression or partial+ * non-summarizing indexes */+ Bitmapset *plainattrs; /* columns of other non-summarizing indexes,+ * and plain columns of expression indexes */
List *indexoidlist;
List *newindexoidlist;
Oid relpkindex;
@@ -5376,6 +5384,8 @@ RelationGetIndexAttrBitmap(Relation relation, IndexAttrBitmapKind attrKind)
return bms_copy(relation->rd_summarizedattr);
case INDEX_ATTR_BITMAP_INDEXED:
return bms_copy(relation->rd_indexedattr);
+ case INDEX_ATTR_BITMAP_EXPRESSION:+ return bms_copy(relation->rd_exprattr);
default:
elog(ERROR, "unknown attrKind %u", attrKind);
}
@@ -5420,6 +5430,8 @@ restart:
idindexattrs = NULL;
hotblockingattrs = NULL;
summarizedattrs = NULL;
+ exprattrs = NULL;+ plainattrs = NULL;
foreach(l, indexoidlist)
{
Oid indexOid = lfirst_oid(l);
@@ -5503,6 +5515,15 @@ restart:
{
*attrs = bms_add_member(*attrs,
attrnum - FirstLowInvalidHeapAttributeNumber);
+ if (attrs == &hotblockingattrs)+ {+ if (indexPredicate != NULL)+ exprattrs = bms_add_member(exprattrs,+ attrnum - FirstLowInvalidHeapAttributeNumber);+ else+ plainattrs = bms_add_member(plainattrs,+ attrnum - FirstLowInvalidHeapAttributeNumber);+ }
if (isKey && i < indexDesc->rd_index->indnkeyatts)
uindexattrs = bms_add_member(uindexattrs,
@@ -5524,6 +5545,12 @@ restart:
/* Collect all attributes in the index predicate, too */
pull_varattnos(indexPredicate, 1, attrs);
+ if (attrs == &hotblockingattrs)+ {+ pull_varattnos(indexExpressions, 1, &exprattrs);+ pull_varattnos(indexPredicate, 1, &exprattrs);+ }+
index_close(indexDesc, AccessShareLock);
}
@@ -5552,12 +5579,26 @@ restart:
bms_free(idindexattrs);
bms_free(hotblockingattrs);
bms_free(summarizedattrs);
+ bms_free(exprattrs);+ bms_free(plainattrs);
goto restart;
}
indexedattrs = bms_union(hotblockingattrs, summarizedattrs);
+ /*+ * Record what attributes non-summarizing indexes reference only through+ * expressions, or only as partial indexes. A change to one of these+ * needs new index entries only if it changes the value of an index's+ * expressions or a row's membership in, or key within, a partial index.+ * An attribute a summarizing index references is left out, as that index+ * needs to see any change to it.+ */+ exprattrs = bms_del_members(exprattrs, plainattrs);+ exprattrs = bms_del_members(exprattrs, summarizedattrs);+ bms_free(plainattrs);+
/* Don't leak the old values of these bitmaps, if any */
relation->rd_attrsvalid = false;
bms_free(relation->rd_keyattr);
@@ -5572,6 +5613,8 @@ restart:
relation->rd_summarizedattr = NULL;
bms_free(relation->rd_indexedattr);
relation->rd_indexedattr = NULL;
+ bms_free(relation->rd_exprattr);+ relation->rd_exprattr = NULL;
/*
* Now save copies of the bitmaps in the relcache entry. We intentionally
@@ -5587,6 +5630,7 @@ restart:
relation->rd_hotblockingattr = bms_copy(hotblockingattrs);
relation->rd_summarizedattr = bms_copy(summarizedattrs);
relation->rd_indexedattr = bms_copy(indexedattrs);
+ relation->rd_exprattr = bms_copy(exprattrs);
relation->rd_attrsvalid = true;
MemoryContextSwitchTo(oldcxt);
@@ -5605,6 +5649,8 @@ restart:
return summarizedattrs;
case INDEX_ATTR_BITMAP_INDEXED:
return indexedattrs;
+ case INDEX_ATTR_BITMAP_EXPRESSION:+ return exprattrs;
default:
elog(ERROR, "unknown attrKind %u", attrKind);
return NULL;
diff --git a/src/include/executor/executor.h b/src/include/executor/executor.hindex 5cbdb092564..c82def87f2a 100644--- a/src/include/executor/executor.h+++ b/src/include/executor/executor.h@@ -819,6 +819,7 @@ extern ResultRelInfo *ExecLookupResultRelByOid(ModifyTableState *node,
bool update_cache);
extern Bitmapset *ExecUpdateModifiedIdxAttrs(ResultRelInfo *relinfo,
TupleTableSlot *old_tts,
- TupleTableSlot *new_tts);+ TupleTableSlot *new_tts,+ EState *estate);
#endif /* EXECUTOR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.hindex e3cea52e7df..220aaa8b965 100644--- a/src/include/nodes/execnodes.h+++ b/src/include/nodes/execnodes.h@@ -502,6 +502,14 @@ typedef struct ResultRelInfo
/* array of key/attr info for indices */
IndexInfo **ri_IndexRelationInfo;
+ /*+ * For UPDATE, the table attributes each index references through+ * expressions or its predicate, offset by+ * FirstLowInvalidHeapAttributeNumber. Built on first use; NULL until+ * then.+ */+ Bitmapset **ri_IndexExprAttrs;+
/*
* For UPDATE/DELETE/MERGE result relations, the attribute number of the
* row identity junk attribute in the source plan's output tuples
diff --git a/src/include/utils/rel.h b/src/include/utils/rel.hindex 55b459adba8..fdd2a1e3213 100644--- a/src/include/utils/rel.h+++ b/src/include/utils/rel.h@@ -165,6 +165,8 @@ typedef struct RelationData
Bitmapset *rd_hotblockingattr; /* cols blocking HOT update */
Bitmapset *rd_summarizedattr; /* cols indexed by summarizing indexes */
Bitmapset *rd_indexedattr; /* cols referenced by any index */
+ Bitmapset *rd_exprattr; /* cols non-summarizing indexes reference only+ * via expressions or as partial indexes */
PublicationDesc *rd_pubdesc; /* publication descriptor, or NULL */
diff --git a/src/include/utils/relcache.h b/src/include/utils/relcache.hindex 417990e8022..955d0186c2d 100644--- a/src/include/utils/relcache.h+++ b/src/include/utils/relcache.h@@ -73,6 +73,7 @@ typedef enum IndexAttrBitmapKind
INDEX_ATTR_BITMAP_HOT_BLOCKING,
INDEX_ATTR_BITMAP_SUMMARIZED,
INDEX_ATTR_BITMAP_INDEXED,
+ INDEX_ATTR_BITMAP_EXPRESSION,
} IndexAttrBitmapKind;
extern Bitmapset *RelationGetIndexAttrBitmap(Relation relation,
diff --git a/src/test/isolation/expected/cic-expression-hot.out b/src/test/isolation/expected/cic-expression-hot.outnew file mode 100644index 00000000000..4128f40c141--- /dev/null+++ b/src/test/isolation/expected/cic-expression-hot.out@@ -0,0 +1,43 @@+Parsed test spec with 2 sessions++starting permutation: s1_lock s2_cic_s s1_upd_s s1_unlock s1_upd2 s1_check+step s1_lock: SELECT pg_advisory_lock(4242);+pg_advisory_lock+----------------+ +(1 row)++step s2_cic_s: CREATE INDEX CONCURRENTLY cic_expr_idx ON cic_expr (cic_lower(s)); <waiting ...>+step s1_upd_s: UPDATE cic_expr SET s = upper(s);+step s1_unlock: SELECT pg_advisory_unlock(4242);+pg_advisory_unlock+------------------+t +(1 row)++step s2_cic_s: <... completed>+step s1_upd2: UPDATE cic_expr SET s = lower(s) || '';+step s1_check: + SET enable_seqscan = off;+ SET enable_bitmapscan = off;+ SELECT count(*) FROM cic_expr WHERE cic_lower(s) LIKE 'abc%';+ SELECT count(*) FROM cic_expr WHERE cic_lower(s) = 'abc42';+ RESET enable_seqscan;+ RESET enable_bitmapscan;+ SELECT indisvalid FROM pg_index WHERE indexrelid = 'cic_expr_idx'::regclass;++count+-----+ 100+(1 row)++count+-----+ 1+(1 row)++indisvalid+----------+t +(1 row)+diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_scheduleindex d7da899aa85..2e99ca04305 100644--- a/src/test/isolation/isolation_schedule+++ b/src/test/isolation/isolation_schedule@@ -95,6 +95,7 @@ test: skip-locked-3
test: skip-locked-4
test: drop-index-concurrently-1
test: multiple-cic
+test: cic-expression-hot
test: alter-domain-validate
test: alter-table-1
test: alter-table-2
diff --git a/src/test/isolation/specs/cic-expression-hot.spec b/src/test/isolation/specs/cic-expression-hot.specnew file mode 100644index 00000000000..ad1b612be01--- /dev/null+++ b/src/test/isolation/specs/cic-expression-hot.spec@@ -0,0 +1,43 @@+# UPDATEs during CREATE INDEX CONCURRENTLY of an expression index.+#+# The build waits for an advisory lock that s1 holds, inside the index+# expression, so s1's UPDATE runs while the index is not yet ready for+# inserts. The update leaves the expression's value alone; the finished+# index must find every row under its current key.++setup+{+ CREATE TABLE cic_expr (id int PRIMARY KEY, s text, x int)+ WITH (fillfactor = 50);+ INSERT INTO cic_expr SELECT g, 'abc' || g, g FROM generate_series(1, 100) g;+ CREATE FUNCTION cic_lower(text) RETURNS text IMMUTABLE LANGUAGE plpgsql AS $$+ BEGIN PERFORM pg_advisory_lock_shared(4242); PERFORM pg_advisory_unlock_shared(4242); RETURN lower($1); END;+ $$;+}++teardown+{+ DROP TABLE cic_expr;+ DROP FUNCTION cic_lower(text);+}++session s1+step s1_lock { SELECT pg_advisory_lock(4242); }+step s1_upd_s { UPDATE cic_expr SET s = upper(s); }+step s1_unlock { SELECT pg_advisory_unlock(4242); }+step s1_upd2 { UPDATE cic_expr SET s = lower(s) || ''; }+step s1_check+{+ SET enable_seqscan = off;+ SET enable_bitmapscan = off;+ SELECT count(*) FROM cic_expr WHERE cic_lower(s) LIKE 'abc%';+ SELECT count(*) FROM cic_expr WHERE cic_lower(s) = 'abc42';+ RESET enable_seqscan;+ RESET enable_bitmapscan;+ SELECT indisvalid FROM pg_index WHERE indexrelid = 'cic_expr_idx'::regclass;+}++session s2+step s2_cic_s { CREATE INDEX CONCURRENTLY cic_expr_idx ON cic_expr (cic_lower(s)); }++permutation s1_lock s2_cic_s s1_upd_s s1_unlock s1_upd2 s1_checkdiff --git a/src/test/regress/expected/hot_updates.out b/src/test/regress/expected/hot_updates.outindex f83a686f3a9..276a845a3e9 100644--- a/src/test/regress/expected/hot_updates.out+++ b/src/test/regress/expected/hot_updates.out@@ -321,7 +321,7 @@ SELECT pg_stat_force_next_flush();
SELECT * FROM get_hot_count('hot_jsonb_test');
updates | hot
---------+-----
- 1 | 0+ 1 | 1
(1 row)
UPDATE hot_jsonb_test SET data = data - 'city' WHERE id = 2;
@@ -334,7 +334,7 @@ SELECT pg_stat_force_next_flush();
SELECT * FROM get_hot_count('hot_jsonb_test');
updates | hot
---------+-----
- 2 | 0+ 2 | 2
(1 row)
UPDATE hot_jsonb_test SET data = jsonb_insert(data, '{country}', '"USA"') WHERE id = 2;
@@ -347,7 +347,7 @@ SELECT pg_stat_force_next_flush();
SELECT * FROM get_hot_count('hot_jsonb_test');
updates | hot
---------+-----
- 3 | 0+ 3 | 3
(1 row)
DROP TABLE hot_jsonb_test;
@@ -379,7 +379,7 @@ SELECT * FROM update_case(
NOTICE: table "hot_expr_test" does not exist, skipping
updated | hot
---------+-----
- 10 | 0+ 10 | 10
(1 row)
SELECT * FROM update_case(
@@ -389,7 +389,7 @@ SELECT * FROM update_case(
$$UPDATE hot_expr_test SET a = a - 1, b = b + 1$$);
updated | hot
---------+-----
- 10 | 0+ 10 | 10
(1 row)
-- Expression value changed.
@@ -424,7 +424,7 @@ SELECT * FROM update_case(
$$UPDATE hot_expr_test SET s = upper(s)$$);
updated | hot
---------+-----
- 10 | 0+ 10 | 10
(1 row)
SELECT * FROM update_case(
@@ -434,7 +434,7 @@ SELECT * FROM update_case(
$$UPDATE hot_expr_test SET p = point(p[0], p[1] + 1)$$);
updated | hot
---------+-----
- 10 | 0+ 10 | 10
(1 row)
-- An expression that is a point, stored by the gist index as a box.
@@ -447,7 +447,7 @@ SELECT * FROM update_case(
$$UPDATE hot_expr_test SET d = d + 7$$);
updated | hot
---------+-----
- 10 | 0+ 10 | 10
(1 row)
-- A plain index on the same column as an expression index.
@@ -472,7 +472,7 @@ SELECT * FROM update_case(
$$UPDATE hot_expr_test SET s = upper(s), b = b + 1$$);
updated | hot
---------+-----
- 10 | 0+ 10 | 10
(1 row)
-- A BEFORE ROW trigger that changes the expression's value.
@@ -501,7 +501,7 @@ SELECT * FROM update_case(
$$UPDATE hot_expr_test SET v = v + 1$$);
updated | hot
---------+-----
- 10 | 0+ 10 | 10
(1 row)
SELECT * FROM update_case(
@@ -511,7 +511,7 @@ SELECT * FROM update_case(
$$UPDATE hot_expr_test SET status = 'archived'$$);
updated | hot
---------+-----
- 10 | 0+ 10 | 10
(1 row)
-- Both versions inside the predicate with the key unchanged.
@@ -522,7 +522,7 @@ SELECT * FROM update_case(
$$UPDATE hot_expr_test SET status = 'b'$$);
updated | hot
---------+-----
- 10 | 0+ 10 | 10
(1 row)
-- Entering or leaving the predicate, or changing the key inside it.
@@ -565,7 +565,7 @@ SELECT * FROM update_case(
$$UPDATE hot_expr_test SET v = v + 100 WHERE status = 'done'$$);
updated | hot
---------+-----
- 5 | 0+ 5 | 5
(1 row)
SET enable_seqscan = off;
--
2.50.1
v43-0006-Treat-an-indexed-value-stored-differently-but-un.patch (text/plain, 24.4K) download
From d88271135ccdf26272abc155743f2bfe85d7ec31 Mon Sep 17 00:00:00 2001
From: Greg Burd <greg@burd.me>
Date: Fri, 16 Oct 2026 12:29:42 -0400
Subject: [PATCH v43 6/8] Treat an indexed value stored differently but
unchanged as unchanged
The executor compares indexed values as stored, as heap_update() always
has, so it sees a change when an UPDATE writes back the value a row
already has but the old value is stored compressed or out of line and
the new one is not compressed yet. Every UPDATE that sends all columns,
as ORMs and logical replication do, is like that for large indexed
values, and so was never HOT.
Two values that are the same once detoasted give every index the same
entries, whatever the access method: index entries are a function of the
values. So when two values differ as stored and one of them is
compressed or stored out of line, compare them with datum_image_eq(),
and if they are the same, treat the column as unchanged. This is done
once for each changed indexed column, before the narrowing for
expression and partial indexes. The narrowing then compares again only
the datums of expressions: a table column an index stores gives it the
same entries exactly when the column is no longer in the changed set, so
its values are neither compared nor fetched a second time. It applies to
btree, GiST, SP-GiST and extension access methods alike, so no access
method needs amcomparedatums for it; the previous version of this patch
gave btree a callback that did exactly this, which Matthias van de Meent
pointed out told us nothing about btree.
datum_image_eq() fetches and decompresses only when the stored forms
differ, one of them is compressed or out of line, and their sizes are the
same: an unchanged value that keeps its toast pointer still compares
equal without being fetched, and values of different sizes are not
fetched either. What remains costs both values' detoasting, where
before the update would have been non-HOT and the index insertion would
have detoasted the new value.
Columns of unique indexes that foreign keys can reference, and of the
replica identity, are still compared as stored: the table AM decides its
lock strength and what it logs for logical decoding from them.
Values that are equal but not the same, such as numeric 1.0 and 1.00 or
strings equal under a nondeterministic collation, still count as
changed: an index-only scan would return the old one.
In builds with assertions, heap_update()'s check of the caller's set of
changed columns allows such a column to be left out, if one of its
stored values is compressed or out of line and it is in no candidate key
or replica identity.
---
contrib/amcheck/expected/check_btree.out | 98 +++++++++++++++++
contrib/amcheck/sql/check_btree.sql | 46 ++++++++
doc/src/sgml/indexam.sgml | 4 +-
doc/src/sgml/storage.sgml | 7 +-
src/backend/access/heap/README.HOT | 7 +-
src/backend/access/heap/heapam.c | 35 ++++++-
src/backend/executor/nodeModifyTable.c | 122 +++++++++++++++++++---
src/test/regress/expected/hot_updates.out | 6 +-
8 files changed, 302 insertions(+), 23 deletions(-)
diff --git a/contrib/amcheck/expected/check_btree.out b/contrib/amcheck/expected/check_btree.outindex 6fc1ee97902..4e02954b6f8 100644--- a/contrib/amcheck/expected/check_btree.out+++ b/contrib/amcheck/expected/check_btree.out@@ -366,6 +366,104 @@ SELECT bt_index_check('hot_expr_multi_vip', heapallindexed => true);
(1 row)
DROP TABLE hot_expr_multi;
+-- An UPDATE that writes back the value a row has, when the old value is+-- stored compressed or out of line and the new one is not yet, is HOT: the+-- index would store the same bytes. Index-only scans must return the new+-- row's value.+CREATE TABLE hot_bt_image (id int, t text, e text) WITH (fillfactor = 50);+ALTER TABLE hot_bt_image ALTER COLUMN e SET STORAGE EXTERNAL;+CREATE INDEX hot_bt_image_t ON hot_bt_image (t);+CREATE INDEX hot_bt_image_e ON hot_bt_image (e) INCLUDE (id);+INSERT INTO hot_bt_image+ SELECT g, repeat('x', 3000) || g, repeat('y', 2500) || g+ FROM generate_series(1, 20) g;+SELECT pg_column_compression(t) IS NOT NULL AS t_compressed,+ pg_column_size(e) > 2500 AS e_out_of_line+ FROM hot_bt_image WHERE id = 1;+ t_compressed | e_out_of_line +--------------+---------------+ t | t+(1 row)++CREATE FUNCTION hot_bt_image_count(OUT updated bigint, OUT hot bigint)+LANGUAGE sql AS $$+ SELECT pg_stat_get_xact_tuples_updated('hot_bt_image'::regclass),+ pg_stat_get_xact_tuples_hot_updated('hot_bt_image'::regclass) $$;+BEGIN;+UPDATE hot_bt_image SET t = repeat('x', 3000) || id;+SELECT * FROM hot_bt_image_count();+ updated | hot +---------+-----+ 20 | 20+(1 row)++UPDATE hot_bt_image SET e = e || '';+SELECT * FROM hot_bt_image_count();+ updated | hot +---------+-----+ 40 | 40+(1 row)++UPDATE hot_bt_image SET t = repeat('x', 3000) || (id + 100) WHERE id <= 5;+SELECT * FROM hot_bt_image_count();+ updated | hot +---------+-----+ 45 | 40+(1 row)++COMMIT;+SET enable_seqscan = off;+SET enable_bitmapscan = off;+CREATE FUNCTION hot_bt_image_plan() RETURNS SETOF text LANGUAGE plpgsql AS+$$ DECLARE l text; BEGIN+ FOR l IN EXPLAIN (COSTS OFF)+ SELECT right(t, 3) FROM hot_bt_image WHERE t > repeat('x', 3000)+ LOOP RETURN NEXT left(l, 40); END LOOP; END $$;+SELECT * FROM hot_bt_image_plan();+ hot_bt_image_plan +------------------------------------------+ Index Only Scan using hot_bt_image_t on + Index Cond: (t > 'xxxxxxxxxxxxxxxxxxxx+(2 rows)++SELECT right(t, 3) FROM hot_bt_image WHERE t > repeat('x', 3000)+ ORDER BY t COLLATE "C" LIMIT 3;+ right +-------+ x10+ 101+ 102+(3 rows)++SELECT count(*) FROM hot_bt_image WHERE t > repeat('x', 3000);+ count +-------+ 20+(1 row)++SELECT count(*) FROM hot_bt_image WHERE e > repeat('y', 2500);+ count +-------+ 20+(1 row)++RESET enable_seqscan;+RESET enable_bitmapscan;+SELECT bt_index_check('hot_bt_image_t', heapallindexed => true);+ bt_index_check +----------------+ +(1 row)++SELECT bt_index_check('hot_bt_image_e', heapallindexed => true);+ bt_index_check +----------------+ +(1 row)++DROP TABLE hot_bt_image;+DROP FUNCTION hot_bt_image_count;+DROP FUNCTION hot_bt_image_plan;
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
diff --git a/contrib/amcheck/sql/check_btree.sql b/contrib/amcheck/sql/check_btree.sqlindex 4c54aad5d9f..0e03eeef9e0 100644--- a/contrib/amcheck/sql/check_btree.sql+++ b/contrib/amcheck/sql/check_btree.sql@@ -233,6 +233,52 @@ SELECT bt_index_check('hot_expr_multi_status', heapallindexed => true);
SELECT bt_index_check('hot_expr_multi_vip', heapallindexed => true);
DROP TABLE hot_expr_multi;
+-- An UPDATE that writes back the value a row has, when the old value is+-- stored compressed or out of line and the new one is not yet, is HOT: the+-- index would store the same bytes. Index-only scans must return the new+-- row's value.+CREATE TABLE hot_bt_image (id int, t text, e text) WITH (fillfactor = 50);+ALTER TABLE hot_bt_image ALTER COLUMN e SET STORAGE EXTERNAL;+CREATE INDEX hot_bt_image_t ON hot_bt_image (t);+CREATE INDEX hot_bt_image_e ON hot_bt_image (e) INCLUDE (id);+INSERT INTO hot_bt_image+ SELECT g, repeat('x', 3000) || g, repeat('y', 2500) || g+ FROM generate_series(1, 20) g;+SELECT pg_column_compression(t) IS NOT NULL AS t_compressed,+ pg_column_size(e) > 2500 AS e_out_of_line+ FROM hot_bt_image WHERE id = 1;+CREATE FUNCTION hot_bt_image_count(OUT updated bigint, OUT hot bigint)+LANGUAGE sql AS $$+ SELECT pg_stat_get_xact_tuples_updated('hot_bt_image'::regclass),+ pg_stat_get_xact_tuples_hot_updated('hot_bt_image'::regclass) $$;+BEGIN;+UPDATE hot_bt_image SET t = repeat('x', 3000) || id;+SELECT * FROM hot_bt_image_count();+UPDATE hot_bt_image SET e = e || '';+SELECT * FROM hot_bt_image_count();+UPDATE hot_bt_image SET t = repeat('x', 3000) || (id + 100) WHERE id <= 5;+SELECT * FROM hot_bt_image_count();+COMMIT;+SET enable_seqscan = off;+SET enable_bitmapscan = off;+CREATE FUNCTION hot_bt_image_plan() RETURNS SETOF text LANGUAGE plpgsql AS+$$ DECLARE l text; BEGIN+ FOR l IN EXPLAIN (COSTS OFF)+ SELECT right(t, 3) FROM hot_bt_image WHERE t > repeat('x', 3000)+ LOOP RETURN NEXT left(l, 40); END LOOP; END $$;+SELECT * FROM hot_bt_image_plan();+SELECT right(t, 3) FROM hot_bt_image WHERE t > repeat('x', 3000)+ ORDER BY t COLLATE "C" LIMIT 3;+SELECT count(*) FROM hot_bt_image WHERE t > repeat('x', 3000);+SELECT count(*) FROM hot_bt_image WHERE e > repeat('y', 2500);+RESET enable_seqscan;+RESET enable_bitmapscan;+SELECT bt_index_check('hot_bt_image_t', heapallindexed => true);+SELECT bt_index_check('hot_bt_image_e', heapallindexed => true);+DROP TABLE hot_bt_image;+DROP FUNCTION hot_bt_image_count;+DROP FUNCTION hot_bt_image_plan;+
-- cleanup
DROP TABLE bttest_a;
DROP TABLE bttest_b;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgmlindex d91192bc011..111494b7b53 100644--- a/doc/src/sgml/indexam.sgml+++ b/doc/src/sgml/indexam.sgml@@ -937,7 +937,9 @@ amcomparedatums (Relation indexRelation,
type of the table column or index expression, which need not be the type
the index stores), whose length and by-value property are
<parameter>typlen</parameter> and <parameter>typbyval</parameter>, and
- are not equal byte for byte. The function is+ are not equal byte for byte, nor once detoasted: the executor itself+ treats two values that are the same, but stored differently, as giving+ the same entries in any index. The function is
called during <command>UPDATE</command>, before the table access method
is, for every index that uses a column whose value changed; if every such
index answers <literal>true</literal> for every changed column, the update
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgmlindex 15b5dae1752..66cfe0e7b14 100644--- a/doc/src/sgml/storage.sgml+++ b/doc/src/sgml/storage.sgml@@ -1126,7 +1126,12 @@ data. Empty in ordinary tables.</entry>
methods, that depends on the method and on the change (see
<xref linkend="indexam"/>). This does not apply to the columns of
unique indexes that a foreign key can reference, of the replica
- identity index, or of exclusion constraints.+ identity index, or of exclusion constraints. Nor is a column changed+ whose new value is the same as the old one, only stored differently, as+ when the old value is compressed or stored out of line+ (see <xref linkend="storage-toast"/>) and an update writes it back;+ except for the columns of those unique and replica identity indexes,+ this holds for every kind of index.
</para>
</listitem>
<listitem>
diff --git a/src/backend/access/heap/README.HOT b/src/backend/access/heap/README.HOTindex 5c2dfc27f58..c317840cb3d 100644--- a/src/backend/access/heap/README.HOT+++ b/src/backend/access/heap/README.HOT@@ -45,8 +45,11 @@ The same holds for a column that indexes also store directly, if every
index that does so has an access method that can tell, from the old and
new values, that it would store the same entries (amcomparedatums), and
the column is in no candidate key, replica identity or exclusion
-constraint. The executor works that out before calling the table AM,-and passes on only the indexed columns whose change matters. An index+constraint. And a column of no candidate key or replica identity is not+changed if its value is the same, only stored differently: compressed or+out of line in the old tuple, say, and not yet in the new one. The+executor works that out before calling the table AM, and passes on only+the indexed columns whose change matters. An index
that is not yet ready for inserts, or whose build failed, counts as
changed whenever a column it references does; see "CREATE INDEX
CONCURRENTLY" below.
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.cindex a0e2d904beb..a224e25bad2 100644--- a/src/backend/access/heap/heapam.c+++ b/src/backend/access/heap/heapam.c@@ -3317,6 +3317,7 @@ heap_update(Relation relation, const ItemPointerData *otid, HeapTuple newtup,
#ifdef USE_ASSERT_CHECKING
Bitmapset *check_attrs;
Bitmapset *narrowable_attrs;
+ Bitmapset *key_attrs;
#endif
TransactionId xmax_new_tuple,
xmax_old_tuple;
@@ -3360,6 +3361,7 @@ heap_update(Relation relation, const ItemPointerData *otid, HeapTuple newtup,
id_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_IDENTITY_KEY);
#ifdef USE_ASSERT_CHECKING
check_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_INDEXED);
+ key_attrs = RelationGetIndexAttrBitmap(relation, INDEX_ATTR_BITMAP_KEY);
narrowable_attrs = RelationGetIndexAttrBitmap(relation,
INDEX_ATTR_BITMAP_EXPRESSION);
{
@@ -3428,6 +3430,7 @@ heap_update(Relation relation, const ItemPointerData *otid, HeapTuple newtup,
#ifdef USE_ASSERT_CHECKING
bms_free(check_attrs);
bms_free(narrowable_attrs);
+ bms_free(key_attrs);
#endif
/* modified_idx_attrs is owned by the caller, don't free it */
@@ -3777,6 +3780,7 @@ l2:
#ifdef USE_ASSERT_CHECKING
bms_free(check_attrs);
bms_free(narrowable_attrs);
+ bms_free(key_attrs);
#endif
/* modified_idx_attrs is owned by the caller, don't free it */
@@ -3810,19 +3814,42 @@ l2:
* wrong tuples, is caught. The caller may leave out an attribute that
* indexes use only through expressions or as partial indexes, or only in
* indexes that can compare entries, having found that no index's entry
- * changes (see ExecUpdateModifiedIdxAttrs()), and may report more:- * comparing slots, it cannot always tell that two values will be stored- * alike.+ * changes (see ExecUpdateModifiedIdxAttrs()), or one whose value is the+ * same, stored compressed or out of line in one of the tuples, unless it+ * is in a candidate key or the replica identity. It may also report+ * more: comparing slots, it cannot always tell that two values will be+ * stored alike.
*/
{
Bitmapset *changed = HeapUpdateModifiedIdxAttrs(relation, check_attrs,
&oldtup, newtup);
+ TupleDesc tupdesc = RelationGetDescr(relation);+ int attidx = -1;
changed = bms_del_members(changed, narrowable_attrs);
- Assert(bms_is_subset(changed, modified_idx_attrs));+ changed = bms_del_members(changed, modified_idx_attrs);+ while ((attidx = bms_next_member(changed, attidx)) >= 0)+ {+ AttrNumber attrnum = attidx + FirstLowInvalidHeapAttributeNumber;+ Datum value1,+ value2;+ bool isnull1,+ isnull2;++ Assert(attrnum > 0);+ Assert(!bms_is_member(attidx, id_attrs));+ Assert(!bms_is_member(attidx, key_attrs));+ Assert(TupleDescCompactAttr(tupdesc, attrnum - 1)->attlen == -1);+ value1 = heap_getattr(&oldtup, attrnum, tupdesc, &isnull1);+ value2 = heap_getattr(newtup, attrnum, tupdesc, &isnull2);+ Assert(!isnull1 && !isnull2);+ Assert(VARATT_IS_EXTENDED(DatumGetPointer(value1)) ||+ VARATT_IS_EXTENDED(DatumGetPointer(value2)));+ }
bms_free(changed);
bms_free(check_attrs);
bms_free(narrowable_attrs);
+ bms_free(key_attrs);
}
#endif
diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.cindex 4054f6da0df..b2f46945637 100644--- a/src/backend/executor/nodeModifyTable.c+++ b/src/backend/executor/nodeModifyTable.c@@ -82,6 +82,7 @@
#include "utils/lsyscache.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
+#include "varatt.h"
typedef struct MTTargetRelLookup
@@ -199,11 +200,94 @@ static TupleTableSlot *ExecMergeNotMatched(ModifyTableContext *context,
ResultRelInfo *resultRelInfo,
bool canSetTag);
+/*+ * Are two varlena values that differ as stored the same value? They can+ * only be if one of them is compressed or stored out of line; otherwise the+ * stored bytes are the value. datum_image_eq() then compares them+ * detoasted, which it skips when their sizes differ. Any index makes the+ * same entries for the same value, so for HOT the two are the same.+ */+static bool+VarlenaImageEqual(Datum value1, Datum value2)+{+ varlena *v1 = (varlena *) DatumGetPointer(value1);+ varlena *v2 = (varlena *) DatumGetPointer(value2);++ if (!VARATT_IS_EXTERNAL(v1) && !VARATT_IS_COMPRESSED(v1) &&+ !VARATT_IS_EXTERNAL(v2) && !VARATT_IS_COMPRESSED(v2))+ return false;+ return datum_image_eq(value1, value2, false, -1);+}++/*+ * Remove from attrs, the indexed attributes whose values differ as stored+ * between old_tts and new_tts, those that are the same value stored+ * differently: the old one compressed or out of line, say, and the new one+ * not yet, as when an UPDATE writes back the value a row has. No index needs+ * a new entry for them. Attributes of candidate keys and the replica+ * identity stay, as the table AM decides its lock strength and what it logs+ * from the values as stored.+ */+static Bitmapset *+DropRestoredAttrs(Relation relation, Bitmapset *attrs,+ TupleTableSlot *old_tts, TupleTableSlot *new_tts,+ EState *estate)+{+ TupleDesc tupdesc = RelationGetDescr(relation);+ Bitmapset *keepattrs = NULL;+ bool fetched = false;+ MemoryContext oldcxt;+ int attidx = -1;++ oldcxt = MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));+ while ((attidx = bms_next_member(attrs, attidx)) >= 0)+ {+ AttrNumber attrnum = attidx + FirstLowInvalidHeapAttributeNumber;+ Datum value1,+ value2;+ bool null1,+ null2;++ if (attrnum <= 0 ||+ TupleDescCompactAttr(tupdesc, attrnum - 1)->attlen != -1)+ continue;+ value1 = slot_getattr(old_tts, attrnum, &null1);+ value2 = slot_getattr(new_tts, attrnum, &null2);+ if (null1 || null2 || !VarlenaImageEqual(value1, value2))+ continue;++ if (!fetched)+ {+ Bitmapset *idattrs;++ MemoryContextSwitchTo(oldcxt);+ keepattrs = RelationGetIndexAttrBitmap(relation,+ INDEX_ATTR_BITMAP_KEY);+ idattrs = RelationGetIndexAttrBitmap(relation,+ INDEX_ATTR_BITMAP_IDENTITY_KEY);+ keepattrs = bms_add_members(keepattrs, idattrs);+ bms_free(idattrs);+ MemoryContextSwitchTo(GetPerTupleMemoryContext(estate));+ fetched = true;+ }+ if (!bms_is_member(attidx, keepattrs))+ attrs = bms_del_member(attrs, attidx);+ }+ MemoryContextSwitchTo(oldcxt);+ bms_free(keepattrs);++ return attrs;+}+
/*
* Do the old and new index datums differ? Compare each column as the type
* FormIndexDatum() produced it in: the table column's type or the
- * expression's, which need not be the type the index stores. Datums that- * differ byte for byte may still give the index the same entries; if its+ * expression's, which need not be the type the index stores. A table column+ * is not compared again: "changed" says which of the index's columns the+ * update changed, both as stored and once detoasted, and the others give the+ * same entries. An expression's datums that differ byte for byte may be the+ * same value stored differently, which gives any index the same entries.+ * Other datums that differ may still give the index the same entries; if its
* access method can tell (amcomparedatums), ask it, but not for a column the
* index can return in index-only scans, where only the same value will do.
*
@@ -214,6 +298,7 @@ static TupleTableSlot *ExecMergeNotMatched(ModifyTableContext *context,
*/
static bool
IndexDatumsDiffer(Relation heapRel, Relation indexRel, IndexInfo *indexInfo,
+ const Bitmapset *changed,
const Datum *old_values, const bool *old_isnull,
const Datum *new_values, const bool *new_isnull)
{
@@ -230,9 +315,12 @@ IndexDatumsDiffer(Relation heapRel, Relation indexRel, IndexInfo *indexInfo,
if (attnum != 0)
{
- Form_pg_attribute att = TupleDescAttr(RelationGetDescr(heapRel),- attnum - 1);+ Form_pg_attribute att;+ if (!bms_is_member(attnum - FirstLowInvalidHeapAttributeNumber,+ changed))+ continue;+ att = TupleDescAttr(RelationGetDescr(heapRel), attnum - 1);
typid = att->atttypid;
typlen = att->attlen;
typbyval = att->attbyval;
@@ -246,8 +334,12 @@ IndexDatumsDiffer(Relation heapRel, Relation indexRel, IndexInfo *indexInfo,
if (old_isnull[i] != new_isnull[i])
return true;
- if (old_isnull[i] ||- datumIsEqual(old_values[i], new_values[i], typbyval, typlen))+ if (old_isnull[i])+ continue;+ if (attnum == 0 &&+ (datumIsEqual(old_values[i], new_values[i], typbyval, typlen) ||+ (typlen == -1 &&+ VarlenaImageEqual(old_values[i], new_values[i]))))
continue;
if (cmp == NULL || index_can_return(indexRel, i + 1) ||
!cmp(indexRel, indexInfo, i + 1, typid, typlen, typbyval,
@@ -259,7 +351,8 @@ IndexDatumsDiffer(Relation heapRel, Relation indexRel, IndexInfo *indexInfo,
/*
* Would the update from old_tts to new_tts give this index a new entry, or
- * take one away?+ * take one away? "changed" are the attributes the update changed that the+ * index uses, as for IndexDatumsDiffer().
*
* The index datums are formed in the per-tuple memory context, which
* callers reset for each row. The predicate's ExprState is kept in the
@@ -268,8 +361,9 @@ IndexDatumsDiffer(Relation heapRel, Relation indexRel, IndexInfo *indexInfo,
*/
static bool
IndexEntryChanges(ResultRelInfo *resultRelInfo, Relation indexRel,
- IndexInfo *indexInfo, TupleTableSlot *old_tts,- TupleTableSlot *new_tts, EState *estate)+ IndexInfo *indexInfo, const Bitmapset *changed,+ TupleTableSlot *old_tts, TupleTableSlot *new_tts,+ EState *estate)
{
ExprContext *econtext = GetPerTupleExprContext(estate);
TupleTableSlot *save_scantuple = econtext->ecxt_scantuple;
@@ -314,7 +408,7 @@ IndexEntryChanges(ResultRelInfo *resultRelInfo, Relation indexRel,
econtext->ecxt_scantuple = save_scantuple;
result = IndexDatumsDiffer(resultRelInfo->ri_RelationDesc, indexRel,
- indexInfo, old_values, old_isnull,+ indexInfo, changed, old_values, old_isnull,
new_values, new_isnull);
MemoryContextSwitchTo(oldcxt);
return result;
@@ -402,7 +496,7 @@ IndexUnaffected(ResultRelInfo *resultRelInfo, int i, Bitmapset *mine,
{
if (IndexEntryChanges(resultRelInfo,
resultRelInfo->ri_IndexRelationDescs[i],
- indexInfo, old_tts, new_tts,+ indexInfo, mine, old_tts, new_tts,
estate))
elog(ERROR, "index \"%s\" proven unaffected by UPDATE, but its entry changed",
RelationGetRelationName(resultRelInfo->ri_IndexRelationDescs[i]));
@@ -415,7 +509,7 @@ IndexUnaffected(ResultRelInfo *resultRelInfo, int i, Bitmapset *mine,
return !IndexEntryChanges(resultRelInfo,
resultRelInfo->ri_IndexRelationDescs[i],
- indexInfo, old_tts, new_tts,+ indexInfo, mine, old_tts, new_tts,
estate);
}
@@ -485,6 +579,10 @@ ExecUpdateModifiedIdxAttrs(ResultRelInfo *resultRelInfo,
attrs = table_slot_modified_idx_attrs(relation, old_tts, new_tts);
+ if (bms_is_empty(attrs))+ return attrs;++ attrs = DropRestoredAttrs(relation, attrs, old_tts, new_tts, estate);
if (bms_is_empty(attrs))
return attrs;
diff --git a/src/test/regress/expected/hot_updates.out b/src/test/regress/expected/hot_updates.outindex 276a845a3e9..fc820671060 100644--- a/src/test/regress/expected/hot_updates.out+++ b/src/test/regress/expected/hot_updates.out@@ -671,7 +671,7 @@ SELECT * FROM update_case(
NOTICE: table "hot_expr_test" does not exist, skipping
updated | hot
---------+-----
- 10 | 0+ 10 | 10
(1 row)
SET enable_seqscan = off;
@@ -692,7 +692,7 @@ SELECT * FROM update_case(
$$UPDATE hot_expr_test SET t = repeat('x', 3000) || id, x = x + 1$$);
updated | hot
---------+-----
- 10 | 0+ 10 | 10
(1 row)
-- The same through an expression that returns the column's value as is.
@@ -703,7 +703,7 @@ SELECT * FROM update_case(
$$UPDATE hot_expr_test SET t = repeat('x', 3000) || id, x = x + 1$$);
updated | hot
---------+-----
- 10 | 0+ 10 | 10
(1 row)
-- ... but not for a column of a unique index, which the table's lock
--
2.50.1
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: greg@burd.me, boekewurm+postgres@gmail.com, pgsql@j-davis.com, nathandbossart@gmail.com
Subject: Re: Expanding HOT updates for expression and partial indexes
In-Reply-To: <atpXQaH1UBS3LY-6@floki>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox